This is a toolset to visualize GitHub archives using Grafana dashboards.
GHA2DB stands for GitHub Archives to DashBoards.
More information about Kubernetes dashboards here.
Please see example Helm chart for an example Helm deployment.
Please see Helm chart for a full Helm deployment.
Please see LF Helm chart for the LF Helm deployment (it is a data deployment, has no Grafana and uses ElasticSearch in addition to Postgres to store data).
Please see GraphQL Helm chart for GraphQL foundation DevStats deployment.
Please see Kubernetes dashboard if you want to enable a local dashboard to explore cluster state.
Please see bare metal example to see example bare metal deployment.
The rest of this document describes current bare metal deployment on metal.equinix.com used by CNCF projects.
Under the hood, DevStats uses the following CNCF projects:
And other projects, including:
Please check this for a detailed architecture description.
We want to create a toolset for visualizing various metrics for the Kubernetes community (and also for all CNCF projects).
Everything is open source so that it can be used by other CNCF and non-CNCF open source projects.
The only requirement is that project must be hosted on a public GitHub repository/repositories.
If you want to hide your data (replace with anon-#) please follow instructions here.
This toolset uses only Open Source tools: GitHub archives, GitHub API, git, Postgres databases and multiple Grafana instances. It is written in Go, and can be forked and installed by anyone.
To work on this project locally please fork the original repository, and:
Please see Development for local development guide.
For more detailed description of all environment variables, tools, switches etc, please see Usage.
We want to support all kind of metrics, including historical ones. Please see requested metrics to see what kind of metrics are needed. Many of them cannot be computed based on the data sources currently used.
There are some groups of repositories that are grouped together as a repository groups. They are defined in scripts/kubernetes/repo_groups.sql.
To setup default repository groups:
This is a part of
kubernetes/psql.sh script and kubernetes psql dump already has groups configured.
We also want to have per company statistics. To implement such metrics we need a mapping of developers and their employers.
There is a project that attempts to create such mapping cncf/gitdm.
DevStats has an import tool that fetches company affiliations from
cncf/gitdm and allows to create per company metrics/statistics. It also uses
companies.yaml file to map company acquisitions (any data generated by company acquired by the another company is assigned to the latter using mapping from
If you see errors in the company affiliations, please open a pull request on cncf/gitdm and the updates will be reflected on https://k8s.devstats.cncf.io a couple days after the PR has been accepted. Note that gitdm supports mapping based on dates, to account for developers moving between companies.
New affiliations are imported into DevStats about 1-2 times/month.
For architecture details please see architecture file.
Detailed usage is here
Please see metrics to see how to add new metrics.
To add new project on a bare metal deployment follow adding new project instructions.
ADDING_NEW_PROJECTS.md for informations about how to add more projects on Kubernetes/Helm deployment.
Please see dashboards to see list of already defined Grafana dashboards.
Please see exporting.
GHA2DB_PROJECTS_OVERRIDE="+cncf" PG_PASS=pwd devstats.
crontab -eto run every hour HH:08.
PG_PASS=... PG_DB=allprj ./devel/activity.sh '1 month,,' > all.txt.