This topic introduces the GitHub public event data in the MaxCompute public dataset, shows how to query public GitHub activity with MaxCompute, and provides sample queries and analysis.
Introduction
Many developers build open source projects on GitHub, generating a vast number of events. GitHub records details about each event, such as its type, the developer, and the associated code repository. GitHub also makes public events, such as starring a repository and committing code, available. For a full list of event types, see GitHub Event Types. The GH Archive project aggregates these public GitHub events hourly and makes them available to developers. For more information, see GH Archive.
MaxCompute processes the large-scale public event data from GH Archive offline and creates the following tables:
Table name | Description | Update cycle |
dwd_github_events_odps | A fact table for GitHub public event data. | Updated hourly (T+1) |
dws_overview_by_repo_month | An aggregate table for monthly metrics of GitHub public event data. | Updated daily (T+1) |
The dwd_github_events_odps and dws_overview_by_repo_month tables are stored in the github_events schema of the MaxCompute public project BIGDATA_PUBLIC_DATASET (To learn more about schemas, see Schema operations). If you have enabled the MaxCompute service, you can query these tables using cross-project access.
Tables
dwd_github_events_odps
The fact table stores the main information for each public GitHub activity and is updated with a T+1 hour delay. The table contains the following fields:
Parameter | Type | Description |
id | bigint | The event ID. |
actor_id | bigint | The ID of the event actor. |
actor_login | string | The login name of the event actor. |
repo_id | bigint | The repository ID. |
repo_name | string | The repository name, in the format owner/repository_name. |
org_id | bigint | The ID of the organization that owns the repository. |
org_login | string | The name of the organization that owns the repository. |
type | string | The event type. For details, see GitHub Events Type. |
created_at | datetime | The timestamp when the event occurred. |
action | string | The event action. |
iss_or_pr_id | bigint | The ID of the issue or pull request. |
number | bigint | The number of the issue or pull request. |
comment_id | bigint | The comment ID. |
commit_id | string | The commit ID. |
member_id | bigint | The member ID. |
rev_or_push_or_rel_id | bigint | The ID of the review, push, or release. |
ref | string | The name of the resource that was created or deleted. |
ref_type | string | The type of the resource that was created or deleted. |
state | string | The state of the issue, pull request, or pull_request_review. |
author_association | string | The relationship between the actor and the repository. |
language | string | The programming language of the pull request. |
merged | boolean | Whether the pull request was merged. |
merged_at | datetime | The timestamp when the pull request was merged. |
additions | bigint | The number of lines of code added. |
deletions | bigint | The number of lines of code deleted. |
changed_files | bigint | The number of files changed in the pull request. |
push_size | bigint | The number of commits in the push. |
push_distinct_size | bigint | The number of distinct commits in the push. |
hr | string | The hour the event occurred. For example, if the time is |
month | string | The month the event occurred. For example, for an event in October 2015, |
year | string | The year the event occurred. For example, for an event in 2015, |
ds | string | The date the event occurred. The format is |
dws_overview_by_repo_month
The aggregation table stores the monthly summary of project-level event metrics and is updated on a T+1 basis. The table contains the following fields:
Parameter | Type | Description |
repo_id | bigint | The repository ID. |
repo_name | string | The repository name, in the format owner/repository_name. |
stars | bigint | The number of stars for the repository. |
commits | bigint | The number of commits to the repository. |
pushes | bigint | The number of pushes to the repository. |
total_prs | bigint | The total number of pull requests for the repository. |
pr_creators | bigint | The number of users who created pull requests for the repository. |
pr_reviews | bigint | The total number of pull request reviews for the repository. |
pr_reviewers | bigint | The number of users who reviewed pull requests for the repository. |
total_issues | bigint | The total number of issues for the repository. |
forks | bigint | The number of forks of the repository. |
month | string | The month the events occurred. For example, for events in October 2015, the value is |
To learn how this data is generated, see batch and stream integration practice based on the GitHub public event dataset.
To perform analysis on specific objects, MaxCompute also stores the db_repos table with the IDs and names of open source database projects, and the programming_language_repos table with the IDs and names of open source programming language projects. (The project list is from ossinsight).
Available regions
Region | Region ID |
China (Hangzhou) | cn-hangzhou |
China (Shanghai) | cn-shanghai |
China (Beijing) | cn-beijing |
China (Zhangjiakou) | cn-zhangjiakou |
China (Ulanqab) | cn-wulanchabu |
China (Shenzhen) | cn-shenzhen |
China (Chengdu) | cn-chengdu |
Disclaimer
The GitHub public event data from MaxCompute is for product testing only. The accuracy of this data is not guaranteed. Do not use this data in a production environment.
Notes
Public datasets are available to all MaxCompute users. When you query these datasets, note the following:
All data in the public dataset is stored in the
BIGDATA_PUBLIC_DATASETproject. However, you are not a member of this project. Therefore, you must use cross-project access to retrieve the data. When you write an SQL script, you must specify the project name and schema name before the table name. Additionally, if tenant-level schema syntax is disabled, you must enable session-level schema syntax to ensure that your commands run properly. An example command is as follows:-- Enable session-level schema syntax SET odps.namespace.schema=true; -- Query 100 rows from the dwd_github_events_odps table SELECT * FROM bigdata_public_dataset.github_events.dwd_github_events_odps WHERE ds='2024-05-10' limit 100;ImportantYou do not incur storage fees for public datasets, but you are charged for the computing fees for your queries. For more information about billing, see Computing fees (pay-as-you-go).
Because public datasets require cross-project access, you cannot find tables from these datasets in the DataWorks Data Map.
The project for public datasets uses schemas. If tenant-level schema syntax is not enabled for your project, you cannot view the public datasets directly in DataWorks DataAnalysis. However, you can still query the data by running SQL statements.
Explore GitHub public event data with MaxCompute
Query example
Activate MaxCompute and create a project. For more information, see Create a MaxCompute project.
Query example
-- Enable session-level schema syntax. SET odps.namespace.schema=true; -- The query in this example is for the tpcds_10g dataset. To run queries on other datasets, replace the schema name with the corresponding name. SELECT * FROM bigdata_public_dataset.tpcds_10g.store_sales limit 100;The following result is returned:
+-----------------+-----------------+------------+----------------+-------------+-------------+------------+-------------+-------------+------------------+-------------+-------------------+---------------+----------------+---------------------+--------------------+-----------------------+-------------------+------------+---------------+-------------+---------------------+---------------+ | ss_sold_date_sk | ss_sold_time_sk | ss_item_sk | ss_customer_sk | ss_cdemo_sk | ss_hdemo_sk | ss_addr_sk | ss_store_sk | ss_promo_sk | ss_ticket_number | ss_quantity | ss_wholesale_cost | ss_list_price | ss_sales_price | ss_ext_discount_amt | ss_ext_sales_price | ss_ext_wholesale_cost | ss_ext_list_price | ss_ext_tax | ss_coupon_amt | ss_net_paid | ss_net_paid_inc_tax | ss_net_profit | +-----------------+-----------------+------------+----------------+-------------+-------------+------------+-------------+-------------+------------------+-------------+-------------------+---------------+----------------+---------------------+--------------------+-----------------------+-------------------+------------+---------------+-------------+---------------------+---------------+ | NULL | NULL | 39073 | NULL | 1420876 | 1738 | 56600 | NULL | NULL | 41171 | 90 | 53.3 | NULL | 72.87 | 0 | NULL | 4797 | 7626.6 | 459.08 | 0 | NULL | NULL | NULL | | NULL | NULL | 22434 | 98163 | NULL | NULL | NULL | 1 | NULL | 8909 | NULL | 15.22 | NULL | 9.2 | NULL | 690 | NULL | 1380.75 | NULL | NULL | NULL | NULL | -451.5 | | NULL | NULL | 82219 | NULL | NULL | 1572 | 209531 | 38 | 285 | 14907 | 48 | 84.64 | 132.03 | NULL | 0 | NULL | NULL | NULL | 51.96 | 0 | NULL | 2650.2 | -1464.48 | | NULL | NULL | 97573 | 214533 | 1298744 | NULL | NULL | NULL | 77 | 26167 | NULL | 92.55 | 143.45 | 91.8 | 0 | 8353.8 | NULL | NULL | NULL | 0 | NULL | NULL | -68.25 | | NULL | NULL | 60120 | 376494 | NULL | 1678 | 13917 | NULL | NULL | 35953 | 9 | 46.97 | NULL | NULL | NULL | NULL | NULL | 714.33 | NULL | NULL | NULL | NULL | 34.38 | | NULL | NULL | 85927 | NULL | NULL | NULL | 22622 | 19 | 122 | 3 | NULL | NULL | 130.66 | NULL | NULL | 3104.46 | NULL | 3527.82 | 124.17 | NULL | NULL | NULL | NULL | | NULL | NULL | 25498 | NULL | 1504134 | 3745 | 96156 | 58 | NULL | 14914 | 63 | NULL | NULL | 0.68 | 10.71 | 42.84 | NULL | 148.05 | NULL | 10.71 | 32.13 | 35.02 | NULL | | NULL | NULL | 62918 | 272574 | 1120896 | 6818 | NULL | NULL | NULL | 14915 | 34 | 18.81 | NULL | NULL | 0 | NULL | NULL | NULL | NULL | 0 | 1240.66 | NULL | 601.12 | | NULL | NULL | 82088 | 496000 | 1330049 | NULL | 204594 | 67 | NULL | 14902 | 50 | NULL | 1.46 | 1.09 | 0 | 54.5 | 62.5 | 73 | 4.36 | 0 | NULL | NULL | NULL | | NULL | NULL | 97843 | 377826 | NULL | 6088 | NULL | 2 | NULL | 7327 | 38 | 73.11 | 97.96 | NULL | NULL | 1600.56 | 2778.18 | 3722.48 | NULL | NULL | 1600.56 | 1728.6 | NULL | | NULL | NULL | 58623 | NULL | NULL | 7029 | 14784 | NULL | 94 | 26170 | NULL | NULL | NULL | NULL | 0 | 1879 | 4202 | 4370 | 150.32 | 0 | 1879 | NULL | NULL | | NULL | NULL | 87829 | 57817 | NULL | 6906 | 28874 | NULL | 378 | 26171 | 22 | 60.67 | NULL | NULL | 118.9 | 990.88 | 1334.74 | NULL | 26.15 | 118.9 | 871.98 | 898.13 | -462.76 | | NULL | NULL | 48322 | NULL | 547185 | 4104 | NULL | NULL | NULL | 31046 | NULL | NULL | 9.15 | 4.39 | 0 | 351.2 | NULL | 732 | NULL | 0 | NULL | 365.24 | NULL | | NULL | NULL | 36184 | 232464 | 314914 | NULL | NULL | NULL | 121 | 20656 | 24 | 53.29 | NULL | NULL | NULL | NULL | 1278.96 | NULL | NULL | NULL | 95.04 | NULL | -1183.92 | | NULL | NULL | 80738 | NULL | 169026 | NULL | 109667 | 38 | 19 | 20655 | 35 | 26.46 | 43.12 | NULL | NULL | 422.45 | 926.1 | NULL | 25.34 | NULL | NULL | NULL | NULL | | NULL | NULL | 46490 | 343482 | NULL | NULL | NULL | 38 | NULL | 20655 | NULL | NULL | NULL | NULL | 0 | NULL | 6275.55 | NULL | 68.27 | 0 | NULL | 6895.47 | 551.65 | | NULL | NULL | 17958 | 206463 | NULL | NULL | NULL | 25 | NULL | 20654 | NULL | 39.8 | NULL | 6.2 | 57.04 | 248 | 1592 | 3104.4 | 3.81 | 57.04 | NULL | 194.77 | -1401.04 | | NULL | NULL | 62707 | NULL | 1903460 | 6988 | 42469 | NULL | NULL | 42570 | 38 | 60.75 | 91.12 | NULL | 0 | 380.76 | 2308.5 | NULL | NULL | 0 | NULL | 407.41 | NULL | | NULL | NULL | 27548 | 343415 | 1289219 | 6223 | NULL | NULL | NULL | 20659 | NULL | NULL | 45.98 | 2.29 | NULL | 43.51 | NULL | 873.62 | 3.91 | NULL | NULL | NULL | -502.55 | | NULL | NULL | 53258 | 40152 | NULL | 3046 | 59693 | 16 | 72 | 26160 | 34 | 88.78 | 166.01 | 154.38 | 0 | NULL | 3018.52 | NULL | NULL | 0 | NULL | NULL | NULL | | NULL | NULL | 15001 | 353029 | NULL | 2092 | NULL | NULL | 68 | 6 | NULL | NULL | NULL | NULL | NULL | 449.88 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | | NULL | NULL | 50496 | 451224 | 963571 | 5071 | NULL | 46 | NULL | 20660 | NULL | 55.31 | 110.06 | 93.55 | 0 | NULL | 442.48 | NULL | 67.35 | 0 | 748.4 | 815.75 | 305.92 | | NULL | NULL | 66573 | 384060 | NULL | NULL | NULL | NULL | NULL | 31043 | NULL | NULL | 55.36 | 3.32 | NULL | 185.92 | 2000.32 | 3100.16 | NULL | NULL | 185.92 | 189.63 | -1814.4 | | NULL | NULL | 90182 | 154742 | 1356667 | NULL | 124517 | NULL | 42 | 14895 | 76 | NULL | NULL | 9.09 | NULL | 690.84 | NULL | 3839.52 | NULL | NULL | 200.35 | NULL | -1993.77 | | NULL | NULL | 89989 | NULL | NULL | NULL | NULL | NULL | NULL | 48982 | NULL | 3.15 | 5.85 | NULL | 0 | 414.96 | NULL | 532.35 | 16.59 | 0 | 414.96 | NULL | NULL | | NULL | NULL | 61099 | 104451 | 71288 | 6925 | 174515 | NULL | NULL | 10 | 92 | NULL | NULL | NULL | NULL | 2092.08 | 2280.68 | 3033.24 | NULL | NULL | 2092.08 | 2280.36 | -188.6 | | NULL | NULL | 92095 | 239093 | 281730 | 2767 | NULL | NULL | NULL | 48978 | 92 | 17.62 | 22.55 | 6.99 | NULL | 643.08 | 1621.04 | NULL | 19.29 | NULL | 643.08 | NULL | NULL | | NULL | NULL | 24430 | NULL | 54906 | 2936 | NULL | 44 | 488 | 35961 | 87 | 38.78 | NULL | 12.92 | NULL | 1124.04 | NULL | 3406.92 | 22.48 | NULL | NULL | 1146.52 | -2249.82 | | NULL | NULL | 84283 | 108617 | 777596 | 6575 | 56754 | NULL | NULL | 14891 | 91 | NULL | NULL | NULL | 0 | 2110.29 | NULL | 3297.84 | 147.72 | 0 | NULL | NULL | 192.92 | | NULL | NULL | 90794 | NULL | 281730 | 2767 | 125004 | 13 | 28 | 48978 | NULL | NULL | 77.47 | NULL | NULL | NULL | 3901.8 | NULL | NULL | NULL | NULL | NULL | NULL | | NULL | NULL | 20656 | 63574 | 324806 | 1238 | 59623 | 67 | NULL | 14890 | 67 | NULL | 91.86 | 27.55 | 0 | NULL | 3140.29 | 6154.62 | NULL | 0 | NULL | 1919.68 | -1294.44 | | NULL | NULL | 54828 | NULL | 781292 | NULL | 72923 | 56 | 246 | 11 | NULL | NULL | NULL | NULL | 0 | NULL | 4141.28 | NULL | 483.99 | 0 | NULL | 5861.67 | 1236.4 | | NULL | NULL | 53799 | NULL | NULL | NULL | 183758 | NULL | NULL | 14888 | 91 | NULL | NULL | 0 | NULL | 0 | NULL | 1955.59 | 0 | NULL | 0 | 0 | NULL | | NULL | NULL | 90484 | 40713 | 717863 | NULL | NULL | NULL | NULL | 14887 | NULL | 15.64 | 17.82 | NULL | NULL | NULL | 641.24 | NULL | NULL | NULL | 34.17 | 36.56 | NULL | | NULL | NULL | 71720 | NULL | 1835166 | NULL | 192536 | 49 | 99 | 35968 | 2 | 47.38 | NULL | NULL | NULL | NULL | NULL | 154.44 | NULL | NULL | NULL | NULL | 30.32 | | NULL | NULL | 30862 | 196636 | 16807 | 6131 | NULL | 25 | 376 | 14880 | 72 | 73.28 | NULL | 78.08 | 0 | 5621.76 | 5276.16 | 5856.48 | 337.3 | 0 | 5621.76 | NULL | 345.6 | | NULL | NULL | 66734 | 378535 | 519961 | NULL | 206013 | 31 | NULL | 35969 | 28 | NULL | 66.74 | NULL | NULL | 1625.68 | NULL | 1868.72 | NULL | NULL | 292.63 | NULL | -1133.97 | | NULL | NULL | 88849 | 267814 | 1705234 | 2301 | 176521 | NULL | NULL | 35948 | 69 | NULL | 7.65 | 5.2 | 0 | NULL | 527.85 | NULL | NULL | 0 | NULL | NULL | NULL | | NULL | NULL | 92298 | 440725 | NULL | 3376 | 102125 | NULL | NULL | 35970 | 80 | NULL | 60.55 | NULL | 847.17 | NULL | 2647.2 | NULL | NULL | 847.17 | 1720.03 | NULL | -927.17 | | NULL | NULL | 7184 | 51104 | NULL | NULL | 138239 | NULL | 169 | 19 | 49 | 61.82 | 110.03 | NULL | NULL | 1401.4 | NULL | 5391.47 | NULL | NULL | NULL | NULL | NULL | | NULL | NULL | 88454 | 197060 | NULL | NULL | 198345 | 26 | 20 | 14925 | 51 | NULL | 95 | 32.3 | 0 | 1647.3 | 3318.57 | 4845 | 0 | 0 | NULL | NULL | NULL | | NULL | NULL | 60380 | NULL | NULL | NULL | 221719 | NULL | NULL | 14926 | 86 | NULL | 24.16 | 2.17 | NULL | NULL | 1855.88 | 2077.76 | NULL | NULL | 186.62 | 201.54 | -1669.26 | | NULL | NULL | 8539 | 248260 | 1547699 | 3138 | 147377 | NULL | 444 | 14927 | NULL | NULL | 6.62 | 2.97 | 0 | NULL | 512.64 | 635.52 | NULL | 0 | 285.12 | 296.52 | NULL | | NULL | NULL | 93595 | 237437 | 1834495 | NULL | 149811 | NULL | 442 | 14929 | 67 | 41.68 | 65.43 | NULL | NULL | NULL | 2792.56 | NULL | NULL | NULL | 1972.48 | NULL | NULL | | NULL | NULL | 15248 | 418743 | NULL | NULL | 204342 | NULL | 100 | 14930 | NULL | 82.42 | NULL | NULL | NULL | 3345 | 8242 | NULL | NULL | NULL | 3345 | 3545.7 | NULL | | NULL | NULL | 95747 | 62791 | NULL | 711 | 51271 | NULL | 38 | 35946 | 86 | NULL | NULL | 52.85 | NULL | NULL | 2732.22 | 4835.78 | 90.9 | NULL | 4545.1 | NULL | 1812.88 | | NULL | NULL | 41587 | NULL | 877829 | 2403 | 132977 | 74 | 255 | 14868 | NULL | NULL | 112.3 | 69.62 | NULL | NULL | NULL | 336.9 | NULL | NULL | NULL | NULL | 31.53 | | NULL | NULL | 72620 | NULL | 936772 | NULL | NULL | 76 | NULL | 35945 | NULL | NULL | 99.91 | NULL | NULL | NULL | 1189.5 | 1498.65 | 6.87 | NULL | 229.28 | 236.15 | NULL | | NULL | NULL | 61078 | NULL | 1695076 | NULL | 116906 | 2 | NULL | 31042 | 2 | 91.01 | 148.34 | 10.38 | 0 | NULL | NULL | 296.68 | NULL | 0 | NULL | NULL | -161.26 | | NULL | NULL | 26476 | 333344 | 221814 | 3084 | 60801 | NULL | NULL | 14933 | 83 | NULL | NULL | NULL | NULL | 2583.79 | NULL | NULL | NULL | NULL | NULL | NULL | 168.49 | | NULL | NULL | 13849 | NULL | 1026254 | NULL | 142191 | 28 | 136 | 26152 | 65 | NULL | 29.44 | NULL | NULL | 95.55 | NULL | 1913.6 | 7.64 | NULL | NULL | NULL | -1010.75 | | NULL | NULL | 60799 | NULL | 96041 | NULL | NULL | NULL | 93 | 26151 | 6 | 30.56 | NULL | 28.21 | 28.77 | NULL | 183.36 | NULL | 11.23 | 28.77 | 140.49 | NULL | -42.87 | | NULL | NULL | 69529 | 325046 | 1633417 | 2141 | NULL | NULL | NULL | 35978 | 51 | 72.42 | 83.28 | 16.65 | 0 | NULL | NULL | 4247.28 | 76.42 | 0 | NULL | 925.57 | -2844.27 | | NULL | NULL | 17822 | NULL | NULL | NULL | 199677 | 76 | 320 | 42573 | NULL | 71.23 | 129.63 | NULL | NULL | NULL | NULL | 8166.69 | NULL | NULL | NULL | 3464.01 | -1057.77 | | NULL | NULL | 24737 | 212139 | NULL | 3208 | 199719 | NULL | NULL | 14934 | NULL | 68.23 | 68.91 | 33.76 | 0 | 1519.2 | NULL | NULL | NULL | 0 | 1519.2 | 1640.73 | -1551.15 | | NULL | NULL | 10439 | 292855 | 500497 | 492 | 24534 | 76 | NULL | 42569 | 74 | 20.32 | 29.87 | NULL | 0 | NULL | NULL | 2210.38 | NULL | 0 | NULL | 1347.64 | -244.2 | | NULL | NULL | 41249 | 495385 | NULL | 4103 | NULL | 7 | NULL | 14866 | NULL | NULL | 25.6 | 16.89 | 0 | 928.95 | 1235.3 | 1408 | 55.73 | 0 | NULL | 984.68 | NULL | | NULL | NULL | 53629 | 426176 | 932143 | NULL | NULL | 14 | NULL | 48975 | 83 | 57.65 | 60.53 | 38.73 | 0 | 3214.59 | NULL | NULL | NULL | 0 | NULL | NULL | -1570.36 | | NULL | NULL | 79070 | NULL | 1718513 | NULL | NULL | 56 | 380 | 42568 | NULL | NULL | 37.03 | NULL | NULL | 675.99 | NULL | NULL | NULL | NULL | NULL | NULL | -1720.91 | | NULL | NULL | 75323 | 338864 | 701453 | 4042 | 93784 | 50 | NULL | 31038 | 85 | NULL | 13 | NULL | NULL | NULL | 884 | 1105 | 19.89 | NULL | NULL | NULL | NULL | | NULL | NULL | 34640 | NULL | NULL | NULL | 17559 | 40 | NULL | 48970 | NULL | 16.53 | 23.14 | 9.95 | NULL | 298.5 | NULL | 694.2 | 2.98 | NULL | NULL | 301.48 | -197.4 | | NULL | NULL | 88971 | NULL | 1568794 | NULL | NULL | 76 | 307 | 7333 | NULL | NULL | 79.74 | 76.55 | 0 | 153.1 | 89.1 | 159.48 | NULL | 0 | 153.1 | NULL | 64 | | NULL | NULL | 77393 | 430396 | 269868 | NULL | 67306 | NULL | 312 | 14862 | NULL | NULL | 175.44 | 3.5 | NULL | 129.5 | 3245.64 | 6491.28 | NULL | NULL | 129.5 | 134.68 | -3116.14 | | NULL | NULL | 31636 | 306407 | 72705 | 1756 | NULL | NULL | 198 | 14940 | 86 | NULL | 146.54 | NULL | 0 | NULL | 7779.56 | 12602.44 | NULL | 0 | 8947.44 | 9394.81 | NULL | | NULL | NULL | 7834 | NULL | 477126 | NULL | 100475 | NULL | NULL | 35984 | NULL | 39.43 | NULL | NULL | NULL | 151.9 | NULL | NULL | 13.67 | NULL | NULL | 165.57 | NULL | | NULL | NULL | 60914 | 464049 | NULL | NULL | NULL | 82 | NULL | 7320 | NULL | NULL | NULL | NULL | NULL | NULL | 3214.17 | 4242.25 | 31.8 | NULL | 1060.03 | NULL | -2154.14 | | NULL | NULL | 21651 | NULL | 1419300 | NULL | NULL | 68 | NULL | 14860 | NULL | 11.18 | NULL | NULL | 0 | 400.64 | NULL | 500.8 | NULL | 0 | NULL | 404.64 | 42.88 | | NULL | NULL | 100164 | 342185 | 564208 | 3478 | 140701 | 8 | 244 | 42557 | 95 | NULL | 26.97 | NULL | 563.35 | NULL | 2373.1 | 2562.15 | 0 | 563.35 | NULL | 0 | NULL | | NULL | NULL | 85194 | NULL | 564208 | 3478 | 140701 | NULL | 292 | 42557 | NULL | 66.43 | 116.25 | 4.65 | 0 | 311.55 | NULL | NULL | 24.92 | 0 | NULL | NULL | NULL | | NULL | NULL | 90834 | NULL | 564208 | NULL | NULL | NULL | NULL | 42557 | 89 | NULL | NULL | NULL | NULL | 402.28 | 2338.03 | NULL | NULL | NULL | 366.08 | 377.06 | -1971.95 | | NULL | NULL | 5084 | NULL | NULL | NULL | 63654 | NULL | 376 | 7336 | 69 | 86.04 | NULL | NULL | 0 | 5479.29 | 5936.76 | 8429.73 | 383.55 | 0 | 5479.29 | NULL | NULL | | NULL | NULL | 81499 | NULL | NULL | NULL | 151451 | 58 | NULL | 20652 | 48 | NULL | 64 | NULL | NULL | NULL | NULL | 3072 | 0 | NULL | NULL | 2181.12 | 261.12 | | NULL | NULL | 676 | 473523 | NULL | 4509 | 94755 | NULL | NULL | 35937 | 30 | 78.49 | 89.47 | NULL | 6.94 | NULL | 2354.7 | NULL | NULL | 6.94 | 46.46 | NULL | -2308.24 | | NULL | NULL | 60001 | NULL | 446072 | 4353 | 238234 | NULL | NULL | 26149 | 7 | NULL | 100.81 | 24.19 | NULL | NULL | 555.66 | NULL | NULL | NULL | NULL | 184.56 | NULL | | NULL | NULL | 57841 | 90177 | 1130369 | 2114 | NULL | 44 | 460 | 31036 | 94 | NULL | 30.51 | NULL | NULL | NULL | 1899.74 | 2867.94 | 0 | NULL | NULL | NULL | NULL | | NULL | NULL | 50752 | 55142 | NULL | NULL | 71154 | 20 | 85 | 35933 | NULL | NULL | 112.29 | NULL | 0 | NULL | 5452.44 | 9432.36 | 207.48 | 0 | NULL | NULL | -1302.84 | | NULL | NULL | 7108 | NULL | NULL | 5423 | 243298 | NULL | 84 | 14852 | NULL | NULL | 31.3 | NULL | 0 | 1516.4 | NULL | 2660.5 | NULL | 0 | NULL | NULL | NULL | | NULL | NULL | 3949 | NULL | NULL | NULL | NULL | 94 | NULL | 14851 | NULL | NULL | 88.44 | NULL | 1010.02 | NULL | 4267.86 | 7340.52 | NULL | 1010.02 | 1338.88 | NULL | NULL | | NULL | NULL | 23335 | 374794 | NULL | NULL | NULL | NULL | 89 | 35932 | NULL | NULL | NULL | NULL | 0 | NULL | NULL | 5926.14 | NULL | 0 | NULL | 6103.92 | NULL | | NULL | NULL | 43855 | 456448 | NULL | 2939 | 246408 | NULL | NULL | 14952 | NULL | 35.59 | 70.46 | NULL | 621.18 | 887.4 | NULL | 3170.7 | 0 | 621.18 | NULL | NULL | NULL | | NULL | NULL | 2896 | 147971 | NULL | 6773 | 180574 | 67 | 486 | 34 | 20 | 22.82 | NULL | 6.74 | NULL | 134.8 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | | NULL | NULL | 3829 | 302788 | NULL | 1341 | NULL | NULL | NULL | 48959 | 30 | 27.05 | 32.46 | NULL | NULL | NULL | 811.5 | NULL | NULL | NULL | 369.9 | 369.9 | NULL | | NULL | NULL | 35413 | 477124 | NULL | NULL | NULL | NULL | 381 | 48957 | 2 | 13.73 | 19.63 | NULL | 0 | NULL | 27.46 | 39.26 | 1.39 | 0 | 34.94 | NULL | 7.48 | | NULL | NULL | 15138 | NULL | NULL | NULL | 123403 | NULL | 66 | 14849 | 94 | 40.87 | 47 | 13.63 | NULL | NULL | 3841.78 | NULL | NULL | NULL | NULL | 96.86 | -3752.09 | | NULL | NULL | 26168 | NULL | NULL | 2171 | NULL | 26 | NULL | 38 | 2 | NULL | NULL | NULL | 0 | NULL | 190.42 | 239.92 | 6.38 | 0 | 91.16 | 97.54 | NULL | | NULL | NULL | 100653 | 426136 | NULL | 2033 | NULL | NULL | NULL | 35924 | 64 | 9.07 | 14.33 | 5.15 | 0 | NULL | 580.48 | 917.12 | 26.36 | 0 | NULL | 355.96 | NULL | | NULL | NULL | 10705 | NULL | NULL | NULL | 203478 | 46 | 293 | 48955 | 65 | 32.09 | 44.28 | NULL | 0 | 1927.9 | NULL | 2878.2 | NULL | 0 | NULL | NULL | NULL | | NULL | NULL | 70316 | 366489 | NULL | 2233 | 249705 | 16 | 58 | 40 | NULL | 38.12 | NULL | 62.75 | 0 | NULL | NULL | 3522.2 | 103.53 | 0 | NULL | 3554.78 | NULL | | NULL | NULL | 28379 | NULL | NULL | 6216 | NULL | 43 | 254 | 35989 | NULL | 13.27 | NULL | NULL | 0 | NULL | 1035.06 | 1345.5 | NULL | 0 | 349.44 | 363.41 | NULL | | NULL | NULL | 55728 | NULL | 590996 | NULL | NULL | 79 | NULL | 26181 | 94 | NULL | 182.78 | NULL | 0 | 15118.96 | NULL | NULL | NULL | 0 | 15118.96 | NULL | NULL | | NULL | NULL | 62502 | 86490 | NULL | NULL | 216915 | NULL | 91 | 26182 | NULL | NULL | NULL | NULL | NULL | 133.92 | NULL | NULL | 0 | NULL | NULL | 133.92 | NULL | | NULL | NULL | 72087 | NULL | NULL | 3278 | NULL | 14 | NULL | 14955 | 61 | NULL | 104.28 | 32.32 | NULL | 1971.52 | 4448.73 | NULL | NULL | NULL | 1971.52 | 2010.95 | NULL | | NULL | NULL | 40093 | 369422 | NULL | 1668 | NULL | 55 | NULL | 14844 | NULL | 84.1 | NULL | NULL | 0 | 2834.52 | NULL | NULL | 0 | 0 | NULL | 2834.52 | NULL | | NULL | NULL | 54979 | 171726 | 577101 | NULL | 40251 | 40 | NULL | 48952 | NULL | 1.1 | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 11.92 | -2.16 | | NULL | NULL | 16181 | NULL | NULL | 6319 | NULL | NULL | NULL | 31035 | NULL | 82.72 | NULL | 62.86 | 0 | NULL | 2564.32 | 2564.32 | 97.43 | 0 | NULL | NULL | -615.66 | | NULL | NULL | 18379 | 445761 | NULL | 5529 | NULL | NULL | NULL | 31052 | NULL | 85.25 | NULL | 33.03 | NULL | NULL | 7587.25 | 11759.57 | NULL | NULL | 2939.67 | 2998.46 | -4647.58 | | NULL | NULL | 72566 | NULL | NULL | 877 | 60361 | NULL | 138 | 14841 | 1 | NULL | 12.11 | 0.6 | NULL | NULL | 7.43 | 12.11 | 0.03 | NULL | 0.6 | NULL | -6.83 | | NULL | NULL | 26398 | NULL | 1781928 | NULL | 18588 | 16 | 128 | 20668 | 64 | NULL | 40.32 | NULL | NULL | NULL | NULL | 2580.48 | NULL | NULL | NULL | NULL | 98.75 | | NULL | NULL | 55346 | NULL | 1781928 | NULL | NULL | 16 | 96 | 20668 | NULL | 65.99 | NULL | 34.1 | 0 | NULL | 3761.43 | NULL | NULL | 0 | 1943.7 | 1982.57 | -1817.73 | | NULL | NULL | 48684 | NULL | 827096 | 1660 | NULL | 68 | NULL | 31056 | NULL | NULL | NULL | NULL | NULL | 3721.74 | 4372.5 | NULL | 260.52 | NULL | NULL | 3982.26 | NULL | +-----------------+-----------------+------------+----------------+-------------+-------------+------------+-------------+-------------+------------------+-------------+-------------------+---------------+----------------+---------------------+--------------------+-----------------------+-------------------+------------+---------------+-------------+---------------------+---------------+
Supported tools and platforms
Appendix: Query examples and analysis
Explore open source databases
Top 10 popular database projects (2015-2022)
query example:
SET odps.namespace.schema = TRUE; SELECT dws.repo_id AS repo_id, repos.name AS repo_name, SUM(dws.stars) AS stars FROM bigdata_public_dataset.github_events.dws_overview_by_repo_month dws JOIN bigdata_public_dataset.github_events.db_repos repos ON dws.repo_id = repos.id WHERE MONTH >= '2015-01' AND MONTH <='2022-12' GROUP BY dws.repo_id, repos.name ORDER BY stars DESC LIMIT 10;Analysis results: In the past five years,
elasticsearchhas been the most popular open source database project, followed byredisandprometheus. After you execute the query, in the Execution Results area, select Stacked Bar Chart, set the X-axis torepo_name, and the Y-axis tosum(stars). The chart shows the top 10 repositories by stars from 2015 to 2022: elastic/elasticsearch (approximately 60000), redis/redis (approximately 55000), prometheus/prometheus (approximately 50000), etcd-io/etcd (approximately 42000), apache/spark (approximately 40000), pingcap/tidb (approximately 35000), clickhouse/clickhouse (approximately 35000), cockroachdb/cockroach (approximately 30000), influxdata/influxdb (approximately 28000), and facebook/rocksdb (approximately 27000).
Ranking changes for top 10 databases (2018-2022)
query example:
SET odps.namespace.schema = TRUE; SET odps.sql.validate.orderby.limit=FALSE; WITH tmp as (SELECT dws.repo_id AS repo_id, repos.name AS repo_name, SUM(dws.stars) AS stars, SUBSTR(MONTH,1,4) AS YEAR, row_number() over (partition BY SUBSTR(MONTH,1,4) ORDER BY SUM(dws.stars) DESC) AS ranknum FROM bigdata_public_dataset.github_events.dws_overview_by_repo_month dws JOIN bigdata_public_dataset.github_events.db_repos repos ON dws.repo_id = repos.id WHERE MONTH>='2018-01' GROUP BY dws.repo_id, repos.name,SUBSTR(MONTH,1,4)) SELECT repo_id, repo_name, stars, ranknum, YEAR FROM tmp WHERE YEAR<=2022 AND ranknum<=10 ORDER BY YEAR ASC,ranknum ASC;Analysis results: Over the past 5 years, the top 10 rankings of open-source databases have fluctuated among developers.
clickhousewas the fastest-rising project, climbing from tenth place in 2018 to first place in 2021, before being surpassed by the steadily risingredisin 2022. In contrast,taosdata/TDengineranked first in 2019 but dropped to ninth place in just one year. After running the query, in the data visualization panel, select line chart as the chart type. Set the X-axis toyear, the Y-axis toranking, and the breakdown field torepo_nameto view the star ranking trends from 2018 to 2022 for repositories such as elastic/elasticsearch, prometheus/prometheus, redis/redis, and pingcap/tidb.
Monthly star growth of top 10 databases (2018-2022)
query example:
SET odps.namespace.schema = true; SET odps.sql.validate.orderby.LIMIT=false; WITH top_10_repos AS ( SELECT dws.repo_id AS repo_id, repos.name AS repo_name, SUM(dws.stars) AS stars FROM bigdata_public_dataset.github_events.dws_overview_by_repo_month dws JOIN bigdata_public_dataset.github_events.db_repos repos ON dws.repo_id = repos.id WHERE MONTH >= '2018-01' AND MONTH <='2022-12' GROUP BY dws.repo_id, repos.name ORDER BY stars DESC LIMIT 10 ), tmp AS ( SELECT MONTH, repo_id, stars, SUM(stars) OVER (PARTITION BY repo_id ORDER BY MONTH ASC ) AS total_stars FROM bigdata_public_dataset.github_events.dws_overview_by_repo_month WHERE MONTH >= '2015-01' AND stars is NOT null AND repo_id IN (SELECT repo_id FROM top_10_repos) GROUP BY repo_id,MONTH,stars ORDER BY MONTH ASC,repo_id) SELECT tmp.MONTH AS MONTH, top_10_repos.repo_name AS repo_name, tmp.total_stars AS total_stars FROM tmp JOIN top_10_repos ON top_10_repos.repo_id = tmp.repo_id GROUP BY MONTH,repo_name,total_stars ORDER BY MONTH ASC,repo_name ;Analysis results: Among the top 10 open source database projects,
elasticsearchhas consistently been the most popular, andclickhousehas grown rapidly since 2021.Select line chart. Set the X-axis to
month, the Y-axis tosum(total_stars), and the breakdown field torepo_name.
Most popular databases in H1 2023
query example:
set odps.namespace.schema = true; SELECT repos.name AS repo_name, sum(dws.stars) AS stars FROM bigdata_public_dataset.github_events.dws_overview_by_repo_month dws JOIN bigdata_public_dataset.github_events.db_repos repos ON repos.id = dws.repo_id WHERE month >= '2023-01' AND month <= '2023-06' GROUP BY repo_name ORDER BY stars DESC LIMIT 10;Analysis results: The
clickhousedatabase was the most popular in the first half of 2023, followed byprometheusandredis.
Most actively maintained databases in H1 2023
query example:
SET odps.namespace.schema = true; SELECT repos.name AS repo_name, COUNT(dwd.id) AS num FROM bigdata_public_dataset.github_events.dwd_github_events_odps dwd JOIN bigdata_public_dataset.github_events.db_repos repos ON repos.id = dwd.repo_id WHERE type = 'PullRequestEvent' AND ds>='2023-01-01' and ds<='2023-06-30' AND action = 'opened' GROUP BY repos.name ORDER BY num DESC LIMIT 10;Analysis results: In the first half of 2023,
StarRockswas the most actively maintained and updated project (where project activity is defined by the opening ofPull Requests).A grouped bar chart shows the number of pull requests (
sum(num)) for each repository. StarRocks/starrocks leads with 7,810, followed by apache/incubator-doris (approx. 4,100), elastic/elasticsearch (approx. 3,100), apache/spark (approx. 2,300), and MaterializeInc/materialize (approx. 2,000).
Main contributors to StarRocks/starrocks in 2021
query example:
SET odps.namespace.schema=true; WITH a AS ( SELECT repo_id ,repo_name ,actor_id ,actor_login ,COUNT(*) AS contribution ,ds FROM bigdata_public_dataset.github_events.dwd_github_events_odps WHERE ds >='2021-01-01' and ds<='2021-12-31' AND ( ( type = 'PullRequestEvent' AND action = 'opened' ) OR ( type = 'IssuesEvent' AND action = 'opened' ) OR ( type = 'IssueCommentEvent' AND action = 'created' ) OR ( type = 'PullRequestReviewEvent' AND action = 'created' ) OR ( type = 'PullRequestReviewCommentEvent' AND action = 'created' ) OR ( type = 'PushEvent' AND action IS NULL ) ) GROUP BY repo_id ,repo_name ,actor_id ,actor_login ,ds ) SELECT repo_name ,actor_login ,SUM(contribution) AS contribution FROM a WHERE repo_name = 'StarRocks/starrocks' AND actor_login NOT LIKE '%[bot]' AND actor_login NOT LIKE 'cockroach%' GROUP BY repo_name ,actor_login ORDER BY contribution DESC LIMIT 10 ;Analysis results: In the first half of 2023, the main individual contributor to the most active open source database project was
kangkaisen. After you execute the preceding SQL statement, use a data visualization tool to display the results as a pie chart. Set the slice toactor_loginand the value tosum(contribution). The query results show that the top 10 individual contributors to the StarRocks/starrocks project and their contribution percentages are: kangkaisen 22.69%, imay 13.51%, sduzh 9.81%, chaoyli 9.66%, dirtysalt 9.64%, Seaven 8.90%, satanson 8.21%, trueeyu 6.79%, Youngwb 5.54%, and stdpain 5.24%.
Explore programming languages
Top 10 used programming languages (past year)
query example:
SET odps.namespace.schema=true; SELECT language, count(*) total FROM bigdata_public_dataset.github_events.dwd_github_events_odps WHERE ds>=date_add(getdate(), -365) AND language IS NOT NULL GROUP BY language ORDER BY total DESC LIMIT 10;Analysis results: The most used programming language is
JavaScript, followed by languages such asTypeScriptandPython. After you run the query, select grouped bar chart in the data visualization tool, drag language to the X-axis, and set sum(total) as the Y-axis. The chart shows that the ten most used programming languages in the past year are, in descending order: JavaScript (approximately 27 million), TypeScript (approximately 25 million), Python (approximately 15 million), Java (approximately 12 million), Go (approximately 10 million), HTML (approximately 7.5 million), C++ (approximately 5 million), Ruby (approximately 4.5 million), C# (approximately 4 million), and Rust (approximately 4 million).
Top 10 popular language projects since 2015
query example:
SET odps.namespace.schema = TRUE; SELECT dws.repo_id AS repo_id, repos.name AS repo_name, SUM(dws.stars) AS stars FROM bigdata_public_dataset.github_events.dws_overview_by_repo_month dws JOIN bigdata_public_dataset.github_events.programming_language_repos repos ON dws.repo_id = repos.id WHERE MONTH >= '2015-01' GROUP BY dws.repo_id, repos.name ORDER BY stars DESC LIMIT 10;Analysis results: The most popular programming language over the past five years is
Go, with 81,642 stars, followed by languages such asTypeScriptandNode.The other top projects, ranked by star count in descending order, are rust-lang/rust, apple/swift, python/cpython, JetBrains/kotlin, PowerShell/PowerShell, JuliaLang/julia, and vlang/v.
Star growth for top 10 language projects (2018-2022)
query example:
SET odps.namespace.schema = true; SET odps.sql.validate.orderby.limit=false; WITH top_10_repos AS ( SELECT dws.repo_id AS repo_id, repos.name AS repo_name, SUM(dws.stars) AS stars FROM bigdata_public_dataset.github_events.dws_overview_by_repo_by_month dws JOIN bigdata_public_dataset.github_events.programming_language_repos repos ON dws.repo_id = repos.id WHERE month >= '2018-01' AND month <='2022-12' GROUP BY dws.repo_id, repos.name ORDER BY stars DESC LIMIT 10 ), tmp AS ( SELECT month, repo_id, stars, SUM(stars) OVER (PARTITION by repo_id ORDER BY month ASC ) AS total_stars FROM bigdata_public_dataset.github_events.dws_overview_by_repo_by_month WHERE month >= '2015-01' AND stars IS NOT NULL AND repo_id IN (SELECT repo_id FROM top_10_repos) GROUP BY repo_id,month,stars ORDER BY month ASC,repo_id) SELECT tmp.month AS month, top_10_repos.repo_name AS repo_name, tmp.total_stars AS total_stars FROM tmp JOIN top_10_repos ON top_10_repos.repo_id = tmp.repo_id GROUP BY month,repo_name,total_stars ORDER BY month ASC,repo_name ;Analysis results: In the past 5 years,
Gowas the fastest-growing programming language in popularity.