The original intention and persistence of open-source code setters in the cloud native era

Recently, Alibaba Cloud donated the Celeborn project (formerly the EMR Remote Shuffle Service project) to the Apache incubator, which is also the first Apache incubation project born on Alibaba Cloud. Alibaba Cloud's open source big data platform EMR can support running mainstream open source big data components in the industry on Alibaba Cloud, such as Hadoop, Spark, Flink, Kafka, etc. It provides a set of open source big data bucket services with very low threshold and easy use on the cloud.

Cloud native architecture and concepts are also being strengthened and promoted. For example, the separation of storage and computing architecture is a unique architecture attribute of the cloud. Under this technical background, we found that Hive, Spark, Flink, etc. all have a need for data shuffle. Therefore, Alibaba Cloud provides a remote shuffle service, which uses a set of data shuffles to support all big data computing engines.

This is how the cloud brings about changes. The combination of cloud and open source produces chemical reactions. After the birth of the Remote Shuffle Service project, many companies represented by and NetEase have been attracted to participate in the co construction, and the project will finally be open source in December 2021. In order to let more companies participate in the co construction and make the project more influential, we decided to donate it to the Apache Foundation. Starting from the incubator project, we also hope to become the top project of Apache.

The cloud provides elastic resources, so that users can run businesses and exert their computing power with great freedom, and run digital businesses, do data analysis, do databases, do AI, SaaS, PaaS and other businesses easily on the cloud. However, running open source software requires a lot of environments, hardware and parameter configuration, and there is a certain threshold.

The emergence of the cloud has promoted the rise of open source, because resources can be more easily obtained on the cloud. Cloud elasticity meets the needs of on-demand use and purchase, so open source software can also be easily deployed. The cloud becomes the operating base of open source software. For example, with EMR, 100 node clusters can be created within 3 minutes, Hadoop, Spark, Flink, Kafka and other mainstream open source components can be used, and a complete set of links such as operation and maintenance, deployment, management and control, monitoring, use, and development can be easily realized.

Cloud makes the operation of open source software more universal, and the relationship between cloud and open source also promotes the development of the entire digital economy and digital transformation. In addition, the cloud can provide PAI DSW consistent Notebook development experience. The cloud also provides a good business model for open source, which benefits more people. At the same time, build your own business model through open source, get more resources to feed the open source community, and form a positive cycle. In a word, running open source software on the cloud and providing open source software commercial services can make the cloud and open source work better, and also make open source work better.

Alibaba has provided great support for open source. It has 3000+open source projects, more than 30000 external contributors, and cooperates with more than 100 open source communities to jointly promote the development of open source projects and open source culture in the industry.

(The above figures are all cloud born and cloud grown open source projects)

Alibaba Lingjie AI open source family adopts open source style, and has achieved many implementations in different industries and fields, such as audio and video, automatic driving, search and recommendation, OCR character recognition, financial quantification, urban brain's intelligent traffic scheduling, etc.

The core engineering platform PAI machine learning platform also actively embraces open source technology, fully compatible with the industry's mainstream TensorFlow, PyTouch and other international standards, as well as PaddlePaddle, MindSpot, OneFlow and other domestic mainstream AI frameworks. At the same time, we have also achieved many optimizations, including Deep series, Blade series compilation optimization, AI algorithm optimization, algorithm framework optimization, AI engine optimization, AI compilation optimization, and many optimizations have also been fully open source. Ali's AI links run from the underlying basic platform to the upper application, all of which are based on the open source technology stack, and practice the open source culture in the whole process.

Flink has become the standard of real-time computing. Various domestic Internet companies, financial companies and traditional industry companies have used Flink extensively for real-time stream computing analysis, and many overseas companies have used Apache Flink for real-time analysis.

In 2015, Taobao faced great data challenges, and users constantly put forward new requirements for the shopping experience, such as the real-time update of goods/prices and the real-time update of personalized sorting models/recommendation models. Especially on the day of the Double 11 Festival, the operation strategy changed very quickly, so it was necessary to make real-time scheduling based on user feedback and market changes, calculate the real-time transaction volume, and adjust the operation strategy. Therefore, Alibaba needs its own real-time big data engine. After comprehensive consideration, we decided to embrace the open source community and use the open source concept to build the next generation real-time big data computing platform. Finally, we chose Flink, a fledgling Apache, as the core technology cornerstone for the next few years.

In 2017, we continued to firmly embrace open source, integrating the original JStom, Blink and Galaxy in Alibaba into open source projects, and using Flink as the future core engine for real-time computing. Since then, Alibaba has built all real-time big data analysis and streaming big data analysis around Flink. In 2017, Alibaba has gradually unified the real-time computing of the whole group, increased investment based on the Flink community, and put our fully accumulated capabilities on the cloud in 18 years to provide Flink based big data services on the cloud.

We began to invest in community building in 2016. In 2017 and 2018, we participated in Flink community conferences in Europe and the United States, constantly promoting the growth of Flink community in China. In 2019, Alibaba fully acquired the commercial company behind Flink to fully support community development. After two years of continuous investment from 2020 to 2021, Alibaba held nearly 100 online and offline Flink MeetUp and Flink Forward conferences in China, investing a lot of manpower, material resources and energy. Ali has always firmly promoted Flink without considering commercialization and returns, making it one of the best projects in the Apache community. More importantly, the achievements of these years can not be separated from the promotion of some open-source enthusiasts and practitioners of open-source culture.

At present, Flink community has more than 1500 developers worldwide, doubling in three years; The number of GitHub Stars is nearly 20000, which has increased three times in three years and is in the top position in the community.

Thanks to Alibaba's continuous investment, Flink's key indicators have reached the position of top 1 or top 2 in the community. Firmly investing in open source also proves that the Chinese team can play a role in international projects, which is a very proud thing.

Through continuous evolution and technological innovation, Flink community has developed from Java API to SQL API, which makes development simpler, lower barriers, and more people can use it; Later, CDC was introduced, which can better integrate data; Launched Flink ML to better cooperate with machine learning; We have introduced a table store to store streaming batch data, and are also building a next-generation streaming data warehouse architecture.

Alibaba has unreservedly opened its technological innovation to the community, benefiting more developers and hoping to attract more developers to promote community development.

Lingjie system is the open source representative of AI, Flink is the open source representative of big data, and SREWorks project is the precipitation of big data and AI operation and maintenance capability accumulated for more than 10 years, and it has also been open source. SREWorks can provide one-stop O&M service suite management. Delivery, O&M, management, monitoring, operation, data assets and other capabilities can be achieved through SREWorks open source projects.

Behind many open source projects are a group of employees and developers who have great enthusiasm for open source. We have unlimited passion. We will have regular open source technology sharing, including machine learning teams, big data teams, European and American teams, and other international cross team exchanges. There are community meetups, and exchanges with different companies. This is also a guarantee for the perfect implementation of open source projects.

At present, our team has participated in and contributed 10+Apache top open source projects, cultivated more than 50 top open source projects, committers and PMCs, and contributed 1.5 million lines of open source code. The team members really contribute their accumulated work and technology to the open source community and provide them for more companies and enterprises to use. We also hope that more people will join us to build an open source community.