How to open source database PolarDB

Li Feifei, vice president of Alibaba Group and head of Alibaba Cloud Database Business Department, attended the salon and delivered an opening speech: PolarDB is a star product of Alibaba Cloud. It takes great courage to make the decision to open up PolarDB. Open source the most core database products and use the most friendly protocols. Alibaba Cloud is the first global head cloud manufacturer. As a commercial company, commercialization must be the enterprise goal. However, we believe that commercialization should be a by-product of enterprises, and the core should still be to create value. When we create value for the society, value will be reflected in commercialization.

Ecological construction and open source under the trend of multi-cloud will greatly help Alibaba Cloud, and it is of great significance to consolidate the position of our cloud database bridgehead. As long as the main version does not split, this path will be successful. We hope to work with colleagues in the industry to strengthen and expand the open source community of PolarDB, so that PolarDB is not only the benchmark of Alibaba Cloud, but also the benchmark of databases in China and the world.

Wang Yuan, the head of Alibaba Cloud database open source and senior technical expert of Alibaba Cloud database, delivered a keynote speech on the current situation and future planning of PolarDB open source. Wang Yuan pointed out that the first principle Alibaba Cloud adheres to when making open source PolarDB is to be compatible with the database ecosystem. From the first day of open source, PolarDB has taken the compatibility ecology as its goal. The compatibility ecology includes not only the compatibility of syntax and SQL, but also the compatibility of experience. For example, PolarDB-X and PolarDB for PostgreSQL are cloud native distributed databases, but this does not mean that their experience will also reveal the complexity of distribution to customers. We hope that the cloud native database is as easy to use and easy to operate and maintain as the stand-alone database.

The second principle of open source PolarDB is to adhere to complete openness. The most distinctive feature of open source PolarDB is that it really opens 100% of its cloud products running on the cloud to customers. Alibaba Cloud's massive customers have tested the maturity of PolarDB products, so we open source the mature and available enterprise-level products of PolarDB. We hope that through open source, more users and technicians can enjoy this technology dividend and participate in the co-construction, so as to promote the development of PolarDB technology on the cloud. At the same time, we also hope that more users and partners can put forward more requirements to us and promote the continuous evolution of PolarDB.

Tang Cheng shared with the title "The Use of PolarDB in Wahaha". He shared that Wahaha has used PostgreSQL for many years, used a lot of logical replication, and the standby database only provides read-only services for some businesses. At the same time, the database of its important business runs on the shared SAN storage. Therefore, it has the pain points of large delay in primary and standby databases, unstable logical replication and large delay. For Wahaha's characteristics and pain points, we choose to replace its original architecture with PolarDB.

After using PolarDB, its architecture is a master node and a read-only node based on shared storage, and the CLup management system is implemented with PolarDB. The database runs on SAN shared storage and uses CLup to manage PolarDB. When the primary node has problems, the database can quickly switch to a read-only node. Later, we will add read-only nodes to facilitate rapid expansion. Another advantage of PolarDB is that it can add a common standby database, which is equivalent to storing multiple copies of data, ensuring that data is not lost. At the same time, the PolarDB will be backed up regularly using CLup.

Han Yi shared with the title of "Building a Data Base for Zhongzheng Intelligent Identity Authentication Business Based on Open Source PolarDB-X". He shared that Zhongzheng Intelligence is one of the world's leading biometric and identity authentication companies, focusing on image processing and pattern recognition for 20 years. At present, Zhongzheng Intelligence is deeply involved in the financial field. With the increase of business data volume, customers' requirements for business reliability have gradually increased, and the requirements for data security and storage have become increasingly stringent. In addition, we are also faced with the problem of problem solving delay and domestic transformation. We put forward four requirements for database selection: high availability, easy maintenance, low cost and sustainability. PolarDB has significant advantages in the above aspects.

So we created a multi-location distributed data base solution based on the open source PolarDB-X. We have achieved multiple activities in two places through the small data center in Hangzhou and the data center in Shenzhen. The bottom layer is built based on K8s container. After completing the PolarDB transformation, our read and write performance, storage performance and response latency have been significantly improved. Among them, TPS performance improved by 55.8%, response time increased by 46.6%, and business availability increased by 64.6%, which made Zhongzheng Intelligent fully recognized by customers and also established its foothold in the commercial field. The open source of PolarDB has brought huge benefits to the development of our small and medium-sized enterprises, which can be called "the light of domestic goods". We look forward to more excellent "domestic" products to be exported to help small and medium-sized enterprises overcome difficulties and occupy a place in the world's pillar middleware.

Next, Li Ruobing, the head of Alibaba Cloud developer community, as the host of the event, invited Li Feifei and Wang Yuan, the heads of Alibaba Cloud database business department, to have a dialogue with Han Yi, the founder of Multiplier Technology, Tang Cheng, the director of research and development of Zhongzheng Intelligent Technology Platform Software Department, and Wang Minjian, the founder of Hangzhou Pengyue, to discuss "open source of databases in the cloud era".

Moderator: When making the open source PolarDB decision, what is the focus of Alibaba Cloud's internal discussions and disputes?

Li Feifei: Alibaba Cloud is a cloud manufacturer, not a third-party database manufacturer, and has invested a huge amount of money and manpower to develop PolarDB for five consecutive years. The decision to open source will inevitably cause controversy. Therefore, open source PolarDB really needs to take huge risks.

But I believe that all changes in the world are accompanied by crises. There are no risks when things are stable, but there will be no opportunities. There are complex changes in the external world. We need to return to our original intention and adhere to value creation. As long as we persist in value creation and make contributions to the society, eventually both organizations and individuals will get corresponding rewards.

Moderator: PolarDB has been open source for more than a year. The full score is 10 points. How do you rate yourself?

Wang Yuan: I want to give the open source team 9 points. The operation of the open source community, the support of partners, and the R&D team have all performed well, especially in technology. Give yourself 7 points. I have higher expectations for open source PolarDB. It should have a broader space. At the same time, I also hope that the R&D team, users, developers and partners can work together to promote PolarDB further and make it a real national light.

Host: From the outside, how would you rate this product, the open source community or the ecological partners?

Wang Minjian: 8.5 points. I have been in contact with PostgreSQL open source database since about 2006, and I am also the first group of preachers in China. At that time, many people did not understand what open source was, whether it was free or not, and the subtle differences between various licensing agreements. The popularization of these knowledge required industry leaders or large factories to show their broad-mindedness and show their courage to break their necks, which was far beyond the reach of small factories and individuals.

With this vision and ambition, it is certain that in the future, it will be able to connect multiple clouds and chips, cover IaaS layer, DaaS layer, PaaS layer and SaaS layer, and form a complete ecological and full-stack data link. However, in the initial stage, it is necessary to have large factories as the leading enterprises to carry the banner. Therefore, I sincerely admire Alibaba Cloud's courage to open source PolarDB, which is due to its confidence and confidence.

Tang Cheng: I give it 9 points. Open source is a long-distance project. Many open source projects declined after a few months, while PolarDB has been open source for more than a year, and its popularity has not decreased, which also proves its success. Previously, domestic companies have been less open source, mostly for individuals. In recent years, large domestic enterprises have taken the responsibility to vigorously promote open source, lead the change of thinking, and also strive to practice open source. I believe that the domestic open source ecosystem will become more and more prosperous.

Han Yi: I gave it 9.9 points. As a user, I have seen and experienced the product power of PolarDB. After the PolarDB product is open source, small and medium-sized enterprises can use it in a short time, which means that the operation team and product team behind it have made great efforts.

Li Feifei: Alibaba Cloud data team has made great investment in PolarDB. Moreover, PoalrDB is a product developed by Alibaba Cloud for five years. It is indeed a very difficult decision to open source. But looking back now, this is definitely a correct decision.

The decision of open source can enable more partners to build with us and create the most dynamic cloud native database community in the world. This is a new track. In the field of cloud native database and storage computing separation, Alibaba Cloud is in the first open source echelon. Moreover, it is based on the existing and mature open source ecosystem, which has more first-mover advantages.

To sum up, I gave 8 points for the open source of PolarDB. I hope to take today as a new starting point, increase market investment and give partners more confidence. At the same time, we also solemnly promise that we will persevere in the open source path of PolarDB.

Moderator: What are the advantages of Alibaba Cloud's commercial products and open source products?

Li Feifei: Today, the interaction model and business model between customers and manufacturers have changed substantially. Commodity is no longer just a commodity, but also a carrier and an interactive smart store. After the open source of PolarDB, its surrounding ecosystem will become richer and richer, and with the development of partners, the whole system will really become as large as the Amazon rainforest. Among them, the development and mutual achievements of a long stream are far more valuable than simple transactions, and also let us have expectations for a richer form of business on the cloud.

Recently, PolarDB launched Tianyi Cloud's selected market. We expect that in the near future, PolarDB will truly become a product of Tianyi Cloud. At the same time, our cooperation with mobile cloud is also advancing. This kind of cooperation is officially facilitated by the open source of PolarDB, which brings us a new business model.

Moderator: As the first group of preachers of open source, how does Min Jian view this business model?

Wang Minjian: I very much agree with this business model. In my early years, I participated in the operation of some online platforms, hoping to realize the SaaS model. However, for a long time, few domestic companies have achieved success in SaaS mode, which is mainly due to the monopoly of traffic oligarchs. There is also a traffic oligopoly effect in the open source industry. Therefore, the brand giants' early development of open source is bound to be of great benefit to the development of enterprises themselves, because they naturally have brand strength and technical strength. In terms of business ecology, as long as there are enough users, it is only a matter of time to realize a new business model.

TV can also be changed from the original black and white TV to the smart store with interactive functions such as subscription, and the charming cloud native database is beyond doubt. It can be used for operation and maintenance ecology, monitoring ecology, replication ecology, migration ecology, BI report ecology, and so on. All kinds of ecology are blooming, and a hundred schools of thought contend. As long as there are users, rich ecology will come naturally.

Moderator: As an ecological partner, the business model is equally important to you. Mr. Tang, what do you think of this business model?

Tang Cheng: A large number of customers want to use a lighter and more flexible way. At the same time, many customers cannot access the public cloud for various reasons. Therefore, the traditional business model is no longer applicable. Many open source products have been made into ecosystems abroad, such as database plug-in ecosystem, but this model is still relatively backward in China.

The reasons for backwardness include not only technical constraints, but also business logic. The traditional way of covering users needs to invest a lot of promotion costs, and open source is another way to attract users. After open source, with the expansion of community influence, it will attract more customers. With customers, business models will become more and more abundant, such as subscription services and remote services, and eventually become sustainable development models.

Moderator: Excuse me, Mr. Wang, what more preparations should be made for the operation, market, product, service and mechanism of PolarDB open source in order to better promote the launch of the new business model?

Wang Yuan: I think that open source and business model complement each other and do not conflict. Open source is a modern and market-oriented means. At present, the trend of entrepreneurship is very obvious. Many small companies need the support of infrastructure and basic software.

The first step for small companies to select models is to quickly obtain information, such as product information through open source, internet or social media. The second step is to quickly experience and verify. However, if we are faced with commercial products at this time, it means that we need to invest capital, which will also become a difficult threshold for many small companies. The third step is to bundle and verify with the business. Many start-ups are innovative in their businesses and have more consideration for the openness of their products. For example, can they add their own features to their products or combine them with self-developed software systems? Open source products have well met such demands. With the growth of startups, it is particularly important that the scalability and availability of products can meet the demands of high concurrency.

One of the attractions of open source products for startups is that they provide self-service capabilities. As long as its document system is good enough to enable users to quickly find the information they need, users can easily expand the product according to their needs.

I think that the subsequent development of the PolarDB open source community needs to adhere to several principles: first, the dissemination is wide enough, and only more people can use it to make the database develop better. A good database is not developed but used. It should grow with partners. Second, ensure the ease of use of the product and enable users to quickly verify their views. Third, the document system is perfect enough, the technical support of the open source community is strong enough, the community and various groups are active enough, and the channels for answering questions and solving problems are smooth enough to enhance users' confidence in using the open source PolarDB.

Moderator: As a customer of PolarDB, did President Han have any internal disputes when he first decided to use PolarDB? What is the focus of controversy?

Han Yi: The database product is a very low level part of the three middleware. Its migration difficulty is not only to overcome technical problems, but also to change the customer's usage habits. In addition, there are certain risks and costs in switching stable products to new databases. But at that time, we were faced with business bottlenecks. Since we had to step out of this step, it should be sooner rather than later. Second, since PolarDB was open source for a year, its performance has been verified by a large number of customers, confirming its vitality and product strength in the market. Moreover, from the technical architecture of open source PolarDB, we can see that it is definitely the future of database. However, we also believe that there is hope only after we have not gone. Finally, we decided to use PolarDB to build the next generation of business products for enterprises.

Host: What is the development trend of database technology in the future?

Wang Yuan: As an IT practitioner, I think cloud computing is the future development. Secondly, the future IT infrastructure will be transformed to cloud computing, so the future of database is cloud native.

From the history of database, we can see that the changes of database architecture and core technology are closely related to the deployment environment from stand-alone version to distributed version to cloud. For example, the BufferPool management and dirty cleaning technologies are derived from the unmatched speed of the stand-alone memory, CPU and hard disk. But on the cloud, resource pooling and resource decoupling have changed the relationship between computing resources, network resources and storage resources. Under the changing trend, database technology must have corresponding changes in order to develop in the long run.

Therefore, I think cloud native database technology is a big direction. How to make the database evolve with the direction of cloud infrastructure architecture is a problem that needs to be considered in the process of landing cloud native database.

Wang Minjian: From the traditional offline or private cloud point of view, I think the future database should be able to do both scale in and scale out. The traditional scale up era has come to an end. The reason why IBM has developed steadily for many years is that it has low failure rate and does not restart, while Linux PC may have some single points of failure. Therefore, we still need to continue to explore high availability.

We expect to be able to connect heterogeneous computing power, such as AI computing power and unstructured computing power, through the network bus, and truly distribute them on different clouds.

In terms of storage, the capabilities of PolarDB can well meet the needs of the actual business, can easily expand and shrink the capacity, can expand seamlessly with the growth of the customer's business volume, can add nodes, and make offline deployment more convenient. The use of fixed experience is similar to the pluggable mode of supercomputing. At present, we are trying to achieve the minimum business unit, which consists of four nodes, three of which are the PolarDB for PG version, and one node is the monitoring system and then the web GUI management, providing convenient use experience through pluggable mode. It has very low power consumption and can provide appropriate computing power.

In the future, this model can also be seamlessly migrated from private cloud to public cloud.

Li Feifei: I think the development of database in the next 3-5 years can be summarized as four modernizations.

First of all, after the separation of cloud original biochemistry and cloud original biochemistry storage and computing, three layers of decoupling will be realized, namely, the separation of storage, memory and computing, and then the Serverless will be developed from the perspective of resources to the perspective of capabilities.

Second, platform-based. In addition to databases, customers' actual use scenarios are not necessarily suitable for public cloud. Platformization means not only providing the read-write ability and high availability of the database, but also encapsulating other trivial and complex "dirty work" to provide end-to-end one-stop data service instead of a single database.

Third, integration. Integration includes processing and analysis integration (HTAP), distributed and centralized integration (smooth scale in, scale out), and offline and online integration. Users want to blur the boundaries between offline and online as much as possible. They don't want multiple business chimneys, multiple data islands, data migration, and better experience.

Fourth, intellectualization. The database must be easy to use, easy to use, simple to operate and maintain, and able to carry out intelligent diagnosis of problems, such as combining ML and AI machine learning technology to do slow SQL governance, index recommendation, problem diagnosis and discovery, and deep mining of data. For example, data can do some simple modeling in the database and push some business scenarios down to the database layer for direct implementation.

Moderator: As an ecological partner, what results do you expect to achieve after five years of co-creation with Alibaba Cloud?

President Tang: The development of open source databases is a long march. I think the database that can survive all the time must be a full-stack database that can cover more business scenarios.

However, with the advent of the cloud era, its technology stack and the heavy database bottom cause that it needs to pay more efforts to catch up with the new technology development. Open source products naturally have the advantage of "turning around" and can cover the Internet of Things, TP, AP and other aspects.

In addition, if you want to get more use scenarios, you need to be excellent in ease of use and reduce the threshold of learning and use for newcomers.

Moderator: As a customer, what kind of database do you expect?

Han Yi: I hope that through five years of development, the PolarDB ecosystem can be truly transformed into the Amazon rainforest, and everyone can win. The head enterprise provides perfect product performance, realizes the highest value of commercialization, and obtains the market; Partners participate in co-creation, share a share of the market segment and get profits; Ordinary users use high-quality products at extremely low cost and get substantial benefits.

And has driven its cloud through one-stop data service. The development mode of AWS is to develop from IaaS and PaaS layers level by level, while Microsoft starts with its enterprise-level software and drives the development of cloud from top to bottom.

Alibaba Cloud database also needs metabolism and young people to take charge of the ship. If the ecological development of PolarDB eventually becomes the Amazon rainforest, I believe that there will also be better people to lead you and give you confidence and courage, and China's database will also stand on the top of the world.

Moderator: Send one sentence to the Chinese database five years later or to PolarDB.

Wang Yuan: I hope that five years later I will still be deeply involved in PolarDB, and that more practitioners, users, developers and partners will work together to create it.

Wang Minjian: As a former giant, is still innovating. We must also keep making progress. We hope that through the continuous efforts of our domestic colleagues, we can have a truly local and original PolarDB five years later.

Tang Cheng: I hope that PolarDB will have a place in the full-stack database in five years.

Han Yi: In the next five years or even 50 years, I hope we can become the most loyal users of PolarDB.

Li Feifei: As long as you have the spirit of continuous exploration and progress, the PolarDB in five years' time will truly become the light of national products, even the light of the world database.

Zhou Zhengzhong, head of open source ecological operation and senior product expert of PolarDB

After the round table, Zhou Zhengzhong (Huaming Dege), the head of open source ecological operation and senior product expert of PolarDB, released the "Recruitment Plan for Communication Ambassador of PolarDB Open Source Community". He pointed out that the PolarDB open source community has a big circle of ecology. Open source products and ecological partners work together to provide value to users. With users, there will be a huge open market. With the market, there will be a whole stack of talents, including developers, consulting experts, top users, etc., who can feed back the market and users. The operation of the PolarDB open source community also revolves around the above cycle. The communication ambassador of PolarDB will need to participate in the organization of PolarDB open source community interaction, platform promotion and content planning and output. At the same time, the communication ambassador will also get rich participation rights, such as the priority to participate in activities, peripheral gifts, awards and open source contacts. We hope that all users, developers and partners will find opportunities, seize opportunities and become the communication ambassador of PolarDB.