Help Vision Technology enter the cloud-native fast lane

In 2021, the commercial value of cloud native is being accelerated to release.

It is a recognized fact that Serverless is an absolute bright spot in the current cloud native direction. It can be seen that its emergence has enabled enterprise users to truly eliminate the burden of operation and maintenance and focus more on the business itself. In other words, enterprises can reduce costs and increase efficiency based on a pay on demand model to maximize the commercial value of technology.

In this wave of complementary technology and business, Vision Digital Science is a clear case. Based on the Alibaba Cloud Serverless application engine (SAE), Vision Digital has been fully upgraded to a cloud native architecture, providing a best practice model for cloud native applications across the financial technology industry.

Founded in 2014, Vision Digital Technology is the first big data financial information service provider in China that focuses on serving the primary market, corporate credit, industrial planning and investment attraction, and multi-level capital markets. Unlike other to C operations, Vision Digital has deeply bound data products to financial industry usage scenarios and introduced a series of business function modules.

With the continuous upgrading of its business, Vision Digital has started from providing data and information services to financial institutions. In recent years, its customers have gradually expanded their services to governments, industrial parks, large state-owned enterprise groups, etc. At the same time, the coverage of data has also gradually extended from financial related data to industrial product data, enterprise operation data, macroeconomic data, government data, policy and public opinion data, geographic information data, etc., focusing on the digital transformation of customers, Provide customers with overall data service solutions for data sales, data integration, data processing, data center, system development, and big data model analysis consulting services, and assist in the digital transformation of China's industry.

In addition to the role positioning of financial data providers, today's Vision Digital is entering a new stage of development with multiple product lines and multiple types of service customers, becoming a true financial industry data service solution provider.

Business pain points should be first, and Alibaba Cloud Serverless solutions should be preferred

In the financial industry system, the application scenario of big data has always been a synonym for sensitivity and complexity.

Taking data invocation as an example, most financial institution systems are isolated from the external network, requiring data service companies to build data service systems that are more suitable for such scenarios or embedded data functions that can meet customer needs. For customers, Vision Digital not only provides conventional data file/data synchronization/API data service methods, but also seamlessly embeds data into the customer's internal system in an SDK+embedded manner.

On the one hand, this simplifies customer development costs, and on the other hand, it greatly reduces the development time that customers invest in the actual use of a large amount of external procurement data.

In addition, like many financial technology companies, Vision Digital is a hybrid deployment model of self built IDC+public cloud. In the exploration of cloud computing, Vision Digital has always been a pioneer in the industry. As early as 14 years ago, Vision Digital Technology built a first generation infrastructure cloud based on Alibaba Cloud ECS servers, using a combination of open source self built and cloud products. The entire architecture is data centric, including a SaaS based data service platform, security access and protection, data service layer, data processing layer, cloud security, and so on.

Vision Digital Business Architecture Chart

As a startup company in the technology industry, at the beginning of its business, it is necessary to quickly run the business. Initially, all applications were based on a single chimney architecture and manual deployment, but based on the shortcomings of the technology side, these architecture optimization efforts have been delayed.

However, in the past two years, such technical architecture based issues have become increasingly prominent. It can be seen that data is the core asset of an enterprise's business, and the security, stability, and efficiency of data are key to serving large customers. Under the inherent model, the Vision Digital Test environment cannot obtain the full range of real customer data, and many cases cannot be covered. Only before going online, did the frequent release and testing process in a grayscale environment (equivalent to pre release) expose some issues:

1) Slow development iteration efficiency: Single chimney architecture, high code coupling, and slow development efficiency.

2) The launch process is complex and costly: SVN code management+manual deployment is used, and there is a lack of standardized DevOps processes. Before each launch, the three teams of research and development, quality inspection, and operation and maintenance need a lot of collaboration in a grayscale environment. They have to juggle 20 to 30 data verifications back and forth, and frequent release testing results in poor happiness for development and operation and maintenance.

3) Capacity estimation cannot be automated: Every time there is a marketing activity/important event on the customer side (such as Xinhua Finance Financial Ranking, etc.), it is necessary to inform the Vision Data Division of the provisioning ECS one week in advance, and there are risks of provisioning inaccuracies and idle waste.

In response to the above issues, the upgrading of the technical architecture of Vision Digital Technology has been put to the forefront. It is understood that two plans have been discussed within the Vision Digital Department:

Solution 1: ECS has built its own Docker+open source microservice, which has been found to be able to quickly container and improve resource utilization. However, the underlying infrastructure operations and maintenance (Docker Daemon upgrade, configuration management, image warehouse management, etc.) and development workload are large (microservice component self research), and the risk of online operations and maintenance is high. After a simple POC, make a collective decision to give up.

Solution 2: Using a commercial microservice PaaS platform to host applications has been found to reduce the threshold for microservices and provide a foundation for the stability of microservice components. However, the need for self operation and maintenance of ECS is still cumbersome, and the overall cost is too high to budget.

Finally, I learned about SAE at a technical communication meeting, and combined with the company's technical background at that time, I found that SAE and the company had a high degree of compatibility in their technical upgrades and transformations. They did not change the code or the existing deployment method of the application, and enjoyed the complete experience of microservices+Serverless+K8s. They were ready to use out of the box, eliminating the need for later operation and maintenance. Vision Digital Technology has opened the way to practice architecture upgrading.

Comprehensive upgrade of technical architecture

Sharp tools make good work. Before officially migrating the business, the first thing Vision Digital did was to standardize the online process, hoping to reduce the burden through continuous integration.

1) Create a cloud native DevOps system of Git+Jenkins+SAE from 0 to 1.

2) Complete the transformation of microservice architecture through SAE's low threshold, and upgrade to microservice+K8s+Serverless architecture in one step.

In the early stage, we selected a new version of the main product - anti crawling identification application, and tried to split micro services. After splitting, it is developed based on the Spring Cloud standard, and then deployed to SAE. During the process, it was found that SAE's support for Java microservices is too comprehensive: it completely eliminates the need for customers to consider data isolation, distributed transactions, circuit breaker design, current limiting degradation, and other issues. It also eliminates the need to worry about limited community maintenance and secondary customization development. It is available out of the box, greatly improving development efficiency. On the basis of open source, SAE provides advanced service governance capabilities such as lossless online and offline, service authentication, and full link grayscale through deep integration of MSE. Help customers shield K8s technical details, make them container free and embrace K8s without feeling.

In the process of practicing SAE, a strategy of independent services+user grayscale was adopted to gradually increase traffic, gradually bring some services online, and then migrate historical inventory applications.

Continuous evolution to create a financial level cloud platform

Due to the particularity of the financial industry, during the process of upgrading the ECS architecture to the Serverless architecture, I was initially concerned that SAE could not match the regulatory requirements for financial security and compliance. However, after communication and confirmation with SAE classmates, as well as the continuous evolution of SAE products, concerns about vision have been completely dispelled.

1) Security compliance: Security protection products such as Yundun, firewall, DDOS, and barrier machines used in the ECS mode continue to be available, and SAE also provides intrusion detection and vulnerability scanning. SAE later also supported the deployment of container mirroring services for enterprise applications, supporting image security scanning and multi-dimensional vulnerability reporting, ensuring storage and content security.

2) Security isolation: SAE students informed the user that there was no tipping in on the traffic side, and selecting JDK as Dragonwell can also support communication encryption in the future. The underlying layer is based on a secure sandbox container+VPC network, enabling multiple security isolation of systems, networks, and data.

3) Operational Audit: For some operations on SAE, the change history can be traced through SAE's unique release form. At the same time, SAE also interfaces with cloud product operation audits, which can query all operational behavior logs and add, delete, and modify events on the cloud.

4) Permission control: SAE has also solved a long-standing problem: permission isolation and approval. In the past, in the ECS mode, especially when newcomers arrive or cross team joint debugging, configuring user groups, RAM permissions, and new machine login and connection methods are very cumbersome, and account management personnel often become bottlenecks. What's more, the operation and maintenance operations are not approved, and the risks are uncontrollable. The development process has a user name and password for the machine, and the release is relatively arbitrary. After using SAE, add permissions based on application granularity. Add an application once, which saves effort and effort. SAE has also designed an operation and maintenance approval process through primary and sub accounts, effectively reducing the quality risks caused by random online publishing.

Operation and maintenance efficiency increased by 60%, with significant effect

Through continuous running-in verification with the SAE platform, some applications of Vision Digital have gradually migrated to SAE. The entire migration process is smooth, without any transformation costs, zero failures, and only one R&D personnel has been invested. Next, we plan to fully migrate the overall architecture to SAE to fully share the dividends of cloud native technology.

Create a benchmark for the financial technology industry, with great potential for enterprise big data

Currently, many industries such as finance, industry, and agriculture are walking on the fast track of digitization.

The financial industry is special enough. On the one hand, its data sources are relatively standardized, on the other hand, the coverage and application of financial data are relatively broad, and the industry has a high recognition of data. Based on its experience in providing services to core financial institutions, Vision Digital has gained widespread reputation in the industry, and its data quality has received unanimous praise in the industry.

It is understood that Vision Digital Technology has created multiple product lines, ranging from "non scenario" data query platforms to business functionality platforms based on specific business scenarios. In addition to industry standard data, it also divides different data into deeper and more comprehensive labels.

It is commendable that Vision Digital Technology has invested a lot of research efforts in over 100 industrial chains, over 6000 segmented industries, and nearly 100000 types of "product service" classification for the national economy, and has produced a relatively accurate and complete industry classification system. With its deep mining of data tags, it has built its deep processing and recalculation capabilities for various types of enterprise related data.