By Zhou Shuo, Product Manager of DataWorks
This article is a part of the One-stop Big Data Development and Governance DataWorks Use Collection.
As an overall platform for big data development and governance, DataWorks has built a comprehensive process solution from data integration, data development, and data service to application development. According to the following figure, in the whole data process, data services connect data warehouses, databases, and data applications in series, forming a bridge between data and applications.
It can provide individuals, teams, and enterprises with comprehensive data opening and sharing capabilities by encapsulating data into data API. Users can use this platform to manage internal and external API services in a unified way. The data service provides an effective connection to connect data sources downwards and support business applications upwards.
In the overall architecture, the top layer contains three modules, the data service desk, OpenAPI, and API gateway. Among them, users can quickly create and publish the API through the product interface of the data service (data service desk) or OpenAPI or make secondary modifications to the API in the API Gateway. The lower layer of the architecture adopts the API development platform as the base. The data service development platform takes API as the core and provides the capabilities of organization management, API development, resource development, data source management, API permission management, and API measurement.
In organization management, users can create workflows and use workflows as logical units to store resource objects, such as API inside the workflow. This way, it can be subdivided further, and a multi-level management structure can be realized through folders.
In API development capabilities, besides diversified development methods, such as wizard mode and script mode, data services also provide interface debugging and one-click publishing capabilities, realizing the lightweight of API output. In addition, other capabilities assist API development and management, such as permission management (API visibility and authorization call) and measurement statistics (visualization charts to show the call situation).
DataWorks and API Gateway are integrated. API published in data services can be viewed and managed in the gateway list during development and management. In the call process, when the user side sends the call request to the gateway, the gateway will forward the call request to the data service background system. Then, the data service will perform request parsing, SQL parsing, and other processing and finally obtain valid data from the data source to return to the user.
From the preceding processes, in the overall process from the user sending the request to the data service returning the data result, the connection and processing process between different products (clients, API gateways, data services, and data sources) are almost unaware. Users only need to focus on the data itself. DataWorks data services encapsulate and provide a series of underlying services for users.
The preceding architecture shows the capabilities, processing logic, and basic dependencies of data services from a global perspective, while the following picture shows the whole processes of API development and use by users with data services from a practical perspective.
First of all, two ways are used to create a new API on the data service platform. If you already have data sources, you can directly connect the data sources to generate an API by filling in the corresponding connection information. If you already have an encapsulated API, the host address of the existing API can be directly registered to the data service platform for unified control. In addition, the data service provides function compute capabilities to assist in API generation. It also supports organizing multiple APIs and functions into workflows to generate compound API.
Whether it is an independent API of the generated or registered type or a compound API of the service orchestration type, the data service can publish these APIs to the gateway with one click. You can call and consume the published data interfaces in applications, reports, and dashboards or put them on the Alibaba Cloud API market for sale and authorize others to realize internal and external data sharing. From this point of view, the application scenarios supported by data services are diversified and can meet your various needs.
In the traditional way, the processes of developing an API interface are complicated and time-consuming. Starting from preparing and connecting the database, you need to go through the steps of API query logic development, API authentication and flow control capability development, server construction, and interface deployment. After the interface goes online, you still need to carry out a series of later O&M work. In comparison, developing an API in a data service requires only two steps, preparing the database and configuring the query logic of the API. As for the subsequent deployment, control, and O&M work, users do not need to develop it alone. The data services will provide complete product capabilities, infrastructure, and underlying resources.
The data services adopt Serverless architecture, which has the advantages of zero code without O&M and flexible expansion. Users only need to pay attention to the query logic of API, which reduces the complicated process of developing API and achieves cost reduction and efficiency improvement.
In addition, if you need to create API and obtain data results, the connectivity of data sources and networks is essential. DataWorks data services also provide corresponding product capabilities to connect multiple data source types under multiple network environments. The network environments supported by Data Services include the VPC network, classic network, and Internet network. The supported data types include common ApsaraDB RDS and non-ApsaraDB RDS, such as MySQL, PostgreSQL, Oracle, OTS, and MongoDB, and big data storage types, such as MC-Hologres. Data services will also continue to enrich the types of accessible data sources.
In summary, the product advantages of data services say data source types are diverse, network types are rich.
You need to make preparations before using the DataWorks data services. Activate the API Gateway service to ensure that the API can be successfully published and a valid domain name can be obtained.
After preparations, you can enter the service development page of the data service, create a workflow as an organizational unit in the left directory, and continue to create the destination API and other objects (such as functions and service orchestration) under this workflow.
The following figure shows the common use processes in data services. Whether it is API generation, API registration, or compound API service orchestration, it can be published to the gateway for an online call as a data interface after debugging.
The core functions of the data services are described in detail below.
Two approaches are used to generate APIs, namely wizard mode and script mode. If you are an analyst or a businessperson, the visualization wizard mode can be adopted. If you are a developer or a power SQL user, you can customize SQL scripts and edit complex query logic through script mode.
In wizard mode, after you select the destination data table name, the system automatically obtains the table structure and displays it on the API editing page. The overall logic of the API can be defined by checking the request parameters and return parameters. The wizard-mode generation API enables a quick single table query. It supports multiple operator types of request parameters (such as equivalent query and fuzzy matching) and sorting the returned results according to specific fields (such as adding a field to the sorted list). The advantages of using wizard mode are visualization and zero code and how easy it is to get started.
In script mode, data services offer an intelligent SQL editor that supports editing multi-table association analysis logic in SQL, adding aggregate function compute and other complex conditional queries. Notably, the advanced SQL mode covering Mybatis syntax has been officially published. Users can flexibly define dynamic label logic.
In addition to generating APIs, you can register the existing APIs to the data service platform, which is convenient for unified management, publishing, and docking. The registration API supports four common request methods: GET, POST, PUT, and DELETE. Three common data formats, FORM, JSON, and XML, are also supported.
In addition to API, data services provide another resource development capability, the Python function. You can develop Python scripts, bind to an API as a pre-filter or post-filter, and process the request parameters and return results of the API respectively, thus enhancing the logical expression ability of the API, adapting to various scenarios, and realizing data deformation and conversion.
The service orchestration capability of data services can visualize several APIs and functions into a workflow in the form of drag nodes to realize serial and parallel calls among APIs.
For example, in the preceding figure, APIs, functions, and Switch conditional branch nodes are integrated to encapsulate a new overall API to provide to the business side. This approach can reduce network overhead and improve the overall API call performance. As shown in the figure, the organization logic of a service orchestration sample works like this: Several APIs are merged through workflows, and the output of the upstream API is used as the input of the downstream API. Then, the branch condition is judged according to different scenarios to obtain the final output.
After the API is published, the data services will automatically generate detailed API documents (the API detail page), eliminating the trouble of manually writing documents. The API detail page can be viewed by developers and callers. They are the version records and call instructions. The information contained in the API details page includes the basic information of the API, request and response parameters, normal and abnormal return examples, and error codes. This helps you understand the API in all aspects.
Data services allow you to set call authorization for data interfaces to achieve secure and reliable data sharing.
In some scenarios, if the permission of the database and data table is directly exposed in multiple frequencies and on a large scale, it will lead to high data redundancy and affect data security. For such scenarios, the authorization capability of data services allows APIs to be called by users and others. The API authorization approach says enter the service management page, locate the destination API in the list, click authorization in the operation column, and select the cloud account ID and destination workspace to be authorized in the pop-up window. Then, set the validity period of the permission so the data can be opened and shared on a small scale but in a better method.
In terms of API calling permissions, data services provide two methods: simple identity authentication and encrypted signature identity authentication. You can select authentication methods according to different scenarios.
The first method is simple identity authentication. Simple identity authentication uses AppCode authentication. You can copy the call address with parameters on the API detail page to obtain identity authentication parameters when an API is called. This call address contains AppCode information by default.
The second method is encrypted signature authentication. Encrypted signature authentication is performed using the AppKey and AppSecret encryption algorithms to calculate the signature, which has higher security.
The two types of call authentication information can be viewed on the detail page of the API call.
Data services can measure statistics and calculate calling statistics over a specified period for published APIs, including:
In addition to the measurement statistics of the data services, users can print detailed API call logs combined with SLS log services.
Besides, APIs published by data services on gateways also support the configuration of corresponding flow control policies and alarm rules to provide effective escort for calling APIs on services lines.
An Introduction and Best Practice of DataWorks Migration Assistant
864 posts | 196 followers
FollowAlibaba Cloud Community - March 29, 2022
Alibaba Cloud Community - March 21, 2022
Alibaba Cloud Community - March 4, 2022
Alibaba Cloud Community - March 29, 2022
Alibaba Cloud MaxCompute - December 22, 2021
Alibaba Clouder - January 6, 2021
864 posts | 196 followers
Follow
API Gateway
API Gateway provides you with high-performance and high-availability API hosting services to deploy and release your APIs on Alibaba Cloud products.
Learn More
Big Data Consulting for Data Technology Solution
Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.
Learn More
Big Data Consulting Services for Retail Solution
Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.
Learn More
Omnichannel Data Mid-End Solution
This all-in-one omnichannel data solution helps brand merchants formulate brand strategies, monitor brand operation, and increase customer base.
Learn MoreMore Posts by Alibaba Cloud Community
