This topic describes how to connect DataV to DataWorks DataService Studio. You can connect Hologres to DataWorks to develop and generate APIs in DataService Studio and call the APIs in DataV. Then, DataV presents the analysis results of MaxCompute data.
Background
MaxCompute is a fast and fully managed computing platform for large-scale data warehousing. It can process terabytes, petabytes, or even exabytes of data. As a wide variety of data collection methods emerge and huge amounts of data is accumulated, traditional software cannot meet data processing requirements. MaxCompute is used to store and compute masses of structured data. MaxCompute provides stable support for all offline business of data analytics for Alibaba Group over the years.
In the past, if you want to present the analysis results of large amounts of data in DataV, you need to create an offline data computing workflow to import data from your system to a MySQL database. This process involves complex operations and requires high costs. DataWorks provides the
services for you to develop data. You can integrate these services with MaxCompute to build data warehouses for your enterprise with ease.DataWorks DataService Studio allows you to use the codeless user interface (UI) to create APIs based on data tables without the need to write code. Then, you can call the APIs in DataV to present data analysis results on dashboards. This way, you can develop a data warehouse and present data in an efficient manner.
Prerequisites
Data sources are prepared, and DataV is activated. For more information, see What is DataV?.
Add a data source
DataService Studio supports connections to various data sources:
Relational databases: ApsaraDB RDS, Distributed Relational Database Service (DRDS), MySQL, PostgreSQL, Oracle, and SQL Server
Analytic databases: AnalyticDB
NoSQL databases: Tablestore and MongoDB
Log on to the DataWorks console. In the left-side navigation pane, click DataService Studio. On the page that appears, select the desired workspace from the drop-down list and click Go to DataService Studio.
Move the pointer over the icon in the Service Development pane and select Create Data Source. The Workspace page appears.
In the left-side navigation pane, click Data Source.
In the upper-right corner of the page that appears, click Add data source. In the Add data source dialog box, click Hologres in the Big Data Storage section.
This example describes how to add a Hologres data source. This allows you to query data in MaxCompute in real time.
In the Add Hologres data source dialog box, configure the parameters that are shown in the following table.
Parameter
Description
Data Source Type
The type of the
data source. You can set it only to Alibaba Cloud Instance Mode.Data Source Name
The name of the data source. The name can contain only letters, digits, and underscores (_), and must start with a letter.
Data Source Description
The description of the data source. The description can be a maximum of 80 characters in length.
Environment
The environment in which the data source is used. Valid values: Development and Production.
NoteThis parameter is displayed only when the workspace is in standard mode.
Instance ID
The ID of the Hologres instance from which you want to synchronize data. You can obtain the instance ID from the Hologres console.
Database Name
The name of the Hologres database.
AccessKey ID
The AccessKey secret that is used to access the Hologres database. You can obtain the AccessKey secret on the Security Management page.
AccessKey Secret
The AccessKey secret that is used to access the Hologres database. You can obtain the AccessKey secret on the Security Management page.
Find the desired resource group and click Test connectivity in the Actions column.
After the data source passes the network connectivity test, click Complete.
Create an API
After you create the data source, go to the DataService Studio page. This example describes how to create an API by using the codeless UI.
Click the More icon in the upper-left corner and choose
.Move the pointer over the icon in the Service Development pane and choose
.In the Generate API dialog box, configure the parameters. In this example, set API Mode to Wizard Mode.
Parameter
Description
API Name
The name of the API. The name must be 4 to 50 characters in length, and can contain letters, digits, and underscores (_). It must start with a letter.
API Path
The path of the API, such as /user.
Protocol
The protocol used by the API. Valid values: HTTP and HTTPS.
Request Method
The request method used by the API. Valid values: GET and POST.
Response Content Type
The type of the API response content. The value is fixed to JSON.
Description
The description of the API.
Click Determine. The configuration tab of the API appears.
Configure the API parameters
In the Select Table section, select Hologres from the Datasource Type drop-down list and configure the Datasource Name and Table Name parameters.
NoteBefore you configure API parameters, you must add and configure the required data source in Data Integration. You can enter a table name in the Table Name field to search for the desired table.
After you create an API, the API configuration tab appears. Then, you can select a table for the API.
In the Environment Configuration section, configure the Memory and Function Timeout parameters.
After you select a table in the Select Table section, all fields in the table appear in the Select Parameters section. Select the fields that need to be specified as request parameters and the fields that need to be specified as response parameters. Add them to the request parameter list and the response parameter list.
Edit request parameters.
In the right-side navigation pane, click the Request Param tab. In the Request Param panel, configure the Parameter Name, Type, Operator, Required, Example Value, Default Value, and Description parameters.
Edit response parameters.
In the right-side navigation pane, click the Response Param tab. In the Response Param panel, configure the Parameter Name, Type, Example Value, and Description parameters. You can also select Pagination and Filter in the Advanced Settings section.
Select Pagination based on your business requirements.
If you do not select Pagination, the API returns a maximum of 2,000 records by default.
If the number of records returned by the API exceeds 2,000, we recommend that you select Pagination.
Test the API
After you configure the API parameters and save the settings, click Test in the upper-right corner. The Test APIs dialog box appears.
Configure the parameters and click Test to send an API request. The request and response details appear on the right. If the API fails the test, check the error message, modify the API settings, and then test the API again.
Publish the API
After the API passes the test, return to the DataService Studio page.
Click Publish in the upper-right corner to publish the API.
After the API is published, click Service Management in the upper-right corner to view the API details.
If you want to call the API, click Service Management. In the left-side navigation pane of the page that appears, click Call APIs. DataService Studio supports quick authentication based on the AppCode and encrypted signature authentication based on the AppKey and AppSecret. The following sections describe how to call DataService Studio APIs in DataV.
Add DataService Studio as a data source
Log on to the DataV console.
Go to the Data Sources tab. On this tab, click Add Source.
In the Add Data Source dialog box, configure the parameters.
Parameter
Description
Type
The type of the data source.
Name
The name of the data source. You can specify a custom name.
Project
The DataWorks workspace from which the required data comes.
AppKey/AppSecret
The AppKey and AppSecret of the account that has access permissions on the workspace in which the DataService Studio service is used.
NoteTo obtain the AppKey and AppSecret, log on to the DataWorks console, go to the DataService Studio page, and then click Service Management in the upper-right corner. In the left-side navigation pane of the page that appears, click Call APIs.
Call an API of DataService Studio in DataV
Log on to the DataV console. In the top navigation bar, click Projects. On the page that appears, click Create Project.
On the page that appears, select a template and click Create Project. In this example, select the Smart Factory template.
Widgets in the template contain static data. This example describes how to call the API to query the growth trend of the transaction amount and present the query results in the Basic Line Chart widget.
Select the Basic Line Chart widget and go to the data panel. In the data panel, set the Data Source Type parameter to DataWorks.
Select the added data source and created API and configure the query parameters. In this example, set the Page Size parameter to 31 to query data of one month.
Click Preview Data Response to view the query results of the API.
Set the x field to date and the y field to amount.
The preceding figure shows that the x and y fields match no data. This is because DataV has requirements for the data format and cannot identify fields with a deep structure. You need to add a data filter to filter out unnecessary fields.
Select Data Filter and click the plus (+) icon on the right to add a data filter. You can also enter JavaScript code in the code editor to filter and process the data analysis results. The data parameter of the filter specifies the JSON objects returned by the API.
In this example, enter the code
return data.data.rows;
to filter out fields except the rows array returned by the API. Click Preview to preview the filtered data in the lower part, and click Complete.After you add a data filter, DataV can match data based on the specified fields.
The date format returned by the API is different from the default date format of the widget. Therefore, DataV does not correctly display the line chart. You must specify the date format for the X axis of the line chart before DataV can correctly display the line chart.
Go to the configuration panel. Click Axis Label under X Axis. Set the data type to date and the data format to a value in the YYYY/MM/DD format, such as 2016/01/01. Then, DataV correctly displays the line chart.
After the preceding steps are performed, DataV can call the API created in DataService Studio based on a MaxCompute table to obtain and display the data analysis results.
Usage notes
After you connect DataV to DataWorks DataService Studio by adding DataService Studio as a data source of DataV, you can directly call an API of DataService Studio in DataV. In this case, you do not need to enter the API URL. You do not need to enter the AppKey and AppSecret for calling each API. In addition, you can configure API parameters by using a table. This is convenient, secure, and reliable.
DataService Studio allows you to directly present the data analysis results processed by MaxCompute in DataV. This seamlessly links data analytics and data presentation.
When you use DataService Studio and DataV together, take note of the following items:
APIs created on the codeless UI in DataWorks DataService Studio can only query data in a single table based on simple criteria. APIs created in the code editor allow you to use SQL statements or functions to query data in multiple tables based on complex criteria. You can create APIs by using the codeless UI or code editor based on your business requirements.
If you want to call an API to query data within milliseconds, we recommend that you use a relational database, a NoSQL database, or an AnalyticDB database as the data source.
DataV widgets only support arrays, but DataService Studio APIs return complete JSON strings that contain error codes. Therefore, you must use filters to process results returned by the APIs. You can add filters in DataV or when you configure APIs in DataService Studio.
If pagination is disabled for an API, add a filter to filter out fields except the data array returned by the API. If pagination is enabled for an API, add a filter to filter out fields except the data.rows array returned by the API.
If you want to present a data analysis result in multiple types of charts, such as in both a line chart and a column chart, DataV treats the data of each type as an object and distinguishes the types by field. In this case, you need to use a filter for format conversion.