This topic describes how to connect DataV to DataWorks DataService Studio. You can connect Hologres to DataWorks to create APIs in DataService Studio and call the APIs in DataV. Then, DataV presents the analysis results of MaxCompute data.

Background

MaxCompute is a fast and fully managed computing platform for large-scale data warehousing. It can process terabytes, petabytes, or even exabytes of data. As a wide variety of data collection methods emerge and huge amounts of data is accumulated, traditional software cannot meet data processing requirements. MaxCompute is used to store and compute masses of structured data. MaxCompute provides stable support for all offline business of data analytics for Alibaba Group over the years.

In the past, if you want to present the analysis results of large amounts of data in DataV, you need to create a workflow to import data from your system to a MySQL database. This process involves complex operations and requires high costs. DataWorks provides the Data Integration, DataStudio, and DataService Studio services for you to develop data. You can integrate these services with MaxCompute to build data warehouses for your enterprise with ease.

DataWorks DataService Studio allows you to use the codeless user interface (UI) to create APIs based on data tables without the need to write code. Then, you can call the APIs in DataV to present data analysis results on dashboards. This way, you can develop a data warehouse and present data in an efficient way.

Prerequisites

A data source is prepared. DataV is activated.

Create a connection

DataService Studio supports connections to various data sources:
  • Relational databases: ApsaraDB RDS, Distributed Relational Database Service (DRDS), MySQL, PostgreSQL, Oracle, and SQL Server
  • Analytic databases: AnalyticDB
  • NoSQL databases: Tablestore and MongoDB
  1. Log on to the DataWorks console. In the left-side navigation pane, click Workspaces. On the Workspaces page, find the workspace that you want to manage and click DataService Studio in the Actions column.
  2. In the Service Development pane, move the pointer over the Create icon icon and select New Data Source. The Data Source page appears.
  3. Click Add data source in the upper-right corner.
  4. In the Add data source dialog box, click Hologres in the Big Data Storage section.

    This example describes how to create a connection to Hologres. This allows you to query data in MaxCompute in real time.

  5. In the Add Hologres data source dialog box, set the parameters as required.
    Parameter Description
    Data source type In this example, a connection to Hologres is used. Only Alibaba Cloud instance mode is supported.
    Data Source Name The name of the connection. The name can contain letters, digits, and underscores (_), and must start with a letter.
    Data source description The description of the connection. The description can be up to 80 characters in length.
    Environment The environment in which the connection is used. Valid values: Development and Production.
    Note This parameter is displayed only when the workspace is in standard mode.
    Instance ID The ID of the Hologres instance from which you want to synchronize data. You can obtain the instance ID from the Hologres console.
    Database Name The name of the database in the Hologres instance.
    AccessKey ID You can obtain the AccessKey ID from the Security Management page.
    AccessKey Secret You can obtain the AccessKey secret from the Security Management page.
  6. Click Test connectivity.
  7. After the connection passes the connectivity test, click Complete.

Create an API

After you create a connection, go to the DataService Studio page. This example describes how to create an API by using the codeless UI.
  1. Click the More icon in the upper-left corner and choose All Products > DataService Studio. The DataService Studio page appears.
  2. In the Service Development pane, move the pointer over the Create icon icon and choose New API > Generate API.
  3. In the Generate API dialog box, set the parameters as required. In this example, set the API mode parameter to Wizard Mode.
    Parameter Description
    API Name The name of the API. The name must be 4 to 50 characters in length, and can contain letters, digits, and underscores (_). It must start with a letter.
    API Group The API group to which the API belongs. An API group is a collection of APIs for a specific feature or scenario. An API group is the API management unit of API Gateway. Each API group corresponds to a specific API product in the Alibaba Cloud API Marketplace.

    To create an API group, move the pointer over the Create icon in the Service Development pane and select New API Group.

    API Path The path in which the API is stored, such as /user.
    Protocol The protocol used by the API. Valid values: HTTP and HTTPS.
    Request Method The request method used by the API. Valid values: GET and POST.
    Response Content Type The return type of the API. The value is fixed to JSON.
    Description The description of the API.
  4. Click OK. The configuration tab of the API appears.

Set the API parameters

  1. In the Select Table section, select Hologres from the Datasource Type drop-down list and set the Datasource Name and Table Name parameters as required.
    Note
    • Before you set API parameters, you must configure a connection in Data Integration. You can enter a table name in the Table Name field to search for the desired table.
    • After you create the API, the table configuration tab automatically appears for you to select a table for the API.
  2. In the Environment Configuration section, set the Memory and Function Timeout parameters as required. Environment Configuration section
  3. After you select a table in the Select Table section, all fields in the table appear in the Select Parameters section. Select the fields that need to be set as request parameters and the fields that need to be set as response parameters. Add them to the request parameter list and the response parameter list. Select Parameters section
  4. Edit request parameters.
    In the right-side navigation pane, click the Request Parameters tab. In the Request Parameters panel, set the Parameter Name, Parameter Type, Operator, Required, Example Value, Default Value, and Description parameters. Request Parameters panel
  5. Edit response parameters.
    In the right-side navigation pane, click the Response Parameters tab. In the Response Parameters panel, set the Parameter Name, Parameter Type, Example Value, and Description parameters. You can also select Pagination and Filter in the Advanced Settings section. Response Parameters panel
    Select Pagination based on the actual requirements.
    • If you do not select Pagination, the API returns a maximum of 2,000 records by default.
    • If the API may return more than 2,000 records, we recommend that you select Pagination.

Test the API

After you set the API parameters and save the settings, click Test in the upper-right corner. The Test APIs dialog box appears. Test button

Set the parameters and click Test to send an API request. The request and response details appear on the right. If the API fails the test, check the error message, modify the API settings, and then test the API again.

Publish the API

  1. After the API passes the test, return to the DataService Studio page.
  2. Click Publish in the upper-right corner. The API is published.
  3. After the API is published, click Service Management in the upper-right corner to view the API details. Service Management tab

If you want to call the API, click Service Management. On the page that appears, click Call APIs in the left-side navigation pane. DataService Studio supports quick authentication based on the AppCode and encrypted signature authentication based on the AppKey and AppSecret. The following sections describe how to call DataService Studio APIs in DataV.

Add DataService Studio as a data source

  1. Log on to the DataV console.
  2. In the top navigation bar, click Data Sources and then Add Source.
  3. In the Add Data Source dialog box, set the parameters as required. Add Data Source dialog box
    Parameter Description
    Type The type of the data source.
    Name The name of the data source. Enter a custom name.
    Project The DataWorks workspace from which the required data comes.
    AppKey and AppSecret The AppKey and AppSecret of the account that has the permissions to access a DataWorks workspace.
    Note To obtain the AppKey and AppSecret, log on to the DataWorks console, go to the DataService Studio page, and then click Service Management in the upper-right corner. On the page that appears, click Call APIs in the left-side navigation pane. Call APIs

Call an API of DataService Studio in DataV

  1. Log on to the DataV console. In the top navigation bar, click Projects. On the page that appears, click Create Project.
  2. On the page that appears, select a template and click Create Project. In this example, select the Smart Factory template. Smart Factory

    Widgets in the template contain static data. This example describes how to call the API to query the growth trend of the transaction amount and present the query results in the Basic Line Chart widget.

  3. Select the Basic Line Chart widget and go to the data panel. In the data panel, set the Data Source Type parameter to DataWorks.
  4. Select the added data source and created API and set the query parameters. In this example, set the Page Size parameter to 31 to query data of one month. Query data
  5. Click Preview Data Response to view the query results of the API.
  6. Set the x field to date and the y field to amount. Mappings

    The preceding figure shows that the x and y fields match no data. This is because DataV has requirements for the data format and cannot identify fields with a deep structure. You need to add a data filter to filter out unnecessary fields.

  7. Select Data Filter and click the plus (+) icon on the right to add a data filter. You can also enter JavaScript code in the code editor to filter and process the data analysis results. The data parameter of the filter specifies the JSON objects returned by the API.
    In this example, enter the code return data.data.rows; to filter out fields except the rows array returned by the API. Click Preview to preview the filtered data in the lower part, and click Complete. Data Filter
    After you add a data filter, DataV can match data based on the specified fields. Data matched

    The date format returned by the API is different from the default date format of the widget. Therefore, DataV does not correctly display the line chart. You must set the date format for the X axis of the line chart before DataV can correctly display the line chart.

  8. Go to the configuration panel. Click Axis Label under X Axis. Set the data type to date and the data format to a value in the YYYY/MM/DD format, such as 2016/01/01. Then, DataV correctly displays the line chart. Display data
After the preceding steps are performed, DataV can call the API created in DataService Studio based on a MaxCompute table to obtain and display the data analysis results. The following figure shows the data display effect. Effects

Usage notes

After you connect DataV to DataWorks DataService Studio by adding DataService Studio as a data source of DataV, you can directly call an API of DataService Studio in DataV. In this case, you do not need to enter the API URL. You do not need to enter the AppKey and AppSecret for calling each API. In addition, you can set API parameters by using a table. This is convenient, secure, and reliable.

DataService Studio allows you to directly present the data analysis results processed by MaxCompute in DataV. This seamlessly links data analytics and data presentation.

When you use DataService Studio and DataV together, take note of the following items:
  • APIs created on the codeless UI in DataWorks DataService Studio can only query data in a single table based on simple criteria. APIs created in the code editor allow you to use SQL statements or functions to query data in multiple tables based on complex criteria. You can create APIs by using the codeless UI or code editor based on your needs.
  • If you want an API to query data within milliseconds, we recommend that you use a relational database, a NoSQL database, or an AnalyticDB database as the data source.
  • DataV widgets only support arrays, but DataService Studio APIs return complete JSON strings that contain error codes. Therefore, you must use filters to process results returned by the APIs. You can add filters in DataV or when you configure APIs in DataService Studio.

    If pagination is disabled for an API, add a filter to filter out fields except the data array returned by the API. If pagination is enabled for an API, add a filter to filter out fields except the data.rows array returned by the API.

  • If you want to present a data analysis result in multiple types of charts, such as in both a line chart and a column chart, DataV treats the data of each type as an object and distinguishes the types by field. In this case, you need to use a filter for format conversion.