All Products
Search
Document Center

DataWorks:Best practices to connect DataV to DataWorks DataService Studio

Last Updated:Jan 16, 2024

This topic describes how to connect DataV to DataWorks DataService Studio. You can connect Hologres to DataWorks to develop and generate APIs in DataService Studio and call the APIs in DataV. Then, DataV presents the analysis results of MaxCompute data.

Background

MaxCompute is a fast and fully managed computing platform for large-scale data warehousing. It can process terabytes, petabytes, or even exabytes of data. As a wide variety of data collection methods emerge and huge amounts of data is accumulated, traditional software cannot meet data processing requirements. MaxCompute is used to store and compute masses of structured data. MaxCompute provides stable support for all offline business of data analytics for Alibaba Group over the years.

In the past, if you want to present the analysis results of large amounts of data in DataV, you need to create an offline data computing workflow to import data from your system to a MySQL database. This process involves complex operations and requires high costs. DataWorks provides the Data Integration > DataStudio > DataService Studio services for you to develop data. You can integrate these services with MaxCompute to build data warehouses for your enterprise with ease.

DataWorks DataService Studio allows you to use the codeless user interface (UI) to create APIs based on data tables without the need to write code. Then, you can call the APIs in DataV to present data analysis results on dashboards. This way, you can develop a data warehouse and present data in an efficient manner.

Prerequisites

Data sources are prepared, and DataV is activated. For more information, see What is DataV?.

Add a data source

DataService Studio supports connections to various data sources:

  • Relational databases: ApsaraDB RDS, Distributed Relational Database Service (DRDS), MySQL, PostgreSQL, Oracle, and SQL Server

  • Analytic databases: AnalyticDB

  • NoSQL databases: Tablestore and MongoDB

  1. Log on to the DataWorks console. In the left-side navigation pane, click DataService Studio. On the page that appears, select the desired workspace from the drop-down list and click Go to DataService Studio.

  2. Move the pointer over the image icon in the Service Development pane and select Create Data Source. The Workspace page appears.

  3. In the left-side navigation pane, click Data Source.

  4. In the upper-right corner of the page that appears, click Add data source. In the Add data source dialog box, click Hologres in the Big Data Storage section.

    This example describes how to add a Hologres data source. This allows you to query data in MaxCompute in real time.

    1. In the Add Hologres data source dialog box, configure the parameters that are shown in the following table.

      Parameter

      Description

      Data Source Type

      The type of the Hologres data source. You can set it only to Alibaba Cloud Instance Mode.

      Data Source Name

      The name of the data source. The name can contain only letters, digits, and underscores (_), and must start with a letter.

      Data Source Description

      The description of the data source. The description can be a maximum of 80 characters in length.

      Environment

      The environment in which the data source is used. Valid values: Development and Production.

      Note

      This parameter is displayed only when the workspace is in standard mode.

      Instance ID

      The ID of the Hologres instance from which you want to synchronize data. You can obtain the instance ID from the Hologres console.

      Database Name

      The name of the Hologres database.

      AccessKey ID

      The AccessKey secret that is used to access the Hologres database. You can obtain the AccessKey secret on the Security Management page.

      AccessKey Secret

      The AccessKey secret that is used to access the Hologres database. You can obtain the AccessKey secret on the Security Management page.

    2. Find the desired resource group and click Test connectivity in the Actions column.

    3. After the data source passes the network connectivity test, click Complete.

Create an API

After you create the data source, go to the DataService Studio page. This example describes how to create an API by using the codeless UI.

  1. Click the More icon in the upper-left corner and choose All Products > Data Service > DataService Studio.

  2. Move the pointer over the image icon in the Service Development pane and choose Create API > Generate API.

  3. In the Generate API dialog box, configure the parameters. In this example, set API Mode to Wizard Mode.

    Parameter

    Description

    API Name

    The name of the API. The name must be 4 to 50 characters in length, and can contain letters, digits, and underscores (_). It must start with a letter.

    API Path

    The path of the API, such as /user.

    Protocol

    The protocol used by the API. Valid values: HTTP and HTTPS.

    Request Method

    The request method used by the API. Valid values: GET and POST.

    Response Content Type

    The type of the API response content. The value is fixed to JSON.

    Description

    The description of the API.

  4. Click Determine. The configuration tab of the API appears.

Configure the API parameters

  1. In the Select Table section, select Hologres from the Datasource Type drop-down list and configure the Datasource Name and Table Name parameters.

    Note
    • Before you configure API parameters, you must add and configure the required data source in Data Integration. You can enter a table name in the Table Name field to search for the desired table.

    • After you create an API, the API configuration tab appears. Then, you can select a table for the API.

  2. In the Environment Configuration section, configure the Memory and Function Timeout parameters.环境配置

  3. After you select a table in the Select Table section, all fields in the table appear in the Select Parameters section. Select the fields that need to be specified as request parameters and the fields that need to be specified as response parameters. Add them to the request parameter list and the response parameter list.选择参数

  4. Edit request parameters.

    In the right-side navigation pane, click the Request Param tab. In the Request Param panel, configure the Parameter Name, Type, Operator, Required, Example Value, Default Value, and Description parameters.请求参数

  5. Edit response parameters.

    In the right-side navigation pane, click the Response Param tab. In the Response Param panel, configure the Parameter Name, Type, Example Value, and Description parameters. You can also select Pagination and Filter in the Advanced Settings section.返回参数

    Select Pagination based on your business requirements.

    • If you do not select Pagination, the API returns a maximum of 2,000 records by default.

    • If the number of records returned by the API exceeds 2,000, we recommend that you select Pagination.

Test the API

After you configure the API parameters and save the settings, click Test in the upper-right corner. The Test APIs dialog box appears.测试

Configure the parameters and click Test to send an API request. The request and response details appear on the right. If the API fails the test, check the error message, modify the API settings, and then test the API again.

Publish the API

  1. After the API passes the test, return to the DataService Studio page.

  2. Click Publish in the upper-right corner to publish the API.

  3. After the API is published, click Service Management in the upper-right corner to view the API details.服务管理

If you want to call the API, click Service Management. In the left-side navigation pane of the page that appears, click Call APIs. DataService Studio supports quick authentication based on the AppCode and encrypted signature authentication based on the AppKey and AppSecret. The following sections describe how to call DataService Studio APIs in DataV.

Add DataService Studio as a data source

  1. Log on to the DataV console.

  2. Go to the Data Sources tab. On this tab, click Add Source.

  3. In the Add Data Source dialog box, configure the parameters.添加数据

    Parameter

    Description

    Type

    The type of the data source.

    Name

    The name of the data source. You can specify a custom name.

    Project

    The DataWorks workspace from which the required data comes.

    AppKey/AppSecret

    The AppKey and AppSecret of the account that has access permissions on the workspace in which the DataService Studio service is used.

    Note

    To obtain the AppKey and AppSecret, log on to the DataWorks console, go to the DataService Studio page, and then click Service Management in the upper-right corner. In the left-side navigation pane of the page that appears, click Call APIs.API调用

Call an API of DataService Studio in DataV

  1. Log on to the DataV console. In the top navigation bar, click Projects. On the page that appears, click Create Project.

  2. On the page that appears, select a template and click Create Project. In this example, select the Smart Factory template.智能工厂

    Widgets in the template contain static data. This example describes how to call the API to query the growth trend of the transaction amount and present the query results in the Basic Line Chart widget.

  3. Select the Basic Line Chart widget and go to the data panel. In the data panel, set the Data Source Type parameter to DataWorks.

  4. Select the added data source and created API and configure the query parameters. In this example, set the Page Size parameter to 31 to query data of one month.查询

  5. Click Preview Data Response to view the query results of the API.

  6. Set the x field to date and the y field to amount.字段映射

    The preceding figure shows that the x and y fields match no data. This is because DataV has requirements for the data format and cannot identify fields with a deep structure. You need to add a data filter to filter out unnecessary fields.

  7. Select Data Filter and click the plus (+) icon on the right to add a data filter. You can also enter JavaScript code in the code editor to filter and process the data analysis results. The data parameter of the filter specifies the JSON objects returned by the API.

    In this example, enter the code return data.data.rows; to filter out fields except the rows array returned by the API. Click Preview to preview the filtered data in the lower part, and click Complete.使用过滤器

    After you add a data filter, DataV can match data based on the specified fields.匹配成功

    The date format returned by the API is different from the default date format of the widget. Therefore, DataV does not correctly display the line chart. You must specify the date format for the X axis of the line chart before DataV can correctly display the line chart.

  8. Go to the configuration panel. Click Axis Label under X Axis. Set the data type to date and the data format to a value in the YYYY/MM/DD format, such as 2016/01/01. Then, DataV correctly displays the line chart.展示

After the preceding steps are performed, DataV can call the API created in DataService Studio based on a MaxCompute table to obtain and display the data analysis results.效果

Usage notes

After you connect DataV to DataWorks DataService Studio by adding DataService Studio as a data source of DataV, you can directly call an API of DataService Studio in DataV. In this case, you do not need to enter the API URL. You do not need to enter the AppKey and AppSecret for calling each API. In addition, you can configure API parameters by using a table. This is convenient, secure, and reliable.

DataService Studio allows you to directly present the data analysis results processed by MaxCompute in DataV. This seamlessly links data analytics and data presentation.

When you use DataService Studio and DataV together, take note of the following items:

  • APIs created on the codeless UI in DataWorks DataService Studio can only query data in a single table based on simple criteria. APIs created in the code editor allow you to use SQL statements or functions to query data in multiple tables based on complex criteria. You can create APIs by using the codeless UI or code editor based on your business requirements.

  • If you want to call an API to query data within milliseconds, we recommend that you use a relational database, a NoSQL database, or an AnalyticDB database as the data source.

  • DataV widgets only support arrays, but DataService Studio APIs return complete JSON strings that contain error codes. Therefore, you must use filters to process results returned by the APIs. You can add filters in DataV or when you configure APIs in DataService Studio.

    If pagination is disabled for an API, add a filter to filter out fields except the data array returned by the API. If pagination is enabled for an API, add a filter to filter out fields except the data.rows array returned by the API.

  • If you want to present a data analysis result in multiple types of charts, such as in both a line chart and a column chart, DataV treats the data of each type as an object and distinguishes the types by field. In this case, you need to use a filter for format conversion.