Integrate DataWorks with EMR Serverless StarRocks - DataWorks

DataWorks connects to EMR Serverless StarRocks through StarRocks data sources, enabling you to integrate, develop, analyze, and serve your StarRocks data from a single platform.

Background

EMR Serverless StarRocks

StarRocks is a next-generation Massively Parallel Processing (MPP) database designed for a fast, unified analytics experience.

EMR Serverless StarRocks is a fully managed service for open-source StarRocks on Alibaba Cloud that lets you create and manage StarRocks instances and data. Compatible with the MySQL protocol, StarRocks delivers high-performance OLAP capabilities for multidimensional analysis, data lake analytics, high-concurrency queries, and real-time data analysis.

Important

For workspaces that use Use Data Studio (New Version), a data source with the same name is automatically generated when you bind an E-MapReduce (EMR) Serverless StarRocks compute resource. You can skip the data source creation steps in this topic.
For workspaces that do not Use Data Studio (New Version), you must create a StarRocks data source as described in this topic to use StarRocks in DataWorks.

DataWorks on EMR Serverless StarRocks

DataWorks connects to EMR Serverless StarRocks through data sources, enabling data integration, periodic job scheduling, interactive analytics, and API generation powered by the StarRocks engine.

Click to view basic concepts and an overview of the main DataWorks components

The following table describes the main DataWorks components and concepts used with EMR Serverless StarRocks.

Concept/component	Description	References
resource group	A DataWorks resource group is required to run tasks.	For more information about the billing of resource groups, see Resource group fees. For more information about how to purchase and configure resource groups, see Resource group management.
data source	A StarRocks data source connects DataWorks to EMR Serverless StarRocks for task development and execution. You must create this data source before you can use StarRocks in DataWorks.	For more information, see StarRocks data source.
Data Integration	Data Integration enables data synchronization between various data sources across multiple scenarios.	For more information, see Data Integration.
Data Development and Operation Center	Data Development and Operation Center work together: develop and debug tasks in Data Development, then submit and deploy them to Operation Center for periodic execution.	For more information about the Data Development component, see Data Development (DataStudio) (Old Version). For more information about the Operation Center component, see Operation Center Overview.
Data Analysis	Data Analysis lets you query, analyze, edit, and share data online.	Data Analysis
DataService Studio	DataService Studio is a platform for building data APIs. It provides data service and sharing capabilities for individuals, teams, and enterprises, and helps you manage internal and external API services in one place.	DataService Studio
Data Map	Data Map is an enterprise data catalog built on metadata. It provides global data search, metadata details, data preview, data lineage, and data category management to help you discover, understand, and use data.	Data Map overview

Prerequisites

You have purchased a resource group and completed workspace binding and network configuration. For more information, see Resource group management.
You have created an EMR Serverless StarRocks instance. For more information, see Quick start for all-in-one instances.
Note
After you create the instance, you can view its details in the EMR console and connect to it through EMR StarRocks Manager to view databases and tables.
You have added the IP address of the DataWorks resource group to the whitelist of the EMR Serverless StarRocks instance.
You can configure the whitelist for an EMR Serverless StarRocks instance by using the following methods:
On the Instance Details page of the EMR Serverless StarRocks instance, click Internal Whitelist to the right of Security Group ID to configure the internal network whitelist. In the FE Details section, click Public Whitelist to the right of Public Endpoint to configure the public whitelist.

Create data source

To use EMR Serverless StarRocks in DataWorks, first create a StarRocks data source that connects to your EMR Serverless StarRocks database. This enables StarRocks across all DataWorks components.

Important

For workspaces that use Use Data Studio (New Version), a data source with the same name is automatically generated when you bind an EMR Serverless StarRocks compute resource. You can skip the data source creation steps in this topic.
For workspaces that do not Use Data Studio (New Version), you must create a StarRocks data source as described in this topic to use StarRocks in DataWorks.

For more information, see StarRocks data source. The entry point and key configuration parameters are described in the following steps.

1. Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.
2. On the Workspace Management page, click Data Sources in the left-side navigation pane to open the data source page.
1. Log on to the DataWorks console. In the target region, click More > Management Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.
2. On the Workspace Management page, click Data Sources in the left-side navigation pane to open the data source page.

Click Add Data Source. Configure the key parameters as described in the following section. You can leave other parameters at their default values.

Choose the appropriate method to create the data source based on the network connectivity between your StarRocks instance and the DataWorks resource group. For more information about network connectivity solutions, see Overview of network connectivity solutions.

Internal network

Parameter	Description
Configuration Mode	Select ApsaraDB for RDS.
Alibaba Cloud Account	If the EMR Serverless StarRocks instance and DataWorks belong to the same account, select Current Alibaba Cloud Account. If the EMR Serverless StarRocks instance belongs to another Alibaba Cloud account, select Another Alibaba Cloud Account. After you select Another Alibaba Cloud Account, you must also configure the UID of Alibaba Cloud Account and the RAM Role. For more information about configuring the RAM role of the other account, see Cross-account authorization (RDS, Hive, or Kafka).
Region	Select the region where your EMR Serverless StarRocks instance resides.
Instance	Select the target Serverless StarRocks instance.
Database Name	The name of the database that you want to connect to. You can find this name in Metadata Management after you connect to the instance by using EMR StarRocks Manager.
Username/Password	The username and password of the instance. When you create a StarRocks instance, a default admin user is created. The password is the one that you specified during instance creation.
Connection Configuration	Test the connectivity between the data source and the selected resource group. A status of Connectable indicates that the connection is successful.

Public network

Parameter	Description
Configuration Mode	Select User-created Data Store with Public IP Addresses.
Host Address/IP Address	The Public network address of the FE in the EMR Serverless StarRocks instance. The public endpoint is in the `fe-c-<instance-id>.starrocks.aliyuncs.com` format, and the HTTP port is `8030`. You can click Public Whitelist to configure an access whitelist, or click Disable Public Access to deactivate public access.
Port	The query port of the FE in the EMR Serverless StarRocks instance. The default value is 9030.
Load URL	The addresses of StarRocks FE nodes used for StreamLoad. Separate multiple addresses with commas. The address format is `FE public endpoint:FE HTTP port`.
Database Name	The name of the database that you want to connect to. You can find this name in Metadata Management after you connect to the instance by using EMR StarRocks Manager.
Username/Password	The username and password of the instance. When you create a StarRocks instance, a default admin user is created. The password is the one that you specified during instance creation.
Connection Configuration	Test the connectivity between the data source and the selected resource group. A status of Connectable indicates that the connection is successful.

Data Integration

DataWorks supports synchronizing data from various sources, such as MySQL, Hive, Kafka, OSS, and HDFS, to EMR Serverless StarRocks tables. The following example shows how to batch synchronize data from MySQL to a StarRocks table.

Note

For more information about how to configure StarRocks data source synchronization tasks, see StarRocks data source.

Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
Create a batch synchronization node. Set the data source to MySQL and the destination to the StarRocks data source.
In the left-side navigation tree, expand Data Integration, and then choose New Node > Batch Synchronization. In the Network and Resource Configuration step, select a resource group and click Test Connectivity for both the source and destination to verify that both are Connected.
After you select the resource group, test the connectivity to the source and destination data sources.
Set a scheduling cycle, and then submit and deploy the node to run periodically.
After you debug the node, click Scheduling in the right-side pane to configure scheduling parameters such as the scheduling cycle and rerun policy, and set the resource group for the task. After the configuration is complete, click Submission and Deploy.

Data Development and scheduling

To schedule EMR Serverless StarRocks tasks periodically, create a StarRocks node in Data Studio where you can write SQL, connect to a StarRocks data source, and configure scheduling. The key steps are as follows:

Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
In Data Studio, create a StarRocks node, select the connected StarRocks data source, and then write your SQL task.
Right-click the database node in your business flow and choose New Node > StarRocks. In the SQL editor that opens, select the corresponding data source and write your SQL statement. Example: SELECT * FROM testdb.testtb;.
To debug an EMR Serverless StarRocks SQL task in Data Studio, select the SQL statements to debug, click the Run button, and then select a resource group for the test run.
Set a scheduling cycle, and then submit and deploy the node to run periodically.
After you debug the node, click Scheduling in the right-side pane to configure scheduling parameters such as the scheduling cycle and rerun policy, and set the resource group for the task. After the configuration is complete, click Submission and Deploy.

Data Analysis

Use Data Analysis to quickly query and analyze data in EMR Serverless StarRocks tables. The key steps are as follows:

Go to the SQL Query page.
Log on to the DataWorks console. In the target region, click Data Analysis and Service > DataAnalysis in the left-side navigation pane. Click Go to DataAnalysis to open the Data Analysis page. In the left-side navigation pane, click SQL Query.
Click the icon in the left-side pane, click More > System Management to go to the System Management page, and then set the query resource group for StarRocks to the one used by your task.
In the SQL Query > Resource Group section, the StarRocks engine uses the Public Scheduling Resource Group by default. Change it to your target resource group and click Save.
Return to the SQL Query page. In the upper-right corner, switch the engine type to StarRocks and select your data source. You can now write and run queries to perform data analysis on EMR Serverless StarRocks.
In the configuration panel on the right, you must also select the Workspace and then click OK.

DataService Studio

DataService Studio supports generating APIs from StarRocks data sources. The key steps are as follows:

Log on to the DataWorks console. In the target region, click Data Analysis and Service > DataService Studio in the left-side navigation pane. Select a workspace from the drop-down list and click Go to DataService Studio.
Generate an API and configure its parameters.
DataService Studio supports both wizard mode and script mode. Script mode can automatically generate API request and response parameters based on an SQL query. The following steps demonstrate how to use wizard mode.
In the left-side navigation tree, expand the API node under the target business flow, right-click and choose New API > Generate API to go to the wizard mode configuration page.
Set the data source type to StarRocks, select the StarRocks data source that you created, select the corresponding table, and then configure the API request parameters, response parameters, and other settings as prompted on the page.
In the right-side pane, click Service Resource Group and set the resource group to an exclusive resource group for data services.
In the Exclusive Service Resource Group drop-down list, select the target resource group, such as test. Set API Gateway Instance Type to Shared Instance, Timeout to 30000 ms, and Maximum Number of Data Rows per Request to 10000.
After you successfully test the API, submit and publish it.

Data Map

Data Map supports metadata collection, search, and table detail viewing for StarRocks data.

Metadata collection

Log on to the DataWorks console. In the target region, click Data Governance > Data Map in the left-side navigation pane. On the page that appears, click Go to Data Map.
In the left-side navigation pane, click the icon, and then click Management in the upper-right corner of the StarRocks module.
Switch to the Data Sources for Which No Crawler Is Created tab and click Metadata Collection in the Operation column.
After you configure the Resource Group Name, pass the Test Connectivity, and set the Collection Schedule, click Confirm to complete the metadata collection configuration.
Note
- For more information about metadata collection, see Metadata collection.
- This task can only run on Serverless resource groups.

Search

Log on to the DataWorks console. In the target region, click Data Governance > Data Map in the left-side navigation pane. On the page that appears, click Go to Data Map.
In the left-side navigation pane, click the icon, select StarRocks as the data source, and then use the search bar at the top to search for tables.
Note
For more information about the search feature, see Metadata retrieval.

Table details

Log on to the DataWorks console. In the target region, click Data Governance > Data Map in the left-side navigation pane. On the page that appears, click Go to Data Map.
Find the target table on the Data Map homepage or in the search results, and click the table name to go to the table details page.
On the table details page, you can view information such as Table Basic Information, Technical Information, Business Information, Details, Output information, and Lineage Information.
Note
- For more information about table details, see Metadata retrieval.
- EMR Serverless StarRocks clusters in versions 3.1.13, 3.2.9, and later support metadata and data lineage analysis. For more information about how to configure this feature, see View data lineage.
The table details page also includes a Usage Guide tab. The Details tab contains two sub-tabs: Field Information and Change History. The Field Information table displays the name, data type, and description of each field.