DataWorks supports StarRocks data sources. Connecting to EMR Serverless StarRocks through a data source enables Data Integration, development, analytics, and DataService capabilities. This topic describes the procedure in DataWorks.
Background
About EMR Serverless StarRocks
StarRocks is a next-generation, high-performance MPP (massively parallel processing) database for fast, unified analytics.
EMR Serverless StarRocks is a fully managed Alibaba Cloud service based on open-source StarRocks. Use it to create and manage StarRocks instances and data. As a MySQL protocol-compatible OLAP engine, StarRocks offers high performance and models for multidimensional OLAP, data lake analytics, highly concurrent queries, and real-time analytics.
Use Data Studio (New Version) workspaces can use Associate an EMR Serverless StarRocks computing resource to generate a data source of the same name, without creating one through the steps in this topic.
Workspaces that have not Use Data Studio (New Version) must create a StarRocks data source as described here to use StarRocks for development in DataWorks.
About DataWorks on EMR Serverless StarRocks
DataWorks is an end-to-end big data development and governance platform. Through a data source connection to EMR Serverless StarRocks, it supports integration and periodic scheduling of StarRocks jobs. Combined with the engine’s fast analytics and DataService capabilities, DataWorks helps you use StarRocks in many business scenarios.
Prerequisites
Activate DataWorks and create a workspace. For the procedure, see Purchase guide.
Purchase a resource group, bind it to the workspace, and configure its network. For details, see Resource Group Overview.
Create an EMR Serverless StarRocks instance. For the procedure, see Get started with a storage-compute integrated instance.
NoteAfter creating the instance, view its details in the EMR console and use EMR StarRocks Manager to connect and inspect databases and tables.
Add the DataWorks resource group IP addresses to the EMR Serverless StarRocks instance allowlist.
To configure the instance allowlists:
On the EMR Serverless StarRocks instance Instance Details page, click Security Group ID and then Private Network Allowlist to configure the private network allowlist. In the FE Details section, click Public Address and then Public Network Allowlist to configure the public network allowlist.
Create a data source
To use EMR Serverless StarRocks in DataWorks, first create a StarRocks data source connected to the existing database. Then use it in DataWorks modules.
Use Data Studio (New Version) workspaces can use Associate an EMR Serverless StarRocks computing resource to generate a data source of the same name without using the following steps.
Workspaces that have not Use Data Studio (New Version) must create a StarRocks data source as described here to use StarRocks for development in DataWorks.
For data source details, see StarRocks data source. The data source entry point and key settings are:
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Management Center.
-
On the Workspace Management page, click Data Sources in the left-side navigation pane to open the data source page.
-
Click Add Data Source. Configure the following key settings; other parameters can retain their defaults.
Choose how to create the data source based on connectivity between the StarRocks instance and DataWorks resource group. For connectivity options, see Overview.
Private network connection
Key parameter
Description
Configuration mode
Select ApsaraDB for RDS.
Alibaba Cloud Account
If the instance and DataWorks are under the same account, select Current Alibaba Cloud Account.
If the instance is under another Alibaba Cloud account, select Another Alibaba Cloud Account. After selecting Another Alibaba Cloud Account, also configure UID of Alibaba Cloud Account and RAM Role. For more about the other party’s RAM role, see Cross-account authorization (RDS, Hive, or Kafka).
Region
Select the region of the EMR Serverless StarRocks instance.
Instance
Select the specific StarRocks instance of the Serverless edition.
Database Name
Name of the database to connect. Use EMR StarRocks Manager and check Metadata Management after connecting to the instance.
Username/Password
Instance username and password.
An admin user is created by default when the StarRocks instance is created. Its password is set at instance creation.
Connection Configuration
Test connectivity between the data source and purchased resource group. The status Connectable indicates that the network connection is available.
Public network connection
Key parameter
Description
Configuration mode
Select User-created Data Store with Public IP Addresses.
Host Address/IP Address
The FE Public network address.
public address format is
fe-c-<instance-id>.starrocks.aliyuncs.com, with HTTP port8030. Click Public Network Allowlist to configure an access allowlist, or Disable Public Access to stop public access.Port
The FE Query port (9030 by default).
Load URL
StarRocks FE addresses are used for Stream Load. You can specify multiple FE addresses in the form
FE public address:FE HTTP port, separated by commas.Database Name
Name of the database to connect. Use EMR StarRocks Manager and check Metadata Management after connecting to the instance.
Username/Password
Instance username and password.
An admin user is created by default when the StarRocks instance is created. Its password is set at instance creation.
Connection Configuration
Test connectivity between the data source and purchased resource group. The status Connectable indicates that the network connection is available.
Data Integration
DataWorks can synchronize MySQL, Hive, Kafka, OSS, HDFS, and other sources into EMR Serverless StarRocks tables. The following example shows how to batch synchronize MySQL data to such a table.
For more StarRocks synchronization task settings, see StarRocks data source.
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
Create a batch synchronization node. Set MySQL as the source and StarRocks as the destination.
In the left navigation tree, expand Data Integration and click Create Node > Batch Synchronization. In the Network and Resource Configuration step, select a resource group and click Test Connectivity for both source and destination. Confirm that both show Connected.
Test connectivity from the selected resource group to the source and destination data sources.
Set a scheduling cycle, then commit and publish the node for periodic runs.
After debugging, click Scheduling in the sidebar. Configure the scheduling cycle, rerun policy, other scheduling parameters, and resource group. Then click Submission and Deploy.
Development and scheduled operations
For EMR Serverless StarRocks jobs that require periodic scheduling, create a StarRocks node in DataStudio and select the connected data source. Write the StarRocks SQL and set a scheduling cycle. Key steps:
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
Create a StarRocks node in DataStudio and select the connected StarRocks data source to write SQL.
Right-click Database Nodes in a business process and select Create Node > StarRocks. In the SQL editor, select the data source and write SQL, such as
SELECT * FROM testdb.testtb;.To debug StarRocks SQL in DataStudio, select the SQL statement, click Run, choose a resource group, and debug it.
Set a scheduling cycle, then commit and publish the node for periodic runs.
After debugging, click Scheduling in the sidebar. Configure the scheduling cycle, rerun policy, other scheduling parameters, and resource group. Then click Submission and Deploy.
Data Analysis
Use Data Analysis to analyze EMR Serverless StarRocks table data quickly:
-
Go to the SQL Query page.
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Click Go to DataAnalysis to open the Data Analysis page. In the left-side navigation pane, click SQL Query.
Click the
icon in the left sidebar, then click to open the System Management page. Set the StarRocks query resource group to the group used by the task.In SQL Query, under Resource group settings, the StarRocks engine uses the public scheduling resource group by default. Change it to the target resource group and click Save.
Return to SQL Query, switch the engine to StarRocks in the upper-right corner, select a data source, and run a query to analyze data.
In the settings pane, also select a workspace and click OK.
DataService Studio
DataService Studio can generate APIs from StarRocks data sources.
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. Select a workspace from the drop-down list and click Go to DataService Studio.
Generate an API and configure its parameters.
DataService Studio offers wizard and script modes. Script mode can generate request and response parameters from SQL. The following wizard example shows key steps.
In the left navigation tree, expand API under the business process, right-click, and select Create API > Generate API to open wizard configuration.
Select StarRocks as the data source type, the created StarRocks data source and table, then configure request and response parameters.
Click Service Resource Group in the right sidebar and select an exclusive resource group for DataService Studio.
In the Exclusive Service Resource Group list, select a group such as test, and set API Gateway Instance Type to Shared Instance and , to
Timeoutto Maximum Rows per Request to10000rows.After testing the API, commit and publish it.
Data Map
Data Map supports metadata collection, search, and table details for StarRocks.
Metadata collection
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
Click the
icon in the left navigation pane, then click Management.Switch to the Data Sources for Which No Crawler Is Created tab. In the Operation column, click Metadata Collection.
ConfigureResource Group Name and Test Connectivity and set Collection Plan. Click Confirm to finish.
NoteFor metadata collection details, see Metadata collection.
This task can run only on a Serverless resource group.
Search
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
Click the
icon. Select the StarRocks data source, then search for tables across data source types at the top.NoteFor search details, see Metadata retrieval.
Table details
-
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
Find a table in Data Map or search and click its name to open details.
On the details page, view Table Basic Information and Technical Information and Business Information and Details and Output information and Lineage Information other details.
NoteFor table details, see Metadata retrieval.
Metadata and lineage analysis can be enabled for StarRocks Serverless clusters version 3.1.13, 3.2.9, and later. For configuration, see View lineage.
The table details page also has a Usage tab. Details contains the Field Information and Change Records subtabs. The Field Information table shows field names, data types, and descriptions.