DataWorks:Prepare the environment

Create an instance

If you don't have an EMR Serverless StarRocks instance, create one with the following configuration:

Create the database

After the instance is created, log on to the instance and run the following SQL statement in the SQL Editor:

CREATE DATABASE user_behavior_analysis;

This database is used throughout the tutorial.

Create a workspace

1. Log on to the DataWorks console. In the upper-left corner, select the China (Shanghai) region.

2. In the left-side navigation pane, click Workspaces. On the Workspaces page, click Create Workspace and configure the parameters. For details, see Create a workspace.

If you already have a workspace, skip this step and use your existing workspace. The MySQL and HttpFile data sources in this tutorial are in the China (Shanghai) region, so use a workspace in the same region.

Create a resource group

A resource group is required to run StarRocks tasks in DataWorks.

Purchase a resource group

Purchase a serverless resource group. For details, see Use serverless resource groups.

Configure network connectivity

Make sure the resource group can reach the StarRocks data source. For an overview of network connectivity options, see Network connectivity solutions.

Complete the following tasks:

1. Check the StarRocks network environment.

   

2. Associate the resource group with the virtual private cloud (VPC) where the StarRocks instance resides.

   

3. Add the resource group's outbound IP address to the StarRocks IP address whitelist.

   1. Get the outbound IP address of the DataWorks serverless resource group.

      

   2. Click the EMR Serverless StarRocks instance name. In the Basic information section, click Add Rule and add the CIDR block of the vSwitch associated with the serverless resource group.

      

4. Configure an Internet NAT gateway so the resource group can access the StarRocks instance over the internet using an elastic IP address (EIP).

   1. Log on to the VPC console and go to the Internet NAT Gateway page. In the top navigation bar, select the China (Shanghai) region.

   2. Click Create NAT Gateway and configure the following parameters:

      Parameter	Value
      Region	China (Shanghai)
      Network and zone	Select the VPC and vSwitch associated with the resource group. To find these values: in the DataWorks console, go to Resource Groups, find the resource group, and click Network Settings in the Actions column. Under the VPC Binding tab, view the VPC and vSwitch in the Data Scheduling & Data Integration section. For details on VPCs, see What is VPC?
      EIP	Select Purchase EIP
      Service-linked role	Click Create Service-linked Role. Required the first time you create an Internet NAT gateway.

      Retain the default values for all other parameters.

   3. Click Buy Now. On the Confirm page, read and accept the terms of service, then click Confirm.

Add a StarRocks data source

1. In the left-side navigation pane of the DataWorks console, click Management Center. Select your workspace from the drop-down list and click Go to Management Center.

2. In the left-side navigation pane of the SettingCenter page, choose Data Sources > Data Sources. Click Add Data Source.

3. In the Add Data Source dialog box, click StarRocks. On the Add StarRocks Data Source page, set Configuration Mode to Alibaba Cloud Instance Mode.

   

4. Configure the basic information. Get the instance details from the EMR console.

   Parameter	Value
   Data source name	Doc_StarRocks_Storage_Compute_Tightly_01
   Configuration mode	Alibaba Cloud Instance Mode
   Region	China East 2 (Shanghai)
   Instance	Select the Serverless instance you created
   Database name	user_behavior_analysis
   Username	Your StarRocks database username
   Password	Your StarRocks database password

5. Click Test Network Connectivity. If the test passes, click Complete Creation.

Add a MySQL data source

1. In the left-side navigation pane of the SettingCenter page, click Data Sources. Click Add Data Source.

2. In the Add Data Source dialog box, select MySQL.

3. On the Add MySQL Data Source page, configure the following parameters. Use the sample values shown — this is a shared test dataset provided for this tutorial.

   Parameter	Value
   Data source name	user_behavior_analysis_mysql
   Data source description	This data source is provided for this tutorial and serves as the source for batch synchronization tasks. Use it for data reading in data synchronization scenarios only.
   Configuration mode	Connection String Mode
   Host IP address	rm-bp1z69dodhh85z9qa.mysql.rds.aliyuncs.com
   Port number	3306
   Database name	workshop
   Username	workshop
   Password	workshop#2017
   Authentication method	No Authentication

4. For each resource group, click Test Network Connectivity in both the Connection Status (Development Environment) and Connection Status (Production Environment) columns. When connectivity is confirmed, Connected appears in each column.

5. Click Complete Creation.

Add an HttpFile data source

1. In the left-side navigation pane of the SettingCenter page, choose Data Sources > Data Sources. Click Add Connection.

2. In the dialog box, click HttpFile.

   

3. Configure the following parameters:

   Parameter	Value
   Data source name	user_behavior_analysis_httpfile
   Description	This data source is provided for this tutorial and serves as the source for batch synchronization tasks. Use it for data reading in data synchronization scenarios only.
   URL	https://dataworks-workshop-2024.oss-cn-shanghai.aliyuncs.com

4. Click Test Network Connectivity. If the test passes, click Complete Creation.