AnalyticDB:Synchronize SLS data with the data synchronization feature

Step 1: Create a data source and data link

1. Log on to the Simple Log Service console.

2. In the Project list, click the target Project. In the left-side navigation pane, click the icon, go to the Logstores tab, and then expand the target Logstore.

3. Click Data Processing > Export, and then click the + icon next to AnalyticDB.

4. In the Shipping Note dialog box, select Create in AnalyticDB for MySQL Console.

   This dialog box indicates that direct shipping sends all newly written data to the destination. If you need to process data, such as by filtering, converting formats, or enriching data, or if you need to ship historical data, you can use the Data Transformation feature. This requires staging the data in a temporary Logstore and then creating a direct shipping task from it.

5. On the AnalyticDB for MySQL Log Synchronization page, configure the parameters in the Source and Destination Settings, Destination Database and Table Settings, and Synchronization Settings sections.

   • The following table describes the parameters for Source and Destination Settings.

     Parameter	Description
     Job Name	The name of the synchronization job. The system generates a name based on the data source type and the creation time. You can modify the name as needed.
     Simple Log Service Project	The source SLS Project.
     Simple Log Service Logstore	The source SLS Logstore.
     Destination AnalyticDB for MySQL Cluster	Select the destination AnalyticDB for MySQL cluster.
     AnalyticDB for MySQL Account	The database account for the AnalyticDB for MySQL cluster.
     AnalyticDB for MySQL Password	The password for the AnalyticDB for MySQL database account.

   • Destination Database and Table Settings

     Parameter	Description
     Database Name	The name of the database in the AnalyticDB for MySQL cluster.
     Table Name	The name of the destination table in the AnalyticDB for MySQL cluster.
     Source Data Preview	Click View Latest 10 Logstore Data Entries to preview the 10 most recent entries from the source SLS logstore.
     Schema Field Mapping	The AnalyticDB for MySQL cluster table fields automatically populate the Destination Field and Source Field. If the mapping between the Destination Field and Source Field is incorrect, modify it manually. For example, if a field in a data table of an AnalyticDB for MySQL cluster is named name and the source data field in SLS is named user_name, the system automatically populates the Source Field and Destination Field with name. In this case, you must manually change the Source Field to user_name.

   • The following table describes the parameters for Synchronization Settings.

     Parameter	Description
     Start Offset	Specifies the point in time from which the synchronization job starts consuming SLS data. For example, if you set the Start Offset to 2024-04-09 13:10, the system starts consuming data from the first record created after 13:10 on April 9, 2024.
     Dirty Data Processing Mode	Synchronization can fail if a source field's data type is incompatible with the corresponding destination field's type. For example, if the source data is abc and the destination field type is int, a conversion error occurs, causing synchronization to fail. The following modes are available for handling dirty data: Stop Synchronization (Default): The synchronization job stops. You must modify the destination field type or select a different dirty data processing mode, and then restart the job. Treat as Null: Dirty data is converted to NULL and written to the destination table. The original dirty data is discarded. For example, if a row in SLS has three fields (col1, col2, and col3) and the data in col2 is dirty, the value of col2 is set to NULL, while the data in col1 and col3 is written as usual.
     Convert UNIX Timestamp into Datetime	If the source SLS field is a Unix timestamp (for example, 1710604800) and the destination field type is DATETIME or TIMESTAMP, enable this feature to convert the value. After you enable this option, select the precision of the source timestamp: Timestamp Accurate to Seconds, Timestamp Accurate to Milliseconds, or Timestamp Accurate to Microseconds.
     Job Resource Group	Select the job-specific resource group to run the incremental sync task. Important This parameter applies only to AnalyticDB for MySQL clusters of the Enterprise Edition, Basic Edition, or Data Lakehouse Edition.
     ACUs for Incremental Synchronization	The initial number of ACUs for the incremental sync task is fixed at 1. After the data link is created, AnalyticDB for MySQL automatically scales the number of ACUs for the task based on the workload, from a minimum of 1 ACU to a maximum of 64 ACUs. Important This parameter applies only to AnalyticDB for MySQL clusters of the Enterprise Edition, Basic Edition, or Data Lakehouse Edition.

6. After you configure the parameters, click Submit.

   The system automatically creates a Simple Log Service data source and a data link in AnalyticDB for MySQL and redirects you to the Simple Log Service/Kafka Data Synchronization page in the AnalyticDB for MySQL console.

Step 2: Start the sync task

1. In the Actions column for the target data link, click Start.

2. After starting the task, click Search in the upper-right corner. The sync task has started successfully when its status changes to Running.

Step 3: Manage the sync task

In the Actions column, you can perform the following operations:

Step 1: Configure RAM authorization (optional)

For cross-account synchronization of Simple Log Service (SLS) data to AnalyticDB for MySQL, you must create a RAM role in the source account, grant the role fine-grained authorization, and modify its trust policy.

1. Create a RAM role. For more information, see Create a RAM role for a trusted Alibaba Cloud account.

   Note

   When you configure the Select Trusted Alibaba Cloud Account parameter, select Other Alibaba Cloud Account and enter the account ID of the Alibaba Cloud account that contains the AnalyticDB for MySQL cluster. You can find the account ID on the Overview page in the Account Center.

2. Use fine-grained authorization to grant the AliyunAnalyticDBAccessingLogRolePolicy permission to the RAM role.

3. Modify the trust policy of the RAM role to allow the AnalyticDB for MySQL cluster in the specified Alibaba Cloud account to assume this role.

   {
     "Statement": [
       {
         "Action": "sts:AssumeRole",
         "Effect": "Allow",
         "Principal": {
           "RAM": [
               "acs:ram::<Alibaba Cloud account ID>:root"
           ],
           "Service": [
               "<Alibaba Cloud account ID>@ads.aliyuncs.com"
           ]
         }
       }
     ],
     "Version": "1"
   }

   Note

   Replace <Alibaba Cloud account ID> with the account ID of the Alibaba Cloud account that contains the AnalyticDB for MySQL cluster, as specified in Step 1. Do not include the angle brackets (<>).

Step 2: Create a data source

1. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

2. In the left-side navigation pane, choose Simple Log Service Data Ingestion > Data Sources.

3. In the upper-right corner, click Create Data Source.

4. On the Create Data Source page, configure the parameters as described in the following table.

   Parameter	Description
   Data Source Type	Select SLS.
   Data Source Name	The system generates a name based on the data source type and the creation time. You can modify the name as needed.
   Data Source Description	An optional description of the data source, such as its use case or business-related constraints.
   Deployment Mode	Only Alibaba Cloud Instance is supported.
   Region of Simple Log Service Project	The region where the SLS project is located. Note You can only select the region where your AnalyticDB for MySQL cluster is located.
   Across Alibaba Cloud Accounts	AnalyticDB for MySQL clusters can synchronize SLS data from the same Alibaba Cloud account or a different one (cross-account). No: Synchronize SLS data from the current Alibaba Cloud account to the AnalyticDB for MySQL cluster. Yes: Synchronize SLS data from a different Alibaba Cloud account to the AnalyticDB for MySQL cluster. If you select Yes, you must configure RAM authorization and specify the Alibaba Cloud Account and RAM Role. For more information about RAM authorization, see Step 1: Configure RAM authorization (optional). Note Alibaba Cloud Account: The account ID of the Alibaba Cloud account that contains the SLS project. RAM Role: The RAM role in the Alibaba Cloud account that contains the SLS project. This is the RAM role that you created in Step 1.
   Simple Log Service Project	The source SLS project.
   Simple Log Service Logstore	The source SLS logstore.

5. After you configure the parameters, click Create.

Step 3: Create a synchronization job

1. In the left-side navigation pane, choose Simple Log Service Data Ingestion > Data Synchronization.

2. In the upper-right corner, click Create Synchronization Job.

3. On the Create Synchronization Job page, configure the Source and Destination Settings, Destination Database and Table Settings, and Synchronization Settings sections. The parameters are described in the following tables.

   • Source and Destination Settings

     Parameter	Description
     Job Name	The name of the synchronization job. The system generates a name based on the data source type and the creation time. You can modify the name as needed.
     Data Source	Select an existing SLS data source, or create a new one.
     Destination Type	For Enterprise Edition, Basic Edition, and Data Lakehouse Edition clusters, select Data Warehouse - ADB Storage. This parameter does not apply to Data Warehouse Edition clusters.
     AnalyticDB for MySQL Account	The database account for the AnalyticDB for MySQL cluster.
     AnalyticDB for MySQL Password	The password for the AnalyticDB for MySQL database account.

   • Destination Database and Table Settings

     Parameter	Description
     Database Name	The name of the database in the AnalyticDB for MySQL cluster.
     Table Name	The name of the destination table in the AnalyticDB for MySQL cluster.
     Source Data Preview	Click View Latest 10 Logstore Data Entries to preview the 10 most recent entries from the source SLS logstore.
     Schema Field Mapping	The AnalyticDB for MySQL cluster table fields automatically populate the Destination Field and Source Field. If the mapping between the Destination Field and Source Field is incorrect, modify it manually. For example, if a field in a data table of an AnalyticDB for MySQL cluster is named name and the source data field in SLS is named user_name, the system automatically populates the Source Field and Destination Field with name. In this case, you must manually change the Source Field to user_name.

   • Synchronization Settings

     Parameter	Description
     Start Offset	Specifies the point in time from which the synchronization job starts consuming SLS data. For example, if you set the Start Offset to 2024-04-09 13:10, the system starts consuming data from the first record created after 13:10 on April 9, 2024.
     Dirty Data Processing Mode	Synchronization can fail if a source field's data type is incompatible with the corresponding destination field's type. For example, if the source data is abc and the destination field type is int, a conversion error occurs, causing synchronization to fail. The following modes are available for handling dirty data: Stop Synchronization (Default): The synchronization job stops. You must modify the destination field type or select a different dirty data processing mode, and then restart the job. Treat as Null: Dirty data is converted to NULL and written to the destination table. The original dirty data is discarded. For example, if a row in SLS has three fields (col1, col2, and col3) and the data in col2 is dirty, the value of col2 is set to NULL, while the data in col1 and col3 is written as usual.
     Convert UNIX Timestamp into Datetime	If the source SLS field is a Unix timestamp (for example, 1710604800) and the destination field type is DATETIME or TIMESTAMP, enable this feature to convert the value. After you enable this option, select the precision of the source timestamp: Timestamp Accurate to Seconds, Timestamp Accurate to Milliseconds, or Timestamp Accurate to Microseconds.

4. After you configure all the parameters, click Submit.

Step 4: Start the data synchronization job

1. On the Data Synchronization page, find the synchronization job you created. In the Actions column, click Start.

2. In the upper-right corner, click Search. The job starts successfully when its status changes to Running.

Step 5: Manage the synchronization job

On the Data Synchronization page, you can perform the following operations in the Actions column.