Configure AnalyticDB for MySQL 3.0 Reader

Data types

The following table lists the data types supported by AnalyticDB for MySQL 3.0 Reader.

Parameters

Configure AnalyticDB for MySQL 3.0 Reader by using the codeless UI

1. Configure the connections.

   Configure the source and destination connections for the sync node.

   Parameter	Description
   Connection	The datasource parameter in the preceding parameter description. Select a connection type, and enter the name of a connection that has been configured in DataWorks.
   Table	The table parameter in the preceding parameter description.
   Filter	The filter condition for the data to be synchronized. Currently, filtering based on the limit keyword is not supported. The SQL syntax is determined by the selected connection.
   Shard Key	The shard key. You can use a column in the source table as the shard key. We recommend that you use the primary key or an indexed column. Only integer fields are supported. If data sharding is performed based on the configured shard key, data can be read concurrently to improve data synchronization efficiency. Note The Shard Key parameter is displayed only when you configure the source connection for a sync node.

2. Configure field mapping, that is, the column parameter in the preceding parameter description.
   Fields in the source table on the left have a one-to-one mapping with fields in the destination table on the right. You can click Add to add a field, or move the pointer over a field and click the Delete icon to delete the field.

   Parameter	Description
   Map Fields with the Same Name	Click Map Fields with the Same Name to establish a mapping between fields with the same name. Note that the data types of the fields must match.
   Fields in the Same Line	Click Map Fields in the Same Line to establish a mapping for fields in the same row. Note that the data types of the fields must match.
   Delete All Mappings	Click Delete All Mappings to remove mappings that have been established.
   Auto Layout	Click Auto Layout. The fields are automatically sorted based on specified rules.
   Change Fields	Click the Change Fields icon. In the Change Fields dialog box that appears, you can manually edit fields in the source table. Each field occupies a row. The first and the last blank rows are included, whereas other blank rows are ignored.
   Add	Click Add to add a field. You can enter constants. Each constant must be enclosed in single quotation marks (' '), such as 'abc' and '123'. You can use scheduling parameters, such as ${bizdate}. You can enter functions supported by relational databases, such as now() and count(1). Fields that cannot be parsed are indicated by Unidentified.

3. Configure channel control policies.

   Parameter	Description
   Expected Maximum Concurrency	The maximum number of concurrent threads to read and write data to data storage within the sync node. You can configure the concurrency for a node on the codeless UI.
   Bandwidth Throttling	Specifies whether to enable bandwidth throttling. You can enable bandwidth throttling and set a maximum transmission rate to avoid heavy read workload of the source. We recommend that you enable bandwidth throttling and set the maximum transmission rate to a proper value.
   Dirty Data Records Allowed	The maximum number of dirty data records allowed.
   Resource Group	The resource group used for running the sync node. If a large number of nodes including this sync node are deployed on the default resource group, the sync node may need to wait for resources. We recommend that you purchase an exclusive resource group for data integration or add a custom resource group. For more information, see DataWorks exclusive resources and Add a custom resource group.

Configure AnalyticDB for MySQL 3.0 Reader by using the code editor

In the following code, a node is configured to read data from an AnalyticDB for MySQL 3.0 database. For more information about the parameters, see the preceding parameter description.

{
    "type": "job",
    "steps": [
        { 
            "stepType": "analyticdb_for_mysql", // The reader type.
             "parameter": {
                "column": [ // The columns to be synchronized.
                     "id",
                    "value",
                    "table"
                ],
                "connection": [
                    {
                        "datasource": "",// The connection name.
                         "table":[// The name of the table to be synchronized.
                             "xxx"
                        ]
                    }
                ],
                "where": "", // The WHERE clause.
                 "splitPk": "", // The shard key.
                 "encoding": "UTF-8" // The encoding format.
             },
            "name": "Reader",
            "category": "reader"
        },
        {// The following template is used to configure Stream Writer. For more information, see the corresponding topic.
             "stepType": "stream",
            "parameter": {},
            "name": "Writer",
            "category": "writer"
        }
    ],
    "version": "2.0",
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {
            "record":"0" // The maximum number of dirty data records allowed.
         },
        "speed": {
            "concurrent": 2,// The maximum number of concurrent threads.
             "throttle": false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
         }
    }
}