HybridDB for MySQL data source - DataWorks - Alibaba Cloud Documentation Center

DataWorks provides HybridDB for MySQL Reader and HybridDB for MySQL Writer for you to read data from and write data to HybridDB for MySQL data sources. This topic describes the capabilities of synchronizing data from or to HybridDB for MySQL data sources.

Limits

Data of views can be read during batch synchronization.
HybridDB for MySQL Reader can read tables and views. If you use HybridDB for MySQL Reader to read data from tables, you can specify all or some of the columns in the tables in sequence, change the column order, specify constant fields, and configure HybridDB for MySQL functions, such as now().

Data type mappings

The following table lists the data type mappings based on which HybridDB for MySQL Reader or HybridDB for MySQL Writer converts data types.

Category	HybridDB for MySQL data type
Integer	INT, TINYINT, SMALLINT, MEDIUMINT, BIGINT, and YEAR
Floating point	FLOAT, DOUBLE, and DECIMAL
String	VARCHAR, CHAR, TINYTEXT, TEXT, MEDIUMTEXT, and LONGTEXT
Date and time	DATE, DATETIME, TIMESTAMP, and TIME
Boolean	BOOL
Binary	TINYBLOB, MEDIUMBLOB, BLOB, LONGBLOB, and VARBINARY

Note

Data types that are not listed in the preceding table are not supported.
HybridDB for MySQL Reader processes TINYINT(1) as an integer data type.

Add a data source

Before you develop a synchronization task in DataWorks, you must add the required data source to DataWorks by following the instructions in Add and manage data sources. You can view the infotips of parameters in the DataWorks console to understand the meanings of the parameters when you add a data source.

Develop a data synchronization task

For information about the entry point for and the procedure of configuring a synchronization task, see the following configuration guides.

Configure a batch synchronization task to synchronize data of a single table

For more information about the configuration procedure, see Configure a batch synchronization task by using the codeless UI and Configure a batch synchronization task by using the code editor.
For information about all parameters that are configured and the code that is run when you use the code editor to configure a batch synchronization task, see Appendix: Code and parameters.

Configure synchronization settings to implement batch synchronization of all data in a database

For more information about the configuration procedure, see Configure a synchronization task in Data Integration.

Appendix: Code and parameters

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Configure a batch synchronization task by using the code editor. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Code for HybridDB for MySQL Reader

Sample code for HybridDB for MySQL Reader to read data from a table that is not sharded:

{
    "type": "job",
    "steps": [
        {
            "parameter": {
                "datasource": "px_aliyun_hymysql",// The name of the data source. 
                "column": [// The names of the columns to which you want to write data. 
                    "id",
                    "name",
                    "sex",
                    "salary",
                    "age",
                    "pt"
                ],
                "where": "id=10001",// The WHERE clause. 
                "splitPk": "id",// The shard key. 
                "table": "person"// The name of the source table. 
            },
            "name": "Reader",
            "category": "reader"
        },
        {
            "parameter": {}
    ],
    "version": "2.0",// The version number. 
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {// The maximum number of dirty data records allowed. 
            "record": ""
        },
        "speed": {
            "concurrent": 7,// The maximum number of parallel threads. 
            "throttle": true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "mbps": 1 // The maximum transmission rate. Unit: MB/s. 
        }
    }
}

Parameters in code for HybridDB for MySQL Reader

Parameter	Description	Required	Default value
datasource	The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor.	Yes	No default value
table	The name of the table from which you want to read data. Each synchronization task can be used to synchronize data to only one table.	Yes	No default value
column	The names of the columns from which you want to read data. Specify the names in a JSON array. The default value is [*], which indicates all the columns in the source table. You can select specific columns to read. The column order can be changed. This indicates that you can specify columns in an order different from the order specified by the schema of the source table. Constants are supported. The column names must be arranged in compliance with the SQL syntax supported by HybridDB for MySQL Reader, such as `["id", "table","1","'mingya.wmy'","'null'", "to_char(a+1)","2.3","true"]`. id: a column name. table: the name of a column that contains reserved keywords. 1: an integer constant. 'mingya.wmy': a string constant, which is enclosed in single quotation marks ('). 'null': the string null. to_char(a+1): a function expression that is used to calculate the length of a string. 2.3: a floating-point constant. true: a Boolean value. The column parameter must explicitly specify all the columns from which you want to read data. The parameter cannot be left empty.	Yes	No default value
splitPk	The field that is used for data sharding when HybridDB for MySQL Reader reads data. If you configure this parameter, the source table is sharded based on the value of this parameter. Data Integration then runs parallel threads to read data. This improves data synchronization efficiency. We recommend that you set the splitPk parameter to the name of a primary key column of the table. Data can be evenly distributed to different shards based on the primary key column, instead of being intensively distributed only to specific shards. The splitPk parameter supports sharding for data only of integer data types. If you set this parameter to a field of an unsupported data type, such as a string, floating point, or date data type, HybridDB for MySQL Reader ignores the setting of the splitPk parameter and uses a single thread to read data. If the splitPk parameter is not provided or is left empty, HybridDB for MySQL Reader uses a single thread to read data.	No	No default value
where	The WHERE clause. For example, you can set this parameter to `gmt_create > $bizdate` to read the data that is generated on the current day. You can use the WHERE clause to read incremental data. If the where parameter is not provided or is left empty, HybridDB for MySQL Reader reads all data. Do not set the where parameter to limit 10, which does not conform to the constraints of HybridDB for MySQL on the SQL WHERE clause.	No	No default value
querySql (advanced parameter, which is available only in the code editor)	The SQL statement that is used for refined data filtering. If you specify this parameter, HybridDB for MySQL Reader filters data based only on the value of this parameter. For example, if you want to join multiple tables for data synchronization, set this parameter to `["id","table","1","'mingya.wmy'","'null'","to_char(a+1)","2.3","true"]`. The priority of the querySql parameter is higher than those of the table, column, where, and splitPk parameters. If you specify the querySql parameter, HybridDB for MySQL Reader ignores the settings of the column, table, where, and splitPk parameters that you specified. The system parses information, such as the username and password, that is required by the data source specified by the datasource parameter from the querySql parameter.	No	No default value
singleOrMulti (available only for a sharded table)	Specifies that the source table is a sharded table. After you switch from the codeless UI to the code editor, the `"singleOrMulti":"multi"` configuration is automatically generated. However, if you use the code editor at the beginning, the configuration is not automatically generated, and you must manually add the configuration in the code editor. If you do not add the configuration, HybridDB for MySQL Reader reads data only from the first shard. The singleOrMulti parameter is used only at the frontend.	Yes	multi

Code for HybridDB for MySQL Writer

{
    "type": "job",
    "steps": [
        {
            "parameter": {},
        {
            "parameter": {
                "postSql": [],// The SQL statement that you want to execute after the synchronization task is run. 
                "datasource": "px_aliyun_hy***",// The name of the data source. 
                "column": [// The names of the columns. 
                    "id",
                    "name",
                    "sex",
                    "salary",
                    "age",
                    "pt"
                ],
                "writeMode": "insert",// The write mode. 
                "batchSize": 256,// The number of data records to write at a time. 
                "encoding": "UTF-8",// The encoding format. 
                "table": "person_copy",// The name of the destination table. 
                "preSql": []// The SQL statement that you want to execute before the synchronization task is run. 
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "version": "2.0",// The version number. 
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    },
    "setting": {
        "errorLimit": {// The maximum number of dirty data records allowed. 
            "record": ""
        },
        "speed": {
            "concurrent": 7,// The maximum number of parallel threads. 
            "throttle": true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "mbps": 1,// The maximum transmission rate. Unit: MB/s. 
        }
    }
}

Parameters in code for HybridDB for MySQL Writer

Parameter	Description	Required	Default value
datasource	The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor.	Yes	No default value
table	The name of the table to which you want to write data.	Yes	No default value
writeMode	The write mode. Valid values: replace: If no primary key conflict or unique index conflict occurs, data is processed in the same way as that when you set this parameter to insert. If a conflict occurs, the specified fields in the rows in the destination table are updated. insert: If no primary key conflict or unique index conflict occurs, data is directly written to the destination table. If a primary key conflict or unique index conflict occurs, data cannot be written to conflicting rows, and the data that is not written to these rows is regarded as dirty data.	No	insert
column	The names of the columns to which you want to write data. Separate the names with commas (,), such as `"column":["id","name","age"]`. If you want to write data to all the columns in the destination table, set the value to an asterisk (), such as `"column":[""]`.	Yes	No default value
preSql	The SQL statement that you want to execute before the batch synchronization task is run. For example, you can set this parameter to the SQL statement that is used to delete outdated data. You can execute only one SQL statement on the codeless UI and multiple SQL statements in the code editor.	No	No default value
postSql	The SQL statement that you want to execute after the batch synchronization task is run. For example, you can set this parameter to the SQL statement that is used to add a timestamp. You can execute only one SQL statement on the codeless UI and multiple SQL statements in the code editor.	No	No default value
batchSize	The number of data records to write at a time. Set this parameter to an appropriate value based on your business requirements. This greatly reduces the interactions between Data Integration and HybridDB for MySQL and increases throughput. If you set this parameter to an excessively large value, an out of memory (OOM) error may occur during data synchronization.	No	1,024