DRDS Reader reads data from Distributed Relational Database Service (DRDS). This topic describes the data types and parameters that are supported by DRDS Reader and how to configure DRDS Reader by using the codeless user interface (UI) and code editor.

Background information

DRDS Reader supports only MySQL engines. DRDS is a distributed MySQL database service. Most of the communication protocols that DRDS uses are the same as those used by MySQL.
Notice Only exclusive resource groups for Data Integration can be used to read data from DRDS instances that run MySQL 8.0.

DRDS Reader connects to a remote DRDS database by using Java Database Connectivity (JDBC), generates an SQL statement based on your configurations, and then sends the statement to the database. The system executes the statement on the database and returns data. Then, DRDS Reader assembles the returned data into abstract datasets of the data types supported by Data Integration and sends the datasets to a writer.

DRDS Reader generates the SQL statement based on the settings of the table, column, and where parameters, and sends the generated statement to the DRDS database. DRDS does not support MySQL specifications, such as JOIN statements.

Data types

DRDS Reader supports most DRDS data types. Make sure that the data types of your database are supported.

The following table describes the data types that are supported by DRDS Reader.
Category DRDS data type
Integer INT, TINYINT, SMALLINT, MEDIUMINT, and BIGINT
Floating point FLOAT, DOUBLE, and DECIMAL
String VARCHAR, CHAR, TINYTEXT, TEXT, MEDIUMTEXT, and LONGTEXT
Date and time DATE, DATETIME, TIMESTAMP, TIME, and YEAR
Boolean BIT and BOOLEAN
Binary TINYBLOB, MEDIUMBLOB, BLOB, LONGBLOB, and VARBINARY

Parameters

Parameter Description Required Default value
datasource The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor. Yes No default value
table The name of the table from which you want to read data. Yes No default value
column The names of the columns from which you want to read data. Specify the names in a JSON array. The default value is [*], which indicates all the columns in the source table.
  • You can select specific columns to read.
  • The column order can be changed. You can configure DRDS Reader to read the specified columns in an order different from that specified by the schema of the table.
  • Constants are supported. The column names must be arranged in compliance with the SQL syntax supported by MySQL, such as ["id", "`table`", "1", "'bazhen.csy'", "null", "to_char(a + 1)", "2.3", "true"]. Description of the parameters in the preceding example:
    • id: a column name.
    • table: the name of a column that contains reserved keywords.
    • 1: an integer constant.
    • bazhen.csy: a string constant.
    • null: a null pointer.
    • to_char(a + 1): a function expression that is used to calculate the length of a string.
    • 2.3: a floating-point constant.
    • true: a Boolean value.
  • The column parameter must explicitly specify all the columns from which you want to read data. The parameter cannot be left empty.
Yes No default value
where The WHERE clause. DRDS Reader generates an SQL statement based on the settings of the column, table, and where parameters, and uses the generated statement to read data.
  • You can use the WHERE clause to read incremental data.
  • If the where parameter is not specified or is left empty, DRDS Reader reads full data.

For example, you can set this parameter to STRTODATE('${bdp.system.bizdate}','%Y%m%d') <= taday AND taday < DATEADD(STRTODATE('${bdp.system.bizdate}', '%Y%m%d'), interval 1 day) to read the data that is generated on the current day.

No No default value

Configure DRDS Reader by using the codeless UI

Create a synchronization node and configure the node. For more information, see Configure a synchronization node by using the codeless UI.

Perform the following steps on the configuration tab of the synchronization node:
  1. Configure data sources.
    Set parameters in the Source and Target sections for the synchronization node. Configure data sources
    Parameter Description
    Connection The name of the data source from which you want to read data. This parameter is equivalent to the datasource parameter that is described in the preceding section.
    Table The name of the table from which you want to read data. This parameter is equivalent to the table parameter that is described in the preceding section.
    Filter The condition that is used to filter the data you want to read. Filtering based on the LIMIT keyword is not supported. The SQL syntax is determined by the selected data source.
    Shard Key The shard key. You can use a column in the source table as the shard key. We recommend that you use the primary key column or an indexed column. Only integer columns are supported.
    If you set this parameter, data sharding is performed based on the value of this parameter, and parallel threads can be used to read data. This improves data synchronization efficiency.
    Note The Shard Key parameter is displayed only after you select the data source for the synchronization node.
  2. Configure field mappings. This operation is equivalent to setting the column parameter that is described in the preceding section.
    Fields in the source on the left have a one-to-one mapping with fields in the destination on the right. You can click Add to add a field. To remove an added field, move the pointer over the field and click the Remove icon. Field mappings
    Operation Description
    Map Fields with the Same Name Click Map Fields with the Same Name to establish mappings between fields with the same name. The data types of the fields must match.
    Map Fields in the Same Line Click Map Fields in the Same Line to establish mappings between fields in the same row. The data types of the fields must match.
    Delete All Mappings Click Delete All Mappings to remove the mappings that are established.
    Auto Layout Click Auto Layout. Then, the system automatically sorts the fields based on specific rules.
    Change Fields Click the Change Fields icon. In the Change Fields dialog box, you can manually edit the fields in the source and destination tables. Each field occupies a row. The first and the last blank rows are included, whereas other blank rows are ignored.
    Add Click Add to add a field. Take note of the following rules when you add a field:
    • You can enter constants. Each constant must be enclosed in single quotation marks ('), such as 'abc' and '123'.
    • You can use scheduling parameters, such as ${bizdate}.
    • You can enter functions that are supported by relational databases, such as now() and count(1).
    • If the field that you entered cannot be parsed, the value of Type for the field is Unidentified.
  3. Configure channel control policies. Channel control
    Parameter Description
    Expected Maximum Concurrency The maximum number of parallel threads that the synchronization node uses to read data from the source or write data to the destination. You can configure the parallelism for the synchronization node on the codeless UI.
    Bandwidth Throttling Specifies whether to enable bandwidth throttling. You can enable bandwidth throttling and specify a maximum transmission rate to prevent heavy read workloads on the source. We recommend that you enable bandwidth throttling and set the maximum transmission rate to an appropriate value based on the configurations of the source.
    Dirty Data Records Allowed The maximum number of dirty data records allowed.
    Distributed Execution

    The distributed execution mode that allows you to split your node into pieces and distribute them to multiple Elastic Compute Service (ECS) instances for parallel execution. This speeds up synchronization. If you use a large number of parallel threads to run your synchronization node in distributed execution mode, excessive access requests are sent to the data sources. Therefore, before you use the distributed execution mode, you must evaluate the access load on the data sources. You can enable this mode only if you use an exclusive resource group for Data Integration. For more information about exclusive resource groups for Data Integration, see Exclusive resource groups for Data Integration and Create and use an exclusive resource group for Data Integration.

Configure DRDS Reader by using the code editor

In the following code, a synchronization node is configured to read data from a DRDS database. For more information about how to configure a synchronization node by using the code editor, see Create a synchronization node by using the code editor.
{
    "type":"job",
    "version":"2.0",// The version number.
    "steps":[
        {
            "stepType":"drds",// The reader type.
            "parameter":{
                "datasource":"",// The name of the data source.
                "column":[// The names of the columns from which you want to read data.
                    "id",
                    "name"
                ],
                "where":"",// The WHERE clause.
                "table":"",// The name of the table from which you want to read data.
                "splitPk": ""// The shard key.
            },
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"stream",// The writer type.
            "parameter":{},
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed.
        },
        "speed":{
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1,// The maximum number of parallel threads.
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}:"Writer"
            }
        ]
    }
}

Additional information

  • Consistent view

    DRDS cannot provide a consistent view of multiple tables in multiple databases because it is a distributed database service. DRDS Reader obtains different snapshots from different table shards, but cannot obtain the snapshot of table shards at the same time slice. As a result, DRDS Reader cannot ensure strong consistency for data queries.

  • Character encoding

    DRDS supports flexible encoding configurations. You can specify the encoding format for an entire instance and specific fields, tables, and databases. The configurations at the field, table, database, and instance levels are prioritized in descending order. We recommend that you use UTF-8 for a database.

    DRDS Reader uses JDBC to read data. This enables DRDS Reader to automatically convert the encoding formats of characters. Therefore, you do not need to specify an encoding format when you use DRDS Reader.

    If you specify an encoding format but data in the source DRDS database is written in a different encoding format, DRDS Reader cannot recognize this inconsistency and may export garbled characters.

  • Incremental data synchronization
    DRDS Reader uses JDBC to connect to a database and uses a SELECT statement with a WHERE clause to read incremental data.
    • For batch data, incremental add, update, and remove operations (including logically remove operations) are distinguished by timestamps. Specify the WHERE clause based on a specific timestamp. The time indicated by the timestamp must be later than the time indicated by the latest timestamp in the previous synchronization.
    • For streaming data, specify the WHERE clause based on the ID of a specific record. The ID must be greater than the maximum ID involved in the previous synchronization.

    If the data that is added or modified cannot be distinguished, DRDS Reader can read only full data.

  • You cannot configure filter conditions for physical tables in the WHERE clause.