edit-icon download-icon

Configure DRDS writer

Last Updated: Apr 03, 2018

The DRDS Writer plug-in provides the ability to write data to DRDS tables. At the underlying implementation level, DRDS Writer connects to the Proxy of a remote DRDS database through JDBC, and writes data into DRDS by running the corresponding SQL statement “replace into…”. Note that the SQL statement you run is “replace into”, and your table must have a primary key or a unique index to avoid data duplication. You must configure the data source before configuring the DRDS Writer plug‑in. For more information, see DRDS data source config.

DRDS Writer is designed for ETL developers to import data from data warehouses to DRDS. DRDS Writer can also be used as a data migration tool by DBA and other users.

DRDS Writer acquires the protocol data generated by Reader by means of the CDP framework, and writes data into DRDS by running the statement “replace into…” (If the primary key does not conflict with the unique index, it performs the same action with “insert into”. When a conflict exists, all the fields in the original line are replaced with the fields in the new line). DRDS Writer submits the accumulated amount of data to DRDS’s Proxy, which then determines whether the data is written into one table or multiple tables, and how to route the data when it is written into multiple tables.

Note: The entire task requires at least the “replace into…” permission. Whether other permissions are required depends on the statements you specified in preSql and postSql when you configure the task.

Similar to MySQL Writer, DRDS Writer currently supports most data types in MySQL. Check whether your data type is supported.

DRDS Writer converts DRDS data types as follows:

Category DRDS data type
Integer int,tinyint, smallint, mediumint, int, bigint, year
Floating point float, double, decimal
String varchar, char, tinytext, text, mediumtext, longtext
Date and time date, datetime, timestamp, time
Boolean bit, bool
Binary tinyblob, mediumblob, blob, longblob, varbinary

Parameter description

  • datasource

    • Description: Data source name. It must be identical to the data source name added. Adding data source is supported in script mode.

    • Required: Yes

    • Default value: None
  • table

    • Description: The table selected for synchronization.

    • Required: Yes

    • Default value: None
  • writeMode

    • Description: Select an import mode. The replace mode and insert ignore modes are supported.

      • replace: If the primary key does not conflict with the unique index, it performs the same operation with “insert”. When a conflict exists, all the fields in the original line are replaced with the fields in the new line.

      • insert ignore: If the primary key conflicts with the unique index, Data Integration ignores and discards the updated data with no logs.

    • Required: No

    • Default value: replace

  • column

    • Description: The fields of the target table into which data is required to be written. These fields are separated by commas. For example: “column”: [“id”, “name”, “age”]. Use * if it is required to write data into all columns in sequence. For example: “column”: [“*”]

    • Required: Yes

    • Default value: None
  • preSql

    • Description: The SQL statement that runs before the data synchronization task runs. Currently, you can run only one SQL statement in wizard mode, and more than one SQL statement in script mode. For example, clear old data.
    • Required: No

    • Default value: None

  • postSql

    • Description: The SQL statement runs after the data synchronization task runs. Currently, you can run only one SQL statement in wizard mode, and more than one SQL statement in script mode. For example, add a timestamp.

    • Required: No

    • Default value: None
  • batchSize

    • Description: The quantity of records submitted in one operation. Setting this parameter can greatly reduce the interactions between Data Integration and MySQL over the network, and increase the overall throughput. However, an excessively large value may cause the running process of Data Integration to become out of memory (OOM).

    • Required: No

    • Default value: 1024

Development in wizard mode

TargetDRDS

  • Data source: datasource in the preceding parameter description. Select the DRDS data source.

  • Table: table in the preceding parameter description. Select the table to be synchronized.

  • Prepared Statement before Import: preSql in the preceding parameter description, namely, the SQL statement that runs before the data synchronization task runs.

  • Prepared Statement after Import: postSql in the preceding parameter description, namely, the SQL statement that runs after the data synchronization task runs.

  • Field Mapping: column in the preceding parameter description.

Development in script mode

Configure a job to write data into DRDS:

  1. {
  2. "type": "job",
  3. "version": "1.0",
  4. "configuration": {
  5. "reader": {},
  6. "writer": {
  7. "plugin": "drds",
  8. "parameter": {
  9. "writeMode": "replace",
  10. "datasource": "datasourceName",
  11. "table": "table",
  12. "column": [
  13. "id", "name", "age"
  14. ],
  15. "preSql": [
  16. "delete from XXX;"
  17. ]
  18. }
  19. }
  20. }
  21. }
Thank you! We've received your feedback.