DataWorks:Origen de datos de ClickHouse

Configurar una tarea de sincronización por lotes de tabla única

• Para conocer el procedimiento, consulte Codeless UI configuration y Script mode configuration.

• Para consultar todos los parámetros del modo script y los scripts de ejemplo, consulte Appendix: Sample scripts and parameters.

Configurar una tarea de sincronización de lectura por lotes de base de datos completa

Para conocer el procedimiento, consulte Batch database synchronization configuration.

Configurar una tarea de sincronización por lotes con el editor de código

Si desea configurar una tarea de sincronización por lotes con el editor de código, defina los parámetros correspondientes en el script según los requisitos unificados de formato. Para obtener más información, consulte Script mode configuration. A continuación se describen los parámetros que debe configurar para los orígenes de datos al utilizar el editor de código.

Script de ejemplo de Reader

{ "type": "job", "version": "2.0", "steps": [ { "stepType": "clickhouse", //Plug-in name. "parameter": { "fetchSize":1024,//Number of records fetched in each batch by the plug-in and database server. "datasource": "example", "column": [ //Column names. "id", "name" ], "where": "", //Filter condition. "splitPk": "", //Split key. "table": "" //Table name. }, "name": "Reader", "category": "reader" }, { "stepType": "clickhouse", "parameter": { "postSql": [ "update @table set db_modify_time = now() where db_id = 1" ], "datasource": "example", //Data source. "batchByteSize": "67108864", "column": [ "id", "name" ], "writeMode": "insert", "encoding": "UTF-8", "batchSize": 1024, "table": "ClickHouse_table", "preSql": [ "delete from @table where db_id = -1" ] }, "name": "Writer", "category": "writer" } ], "setting": { "executeMode": null, "errorLimit": { "record": "0" //Maximum number of error records during synchronization. }, "speed": { "throttle":true,//When throttle is false, mbps is ignored and throughput is unlimited; when true, throughput is limited. "concurrent":1 //Job concurrency. "mbps":"12",//Rate limit. Here, 1 mbps = 1 MB/s. } }, "order": { "hops": [ { "from": "Reader", "to": "Writer" } ] } }

Parámetros del script de Reader

Script de ejemplo de Writer

{ "type":"job", "version":"2.0",//Version number. "steps":[ { "stepType":"stream", "parameter":{}, "name":"Reader", "category":"reader" }, { "stepType":"clickhouse",//Plug-in name. "parameter":{ "username": "", "password": "", "column": [//Fields. "id", "name" ], "connection": [ { "table": [//Table name. "ClickHouse_table" ], "jdbcUrl": "jdbc:clickhouse://ip:port/database" } ], "preSql": [ //SQL statements to run before the synchronization task. "TRUNCATETABLEIFEXISTStablename" ], "postSql": [//SQL statements to run after the synchronization task. "ALTERTABLEtablenameUPDATEcol1=1WHEREcol2=2" ], "batchSize": "1024", "batchByteSize": "67108864", "writeMode": "insert" }, "name":"Writer", "category":"writer" } ], "setting":{ "errorLimit":{ "record":"0"//Number of error records. }, "speed":{ "throttle":true,//When throttle is false, mbps is ignored and throughput is unlimited; when true, throughput is limited. "concurrent":1, //Job concurrency. "mbps":"12"//Rate limit. Here, 1 mbps = 1 MB/s. } }, "order":{ "hops":[ { "from":"Reader", "to":"Writer" } ] } }

Parámetros del script de Writer