DataWorks:TiDB

Confirm the TiDB version

Data Integration requires TiDB version 7.x or 8.x. Ensure that your TiDB database meets this version requirement. To check the current version, you can run the following statement in your TiDB database.

SELECT TIDB_VERSION()\G

*************************** 1. row ***************************
TIDB_VERSION(): Release Version: v8.1.1
Edition: Community
Git Commit Hash: 821e491a20fbab36604b36b647b5bae26a2c1418
Git Branch: HEAD
UTC Build Time: 2024-08-27 19:16:25
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: tikv
1 row in set (0.00 sec)

Configure account permissions

Create a dedicated TiDB account for DataWorks to access the data source.

1. (Optional) Create an account. For more information, see User Account Management.

2. Configure permissions.

   For offline data reads from TiDB, this account must have the read (SELECT) permission on the tables that you want to synchronize.

   You can run the following commands to grant permissions to the account or grant the SUPER permission directly. In the following statements, replace 'sync_account' with the account that you created.

   -- CREATE USER 'sync_account'@'%' IDENTIFIED BY 'password'; // Create a sync account and set a password. This allows the account to log on to the database from any host. The percent sign (%) indicates any host.
   GRANT SELECT ON *.* TO 'sync_account'@'%'; // Grant the SELECT permission on the database to the sync account.

   *.* grants the account the specified permissions on all tables in all databases. You can also grant the account the permissions on a specific table in a target database. For example, to grant the account the permissions on the user table in the test database, you can run the GRANT SELECT ON test.user TO 'sync_account'@'%'; statement.

   Note

   The REPLICATION SLAVE statement grants a global permission. You cannot use this statement to grant permissions on a specific table in a target database to the sync account.

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Configure a task in the code editor. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Reader script demo

The following code provides a sample script for reading data:

{
  "type": "job",
  "version": "2.0",
  "steps":
  [
    {
      "stepType": "tidb",
      "parameter":
      {
        "column":
        [
          "id",
          "name"
        ],
        "where": "",
        "splitPk": "id",
        "connection":
        [
          {
            "selectedDatabase": "test_database",
            "datasource": "test_datasource",
            "table":
            [
              "test_table"
            ]
          }
        ]
      },
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "odps",
      "parameter":
      {
      },
      "name": "Writer",
      "category": "writer"
    }
  ],
  "setting":
  {
    "errorLimit":
    {
      "record": "0"
    },
    "speed":
    {
      "throttle": false,
      "concurrent": 3
    }
  },
  "order":
  {
    "hops":
    [
      {
        "from": "Reader",
        "to": "Writer"
      }
    ]
  }
}

Reader script parameters