DataWorks:Databricks

Resource group

Databricks Reader supports only Serverless resource groups. The virtual private cloud (VPC) bound to the resource group must have a public NAT gateway and an elastic IP address (EIP) configured.

Read mode

Data synchronization tasks read data in JDBC mode only.

Catalog type

When testing data source connectivity or configuring a task in wizard mode, DataWorks uses databricks-sdk to call the Databricks REST API. This API supports only Unity Catalog. Catalogs other than Unity Catalog—such as hive_metastore—cannot use these features.

To work around this limitation, choose one of the following:

• Migrate to Unity Catalog (recommended). Migrate your data and metadata to Unity Catalog to use all DataWorks features. See Migrate to Unity Catalog.

• Use script mode directly. After adding the data source, skip the Test Connectivity step and configure the task in script mode.

Concurrent reads and data consistency

Databricks Reader uses the splitPk parameter to partition data across multiple concurrent tasks, which improves synchronization throughput. Be aware of the following:

• Concurrent tasks do not share a database transaction and have time intervals between them.

• If data is continuously written to the source during synchronization, concurrent reads can produce an incomplete or inconsistent snapshot.

A perfectly consistent snapshot across concurrent reads is not possible. To manage this trade-off:

• Prioritize consistency. Use single-threaded synchronization without splitPk. This guarantees strict data consistency but reduces throughput.

• Prioritize speed. Keep the data source static during synchronization—for example, use table locking, pause application writes, or stop standby database synchronization. This is faster but may affect online services.

Encoding

Databricks Reader extracts data over JDBC, which automatically detects and converts character encodings. No manual encoding configuration is needed.

Incremental synchronization

Databricks Reader extracts data using SELECT ... WHERE ... statements. The key to incremental synchronization is constructing the WHERE clause correctly.

Recommended approach: Design a timestamp column (such as modify_time) in the source table. Update this column whenever a row is added, modified, or logically deleted. In the synchronization task, use this column in the WHERE clause to pull rows changed since the last synchronization point.

Not supported: Tables without a column that distinguishes new or modified rows—such as a timestamp or an auto-incrementing ID—cannot use incremental synchronization. Only full synchronization is possible for these tables.

Single-table offline synchronization

• For the configuration procedure, see Configure an offline synchronization task in wizard mode and Configure an offline synchronization task in script mode.

• For the full parameter list and a script sample, see Appendix: Script sample and parameters.

Reader script sample

Use the following JSON as a starting point for script mode. See Configure an offline synchronization task in script mode for the full script structure.

{
  "type": "job",
  "version": "2.0",
  "steps": [
    {
      "stepType": "databricks",
      "parameter": {
        "datasource": "databricks",
        "schema": "schema1",
        "table": "table1",
        "readMode": "jdbc",
        "where": "id>1",
        "splitPk": "id",
        "column": [
          "c1",
          "c2"
        ]
      },
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "stream",
      "parameter": {},
      "name": "Writer",
      "category": "writer"
    }
  ],
  "setting": {
    "errorLimit": {
      "record": "0"
    },
    "speed": {
      "concurrent": 1
    }
  }
}

Reader parameters