All Products
Search
Document Center

DataWorks:Databricks data source

Last Updated:Oct 09, 2026

This topic describes the capabilities and support for synchronizing Databricks data with DataWorks.

Usage notes

  • Limits

    • Resource group: Databricks Reader supports only Serverless resource groups. Configure Internet NAT gateway and an EIP for the VPC bound to the resource group.

    • Read mode: Data synchronization supports only reading through JDBC.

    • Catalog type and feature limits: To test data source connectivity or configure a synchronization task in wizard mode, DataWorks uses databricks-sdk to call the Databricks REST API, which supports only Unity Catalog. If your catalog, such as hive_metastore, is not Unity Catalog, these features are unavailable. Choose either of the following options:

      • Option 1 (recommended): Migrate to Unity Catalog. Databricks recommends migrating your data and metadata to Unity Catalog to use all DataWorks features. For instructions, see Migrate to Unity Catalog.

      • Option 2: Use script mode directly. After configuring the data source in DataWorks, skip Test Connectivity and use script mode to develop the synchronization task.

  • Concurrent reads and data consistency

    Databricks Reader uses splitPk to split data and start concurrent tasks for faster synchronization. Note the following:

    • Concurrent tasks are not part of the same database transaction and do not start at exactly the same time.

    • If the data source is still receiving writes, concurrent reads can produce an incomplete or inconsistent snapshot.

    Workarounds: Technical limitations prevent a perfectly consistent snapshot across multiple threads. Choose one of these approaches based on your workload:

    • Option 1: Use single-threaded synchronization without data splitting (splitPk). This provides strict consistency but is slower.

    • Option 2: Keep the data source static during synchronization, for example by locking the table, temporarily disabling application writes, or pausing replica synchronization. This is faster but may affect production traffic.

  • Encoding

    Databricks Reader extracts data with JDBC, which automatically detects and converts encodings across databases. You do not need to configure encoding manually.

  • Incremental synchronization

    Databricks Reader extracts data by running a SELECT ... WHERE ... statement. Incremental synchronization depends on constructing the WHERE condition.

    1. Recommended approach (timestamp field):

      1. Add a timestamp field, such as modify_time, to the source table.

      2. Whenever the application inserts or updates a record, including a logical deletion, update the timestamp field too.

      3. Use the timestamp field in the task's WHERE condition to fetch changes since the last synchronization point.

    2. Unsupported case: If the business table has no field that distinguishes inserted or updated records, such as a timestamp or auto-incrementing ID, Databricks Reader cannot synchronize incrementally and must perform full synchronization.

Supported field types

For offline reads, Databricks Reader supports most Databricks types, but some types are unsupported. Check your field types.

The following table lists the type conversions supported by Databricks Reader.

Type category

Databricks data type

Integer

TINYINT, SMALLINT, INT, BIGINT

Floating point

FLOAT, DOUBLE, DECIMAL

String

STRING

Date and time

DATE, TIMESTAMP, TIMESTAMP_NTZ

Boolean

BOOLEAN

Complex type

ARRAY, MAP, STRUCT

Other types

INTERVAL, BINARY, GEOGRAPHY(srid), GEOMETRY(srid)

Create a data source

Before developing a data synchronization task, create the corresponding data source in DataWorks. For the procedure, see Data source configuration.Explanations of the configuration parameters are available in the configuration interface..

Develop a data synchronization task

See the following guides for entry points and common steps for configuring a synchronization task.

Single-table offline synchronization configuration guide

FAQ

  • Q: A read fails with [Databricks][JDBCDriver](500313) Error getting the data value from result set: Column13:[Databricks][JDBCDriver](500312) Error in fetching data rows: Timestamp Conversion has failed.

    A: Databricks TIMESTAMP values can fall outside the range supported by Java Timestamp, causing a JDBC driver error. Modify the column configuration, for example "column": ["CAST(col_timestamp AS STRING)"].

Appendix: Sample script and parameter descriptions

Offline task script configuration

If you use script mode for an offline task, write the required parameters in the standard script format. For details, see Configure an offline synchronization task in script mode. The following describes the data source parameters in script mode.

Sample Reader script

{
  "type": "job",
  "version": "2.0",
  "steps": [
    {
      "stepType": "databricks",
      "parameter": {
        "datasource": "databricks",
        "schema": "schema1",
        "table": "table1",
        "readMode": "jdbc",
        "where": "id>1",
        "splitPk": "id",
        "column": [
          "c1",
          "c2"
        ]
      },
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "stream",
      "parameter": {},
      "name": "Writer",
      "category": "writer"
    }
  ],
  "setting": {
    "errorLimit": {
      "record": "0"
    },
    "speed": {
      "concurrent": 1
    }
  }
}

Reader script parameters

Parameter

Description

Required

Default value

datasource

Data source name. Script mode supports adding data sources; this value must match the name of the added data source.

Yes

None

column

List of column names to synchronize from the configured table, expressed as a JSON array.

  • Column pruning is supported, so you can export only some columns.

  • Column reordering is supported, so the export order can differ from the table schema.

  • Constant values are supported if they follow Databricks SQL syntax.

For example:"column":["id", "1", "'const name'", "null", "upper('abc_lower')", "2.3" , "true"]

  • id is a regular column name.

  • 1 is an integer constant.

  • 'const name' is a string constant and must be enclosed in single quotation marks.

  • null is a null value.

  • upper('abc_lower') is a function expression.

  • 2.3 is a floating-point number.

  • true is a Boolean value.

You must explicitly specify a nonempty set of columns in column.

Yes

None

splitPk

If you specify splitPk, Databricks Reader uses the splitPk field to split data and starts concurrent synchronization tasks for better performance:

  • We recommend using the table's primary key as splitPk because primary keys are usually distributed fairly evenly, reducing data skew in the splits.

  • Currently, splitPk supports splitting only on integer fields. Floating-point, string, date, and other types are unsupported; specifying them causes an error.

No

None

where

Filter condition. Databricks Reader combines it with the specified column and table and where to form the complete query and extract data. To synchronize today's data in a typical workload, set where to gmt_create>${bizdate}.where. The condition supports incremental synchronization. If left empty, the entire table is synchronized.

No

None

schema

Schema containing the data to synchronize.

Yes

None

table

Selected table to synchronize. Each job supports one table.

Yes

None

readMode

Read mode. Only JDBC is currently supported.

No

jdbc