DataWorks:BigQuery data source

Supported BigQuery versions and supported regions

• BigQuery uses the SDK google-cloud-bigquery 2.29.0. For more information about the SDK capabilities supported by BigQuery, see the official documentation.

• BigQuery data sources are supported in the following regions:

  China (Hong Kong), Japan (Tokyo), Singapore, Australia (Sydney), Malaysia (Kuala Lumpur), Indonesia (Jakarta), India (Mumbai), Germany (Frankfurt), UK (London), US (Silicon Valley), and US (Virginia).

Data type mappings

For more information about BigQuery data types, see the BigQuery official documentation. The following table provides the support status of main data types in BigQuery.

Establish a network connection between a BigQuery data source and an exclusive resource group for Data Integration

Before you use a BigQuery data source for data synchronization in Data Integration, you must establish a network connection between an exclusive resource group for Data Integration and the data source to enable the resource group to access the data source over an internal network address. For more information, see Establish a network connection between a resource group and a data source.

Develop a data synchronization task

For information about the entry point for and the procedure of configuring a data synchronization task, see the following subsections. For information about the parameter settings, view the infotip of each parameter on the configuration tab of the task.

Add a data source

Before you configure a data synchronization task to synchronize data from or to a specific data source, you must add the data source to DataWorks. For more information, see Add and manage data sources.

When you add a BigQuery data source, you must configure the following items:

• BigQuery Project ID: the name of a BigQuery project of Google Cloud.

• BigQuery authorization and authentication: You must upload the authentication files of Google Cloud.

Configure a batch synchronization task to synchronize data of a single table

• For more information about the configuration procedure, see Configure a batch synchronization task by using the codeless UI and Configure a batch synchronization task by using the code editor.

• For information about all parameters that are configured and the code that is run when you use the code editor to configure a batch synchronization task, see Appendix: Code and parameters.

Appendix: Code and parameters

Configure a batch synchronization task by using the code editor

If you use the code editor to configure a batch synchronization task, you must configure parameters for the reader of the related data source based on the format requirements in the code editor. For more information about the format requirements, see Configure a batch synchronization task by using the code editor. The following information describes the configuration details of parameters for the reader in the code editor.

Code for BigQuery Reader

{
  "stepType": "bigquery"
  "parameter":
  {
    "datasource":"bq_test1",
    "table": "partition_1107",
    "where": "xxx=3",
    "dataSet": "database_0724",
    "partition": [
      "_PARTITIONTIME='2023-11-07'"
     ],
    "column":
    [
      "id",
      "table_id",
      "table_no",
      "table_name",
      "table_status"
    ]
  },
  "name": "Reader",
  "category": "reader"
}

Parameters in code for BigQuery Reader