Azure Blob Storage data synchronization support - DataWorks

DataWorks provides Azure Blob Storage Reader and Azure Blob Storage Writer for you to read data from and write data to files that are stored in Azure Blob Storage. You can use Azure Blob Storage Reader to access files stored in Azure Blob Storage, parse the data in the files, and then synchronize the data to a destination. You can also use Azure Blob Storage Writer to write data from any source to Azure Blob Storage. This topic describes the capabilities of synchronizing data from and to Azure Blob Storage data sources.

Limitations

The Azure Blob Storage data source supports the following data types.

Data type	Description
STRING	Text.
LONG	Integer.
BYTES	Byte array. The text that is read is converted into a byte array. The encoding format is UTF-8.
BOOL	Boolean.
DOUBLE	Floating point.
DATE	Date and time. The following date and time formats are supported: YYYY-MM-dd HH:mm:ss yyyy-MM-dd HH:mm:ss

Add a data source

Before you develop a synchronization task in DataWorks, you must add the required data source to DataWorks by following the instructions in Data source configuration. You can view parameter descriptions in the DataWorks console to understand the meanings of the parameters when you add a data source.

Develop a data synchronization task

For information about the entry point for and the procedure of configuring a synchronization task, see the following configuration guides.

Configure a batch synchronization task to synchronize data of a single table

For the procedure, see Configure a synchronization task in the codeless UI and Configure a synchronization task by using the code editor.
For the complete parameters and script demo for the code editor, see Appendix: Script demo and parameter description below.

Appendix: Script demo and parameter description

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Script mode configuration. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Reader script demo

The following is a code editor configuration example for reading data from Azure Blob Storage. The writer side uses stream as a placeholder. Replace it with the configuration of the actual destination data source.

{
  "type": "job",
  "version": "2.0",
  "steps": [
    {
      "stepType": "azureblob",
      "parameter": {
        "datasource": "",
        "object": ["f/z/1.csv"],
        "fileFormat": "csv",
        "encoding": "utf8/gbk/...",
        "fieldDelimiter": ",",
        "useMultiCharDelimiter": true,
        "lineDelimiter": "\n",
        "skipHeader": true,
        "compress": "zip/gzip",
        "column": [
          {
            "index": 0,
            "type": "long"
          },
          {
            "index": 1,
            "type": "boolean"
          },
          {
            "index": 2,
            "type": "double"
          },
          {
            "index": 3,
            "type": "string"
          },
          {
            "index": 4,
            "type": "date"
          }
        ]
      },
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "stream",
      "parameter": {},
      "name": "Writer",
      "category": "writer"
    }
  ],
  "setting": {
    "errorLimit": {
      "record": "0"
    },
    "speed": {
      "concurrent": 1
    }
  },
  "order": {
    "hops": [
      {
        "from": "Reader",
        "to": "Writer"
      }
    ]
  }
}

Reader script parameters

Parameter	Description	Required	Default value
datasource	The data source name. The code editor allows you to add data sources. The value of this parameter must be the same as the name of the data source that you add.	Yes	N/A
fileFormat	The type of the source file. Valid values: `csv`, `text`, `parquet`, `orc`, `json`, and `jsonl`.	Yes	N/A
object	The file path. This parameter is used when fileFormat is set to csv, text, json, or jsonl. Note This parameter supports the `` wildcard and can be set to an array. For example, to synchronize the `a/b/1.csv` and `a/b/2.csv` files, you can set this parameter to `a/b/.csv`.	Yes Required when fileFormat is set to csv, text, json, or jsonl.	None
path	The file path. This parameter is used when fileFormat is set to parquet or orc. Note This parameter supports the `` wildcard and can be set to an array. For example, to synchronize the `a/b/1.orc` and `a/b/2.orc` files, you can set this parameter to `a/b/.orc`.	Yes Required when fileFormat is set to parquet or orc.	None
column	The list of fields to read. The type parameter specifies the data type of the source data: CSV/TEXT format: The index parameter specifies the column number (starting from 0) from which the current column is read, and the value parameter specifies a constant value. JSON/JSONL format: Use jsonPath to specify a JSONPath expression to extract fields, and the value parameter specifies a constant value. By default, you can read all data as the STRING type. Example configuration: `column": [""]` You can specify column field information. Example configuration: `// CSV/TEXT format "column": { "type": "long", "index": 0 //Obtain the int field from the first column of the Azure Blob Storage text. }, { "type": "string", "value": "alibaba" //Azure Blob Storage Reader internally generates the string field alibaba as the current field. }` `// JSON/JSONL format "column": { "name": "id", "jsonPath": "$.id", "type": "LONG" }, { "name": "name", "jsonPath": "$.user.name", "type": "STRING" }, { "name": "source", "value": "azure", "type": "STRING" }` Note* For the column information that you specify, the type field is required. For the CSV/TEXT format, specify either index or value. For the JSON/JSONL format, specify either jsonPath or value.	Yes	All data is read as the STRING type.
fieldDelimiter	The field delimiter used to read data. Note Azure Blob Storage Reader requires a field delimiter to read data. If you do not specify a delimiter, a comma (,) is used by default. The comma (,) is also the default value in the console. If the delimiter is invisible, enter its Unicode encoding. Examples: `\u001b` and `\u007c`.	Yes	,
lineDelimiter	The line delimiter used to read data. Note This parameter takes effect only when fileFormat is set to text.	No	N/A
compress	The compression type for text files. By default, this parameter is left empty, which indicates that no compression is applied. Supported compression types are `gzip`, `bzip2`, and `zip`.	No	No compression
encoding	The encoding configuration for reading files.	No	utf-8
nullFormat	Standard strings cannot be used to define null (null pointer) in text files. The data synchronization system provides nullFormat to define which strings can represent null. Examples: If you set `nullFormat:"null"`, it is equivalent to a "visible character". If the source data is null, the data synchronization system treats it as a null field. If you set `nullFormat:"\u0001"`, it is equivalent to an "invisible character". If the source data is the string "\u0001", the data synchronization system treats it as a null field. If you do not specify the `"nullFormat"` parameter, it is equivalent to "not configured", which means the source data is written to the destination as-is without any conversion.	No	N/A
skipHeader	For CSV files, use skipHeader to specify whether to read the header. True: The header is read when data is synchronized from the data source. False: The header is not read when data is synchronized from the data source. Note skipHeader is not supported in compressed file mode.	No	false
parquetSchema	This parameter is configured when you read data from Azure Blob Storage in Parquet file format. It takes effect only when fileFormat is set to parquet and specifies the type description of the Parquet storage. Make sure that the overall configuration conforms to JSON syntax after you specify parquetSchema. `message MessageTypeName { Required or not, DataType, ColumnName; ......................; }` The configuration format of parquetSchema is described as follows: MessageType name: Specify a name. Required or not: required indicates that the field cannot be empty, and optional indicates that the field can be empty. We recommend that you set all fields to optional. Data type: Parquet files support the following data types: BOOLEAN, Int32, Int64, Int96, FLOAT, DOUBLE, BINARY (specify BINARY for string types), and fixed_len_byte_array. Each column setting in a row must end with a semicolon, including the last row. The following example shows the configuration. `"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"`	No	N/A
csvReaderConfig	The parameter configuration for reading CSV files. The value is of the Map type. CsvReader is used to read CSV files. If this parameter is not configured, the default values are used.	No	N/A
maxRetryTimes	The maximum number of retries when a file fails to be downloaded. Note Set this parameter to 0 to disable this feature. This parameter can be configured only in code editor mode. It is not supported in the codeless UI.	No	0
retryIntervalSeconds	The retry interval when a file fails to be downloaded. Unit: seconds. Note This parameter can be configured only in code editor mode. It is not supported in the codeless UI.	No	5

Writer script demo

The following example shows a script mode configuration for writing data to Azure Blob Storage. The reader side uses stream as a placeholder. Replace it with the actual source data source configuration.

{
  "type": "job",
  "version": "2.0",
  "steps": [
    {
      "stepType": "stream",
      "parameter": {},
      "name": "Reader",
      "category": "reader"
    },
    {
      "stepType": "azureblob",
      "parameter": {
        "datasource": "",
        "fileFormat": "csv",
        "object": "dir/example.csv",
        "fieldDelimiter": ",",
        "lineDelimiter": "\n",
        "encoding": "UTF-8",
        "nullFormat": "null",
        "dateFormat": "yyyy-MM-dd",
        "writeMode": "truncate",
        "maxFileSize": 100,
        "writeSingleObject": false
      },
      "name": "Writer",
      "category": "writer"
    }
  ],
  "setting": {
    "errorLimit": {
      "record": "0"
    },
    "speed": {
      "concurrent": 1
    }
  },
  "order": {
    "hops": [
      {
        "from": "Reader",
        "to": "Writer"
      }
    ]
  }
}

Writer script parameters

Parameter	Description	Required	Default value
datasource	The data source name. Script mode allows you to add data sources. The value of this parameter must be the same as the name of the data source that you add. The data source must be configured with the SAS URL and signature (sig) of the Azure Blob Storage container.	Yes	N/A
object	The name of the file to which Azure Blob Storage Writer writes data, including the path prefix. Directories are simulated by using file names, and the delimiter is a forward slash (/). If you use `"object": "datax.csv"`, the written object starts with datax. When multiple files are written, a random UUID string is appended as a suffix. If you use `"object": "cdo/datax.csv"`, the written object starts with `cdo/datax`. If you do not want a random UUID suffix, configure `"writeSingleObject": "true"`. For more information, see the description of writeSingleObject.	Yes	N/A
writeMode	Specifies how Azure Blob Storage Writer processes existing data before writing: truncate: Before writing, all objects whose names match the specified object name prefix are deleted. For example, if you set `"object":"abc"`, all objects whose names start with abc are deleted. append: No processing is performed before writing. Data is directly written by using the object name. When multiple files are written, random UUID suffixes are used to prevent file name conflicts. nonConflict: Before writing, the system checks whether the specified object already exists. If the object exists, an error is returned.	Yes	None
writeSingleObject	Specifies whether to write data to a single file: true: All concurrent tasks write to the same object by using Block Blob chunked upload. No empty file is generated when no data is read. false: Each concurrent task writes to a separate object. A random UUID suffix is appended to the file name. When no data is read, an empty file that contains only the file header is generated if the header parameter is configured. Otherwise, no file is generated. Note This parameter does not take effect when you write data in Parquet or ORC format.	No	false
fileFormat	The format in which files are written. Valid values: csv: Only strict CSV format is supported. If the data to be written contains column delimiters, the data is escaped based on the CSV escape syntax. The escape character is a double quotation mark ("). text: The data to be written is separated by using column delimiters. If the data to be written contains column delimiters, no escaping is performed. jsonl: Data is written in JSON Lines format, with one JSON object per line. parquet: You must add the parquetSchema parameter to define data types. Only the script mode is supported. orc: You must switch to the script mode for configuration.	No	text
fieldDelimiter	The field delimiter used for writing. This parameter is valid for the csv and text formats. If you do not specify this parameter, the default delimiter comma (,) is used. If the delimiter is invisible, enter the Unicode encoding, such as `\u001b` or `\u007c`.	No	,
encoding	The encoding of the output file.	No	utf-8
nullFormat	Null (empty pointer) cannot be defined by using standard strings in text files. The data synchronization system provides the nullFormat parameter to define which strings can be treated as null. For example, if you configure `nullFormat:"null"` and the source data is null, the data synchronization system treats the field as a null field.	No	\N
dateFormat	The format used to format date-type data.	No	`yyyy-MM-dd HH:mm:ss`
header	The header of the output file. Example: `["id", "name", "age"]`. This parameter is valid for the csv and text formats. The header is written at the beginning of the first block.	No	N/A
blockSizeInMB	The size of each block in MB when Block Blob data is uploaded in blocks. Azure Block Blob supports a maximum of 50,000 blocks, and the size of each block ranges from 4 MB to 100 MB. If the number of blocks exceeds the limit, you can increase the block size to support larger file uploads. Note * This parameter is supported only in advanced mode, not in codeless UI mode. * This parameter takes effect only for the text, csv, and jsonl formats.	No	64
parquetSchema	This parameter is required when data is written in Parquet file format. It describes the structure of the destination file and takes effect only when fileFormat is set to parquet. The format is as follows. The configuration format of parquetSchema is described as follows: MessageType name: Enter a name. Required or not: required indicates that the field cannot be null, and optional indicates that the field can be null. We recommend that you set all fields to optional. Data type: Parquet files support the following data types: BOOLEAN, Int32, Int64, Int96, FLOAT, DOUBLE, BINARY (use BINARY for string types), and fixed_len_byte_array. Each column setting in a row must end with a semicolon, including the last row. The following example shows a sample configuration. `"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"`	No	N/A
column	The destination column information configured when writing data in Parquet or ORC format. Format: `{"name":"column name","type":"column type"}`.	No	N/A
compress	The compression format for writing Parquet or ORC files, such as `SNAPPY` and `NONE`. Compression is not supported for text, CSV, or JSONL formats.	No	N/A
maxRetryTimes	The maximum number of retries upon write failures, including uploading blocks, committing block lists, and deleting objects. Set this parameter to 0 to disable retries. Note This parameter is available only in script mode and cannot be configured in codeless UI mode.	No	30
retryIntervalSeconds	The interval between retries upon write failures. Unit: seconds. This parameter is available only in script mode and cannot be configured in codeless UI mode.	No	5
requestTimeoutSeconds	The timeout period for a single request. Unit: seconds. This parameter is available only in script mode and cannot be configured in codeless UI mode.	No	60