DataWorks:Redis data source

Parameter

Description

Required

Default

expireTime

The expiration time for the key in Redis. If you do not specify this parameter, the default value 0 is used, which means the key never expires.

No

0 (The key never expires.)

keyFieldDelimiter

The delimiter used to concatenate multiple columns to form a Redis key, for example, key=key1\u0001id. This parameter is required when you combine multiple columns to form a key. You can omit this parameter if the key consists of a single column.

No

\u0001

dateFormat

The format for Date-type values when written to Redis, such as yyyy-MM-dd HH:mm:ss.

No

None

datasource

The name of the data source. This name must match the one you configured in the DataWorks console.

Yes

None

selectDatabase

The index of the destination database. Valid values range from "0" to "N-1", where N is the number of `databases` configured in Redis. This parameter is unavailable for Redis clusters.

No

Defaults to database 0.

writeMode

The data type of the value to be written to Redis. Redis Writer supports the following five types:

• String (string)

• List (list)

• Set (set)

• Sorted Set (zset)

• Hash (hash)

The configuration of writeMode varies by data type. For details, see the writeMode parameters section below.

No

string

keyIndexes

The zero-based indexes of the source columns to use for the Redis key.

• To use a single column from the source as the Redis key, specify its index. For example, to use the first column as the key, set the value to 0.

• To use multiple source columns as a composite Redis key, specify their indexes in an array. For example, to use the second and third columns as a composite key, set this to [1,2].

Yes

None

batchSize

The number of records to write in a single batch. A larger value can reduce network interactions with Redis and improve throughput. However, setting this value too high may cause an out-of-memory (OOM) error in the synchronization job process.

No

1000

timeout

The write operation timeout in milliseconds.

No

30000

redisMode

The Redis deployment mode. Valid values:

• Cluster mode: Set redisMode to ClusterMode.

  In this mode, Redis Writer connects directly to the Redis cluster. This mode is typically used for self-managed Redis clusters and Alibaba Cloud Redis instances with direct connection addresses. Cluster mode supports batch writing.

• Non-cluster mode: Leave redisMode empty or do not configure it.

  This mode is typically used for Alibaba Cloud Redis instances with proxy addresses, read/write splitting addresses, or standard edition addresses. Non-cluster mode does not support batch writing.

No

None

column

The configuration for columns to be written to Redis. This parameter applies when `writeMode.type` is `string` and `writeMode.mode` is `set`.

• If you do not configure this parameter, the value is written as a string of concatenated values separated by a delimiter (CSV format). For example, if a record has values `18` and `male`, the value in Redis might be `"18::male"`.

• If you configure this parameter, for example, "column": [{"index":"0", "name":"id"}, {"index":"1", "name":"name"}], the value is written in JSON format, such as {"id":"","name":""}. For example, if the `id` is `1` and `name` is `John`, the value in Redis is {"id":"1","name":"John"}.

No

None

Value type

Type parameter

Mode parameter

valueFieldDelimiter parameter

Sample configuration

String

Set type to string.

For the String value type, mode specifies the write mode:

• Set mode to set.

• If a key already exists, its value is overwritten.

valueFieldDelimiter is the delimiter for concatenating values. The default value is \u0001.

• This parameter is used when a source data row has more than one value column. For example, if there are three value columns, they are concatenated by the delimiter, such as value1\u0001value2\u0001value3.

• If the source data has only one key column and one value column, you do not need to configure this parameter.

"writeMode":{
        "type": "string",
        "mode": "set",
        "valueFieldDelimiter": "\u0001"
        }

List

Set type to list.

For the List value type, mode can be one of the following:

• lpush: Pushes an element to the head (left) of the list.

• rpush: Pushes an element to the tail (right) of the list.

"writeMode":{
    "type": "list",
    "mode": "lpush|rpush",
    "valueFieldDelimiter": "\u0001"
}

Set

Set type to set.

For the Set value type, mode specifies the write mode:

• Set mode to sadd to add members to the set.

• If a key with the same name exists but is of a different type, it is overwritten.

"writeMode":{
        "type": "set",
        "mode": "sadd",
        "valueFieldDelimiter": "\u0001"
        }

Sorted Set

Set type to zset.

For the Sorted Set value type, mode specifies the write mode:

• Set mode to zadd to add members to the sorted set.

• If a key with the same name exists but is not a sorted set, its value is overwritten.

Not required.

"writeMode":{
        "type": "zset",
        "mode": "zadd"
        }

Hash

Set type to hash.

For the Hash value type, mode specifies the write mode:

• Set mode to hset to add data to the hash.

• If a key with the same name exists but is not a hash, its value is overwritten.

Not required.

"writeMode":{
        "type": "hash",
        "mode": "hset"
        }