DataWorks:Redis

Parameter

Description

Required

Default value

expireTime

The cache expiration time for Redis values, in seconds. If this parameter is not specified, the default value 0 is used, which indicates that the data never expires.

No

0 (0 indicates that the data never expires)

keyFieldDelimiter

The delimiter for Redis keys. For example, key=key1\u0001id. This parameter is required if multiple keys need to be concatenated. If only one key is used, you can skip this parameter.

No

\u0001

dateFormat

The date format used when data is written to Redis: yyyy-MM-dd HH:mm:ss.

No

N/A

datasource

The data source name. The value must be the same as the name of the data source you added.

Yes

N/A

selectDatabase

The Redis database to write to ("0" to "N-1", where N is the number of databases configured in Redis). Database selection is not supported for Redis clusters.

No

Database 0 by default

writeMode

Redis Writer supports the following five value types for writing data to Redis:

• String

• List

• Set

• ZSet (sorted set)

• Hash

The writeMode configuration varies slightly depending on the value type. For more information, see writeMode parameter description below.

No

string

keyIndexes

Specifies the column indexes of the source columns to be used as the key. Column indexes start from 0 (the first column has an index of 0, the second column has an index of 1, and so on).

• If a single source column is used as the Redis key, set this parameter to the index of that column. For example, if the first column is used as the key, set this parameter to 0.

• If multiple consecutive source columns are combined as the Redis key, set this parameter to an array of the corresponding column indexes. For example, if the second through fourth columns are combined as the key, set this parameter to [1,3].

Yes

N/A

batchSize

The number of records in each batch. This parameter can significantly reduce the number of network interactions between the data synchronization system and Redis, and improve overall throughput. If this value is set too large, the data synchronization process may encounter an out-of-memory (OOM) error.

No

1,000

timeout

The timeout for writing data to Redis, in milliseconds.

No

30,000

redisMode

The running mode of Redis. Valid values:

• Cluster mode: Set redisMode to ClusterMode to indicate the cluster mode.

  In cluster mode, other data sources communicate directly with the Redis cluster when writing data to Redis. Typically, self-managed Redis cluster addresses and Alibaba Cloud Redis direct connection addresses use this mode. Cluster mode supports batch writes.

• Non-cluster mode: Leave redisMode empty (do not configure any value) to indicate non-cluster mode.

  Typically, Alibaba Cloud Redis cluster proxy addresses, read/write splitting addresses, and Standard Edition addresses use this mode. Non-cluster mode does not currently support batch writes.

No

N/A

column

The column configuration for writing data to Redis. For Redis type string with the set operation:

• If this column is not configured, the value format is a delimiter-separated string (CSV format. For example, if ID=1, name="John", age=18, and sex=male, the Redis value example is: "18::male").

• If column is configured in the following format, such as "column": [{"index":"0", "name":"id"}, {"index":"1", "name":"name"}], the Redis value data is stored in JSON format after being written to Redis, in the format of {"id":"value of the corresponding source column","name":"value of the corresponding source column"} . For example, if ID=1 and name="John", the Redis value example is {"id":"1","name":"John"}.

No

N/A

Value type

type parameter (required)

mode parameter (required)

valueFieldDelimiter parameter (optional)

writeMode configuration example

String

Set type to string.

mode is the write mode parameter. When the value is a string:

• Set mode to set.

• If the data to be stored already exists, the existing data is overwritten.

valueFieldDelimiter is the delimiter between values. The default value is \u0001.

• This parameter is mainly used when there are more than two columns in each row of source data. For example, if there are three columns, the values are separated by the delimiter as follows: value1\u0001value2\u0001value3.

• If the source data has only two columns (key and value), you do not need to configure this parameter.

"writeMode":{
        "type": "string",
        "mode": "set",
        "valueFieldDelimiter": "\u0001"
        }

List

Set type to list.

mode is the write mode parameter. When the value is a list, the following options are available:

• lpush: stores data at the leftmost position of the list.

• rpush: stores data at the rightmost position of the list.

"writeMode":{
    "type": "list",
    "mode": "lpush|rpush",
    "valueFieldDelimiter": "\u0001"
}

Set

Set type to set.

mode is the write mode parameter. When the value is a set:

• Set mode to sadd to store data in the set.

• If the data to be stored already exists, the existing data is overwritten.

"writeMode":{
        "type": "set",
        "mode": "sadd",
        "valueFieldDelimiter": "\u0001"
        }

ZSet (sorted set)

Set type to zset.

mode is the write mode parameter. When the value is a ZSet (sorted set):

• Set mode to zadd to store data in the sorted set.

• If the data to be stored already exists, the existing data is overwritten.

This parameter does not need to be configured.

"writeMode":{
        "type": "zset",
        "mode": "zadd"
        }

Hash

Set type to hash.

mode is the write mode parameter. When the value is a hash:

• Set mode to hset to store data in the hash.

• If the data to be stored already exists, the existing data is overwritten.

This parameter does not need to be configured.

"writeMode":{
        "type": "hash",
        "mode": "hset"
        }