Redis Writer is a writer that is developed based on the Data Integration framework. It can be used to import data from data stores such as data warehouses to Redis.
REmote DIctionary Server (Redis) is a network-enabled key-value storage system that is either in-memory or permanent. It supports logs and delivers high performance. It can be used as a database, cache, and message broker. Redis supports diverse data types for values, including STRING, LIST, SET, ZSET (sorted set), and HASH. For more information about Redis, see redis.io.
- Only sync nodes that are run on exclusive resource groups for Data Integration can write data to Redis clusters.
- You must configure a Redis connection before you configure Redis Writer. For more information, see Configure a Redis connection.
- If you write values of the LIST type to Redis by using Redis Writer, the result of rerunning a sync node is not idempotent. If the data type of the values is LIST, you must manually delete the data from Redis when you rerun a sync node.
Parameters
Parameter | Description | Required | Default value |
---|---|---|---|
redisMode | The deployment mode of Redis. A value of ClusterMode indicates the cluster mode.
Note This parameter is applicable only when the sync node is run on an exclusive resource
group for Data Integration.
|
No | N/A |
datasource | The connection name. It must be the same as the name of the added connection. You can add connections in the code editor. | Yes | N/A |
keyIndexes | The columns that are used as the key. The index of the first column is 0. For example, if you want to set the first and second columns of the source data as
the key, set the keyIndexes parameter to [0,1].
Note After you specify the keyIndexes parameter, Redis Writer specifies the remaining columns as the value. If you do not
want to synchronize all the columns, filter columns when you configure the reader.
|
Yes | N/A |
keyFieldDelimiter | The delimiter that is used to separate keys when data is written to Redis. Example: key=key1\u0001id. If multiple keys need to be concatenated, this parameter is required. If only one key exists, this parameter is not required. | No | \u0001 |
batchSize | The number of data records to write at a time. Set this parameter to a proper value based on your actual needs. This can greatly reduce the interactions between Data Integration and Redis on the network, and increase the throughput. However, an excessively large value may lead to the out of memory (OOM) error during the data synchronization process. | No | 1,000 |
expireTime | The expiration time of the values to be cached in Redis. Unit: seconds. The data is
valid permanently if you do not specify this parameter.
|
No | 0, which indicates that the values never expire |
timeout | The timeout period to connect to Redis when data is written to Redis. Unit: milliseconds. | No | 30,000 |
dateFormat | The format in which the data of the DATE type is written to Redis. Set the value to yyyy-MM-dd HH:mm:ss. | No | N/A |
writeMode | The write mode. Redis supports diverse data types for values, including STRING, LIST,
SET, ZSET (sorted set), and HASH. Redis Writer allows you to write values of the preceding
types to Redis. The value of the writeMode parameter varies based on the specified
data type of the values.
Note When you configure Redis Writer, you can choose only one of the five data types that
are described in the following table. If you do not specify a data type, the data
type is STRING by default.
|
No | string |
Data type | Parameter | Description | Required |
---|---|---|---|
STRING
|
type | The data type of the values is STRING. | Yes |
mode | The mode in which data of the STRING type is written to Redis. | Yes. Valid value: set (overwrites the existing data). | |
valueFieldDelimiter | This parameter is required if two or more columns are specified as the values. This
parameter is not required if only one column is specified as the values.
The delimiter that is used to separate values if the data is of the STRING type. Example: value1\u0001value2\u0001value3. |
No. Default value: \u0001. | |
LIST
|
type | The data type of the values is LIST. | Yes |
mode | The mode in which data of the LIST type is written to Redis. | Yes. Valid values: lpush (stores the data at the leftmost of the list) and rpush (stores the data at the rightmost of the list). | |
valueFieldDelimiter | The delimiter that is used to separate values if the data is of the STRING type. Example: value1\u0001value2\u0001value3. | No. Default value: \u0001. | |
SET
|
type | The data type of the values is SET. | Yes |
mode | The mode in which data of the SET type is written to Redis. | Yes. Valid value: sadd (stores the data to a set, or overwrites the existing data). | |
valueFieldDelimiter | The delimiter that is used to separate values if the data is of the STRING type. Example: value1\u0001value2\u0001value3. | No. Default value: \u0001. | |
ZSET (sorted set)
|
type | The data type of the values is ZSET.
Note If the data type of the values is ZSET, each data record must follow the following
standard: Except for the key, a data record can contain only one score and one value.
The score must be placed before the value. This way, Redis Writer can identify which
column is the score and which column is the value.
|
Yes |
mode | The mode in which data of the ZSET type is written to Redis. | Yes. Valid value: zadd (stores data to a sorted set, or overwrites the existing data). | |
HASH
|
type | The data type of the values is HASH.
Note If the data type of the values is HASH, each data record must follow the following
standards: Except for the key, a data record can contain only one attribute and one
value. The attribute must be placed before the value. This way, Redis Writer can identify
which column is the attribute and which column is the value.
|
Yes |
mode | The mode in which data of the HASH type is written to Redis. | Yes. Valid value: hset (stores data to a hash sorted set, or overwrites the existing
data).
If you do not specify a data type, the data type is STRING by default. |
Configure Redis Writer by using the codeless UI
The codeless user interface (UI) is not supported for Redis Writer.
Configure Redis Writer by using the code editor
For more information about how to configure Redis Writer by using the code editor, see Create a sync node by using the code editor.
{
"type":"job",
"version":"2.0", // The version number.
"steps":[
{
"stepType":"stream",
"parameter":{},
"name":"Reader",
"category":"reader"
},
{
"stepType":"redis",// The writer type.
"parameter":{
"expireTime":{// The expiration time of the values to be cached in Redis.
"seconds":"1000"
},
"keyFieldDelimiter":"u0001",// The delimiter that is used to separate keys when data is written to Redis.
"dateFormat":"yyyy-MM-dd HH:mm:ss",// The format in which the data of the DATE type is written to Redis.
"datasource":"",// The connection name.
"writeMode":{// The write mode.
"mode":"",// The mode that is used to write data of a specified data type.
"valueFieldDelimiter":"",// The delimiter that is used to separate values.
"type":""// The data type of the values.
},
"keyIndexes": [// The columns that are used as the key.
0,
1
],
"batchSize":"1000",// The number of data records to write at a time.
},
"name":"Writer",
"category":"writer"
}
],
"setting":{
"errorLimit":{
"record":"0"// The maximum number of dirty data records allowed.
},
"speed":{
"throttle":false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
"concurrent":1// The maximum number of concurrent threads.
}
},
"order":{
"hops":[
{
"from":"Reader",
"to":"Writer"
}
]
}
}