Redis Writer is a writer that is developed based on the Data Integration framework. It can write the data that is obtained from a reader to Redis.

REmote DIctionary Server (Redis) is a key-value storage system that can be accessed over the network and can permanently store data or store data in memory. Redis supports logs and provides high performance. You can use Redis as a database, a cache, or a message broker. Redis supports values of the following data types: STRING, LIST, SET, ZSET (sorted set), and HASH. For more information about Redis, visit redis.io.

Redis Writer interacts with a Redis server by using Jedis. Jedis is a Java client development kit that is provided and recommended by Redis.
Note
  • Redis Writer can write data only to Redis clusters. The synchronization nodes that use Redis Writer must run on exclusive resource groups for Data Integration.
  • Before you configure Redis Writer, you must add a Redis data source. For more information, see Configure a Redis connection.
  • If you rerun a synchronization node that uses Redis Writer to write LIST values to Redis, the result is not idempotent. In this case, you must manually delete the data written by Redis Writer in the previous run from Redis before you rerun the synchronization node.

Parameters

Parameter Description Required Default value
expireTime The expiration time of the values that are cached in Redis. Unit: seconds. If you do not specify this parameter, the default value 0 is used, which indicates that the values are permanently valid.
You can specify the value of the expireTime parameter in one of the following modes:
  • seconds: the relative period of time in seconds starting from the current point in time. It specifies the period during which the values are valid.
  • unixtime: the number of seconds that have elapsed since 00:00:00 on January 1, 1970. It specifies the point in time when the values expire.
    Note If you set the expireTime parameter to a value that is greater than 2,592,000 (30 days), the value is interpreted as a UNIX timestamp.
No 0
keyFieldDelimiter The delimiter that you want to use to separate keys when Redis Writer writes data to Redis. Example: key=key1\u0001id. If multiple keys need to be concatenated, this parameter is required. If the data contains only one key, this parameter is not required. No \u0001
dateFormat The format in which Redis Writer writes the data of the DATE type to Redis. Set the value to yyyy-MM-dd HH:mm:ss. No No default value
datasource The name of the data source. The name must be the same as that of the data source that you added. Yes No default value
writeMode The mode in which Redis Writer writes data to Redis. Redis Writer can write values of the following types:
  • STRING
  • LIST
  • SET
  • ZSET (sorted set)
  • HASH
The value of the writeMode parameter varies based on the data types of the values that you want to write to Redis. For more information, see Configure writeMode.
Note When you configure Redis Writer, you can specify only one data type for the writeMode parameter. If you do not specify this parameter, the default value string is used.
No string
keyIndexes The indexes of source columns that are used as the key. The index starts from 0. For example, the index of the first column is 0, and the index of the second column is 1, and so on.
  • If you want to specify a specific column of the source as the key, set this parameter to the index of the column. For example, if you want to specify the first column as the key, set this parameter to 0.
  • If you want to specify a range of contiguous columns of the source as the key, set this parameter to a closed interval that specifies the indexes of these columns. For example, if you want to specify the second, third, and fourth columns as the key, set this parameter to [1,3].
Note After you specify the keyIndexes parameter, Redis Writer specifies the remaining columns as the value. If you want to synchronize only some of the columns in the source, specify the names of the columns when you configure a reader.
Yes No default value
batchSize The number of data records to write at a time. Set this parameter to an appropriate value based on your business requirements. This greatly reduces the interactions between Data Integration and Redis and increases throughput. If you set this parameter to an excessively large value, an out of memory (OOM) error may occur during data synchronization. No 1,000
timeout The timeout period of the connection to Redis when data is written to Redis. Unit: milliseconds. No 30,000
redisMode The deployment mode of Redis. Redis Writer can write data only to Redis clusters. You must set this parameter to ClusterMode.
Note Synchronization nodes that use Redis Writer must run on exclusive resource groups for Data Integration.
No No default value

Configure writeMode

When you configure Redis Writer, you can specify only one data type for the writeMode parameter. If you do not specify this parameter, the default value string is used.
Data type of values type parameter (required) mode parameter (required) valueFieldDelimiter parameter (optional) Configuration example
STRING Set the type parameter to string. The mode in which Redis Writer writes the data of the STRING type to Redis. Take note of the following rules when you set the type parameter to string:
  • Set the mode parameter to set.
  • If the data that you want to write to Redis already exists in Redis, existing data is overwritten.
The delimiter that you want to use to separate column values. Default value: \u0001.
  • This parameter is required if you specify two or more columns as the value. For example, you specify three columns of the source data as the value and use the default delimiter. In this case, the column values are concatenated in the value1\u0001value2\u0001value3 format.
  • This parameter is not required if you specify only one column as the value. 
"writeMode":{
        "type": "string",
        "mode": "set",
        "valueFieldDelimiter": "\u0001"
        }
LIST Set the type parameter to list. The mode in which Redis Writer writes the data of the LIST type to Redis. Valid values:
  • lpush: indicates that data is stored at the leftmost of the list.
  • rpush: indicates that data is stored at the rightmost of the list. 
"writeMode":{
    "type": "list",
    "mode": "lpush|rpush",
    "valueFieldDelimiter": "\u0001"
}
SET Set the type parameter to set. The mode in which Redis Writer writes the data of the SET type to Redis. Take note of the following rules when you set the type parameter to set:
  • Set the mode parameter to sadd, which indicates that data is stored to a set.
  • If the data that you want to write to Redis already exists in Redis, existing data is overwritten. 
"writeMode":{
        "type": "set",
        "mode": "sadd",
        "valueFieldDelimiter": "\u0001"
        }
ZSET (sorted set) Set the type parameter to zset. The mode in which Redis Writer writes the data of the ZSET type to Redis. Take note of the following rules when you set the type parameter to zset:
  • Set the mode parameter to zadd, which indicates that data is stored to a sorted set.
  • If the data that you want to write to Redis already exists in Redis, existing data is overwritten.
You do not need to specify this parameter. 
"writeMode":{
        "type": "zset",
        "mode": "zadd"
        }
Note If the data type is set to ZSET, each row of the source data must meet the following requirements: A row can contain only one score and one value except for the key. The score must be placed before the value. This way, Redis Writer can distinguish between the score and value.
HASH Set the type parameter to hash. The mode in which Redis Writer writes the data of the HASH type to Redis. Take note of the following rules when you set the type parameter to hash:
  • Set the mode parameter to hset, which indicates that data is stored to a hash sorted set.
  • If the data that you want to write to Redis already exists in Redis, existing data is overwritten.
You do not need to specify this parameter. 
"writeMode":{
        "type": "hash",
        "mode": "hset"
        }
Note If the data type is set to HASH, each row of the source data must comply with the following standards: A row can contain only one attribute and one value except for the key. The attribute must be placed before the value. This way, Redis Writer can distinguish between the attribute and value.

Configure Redis Writer by using the codeless UI

This method is not supported.

Configure Redis Writer by using the code editor

In the following code, a synchronization node is configured to write data from a MySQL database to Redis. For more information about the parameters, see Parameters.
Note For more information about how to configure a synchronization node by using the code editor, see Create a sync node by using the code editor. 
{
    "type":"job",
    "version":"2.0",  // The version number.
    "steps":[
        { // The following sample code is used to configure the reader. For more information about the parameters of the reader, see the topic of the related reader. 
            "stepType":"mysql",   
            "parameter": {
                "envType": 0,
                "datasource": "xc_mysql_demo2",
                "column": [
                    "id",
                    "value",
                    "table"
                ],
                "connection": [
                    {
                        "datasource": "xc_mysql_demo2",
                        "table": []
                    }
                ],
                "where": "",
                "splitPk": "",
                "encoding": "UTF-8"
            },,
            "name":"Reader",
            "category":"reader"
        },
        {// The following sample code is used to configure the writer: 
            "stepType":"redis",                    // The writer type. Set the value to redis. 
            "parameter":{                          // Configure the following parameters of Redis Writer. For more information about the parameters, see Parameters. 
                "expireTime":{                     // The expiration time of the values that are cached in Redis. Set the value to a period of time in seconds or a UNIX timestamp.seconds":"1000"
                            }, 
                "keyFieldDelimiter":"u0001",       // The delimiter that you want to use to separate keys when Redis Writer writes data to Redis. 
                "dateFormat":"yyyy-MM-dd HH:mm:ss",// The format in which Redis Writer writes the data of the DATE type to Redis. 
                "datasource":"xc_mysql_demo2",     // The name of the data source. The name must be the same as that of the data source that you added. 
                "envType": 0,                      // The environment type. Valid values: 0 and 1. The value 1 indicates the development environment. The value 0 indicates the production environment. 
                "writeMode":{                      // The mode in which Redis Writer writes data to Redis. 
                    "type": "string",                 // The data type of the value that you want to write to Redis. 
                    "mode":"set",                  // The mode in which the data of a type specified by the type parameter is written to Redis. 
                    "valueFieldDelimiter":"u0001", // The delimiter that you want to use to separate column values. 
                             },
                "keyIndexes":[0,1],                // The indexes of source columns that are used as the key when Redis Writer writes data to Redis. In this example, the value is set to [0,1], which indicates that the first and second columns of the source data are used as the key. 
                "batchSize":"1000"                 // The number of data records to write at a time. 
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"                           // The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}