Redis Writer is a writer that is developed based on the Data Integration framework. It can write the data that is obtained from a reader to Redis.

REmote DIctionary Server (Redis) is a key-value storage system that can be accessed over the network and can permanently store data or store data in memory. Redis supports logs and provides high performance. You can use Redis as a database, a cache, or a message broker. Redis supports values of the following data types: STRING, LIST, SET, ZSET (sorted set), and HASH. For more information about Redis, visit redis.io.

Redis Writer interacts with a Redis server by using Jedis. Jedis is a Java client development kit that is provided and recommended by Redis.
Note
  • Redis Writer can write data only to Redis clusters. The synchronization nodes that use Redis Writer must run on exclusive resource groups for Data Integration.
  • Before you configure Redis Writer, you must add a Redis data source. For more information, see Add a Redis data source.
  • If you rerun a synchronization node that uses Redis Writer to write LIST values to Redis, the result is not idempotent. In this case, you must manually delete the data written by Redis Writer in the previous run from Redis before you rerun the synchronization node.

Parameters

Parameter Description Required Default value
expireTime The expiration time of the values that are cached in Redis. Unit: seconds. If you do not configure this parameter, the default value 0 is used, which indicates that the values are permanently valid.
You can specify the value of the expireTime parameter in one of the following modes:
  • seconds: the relative period of time in seconds starting from the current point in time. It specifies the period during which the values are valid.
  • unixtime: the number of seconds that have elapsed since 00:00:00 on January 1, 1970. It specifies the point in time when the values expire.
    Note If you set the expireTime parameter to a value that is greater than 2,592,000 (30 days), the value is interpreted as a UNIX timestamp.
No 0
keyFieldDelimiter The delimiter that you want to use to separate keys when Redis Writer writes data to Redis. Example: key=key1\u0001id. If multiple keys need to be concatenated, this parameter is required. If the data contains only one key, this parameter is not required. No \u0001
dateFormat The format in which Redis Writer writes the data of the DATE type to Redis. Set the value to yyyy-MM-dd HH:mm:ss. No No default value
datasource The name of the data source. The name must be the same as that of the data source that you added. Yes No default value
writeMode The mode in which Redis Writer writes data to Redis. Redis Writer can write values of the following types:
  • STRING
  • LIST
  • SET
  • ZSET (sorted set)
  • HASH
The value of the writeMode parameter varies based on the data types of the values that you want to write to Redis. For more information, see Configure writeMode.
Note When you configure Redis Writer, you can specify only one data type for the writeMode parameter. If you do not configure this parameter, the default value string is used.
No string
keyIndexes The indexes of source columns that are used as the key. The indexes start from 0. This indicates that the index of the first column is 0, and the index of the second column is 1, and so on.
  • If you want to specify a specific column of the source as the key, set this parameter to the index of the column. For example, if you want to specify the first column as the key, set this parameter to 0.
  • If you want to specify a range of contiguous columns of the source as the key, set this parameter to a closed interval that specifies the indexes of these columns. For example, if you want to specify the second, third, and fourth columns as the key, set this parameter to [1,3].
Note After you specify the keyIndexes parameter, Redis Writer specifies the remaining columns as the value. If you want to synchronize only some of the columns in the source, specify the names of the columns when you configure a reader.
Yes No default value
batchSize The number of data records to write at a time. Set this parameter to an appropriate value based on your business requirements. This greatly reduces the interactions between Data Integration and Redis and increases throughput. If you set this parameter to an excessively large value, an out of memory (OOM) error may occur during data synchronization. No 1,000
timeout The timeout period of the connection to Redis when data is written to Redis. Unit: milliseconds. No 30,000
redisMode The deployment mode of Redis. Redis Writer can write data only to Redis clusters. You must set this parameter to ClusterMode.
Note Synchronization nodes that use Redis Writer must run on exclusive resource groups for Data Integration.
No No default value
column

The names of the columns to which you want to write data. If you set the type parameter to string and the mode parameter to set, take note of the following points:

  • If the column parameter is not configured, the values stored in Redis are strings that are connected by delimiters. For example, in a CSV source file, the value of id is 1, the value of name is Bob, the value of age is 18, and the value of sex is male. In this case, the values to be written to Redis are "18::male".
  • If the column parameter is configured in the "column": [{"index":"0", "name":"id"}, {"index":"1", "name":"content"}] format, the values are stored in Redis in JSON format {"id":"Value of a source column","name":"Value of a source column"}. For example, the value of id is 1, and the value of name is Bob. In this case, the values stored in Redis are {"id":1,"name":"Bob"}.
No No default value

Configure writeMode

When you configure Redis Writer, you can specify only one data type for the writeMode parameter. If you do not configure this parameter, the default value string is used.
Data type of values type parameter (required) mode parameter (required) valueFieldDelimiter parameter (optional) Configuration example
STRING Set the type parameter to string. The mode in which Redis Writer writes the data of the STRING type to Redis. Take note of the following rules when you set the type parameter to string:
  • Set the mode parameter to set.
  • If the data that you want to write to Redis already exists in Redis, existing data is overwritten.
The delimiter that you want to use to separate column values. Default value: \u0001.
  • This parameter is required if you specify two or more columns as the value. For example, you specify three columns of the source data as the value and use the default delimiter. In this case, the column values are concatenated in the value1\u0001value2\u0001value3 format.
  • This parameter is not required if you specify only one column as the value.
"writeMode":{
        "type": "string",
        "mode": "set",
        "valueFieldDelimiter": "\u0001"
        }
LIST Set the type parameter to list. The mode in which Redis Writer writes the data of the LIST type to Redis. Valid values:
  • lpush: indicates that data is stored at the leftmost of the list.
  • rpush: indicates that data is stored at the rightmost of the list.
"writeMode":{
    "type": "list",
    "mode": "lpush|rpush",
    "valueFieldDelimiter": "\u0001"
}
SET Set the type parameter to set. The mode in which Redis Writer writes the data of the SET type to Redis. Take note of the following rules when you set the type parameter to set:
  • Set the mode parameter to sadd, which indicates that data is stored to a set.
  • If the data that you want to write to Redis already exists in Redis, existing data is overwritten.
"writeMode":{
        "type": "set",
        "mode": "sadd",
        "valueFieldDelimiter": "\u0001"
        }
ZSET (sorted set) Set the type parameter to zset. The mode in which Redis Writer writes the data of the ZSET type to Redis. Take note of the following rules when you set the type parameter to zset:
  • Set the mode parameter to zadd, which indicates that data is stored to a sorted set.
  • If the data that you want to write to Redis already exists in Redis, existing data is overwritten.
You do not need to configure this parameter.
"writeMode":{
        "type": "zset",
        "mode": "zadd"
        }
Note If the data type is set to ZSET, each row of the source data must meet the following requirements: A row can contain only one score and one value except for the key. The score must be placed before the value. This way, Redis Writer can distinguish between the score and value.
HASH Set the type parameter to hash. The mode in which Redis Writer writes the data of the HASH type to Redis. Take note of the following rules when you set the type parameter to hash:
  • Set the mode parameter to hset, which indicates that data is stored to a hash sorted set.
  • If the data that you want to write to Redis already exists in Redis, existing data is overwritten.
You do not need to configure this parameter.
"writeMode":{
        "type": "hash",
        "mode": "hset"
        }
Note If the data type is set to HASH, each row of the source data must meet the following requirements: A row can contain only one attribute and one value except for the key. The attribute must be placed before the value. This way, Redis Writer can distinguish between the attribute and value.

Configure Redis Writer by using the codeless UI

  1. Configure data sources.
    Configure Source and Target for the synchronization node. Configure data sources
    Parameter Description
    Connection The name of the data source to which you want to write data. This parameter is equivalent to the datasource parameter that is described in the Parameters section.
    keyIndexes This parameter is equivalent to the keyIndexes parameter that is described in the Parameters section. This parameter specifies the indexes of source columns that are used as the key. The indexes start from 0. This indicates that the index of the first column is 0, and the index of the second column is 1, and so on.
    Key separator This parameter is equivalent to the keyFieldDelimiter parameter that is described in the Parameters section. This parameter specifies the delimiter that you want to use to separate keys when Redis Writer writes data to Redis.
    Redis key prefix The prefix for the key. For example, the prefix is prefix::, and the key is 1::2. In this case, the key that is generated is prefix::1::2.
    BatchSize This parameter is equivalent to the batchSize parameter that is described in the Parameters section.
    ExpireTime This parameter is equivalent to the expireTime parameter that is described in the Parameters section.
    Timeout This parameter is equivalent to the timeout parameter that is described in the Parameters section.
    DateFormat This parameter is equivalent to the dateFormat parameter that is described in the Parameters section.
    Redis Mode This parameter is equivalent to the redisMode parameter that is described in the Parameters section.
    Redis WriteMode Type This parameter is equivalent to the writeMode parameter that is described in the Parameters section.
    Redis WriteMode The mode in which Redis Writer writes data to Redis. Valid values: set, Ipush, rpush, sadd, zadd, and hset. This parameter is equivalent to the writeMode parameter that is described in the Parameters section. For more information, see Configure writeMode.
    Redis Write delimiter This parameter is equivalent to the keyFieldDelimiter parameter that is described in the Parameters section.
  2. Configure field mappings. This operation is equivalent to setting the column parameter that is described in the Parameters section. The system maps the field in a row of the source to the field in the same row of the destination. You can click the Icon icon to edit fields in the source. Field mappings

Configure Redis Writer by using the code editor

In the following code, a synchronization node is configured to write data from a MySQL database to Redis. For more information about the parameters, see Parameters.
Note For more information about how to configure a synchronization node by using the code editor, see Create a synchronization node by using the code editor.
{
    "type":"job",
    "version":"2.0", // The version number.
    "steps":[
        { // The following sample code is used to configure the reader. For more information about the parameters of the reader, see the topic of the related reader. 
            "stepType":"mysql",   
            "parameter": {
                "envType": 0,
                "datasource": "xc_mysql_demo2", 
                "column": [
                    "id",
                    "value",
                    "table"
                ],
                "connection": [
                    {
                        "datasource": "xc_mysql_demo2",
                        "table": []
                    }
                ],
                "where": "",
                "splitPk": "",
                "encoding": "UTF-8"
            },,
            "name":"Reader",
            "category":"reader"
        },
        {// The following sample code is used to configure the writer: 
            "stepType":"redis",                    // The writer type. Set the value to redis. 
            "parameter":{                          // Configure the following parameters of Redis Writer. For more information about the parameters, see Parameters. 
                "expireTime":{                     // The expiration time of the values that are cached in Redis. Set the value to a period of time in seconds or a UNIX timestamp. For example, you can set this parameter to "seconds":"1000".
                            }, 
                "keyFieldDelimiter":"u0001",       // The delimiter that you want to use to separate keys when Redis Writer writes data to Redis. 
                "dateFormat":"yyyy-MM-dd HH:mm:ss",// The format in which Redis Writer writes the data of the DATE type to Redis. 
                "datasource":"xc_mysql_demo2",     // The name of the data source. The name must be the same as that of the data source that you added. 
                "envType": 0,                      // The type of the environment. Set the value to 1 for the development environment. Set the value to 0 for the production environment. 
                "writeMode":{                      // The mode in which data is written to Redis. 
                    "type": "string",                 // The data type of the value that you want to write to Redis. 
                    "mode":"set",                  // The mode in which the data of a type specified by the type parameter is written to Redis. 
                    "valueFieldDelimiter":"u0001", // The delimiter that you want to use to separate column values. 
                             },
                "keyIndexes":[0,1],                // The indexes of source columns that are used as the key when Redis Writer writes data to Redis. In this example, the value is set to [0,1], which indicates that the first and second columns of the source data are used as the key. 
                "batchSize":"1000"                 // The number of data records to write at a time. 
        "column": [                        // If you set the type parameter to string and the mode parameter to set and do not configure the column parameter, the values to be written to Redis are strings that are connected by delimiters. For example, for a CSV source file, if the value of id is 1, the value of name is Bob, the value of age is 18, and the value of sex is male, the values to be written to Redis are "18::male". If you set the type parameter to string and the mode parameter to set and configure the column parameter in the following format, the values are written to Redis in JSON format, such as {"id":1,"name":"Bob","age":18,"sex":"male"}.
                {
                "name": "id",
                "index": "0"

                },
                {
                "name": "name",
                "index": "1"
                },
                {
                "name": "age",
                "index": "2"
                },
                {
                "name": "sex",
                "index": "3"
                }
            ]
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"                           // The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}