edit-icon download-icon

Configure Redis writer

Last Updated: Apr 03, 2018

Redis (REmote DIctionary Server) is a high‑performance, networked, in-memory, key‑value store system or a persistent log, which can be used as a database, high‑speed cache, and message queue broker. Redis supports various types of storage value, including string, list, set, zset (sorted set), and hash. For more information on Redis, see redis.io.

Redis Writer is a Redis writing plug‑in built on the Data Integration framework. It is used to import data from a warehouse or other data sources into Redis. Redis Writer interacts with Redis Server through Jedis, which is a preferred Java client development kit officially provided by Redis, and covers almost all of the features of Redis. You must configure the data source before configuring the Redis Writer plug‑in. For more information, see Redis data source config.

Note:

When you write data into Redis using Redis Writer, if the value type is list, the result of the re-run synchronization task is not idempotent. So if the value type is list, you must manually clear the corresponding data on Redis when re-running the synchronization task.

Parameter description

  • redisServer

    • Description: It is configured with the Host, port, and password information of all Redis servers, which can be a single Redis server or a Redis cluster.

    • Required: Yes

  • host

    • Description: The ip or host of a Redis server.

    • Required: Yes

  • port

    • Description: The connection port of a Redis server. Default is 6379.

    • Required: No

    • Default value: 6379

  • password

    • Description: The password for accessing Redis. If it does not exist, you can leave it blank.

    • Required: No

  • keyIndexes

    • Description: keyIndexes indicates which columns of the source table are used as key (starts with 0 for the first column). If the key is a combination of the first and second columns, the value of keyIndexes is [0,1].

      Note: After the keyIndexes are configured, Redis Writer sets the remaining columns as the value. If you only want to synchronize several columns of the source table as the key and several other columns as value, but not all fields, configure the column filtering conditions in the Reader plug‑in.

    • Required: Yes

    • Default value: None

  • keyFieldDelimiter

    • Description: The delimiter of the keys to be written into Redis. For example: key=key1\u0001id. If multiple keys are be combined, this value is required. If only one key exists, this configuration item can be ignored.

    • Required: No

    • Default value: \u0001

  • batchSize

    • Description: The quantity of records submitted in one operation. Setting this parameter can greatly reduce the interactions between Data Integration and Redis, and increase the overall throughout. However, an excessively large value may cause the running process of Data Integration to become out of memory (OOM).

    • Required: No

    • Default value: 1000

  • expireTime

    • Description: Cache invalidation time for Redis value (which is valid permanently if this configuration item is left empty).

      • seconds: The relative time (in seconds) starting from the current time point, which indicates the time length from the current time before data is invalid.
      • unixtime: Unix time (number of seconds since January 1, 1970) indicates that data is invalid at a certain time point in the future.

        Note: If the invalidation time is larger than 60*60*24*30 (30 days), the server identifies the invalidation time as the Unix time.

    • Unit: Second

  • Required: No

    • Default value: 0 (valid permanently)
  • timeout

    • Description: Length of time-out when data is written into Redis.

    • Unit: Millisecond

    • Required: No

    • Default value: 30000 (which can cover 30 seconds after network disconnection)

  • dateFormat

    • Description: The time when data is written into Redis in date format: “yyyy-MM-dd HH:mm:ss”.

    • Required: No

    • Default value: None

  • writeMode

    • Description: Redis supports various types of value, including string, list, set, zset, and hash. Redis Writer can also write these types of data into Redis. However, the configuration of writeMode varies slightly with the value type. writeMode is configured as follows. Only one of the following five types can be configured when you configure Redis Writer:

      • string

        1. "writeMode":{
        2. "type": "string",
        3. "mode": "set",
        4. "valueFieldDelimiter": "\u0001"
        5. }

        Configuration item description:

        • type

          • Description: value type: string

          • Required: Yes

        • mode

          • Description: The write mode when the value type is string.

          • Required: Yes. Available value: set (store the data, and overwrite this data if it already exists)

        • valueFieldDelimiter

          • Description: The delimiter between values when the value type is string. For example, value1\u0001value2\u0001value3. This configuration item applies to the scenario where source data has more than two columns in each row (ignore this configuration item if the source data only contains two columns: “key” and “value”).

          • Required: No

          • Default value: \u0001

      • list

        1. "writeMode":{
        2. "type": "list",
        3. "mode": "lpush|rpush",
        4. "valueFieldDelimiter": "\u0001"
        5. }

        Configuration item description:

        • type

          • Description: value type: list

          • Required: Yes

        • mode

          • Description: The write mode when the value type is list.

          • Required: Yes. Available value: lpush (store the data on the far left of list) | rpush (store the data on the far right of list)

        • valueFieldDelimiter

          • Description: The delimiter between values when the value type is string. For example, value1\u0001value2\u0001value3.

          • Required: No

          • Default value: \u0001
      • set

        1. "writeMode":{
        2. "type": "set",
        3. "mode": "sadd",
        4. "valueFieldDelimiter": "\u0001"
        5. }

        Configuration item description:

        • type

          • Description: value type: set

          • Required: Yes

        • mode

          • Description: The write mode when the value type is set.

          • Required: Yes. Available value: sadd (store the data into set, and overwrite this data if it already exists)

        • valueFieldDelimiter

          • Description: The delimiter between values when the value type is string. For example, value1\u0001value2\u0001value3.

          • Required: No

          • Default value: \u0001

      • zset

        Note:If the value type is zset, each row of the data source must follow this rule: apart from the key, each row only contains one pair of score and value, and score must be located before value, so that Redis Writer can parse the score column and the value column.

        1. "writeMode":{
        2. "type": "zset",
        3. "mode": "zadd"
        4. }

        Configuration item description:

        • type

          • Description: value type: zset

          • Required: Yes;

        • mode

          • Description: The write mode when the value type is zset

          • Required: Yes. Available value: zadd (store the data into zset, and overwrite this data if it already exists)

      • hash

        Note:If the value type is hash, each row of the data source must follow this rule: apart from the key, each row only contains one pair of attribute and value, and attribute must be located before value, so that Redis Writer can parse the attribute column and the value column.

        1. "writeMode":{
        2. "type": "hash",
        3. "mode": "hset"
        4. }

        Configuration item description:

        • type

          • Description: value type: hash

          • Required: Yes

        • mode

          • Description: The write mode when the value type is hash.

          • Required: Yes. Available value: hmset (store the data into hash, and overwrite this data if it already exists)

        You must specify one of the data types. If you leave it blank, the data type is string by default.

    • Required: No

    • Default value: string

Development in wizard mode

Currently, development in wizard mode is not supported.

Development in script mode

Configure the data synchronization job to write data to Redis:

  1. {
  2. "type": "job",
  3. "version": "1.0",
  4. "configuration": {
  5. "reader": {},
  6. "writer": {
  7. "plugin": "redis",
  8. "parameter": {
  9. "redisServer": [
  10. {
  11. "host": "ip1",
  12. "port": "port1"
  13. },
  14. {
  15. "host": "ip2",
  16. "port": "port2"
  17. }
  18. ],
  19. "password": "password",
  20. "keyIndexes": [
  21. 0,
  22. 2
  23. ],
  24. "keyFieldDelimiter": "\u0001",
  25. "batchSize": 1000,
  26. "expireTime": {
  27. "seconds": 1000
  28. },
  29. "timeout": 10000,
  30. "dateFormat": "yyyy-MM-dd HH:mm:ss",
  31. "writeMode": {
  32. "type": "string",
  33. "mode": "set",
  34. "valueFieldDelimiter": "\u0001"
  35. }
  36. }
  37. }
  38. }
  39. }
Thank you! We've received your feedback.