All Products
Search
Document Center

DataWorks:Memcache data source

Last Updated:Nov 15, 2023

DataWorks provides Memcache Writer for you to write data to Memcache data sources. You can configure a batch synchronization task for a Memcache data source only by using the code editor. This topic describes the capabilities of synchronizing data to Memcache data sources.

Limits

You can configure a batch synchronization task to synchronize data to a Memcache data source only by using the code editor.

Data type mappings

Memcache Writer writes data only in the text format. The method of converting data types varies based on the format in which Memcache Writer writes data.

  • text: Memcache Writer uses the specified column delimiter to serialize source data to a string.

  • binary: This format is not supported.

Develop a data synchronization task

For information about the entry point for and the procedure of configuring a data synchronization task, see the following sections. For information about the parameter settings, view the infotip of each parameter on the configuration tab of the task.

Add a data source

Before you configure a data synchronization task to synchronize data from or to a specific data source, you must add the data source to DataWorks. For more information, see Add and manage data sources.

Configure a batch synchronization task to synchronize data of a single table

Appendix: Code and parameters

Appendix: Configure a batch synchronization task by using the code editor

If you use the code editor to configure a batch synchronization task, you must configure parameters for the reader and writer of the related data source based on the format requirements in the code editor. For more information about the format requirements, see Configure a batch synchronization task by using the code editor. The following information describes the configuration details of parameters for the reader and writer in the code editor.

Code for Memcache Writer

{
    "type":"job",
    "version":"2.0",// The version number. 
    "steps":[
        { 
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"ocs",// The plug-in name.
            "parameter":{
                "writeFormat":"text",// The format in which Memcache Writer writes the source data. 
                "expireTime":1000,// The expiration time of the source data that is cached in Memcache databases. 
                "indexes":0,
                "datasource":"",// The name of the data source. 
                "writeMode":"set",// The write mode. 
                "batchSize":"256"// The number of data records to write at a time. 
            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate. Unit: MB/s. 
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

Parameters in code for Memcache Writer

Parameter

Description

Required

Default value

datasource

The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor.

Yes

No default value

writeMode

The write mode. Valid values:

  • set: stores the source data.

  • add: stores the source data only when its key does not exist in the destination. This mode is not supported.

  • replace: uses the source data to replace the data record that has the same key as the source data in the destination. This mode is not supported.

  • append: adds the source data to the end of an existing data record that has the same key as the source data, but does not update the expiration time of the existing data record. This mode is not supported.

  • prepend: adds the source data to the beginning of an existing data record that has the same key as the source data, but does not update the expiration time of the existing data record. This mode is not supported.

Yes

No default value

writeFormat

The format in which Memcache Writer writes the source data. Only the text format is supported.

text: serializes the source data to the text format. Memcache Writer uses the first column of the source data as the key and serializes the subsequent columns to strings by using the specified delimiter. Then, Memcache Writer writes key-value pairs to Memcache.

For example, the following source data exists:

| ID   | NAME  | COUNT|
| ---- |:------|:-----|
| 23   | "CDP" | 100  |

If you set the column delimiter to \^, data is written to Memcache in the following format:

| KEY (OCS) | VALUE(OCS) |
| --------- |:---------- |
| 23        | CDP\^100   |

No

No default value

expireTime

The expiration time of the source data that is cached in Memcache. Memcache supports the expiration time in the following formats:

  • unixtime: the UNIX timestamp, which indicates a specific point in time in the future when the data expires. The UNIX timestamp represents the number of seconds that have elapsed since 00:00:00 on January 1, 1970.

  • seconds: the period of time in seconds starting from the current point in time. It specifies the period during which data is valid.

Note

If the specified expiration time is greater than 30 days, the server identifies the time as a UNIX timestamp.

No

0, which indicates that the data never expires.

batchSize

The number of data records to write at a time. Set this parameter to an appropriate value based on your business requirements. This greatly reduces the interactions between Data Integration and Memcache and increases throughput. If you set this parameter to an excessively large value, an out of memory (OOM) error may occur during data synchronization.

No

1,024