Tablestore:Export full data from Tablestore to OSS

(Recommended) Use the codeless UI

1. In the Configure Source and Destination section of the Configure tasks step, configure the source and destination data sources based on the actual case.

   Configure the source data source

   Parameter	Description
   Table	The name of the data table in Tablestore.
   Range of Primary Key(begin)	The start and end primary keys that are used to specify the range of the data to be read. The value is a JSON array. The start and end primary keys must be valid primary keys or virtual points that consist of INF_MIN type values and INF_MAX type values. The number of columns in the virtual points must be the same as that in the primary key. INF_MIN indicates an infinitely small value. All values of other types are greater than the INF_MIN type value. INF_MAX indicates an infinitely great value. All values of other types are smaller than the INF_MAX type value. The rows in the data table are sorted in ascending order by primary key. The reading range is a left-closed and right-open interval. All rows that are greater than or equal to the start primary key and less than the end primary key are returned. For example, a table contains the pk1 and pk2 primary key columns. The pk1 column is of the STRING type. The pk2 column is of the INTEGER type. To export full data from the table, specify the following parameters: Example of the Range of Primary Key(begin) parameter [ { "type": "INF_MIN" }, { "type": "INF_MIN" } ] Example of the Range of Primary Key(end) parameter [ { "type": "INF_MAX" }, { "type": "INF_MAX" } ] To export data from the rows in which the value of the pk1 column is tablestore, specify the following parameters: Example of the Range of Primary Key(begin) parameter [. { "type": "STRING", "value": "tablestore" }, { "type": "INF_MIN" } ] Example of the Range of Primary Key(end) parameter [ { "type": "STRING", "value": "tablestore" }, { "type": "INF_MAX" } ]
   Range of Primary Key(end)
   Split configuration information	The custom rule to split data. We recommend that you do not specify this parameter in common scenarios. If data is unevenly distributed in a Tablestore table, you can customize a rule to split data. You can configure a split key within the range between the start and end primary keys. You do not need to specify all primary keys. The value is a JSON array.

   Configure the destination data source

   Parameter	Description
   Text type	The format of the object that is written to OSS, such as .csv or .txt. Note Different object formats require different configurations. Select an object format and specify the specific parameters that are displayed.
   File name (including path)	This parameter is displayed only when you set the Text type parameter to text, csv, or orc. The name of the object in OSS. The name can contain a path, such as tablestore/20231130/myotsdata.csv.
   path	This parameter is displayed only when you set the Text type parameter to parquet. The path of the object in OSS, such as tablestore/20231130/.
   fileName	This parameter is displayed only when you set the Text type parameter to parquet. The name of the object in OSS.
   Column Delimiter	This parameter is displayed only when you set the Text type parameter to text or csv. The delimiter that is used to separate columns when data is written to the OSS object.
   Row Delimiter	This parameter is displayed only when you set the Text type parameter to text. The custom delimiter that is used to separate rows. Example: \u0001. You must use a delimiter that does not exist in the data as the row delimiter. If you want to use the default row delimiter \n for Linux or \r\n for Windows, we recommend that you leave this parameter empty. The system automatically reads data based on the default row delimiter.
   Coding	This parameter is displayed only when you set the Text type parameter to text or csv. The encoding format of the OSS object to which you write data.
   null value	This parameter is displayed only when you set the Text type parameter to text, csv, or orc. The string that can be interpreted as null in the source data source. For example, if you set this parameter to null and the value of a field in the source data source is null, the system recognizes the field as null.
   Time format	This parameter is displayed only when you set the Text type parameter to text or csv. The time format in which date data is written to the OSS object, such as yyyy-MM-dd.
   Prefix conflict	The processing method that is used when the specified object name is the same as the name of an existing object in OSS. Valid values: Replace: deletes the existing object and creates another object of the same name. Retain: retains the existing object and creates another object whose name combines the existing object name and a random suffix. Error: reports an error and stops the batch synchronization task.
   Split File	This parameter is displayed only when you set the Text type parameter to text or csv. The maximum size of a single OSS object to which data is written. Unit: MB. The maximum size is 100 GB. When the size of an object that is written reaches the value of this parameter, the system generates a new object to continue writing data until all data is written.
   Write as Single File	This parameter is displayed only when you set the Text type parameter to text or csv. Specifies whether to write a single object to OSS at a time. By default, multiple objects are written to OSS at a time. If no data is read during data writing, the following cases may occur: If an object header is configured, an empty object that contains only the object header is generated. If no object header is configured, a completely empty object is generated. If you want to write a single object to OSS at a time, select Write as Single File. In this case, if no data is read during data writing, no empty file is generated.
   First Row as Table Header	This parameter is displayed only when you set the Text type parameter to text or csv. Specifies whether to write the first row as table headers when data is written to an object. By default, no table headers are generated. If you want to write the first row as table headers, select First Row as Table Header.

2. In the Field Mapping section, click the icon next to Source field and manually edit the source fields that you want to synchronize to OSS. You do not need to specify the destination fields.

3. In the Channel Control section, configure the parameters for task execution, such as the Synchronization rate and Dirty Data Records Allowed parameters. For more information about the parameters, see Configure a batch synchronization node by using the codeless UI.

4. Click the icon to save the configuration.

   Note

   If you do not save the configuration, a message appears when you perform subsequent operations. In this case, click OK in the message to save the configuration.

Use the code editor

To synchronize full data, you must use Tablestore Reader and OSS Writer. For more information about how to configure the task by using the code editor, see Tablestore data source and OSS data source.

1. In the Configure tasks step, click the icon. In the dialog box that appears, click OK.

2. In the code editor, specify the parameters based on the following sample code.

   Important

   • In most cases, a task that exports full data is run only once. Therefore, you do not need to configure scheduling parameters for the task. For information about how to configure scheduling parameters, see Synchronize incremental data to OSS.

   • If the script configurations contain variables such as ${date}, set each variable to a specific value when you run the task to synchronize data.

   • Comments are provided in the sample code to help you understand the configurations. Delete all comments when you use the sample code.

   {
       "type": "job", // You cannot change the value of this parameter. 
       "version": "2.0", // The version number. You cannot change the value of this parameter. 
       "steps": [
           {
               "stepType":"ots",// The name of the reader. You cannot change the value of this parameter. 
               "parameter":{
                   "datasource":"otssource",// The name of the Tablestore data source. Specify this parameter based on your business requirements. 
                   "newVersion":"true",// Use Tablestore Reader of the latest version. 
                   "mode": "normal",// Read data in row mode. 
                   "isTimeseriesTable":"false",// Configure the table as a wide table rather than a time series table. 
                   "column":[// Required. The names of the columns you want to export to OSS. 
                       {
                           "name":"column1"// The name of the column that you want to export from the Tablestore data source to OSS. 
                       },
                       {
                           "name":"column2"
                       },
                       {
                           "name":"column3"
                       },
                       {
                           "name":"column4"
                       },
                       {
                           "name":"column5"
                       }
                   ],
                   "range":{
                       "split":[ // The partition configurations of the data table in Tablestore. You can specify this parameter to accelerate the export. In most cases, you do not need to specify this parameter. 
                       ],
                       "end":[// The information about the end primary key column in Tablestore. To export full data, set this parameter to INF_MAX. To export only part of data, specify this parameter based on your business requirements. If the data table contains multiple primary key columns, configure the information about the primary key columns for the end parameter. 
                           {
                               "type":"STRING",
                               "value":"endValue"
                           },
                           {
                               "type":"INT",
                               "value":"100"
                           },
                           {
                               "type":"INF_MAX"
                           },
                           {
                               "type":"INF_MAX"
                           }
                       ],
                       "begin":[// The information about the start primary key column in Tablestore. To export full data, set this parameter to INF_MIN. To export only part of data, specify this parameter based on your business requirements. If the data table contains multiple primary key columns, configure the information about the primary key columns for the begin parameter. 
                           {
                               "type":"STRING",
                               "value":"beginValue"
                           },
                           {
                               "type":"INT",
                               "value":"0"
                           },
                           {
                               "type":"INF_MIN"
                           },
                           {
                               "type":"INF_MIN"
                           }
                       ]
                   },
                   "table":"datatable"// The name of the data table in Tablestore. 
               },
               "name":"Reader",
               "category":"reader"
           },
           {
               "stepType": "oss",// The name of the writer. You cannot change the value of this parameter. 
               "parameter": {
                   "nullFormat": "null", // The string used to identify the null field value. The value can be an empty string. 
                   "dateFormat": "yyyy-MM-dd HH:mm:ss",// The format of the time. 
                   "datasource": "osssource", // The name of the OSS data source. Specify this parameter based on your business requirements. 
                   "envType": 1,
                   "writeSingleObject": true, // Write a single object to OSS at a time. 
                   "writeMode": "truncate", // The operation to be performed by the system if an object that has the specified object name exists in the OSS data source. Valid values: truncate, append, and nonConflict. To export full data, set this parameter to truncate. A value of truncate specifies that the system clears the object in the OSS data source. A value of append specifies that the system appends the data to the object in the OSS data source. A value of nonConflict specifies that an error is reported. 
                   "encoding": "UTF-8", // The encoding format. 
                   "fieldDelimiter": ",", // The delimiter used to separate columns. 
                   "fileFormat": "csv", // The exported file format. Valid values: csv, text, parquet, and orc. 
                   "object": "tablestore/20231130/myotsdata.csv" // The prefix of the object name in the OSS data source. You do not need to include the bucket name in the prefix. Example: tablestore/20231130/. To perform scheduled export, you must use variables in the prefix such as tablestore/${date}. Then, specify the ${date} variable when you configure scheduling parameters. 
               },
               "name": "Writer",
               "category": "writer"
           },
           {
               "name": "Processor",
               "stepType": null,
               "category": "processor",
               "copies": 1,
               "parameter": {
                   "nodes": [],
                   "edges": [],
                   "groups": [],
                   "version": "2.0"
               }
           }
       ],
       "setting": {
           "executeMode": null,
           "errorLimit": {
               "record": "0" // When the number of errors exceeds the value of this parameter, the data fails to be imported to OSS. 
           },
           "speed": {
               "concurrent": 2, // The concurrency. 
               "throttle": false
           }
       },
       "order": {
           "hops": [
               {
                   "from": "Reader",
                   "to": "Writer"
               }
           ]
       }
   }

3. Click the icon to save the configuration.

   Note

   If you do not save the script, a message appears when you perform subsequent operations. In this case, click OK in the message to save the script.