edit-icon download-icon

Configure Table Store writer

Last Updated: Apr 03, 2018

Table Store is a NoSQL database service built upon Alibaba Cloud’s Apsara distributed system, enabling you to store and access massive structured data in real time. Table Store organizes data into instances and tables. Using data partition and server load balancing technology, it provides seamless scaling.

In short, Table Store Writer connects to Table Store server using the official Table Store Java SDK and writes data into Table Store server using SDK. Table Store Writer has greatly optimized the write process, including retry upon write timeout, retry upon exception in writing, batch submission, and other features.

The Table Store Writer plug‑in provides the ability to write data into Table Store in two modes:

  • PutRow: PutRow for Table Store API, which is used to insert data to a specified row. If this row does not exist, a new row is added. Otherwise, the original row is overwritten.

  • UpdateRow: UpdateRow for Table Store API, which is used to update the data of a specified row. If the row does not exist, a new row is added. Otherwise, the values of the specified columns are added, modified, or deleted as requested.

Currently, Table Store Writer supports all Table Store data types and converts the data types in Table Store as follows:

Category Table Store data type
Integer Integer
Floating point Double
String String
Boolean Boolean
Binary Binary

Note:

You must configure the Integer category to Int in script mode so that it can be converted to the Integer type for Table Store. If you directly configure it to the Integer type for Table Store, an error is reported in the log and causes task failure.

Parameter description

  • datasource

    • Description: Data source name. It must be identical to the data source name added. Adding data source is supported in script mode.

    • Required: Yes

    • Default value: None

  • endPoint

    • Description: The EndPoint of OTS Server. For more information, see Access control.

    • Required: Yes

    • Default value: None

  • accessId

    • Description: accessId of Table Store

    • Required: Yes

    • Default value: None
  • accessKey

  • instanceName

    • Description: The instance name for accessing Table Store

      An instance is an entity for using and managing the Table Store service. After you enable the OTS service, you can create an instance in the Console to create and manage tables. Instance is the basic unit for Table Store resource management. All access control and resource measurement made by Table Store for applications are completed at the instance level.

    • Required: Yes

    • Default value: None

  • table

    • Description: The name of the table to be extracted. Only one table can be filled in. Multi-table synchronization is not required for Table Store.

    • Required: Yes

    • Default value: None

  • primaryKey

    • Description: Primary key information of Table Store. Field information is described using JSON array. Table Store itself is a NoSQL system, so the corresponding field name must be specified when Table Store Writer imports data.

      PrimaryKey of Table Store only supports STRING and INT types, so only these two types can be entered for Table Store Writer.

      Data synchronization system supports data type conversion, so Table Store Writer can convert the non‑String and non-Int source data.

      Configuration example:

      1. json
      2. "primaryKey" : [
      3. {"name":"pk1", "type":"string"},
      4. {"name":"pk2", "type":"int"}
      5. ],
    • Required: Yes

    • Default value: None

  • column

    • Description: The column name set to be synchronized in the configured table. Field information is described with arrays in JSON.

      The format is as follows:

      1. {"name":"col2", "type":"INT"},

      “name” specifies the name of Table Store’s column to be written, and “type” specifies the type of data to be written. Data types supported by Table Store include STRING, INT, DOUBLE, BOOL, and BINARY.

      Constants, functions, or custom statements are not supported during write process.

    • Required: Yes

    • Default value: None

  • writeMode

    • Description: Write mode. The following three modes are supported:

      • Single-row operations

        1. GetRow: Read data from a single row.
        2. PutRow: PutRow for Table Store API, which is used to insert data to a specified row. If this row does not exist, a new row is added. Otherwise, the original row is overwritten.
        3. UpdateRow: UpdateRow for Table Store API, which is used to update the data of a specified row. If the row does not exist, a new row is added. Otherwise, the values of the specified columns are added, modified, or deleted as request.
        4. DeleteRow: Delete a row.
      • Batch operation

        1. BatchGetRow: Read data from multiple rows.
      • Read range

        1. GetRange: Read table data within a certain range.
    • Required: Yes

    • Default value: None

Development in wizard mode

Currently, development in wizard mode is not supported.

Development in script mode

Configure a job to write data to Table Store:

  1. {
  2. "type": "job",
  3. "version": "1.0",
  4. "configuration": {
  5. "reader": {},
  6. "writer": {
  7. "plugin": "ots",
  8. "parameter": {
  9. "endpoint":"",
  10. "accessId":"",
  11. "accessKey":"",
  12. "instanceName":"",
  13. "table": "",
  14. "primaryKey": [
  15. {
  16. "name": "pk1",
  17. "type": "string"
  18. },
  19. {
  20. "name": "pk2",
  21. "type": "int"
  22. }
  23. ],
  24. "column": [
  25. {
  26. "name": "col2",
  27. "type": "INT"
  28. },
  29. {
  30. "name": "col3",
  31. "type": "STRING"
  32. },
  33. {
  34. "name": "col4",
  35. "type": "STRING"
  36. },
  37. {
  38. "name": "col5",
  39. "type": "BINARY"
  40. },
  41. {
  42. "name": "col6",
  43. "type": "DOUBLE"
  44. }
  45. ],
  46. "writeMode": "PutRow"
  47. }
  48. }
  49. }
  50. }
Thank you! We've received your feedback.