edit-icon download-icon

Configure Table Store Writer-Internal

Last Updated: Mar 21, 2018

Table Store (previously known as OTS) is a NoSQL database service built on Alibaba Cloud’s Apsara distributed system, enabling you to store and access massive structured data in real time. Table Store organizes data into instances and tables. Using data partition and server load balancing technology, it provides seamless scaling.

Table Store Writer-Internal is used to import data to the table of Table Store Internal model while Table Store Writer is used to import data to the table of Table Store Public model.

Table Store Internal model supports multi‑version columns, so this plug‑in provides two modes for data import.

  • Multi-version mode: Sparse columns are allowed to be imported. You can specify a version for each column. Multi‑version mode has a special requirement for the data output by Reader. Currently, only HBaseReader is supported.

  • Normal mode: Fixed columns and data with specified types are allowed to be imported. You can specify a uniform version for all columns, or choose not to specify any version. Two write modes are provided in normal mode.

    • PutRow: PutRow for Table Store API, which is used to insert data to a specified row. If this row does not exist, a new row is added. Otherwise, the original row is overwritten.

    • UpdateRow: UpdateRow for Table Store API, which is used to update the data of a specified row. If the row does not exist, a new row is added. Otherwise, the values of the specified columns are added, modified, or deleted as request.

In short, Table Store Writer-Internal connects to the Table Store server by using the official Table Store Java SDK and writes data into the Table Store server by using the SDK. Table Store Writer-Internal has greatly optimized the write process, including retry upon write timeout, retry upon exception in writing, batch submission, and other features.

Currently, Table Store Writer-Internal supports all Table Store data types and converts the data types in Table Store as follows.

Internal data integration type Table Store data type
Long Integer
Double Double
String String
Boolean Boolean
Bytes Binary

Note:

  • Table Store itself does not support “date” type.
  • Long value is generally used as Unix TimeStamp at application layer when an error is reported.

Parameter description

  • endpoint

    • Description: The EndPoint of Table Store Server.

    • Required: Yes

    • Default value: None

  • accessId

    • Description: accessId of Table Store

    • Required: Yes

    • Default value: None

  • accessKey

    • Description: AccessKey of Table Store

    • Required: Yes

    • Default value: None

  • instanceName

    • Description: The instance name for accessing Table Store

      An instance is an entity for using and managing the Table Store service. After you enable the OTS service, you can create an instance in the Console to create and manage tables. Instance is the basic unit for Table Store resource management. All access control and resource measurement made by Table Store for applications are completed at the instance level.

    • Required: Yes

    • Default value: None

  • table

    • Description: The name of the table to be extracted. Only one table can be filled in. Multi-table synchronization is not required for Table Store.

    • Required: Yes

    • Default value: None

  • primaryKey

    • Description: The primary key information of Table Store. Only the primary key name is required. Configuration example is as follows.

      1. "primaryKey" : ["pk1", "pk2"]
    • Required: Yes

    • Default value: None

  • mode

    • Description: Specify multi‑version mode or normal mode. Available value: MultiVersion or Normal.

    • Required: Yes

    • Default value: None

  • columnNamePrefixFilter

    • Description: If this value is a non‑null string, all column names are checked in multi‑version mode. All columns must start with this prefix which is then removed when these columns are imported into Table Store. Any non‑compliant column is considered as dirty data.

      Use case: When HBase data is imported into Table Store, ColumnFamily in column name requires to be removed.

    • Required: No

    • Default value: Null string

  • column

    • Description: The column name set to be synchronized in the configured table. Field information is described with arrays in JSON. The format is as follows.

      1. {"name":"col2", "type":"INT"},

      “name” specifies the name of Table Store’s column to be written, and “type” specifies the type of data to be written. Data types supported by Table Store include STRING, INT, DOUBLE, BOOL, and BINARY.

      Constants, functions, or custom statements are not supported during write process. This configuration only takes effect in normal mode.

    • Required: Yes

    • Default value: None

  • writeMode

    • Description: Write mode. The following two modes are supported.

      • PutRow: PutRow for Table Store API, which is used to insert data to a specified row. If this row does not exist, a new row is added. Otherwise, the original row is overwritten.

      • UpdateRow: UpdateRow for Table Store API, which is used to update the data of a specified row. If the row does not exist, a new row is added. Otherwise, the values of the specified columns are added, modified, or deleted as request. This configuration only takes effect in normal mode.

    • Required: Yes

    • Default value: None

  • defaultTimestampInMillionSecond

    • Description: The version of a column written into Table Store in normal mode. If not specified, the version is determined based on the time when the column is written into Table Store by the server.

    • Required: No

    • Default value: None

Development in wizard mode

Currently, development in wizard mode is not supported.

Development in script mode

Examples of multi‑version mode are shown as follows.

  1. {
  2. "type": "job",
  3. "version": "1.0",
  4. "configuration": {
  5. "reader": {
  6. "plugin": "mysql",
  7. "parameter": {}
  8. },
  9. "writer": {
  10. "plugin": "otswriter-internalwriter",
  11. "parameter": {
  12. "endpoint":"", # The address for accessing Table Store service
  13. "accessId":"", # AccessId required for accessing Table Store service
  14. "accessKey":"", # AccessKey required for accessing Table Store service
  15. "instanceName":"", # The instance name for accessing Table Store
  16. "table":"", # The name of the table to be imported into Table Store
  17. "mode":"multiVersion", # Import mode. Choose multiVersion for multi-version mode.
  18. "primaryKey":[ # The primary key of the table. Only the name of the primary key is required.
  19. "userid",
  20. "groupid"
  21. ],
  22. "columnNamePrefixFilter":"cf:" # Filter the prefix of the column name to remove the ColumnFamily information from the column name in the data exported by HBase.
  23. }
  24. }
  25. }
  26. }

Examples of normal mode are shown as follows.

  1. {
  2. "type": "job",
  3. "version": "1.0",
  4. "configuration": {
  5. "reader": {
  6. "plugin": "mysql",
  7. "parameter": {}
  8. },
  9. "writer": {
  10. "plugin": "otswriter-internalwriter",
  11. "parameter":{
  12. "endpoint":"", # The address for accessing Table Store service
  13. "accessId":"", # AccessId required for accessing Table Store service
  14. "accessKey":"", # AccessKey required for accessing Table Store service
  15. "instanceName":"", # The instance name for accessing Table Store
  16. "table":"", # The name of the table to be imported into Table Store
  17. "mode":"normal", # Import mode. Choose normal for normal mode.
  18. "primaryKey":[ # The primary key of the table. Only the name of the primary key is required.
  19. "userid",
  20. "groupid"
  21. ],
  22. "column":[ # The name and type of the column to be imported
  23. {"name":"addr", "type":"string"},
  24. {"name":"height", "type":"int"}
  25. ],
  26. "defaultTimestampInMillionSecond": 142722431, # The uniform version of the column to be imported
  27. "writeMode":"PutRow" # Write mode. PutRow and UpdateRow are available.
  28. }
  29. }
  30. }
  31. }
Thank you! We've received your feedback.