Tablestore SDK for Java can create a delivery task to deliver full data, incremental data, or both from a data table to an OSS bucket in the same region.
Prerequisites
Before you begin, make sure that the following prerequisites are met:
Install the Tablestore SDK for Java and initialize a client. Data delivery requires version 5.10.3 or later. We recommend that you use the latest version.
Activate Object Storage Service (OSS) and create an OSS bucket in the same region as the Tablestore instance. For more information, see Get started with OSS.
Create the
AliyunServiceRoleForOTSDataDeliveryservice-linked role and obtain the role ARN. For more information, see Create a delivery task.
Description
Call createDeliveryTask to create a delivery task. A task can deliver only full or incremental data, or deliver full data first and then continuously deliver incremental data.
public CreateDeliveryTaskResponse createDeliveryTask(CreateDeliveryTaskRequest request)
throws TableStoreException, ClientExceptionAfter you create a delivery task, the task must be initialized. You can call describeDeliveryTask to query delivery task information.
The following sample creates a task that delivers full and incremental data and creates daily OSS partitions based on the time when data is written to Tablestore. In the sample, client is an initialized client. Replace the placeholders with actual values and make sure that the pk, event_time, and active columns are of the String, String, and Boolean types, respectively.
String tableName = "<TABLE_NAME>";
String taskName = "<TASK_NAME>";
OSSTaskConfig taskConfig = new OSSTaskConfig();
taskConfig.setOssPrefix("delivery/year=$yyyy/month=$MM/day=$dd");
taskConfig.setOssBucket("<OSS_BUCKET>");
taskConfig.setOssEndpoint("<OSS_ENDPOINT>");
taskConfig.setOssStsRole("<ROLE_ARN>");
taskConfig.addParquetSchema(new ParquetSchema("pk", "pk", DataType.UTF8));
taskConfig.addParquetSchema(
new ParquetSchema("event_time", "event_time", DataType.UTF8));
taskConfig.addParquetSchema(new ParquetSchema("active", "active", DataType.BOOL));
CreateDeliveryTaskRequest request =
new CreateDeliveryTaskRequest(tableName, taskName, taskConfig);
request.setTaskType(DeliveryTaskType.BASE_INC);
client.createDeliveryTask(request);Parameters
Delivery request
request is of the CreateDeliveryTaskRequest type and contains the following parameters.
|
Name |
Type |
Description |
|
tableName (required) |
String |
The name of the data table. |
|
taskName (required) |
String |
The name of the delivery task. The name can contain only lowercase letters, digits, hyphens ( |
|
taskConfig (required) |
OSSTaskConfig |
The OSS delivery configuration. |
|
taskType (required) |
DeliveryTaskType |
The delivery task type. Valid values: |
OSS delivery configuration
request.taskConfig is of the OSSTaskConfig type and contains the following parameters.
|
Name |
Type |
Description |
|
ossPrefix (required) |
String |
The directory prefix in the OSS bucket. You can use the |
|
ossBucket (required) |
String |
The name of the OSS bucket. The bucket must be in the same region as the Tablestore instance. |
|
ossEndpoint (required) |
String |
The endpoint of the region in which the OSS bucket is located. |
|
ossStsRole (required) |
String |
The ARN of the |
|
parquetSchema (required) |
List<ParquetSchema> |
The fields to deliver. You can select fields and customize their names and order in OSS. The order of elements determines the field order in Parquet files. Call |
|
eventTimeColumn (optional) |
EventColumn |
The event time column. If this parameter is specified, the time variables in |
|
format (optional) |
OSSFileFormat |
The OSS file format. The default and only supported value is |
|
timeFormatter (optional) |
TimeFormatter |
A reserved partition format parameter. The current SDK does not include this parameter in requests. Do not configure it. |
Delivered columns
Each element in request.taskConfig.parquetSchema[] is of the ParquetSchema type and contains the following parameters.
|
Name |
Type |
Description |
|
columnName (required) |
String |
The name of the source field in the Tablestore data table. |
|
ossColumnName (required) |
String |
The name of the field after the field is delivered to OSS. |
|
type (required) |
DataType |
The destination type of the field in the Parquet file. The type must match the data type of the source field. Otherwise, the field value is discarded as dirty data. For more information, see Data type mappings. |
|
encode (optional) |
OSSFileEncoding |
The Parquet encoding. Default value: |
|
typeExtend (optional) |
String |
A reserved Parquet extended type parameter. This parameter is not supported. Do not configure it. |
Event time column
request.taskConfig.eventTimeColumn is of the EventColumn type and contains the following parameters.
|
Name |
Type |
Description |
|
columnName (required) |
String |
The name of the source field that is used as the event time. |
|
timeFormat (required) |
EventTimeFormat |
The event time format. Valid values: |
Examples
Partition data by event time
To create OSS partitions based on the event_time column, configure the event time column in taskConfig before you create the request. In the following sample, values in the column must conform to RFC 3339.
EventColumn eventColumn =
new EventColumn("event_time", EventTimeFormat.RFC3339);
taskConfig.setEventTimeColumn(eventColumn);