Bulk export offline data - Tablestore SDK for Java - Tablestore

Prerequisites

Install the Tablestore SDK for Java and initialize the client.

Description

Scans contiguous rows from a data table by primary key range. The response includes rows whose primary keys are greater than or equal to the start key and less than the end key. Use BulkExportQueryCriteria (in the com.alicloud.openservices.tablestore.model.tunnel subpackage) to set the start and end primary keys, the columns to return, the filter, and the data block encoding type. The response returns row data as ByteBuffer-encoded bytes that you decode into a row list with the matching parser: DBT_PLAIN_BUFFER pairs with PlainBufferBlockParser, and DBT_SIMPLE_ROW_MATRIX pairs with SimpleRowMatrixBlockParser. If getNextStartPrimaryKey() returns a non-null value, more rows remain; continue the scan from that key.

public BulkExportResponse bulkExport(BulkExportRequest bulkExportRequest) throws TableStoreException, ClientException

The following example reads from the bulk_export_demo table over the primary key range [row00, row99) in a single call. It uses DBT_PLAIN_BUFFER encoding and decodes the result with the matching parser.

String tableName = "bulk_export_demo";

// Start primary key (inclusive)
PrimaryKey startPk = PrimaryKeyBuilder.createPrimaryKeyBuilder()
        .addPrimaryKeyColumn("pk", PrimaryKeyValue.fromString("row00"))
        .build();
// End primary key (exclusive)
PrimaryKey endPk = PrimaryKeyBuilder.createPrimaryKeyBuilder()
        .addPrimaryKeyColumn("pk", PrimaryKeyValue.fromString("row99"))
        .build();

// BulkExportQueryCriteria is located in the model.tunnel subpackage
BulkExportQueryCriteria criteria = new BulkExportQueryCriteria(tableName);
criteria.setInclusiveStartPrimaryKey(startPk);
criteria.setExclusiveEndPrimaryKey(endPk);
criteria.setDataBlockType(DataBlockType.DBT_PLAIN_BUFFER);
criteria.addColumnsToGet("pk");
criteria.addColumnsToGet("col1");

BulkExportRequest request = new BulkExportRequest(criteria);
BulkExportResponse response = client.bulkExport(request);

// response.getRows() returns a ByteBuffer; parse it into List<Row> using the corresponding parser
// DBT_PLAIN_BUFFER       -> PlainBufferBlockParser
// DBT_SIMPLE_ROW_MATRIX  -> SimpleRowMatrixBlockParser
PlainBufferBlockParser parser = new PlainBufferBlockParser(response.getRows());
List<Row> rows = parser.getRows();
System.out.println("Rows returned: " + rows.size());
for (Row row : rows) {
    System.out.println(row);
}

// A non-null nextStartPrimaryKey indicates that more data remains and pagination must continue
System.out.println("Has next: " + (response.getNextStartPrimaryKey() != null));

Parameters

Request configuration

BulkExportRequest contains the following parameter.

Name	Type	Description
bulkExportQueryCriteria (required)	BulkExportQueryCriteria	The range read criteria.

Range read criteria

BulkExportRequest.bulkExportQueryCriteria is of the BulkExportQueryCriteria type and contains the following parameters.

Name	Type	Description
tableName (required)	String	The name of the data table.
inclusiveStartPrimaryKey (required)	PrimaryKey	The start primary key of the scan range. If a row with this key exists, it is included in the response. Accepts a valid primary key or a virtual point built from `INF_MIN` and `INF_MAX`. A virtual point must have the same number of columns as the table primary key. `INF_MIN` denotes negative infinity and compares smaller than any value of any column type; `INF_MAX` denotes positive infinity and compares larger than any value of any column type.
exclusiveEndPrimaryKey (required)	PrimaryKey	The end primary key of the scan range. The row with this key is never included in the response, whether it exists or not. Accepts a valid primary key or a virtual point built from `INF_MIN` and `INF_MAX`. A virtual point must have the same number of columns as the table primary key.
columnsToGet (optional)	Set<String>	The set of column names to return. If not set, all columns of each row are returned. When set, the response includes only the existing columns of each row. If a row contains none of the specified columns, the response excludes that row.
filter (optional)	Filter	A server-side filter applied after column selection. If you specify both `columnsToGet` and `filter`, the server first retrieves the columns named in `columnsToGet`, then applies `filter` to the result. For information about how to configure the filter, see Use filters.
dataBlockType (optional)	DataBlockType	The data block encoding of the response. Valid values: `DBT_PLAIN_BUFFER` or `DBT_SIMPLE_ROW_MATRIX` (API default: `DBT_SIMPLE_ROW_MATRIX`). The encoding must match the parser used to decode the result.

Response

Read result

BulkExportResponse contains the following operation-specific fields.

Name	Type	Description
rows	ByteBuffer	The encoded row data. Use the parser that matches `dataBlockType` to decode the rows.
nextStartPrimaryKey	PrimaryKey	The start primary key for the next read. A `null` value indicates that all data in the specified range has been returned.
dataBlockType	DataBlockType	The encoding of the returned row data.
bodyBytes	long	The number of bytes in the response body.

Examples

Paginate through an entire table

A single bulkExport call often returns only part of the range. To scan the entire table, pass the nextStartPrimaryKey from each response as the start primary key of the next call. Stop when nextStartPrimaryKey is null.

String tableName = "bulk_export_demo";

PrimaryKey startPk = PrimaryKeyBuilder.createPrimaryKeyBuilder()
        .addPrimaryKeyColumn("pk", PrimaryKeyValue.fromString("row00"))
        .build();
PrimaryKey endPk = PrimaryKeyBuilder.createPrimaryKeyBuilder()
        .addPrimaryKeyColumn("pk", PrimaryKeyValue.fromString("row99"))
        .build();

int totalRows = 0;
while (startPk != null) {
    BulkExportQueryCriteria criteria = new BulkExportQueryCriteria(tableName);
    criteria.setInclusiveStartPrimaryKey(startPk);
    criteria.setExclusiveEndPrimaryKey(endPk);
    criteria.setDataBlockType(DataBlockType.DBT_PLAIN_BUFFER);

    BulkExportResponse response = client.bulkExport(new BulkExportRequest(criteria));

    PlainBufferBlockParser parser = new PlainBufferBlockParser(response.getRows());
    List<Row> rows = parser.getRows();
    totalRows += rows.size();

    // A null nextStartPrimaryKey indicates that all data has been read
    startPk = response.getNextStartPrimaryKey();
}
System.out.println("Total rows scanned: " + totalRows);