JindoTable collects frequent-access statistics on tables and partitions, provides tiered storage management, and optimizes table data organization on OSS and JindoFS.
Prerequisites
Before you begin, ensure that you have:
-
Java Development Kit (JDK) 8 installed on your on-premises machine
-
An E-MapReduce (EMR) cluster of V3.30.0 or later. See Create a cluster
Commands
Use the following format conventions for all commands:
-
Tables:
database.table -
Partitions:
partitionCol1=1,partitionCol2=2,...
The following table lists all available jindo table commands.
| Command | Description |
|---|---|
-accessStat |
Query the most frequently accessed tables or partitions within a time window |
-cache |
Cache table or partition data to local disks |
-uncache |
Remove cached table or partition data from local disks |
-archive |
Move table or partition data to a lower-cost OSS storage class |
-unarchive |
Restore archived table or partition data to Standard storage |
-status |
View the storage status of a table or partition |
-optimize |
Optimize data organization at the storage layer |
-showTable |
List all partitions in a partitioned table, or show storage details of a non-partitioned table |
-showPartition |
Show storage details of specific partitions |
-listTables |
List all tables in a database |
-dumpmc |
Dump a MaxCompute table to an EMR cluster or OSS |
-leastUseStat |
Query the least recently accessed tables or partitions |
-archiveTable |
Archive table or partition data in SDK mode (no Jindo Namespace Service dependency) |
-unarchiveTable |
Unarchive table or partition data in SDK mode |
-moveTo |
Migrate table or partition data to a different location |
-accessStat
Query the tables or partitions with the highest access frequency within a specified number of days, along with their visit counts.
Syntax
jindo table -accessStat {-d} <days> {-n} <topNums>
Parameters
| Parameter | Description | Required |
|---|---|---|
-d <days> |
Time window in days. Must be a positive integer. | Yes |
-n <topNums> |
Number of top results to return. Must be a positive integer. If omitted, all results are returned. | No |
If-dis set to1and-nis omitted, the command returns access statistics from 00:00 to the current time on the current day.
Example
Query the 20 most frequently accessed tables or partitions over the last 7 days:
jindo table -accessStat -d 7 -n 20
-cache
Cache the data of a specified table or partition to local disks.
Syntax
jindo table -cache {-t} <dbName.tableName> [-p] <partitionSpec> [-pin]
Parameters
| Parameter | Description | Required |
|---|---|---|
-t <dbName.tableName> |
The table to cache. Format: database.table. |
Yes |
-p <partitionSpec> |
The partition to cache. Format: col1=val1,col2=val2,.... If omitted, all partitions are cached. |
No |
-pin |
Prevents cached data from being evicted when cache space runs low. | No |
The table or partition data must reside in Object Storage Service (OSS) or JindoFS.
Example
Cache the date=2020-03-16 partition of db1.t1:
jindo table -cache -t db1.t1 -p date=2020-03-16
-uncache
Remove the cached data of a specified table or partition from local disks.
Syntax
jindo table -uncache {-t} <dbName.tableName> [-p] <partitionSpec>
Parameters
| Parameter | Description | Required |
|---|---|---|
-t <dbName.tableName> |
The table to uncache. Format: database.table. |
Yes |
-p <partitionSpec> |
The partition to uncache. Format: col1=val1,col2=val2,.... If omitted, all cached data for the table is removed. |
No |
The table or partition data must reside in OSS or JindoFS.
Examples
Remove all cached data for db1.t2:
jindo table -uncache -t db1.t2
Remove cached data for a specific partition of db1.t1:
jindo table -uncache -t db1.t1 -p date=2020-03-16,category=1
-archive
Move table or partition data to a lower-cost OSS storage class. The default target class is Archive.
Syntax
jindo table -archive {-a|-i} {-t} <dbName.tableName> [-p] <partitionSpec>
Parameters
| Parameter | Description | Required |
|---|---|---|
-a |
Archive to the Archive storage class (default). | No |
-i |
Archive to the Infrequent Access (IA) storage class instead of Archive. | No |
-t <dbName.tableName> |
The table to archive. Format: database.table. |
Yes |
-p <partitionSpec> |
The partition to archive. Format: col1=val1,col2=val2,.... If omitted, all partitions are archived. |
No |
Example
Archive the date=2020-10-12 partition of db1.t1 to the Archive storage class:
jindo table -archive -t db1.t1 -p date=2020-10-12
-unarchive
Restore archived table or partition data to Standard storage, or change it to Infrequent Access (IA) storage.
Syntax
jindo table -unarchive [-o|-i] {-t} <dbName.tableName> [-p] <partitionSpec>
Parameters
| Parameter | Description | Required |
|---|---|---|
-o |
Temporarily restore an archived object to Standard storage. | No |
-i |
Change an archived object to IA storage instead of Standard. | No |
-t <dbName.tableName> |
The table to unarchive. Format: database.table. |
Yes |
-p <partitionSpec> |
The partition to unarchive. Format: col1=val1,col2=val2,.... If omitted, all archived data for the table is restored. |
No |
Examples
Temporarily restore a specific partition of db1.t1 to Standard storage:
jindo table -unarchive -o -t db1.t1 -p date=2020-03-16,category=1
Change all archived data in db1.t2 to IA storage:
jindo table -unarchive -i -t db1.t2
-status
View the storage status of a specified table or partition.
Syntax
jindo table -status {-t} <dbName.tableName> [-p] <partitionSpec>
Parameters
| Parameter | Description | Required |
|---|---|---|
-t <dbName.tableName> |
The table to inspect. Format: database.table. |
Yes |
-p <partitionSpec> |
The partition to inspect. Format: col1=val1,col2=val2,.... If omitted, the status of the entire table is shown. |
No |
Examples
View the storage status of db1.t2:
jindo table -status -t db1.t2
View the storage status of the date=2020-03-16 partition of db1.t1:
jindo table -status -t db1.t1 -p date=2020-03-16
-optimize
Optimize the data organization of a table at the storage layer.
Syntax
jindo table -optimize {-t} <dbName.tableName>
Parameters
| Parameter | Description | Required |
|---|---|---|
-t <dbName.tableName> |
The table to optimize. Format: database.table. |
Yes |
Example
Optimize db1.t1:
jindo table -optimize -t db1.t1
-showTable
List all partitions in a partitioned table, or show the storage details of a non-partitioned table.
Syntax
jindo table -showTable {-t} <dbName.tableName>
Parameters
| Parameter | Description | Required |
|---|---|---|
-t <dbName.tableName> |
The table to inspect. Format: database.table. |
Yes |
Example
List all partitions in the db1.t1 partitioned table:
jindo table -showTable -t db1.t1
-showPartition
Show the storage details of a specific partition.
Syntax
jindo table -showPartition {-t} <dbName.tableName> [-p] <partitionSpec>
Parameters
| Parameter | Description | Required |
|---|---|---|
-t <dbName.tableName> |
The table to inspect. Format: database.table. |
Yes |
-p <partitionSpec> |
The partition to inspect. Format: col1=val1,col2=val2,.... |
No |
Example
Show storage details for the date=2020-10-12 partition of db1.t1:
jindo table -showPartition -t db1.t1 -p date=2020-10-12
-listTables
List all tables in a database.
Syntax
jindo table -listTables [-db] <dbName>
Parameters
| Parameter | Description | Required |
|---|---|---|
-db <dbName> |
The database name. If omitted, tables in the default database are listed. | No |
Examples
List tables in the default database:
jindo table -listTables
List tables in db1:
jindo table -listTables -db db1
-dumpmc
Dump a MaxCompute table to an EMR cluster or OSS. Supported output formats are CSV and TFRecord.
Syntax
jindo table -dumpmc {-i} <accessId> {-k} <accessKey> {-m} <numMaps> {-t} <tunnelUrl> {-project} <projectName> {-table} <tableName> [-p] <partitionSpec> {-f} <csv|tfrecord> {-o} <outputPath>
Parameters
| Parameter | Description | Required |
|---|---|---|
-i <accessId> |
The AccessKey ID of your Alibaba Cloud account. | Yes |
-k <accessKey> |
The AccessKey secret of your Alibaba Cloud account. | Yes |
-m <numMaps> |
The number of map tasks. | Yes |
-t <tunnelUrl> |
The Tunnel endpoint of the virtual private cloud (VPC) where the MaxCompute project resides. | Yes |
-project <projectName> |
The name of the MaxCompute project. | Yes |
-table <tableName> |
The name of the MaxCompute table. | Yes |
-p <partitionSpec> |
Partition information. Example: pt=xxx. For multiple partitions: pt=xxx,dt=xxx. |
No |
-f <csv|tfrecord> |
Output file format. Valid values: csv, tfrecord. |
Yes |
-o <outputPath> |
The destination path. Use a local path for EMR cluster output, or an oss:// path for OSS output. |
Yes |
Examples
Dump a MaxCompute table in TFRecord format to an EMR cluster:
jindo table -dumpmc -m 10 -project mctest_project -table t1 -t http://dt.xxx.maxcompute.aliyun-inc.com -k xxxxxxxxx -i XXXXXX -o /tmp/outputtf1 -f tfrecord
Dump a MaxCompute table in CSV format to OSS:
jindo table -dumpmc -m 10 -project mctest_project -table t1 -t http://dt.xxx.maxcompute.aliyun-inc.com -k xxxxxxxxx -i XXXXXX -o oss://bucket1/tmp/outputcsv -f csv
-leastUseStat
Query the tables or partitions that have not been accessed for the longest time.
Syntax
jindo table -leastUseStat -n <num> [-i|-ignoreNever]
Parameters
| Parameter | Description | Required |
|---|---|---|
-n <num> |
Number of results to return. Must be a positive integer. | Yes |
-i / -ignoreNever |
Exclude tables or partitions that have never been accessed. | No |
Example
Query the 20 tables or partitions with the longest period of inactivity:
jindo table -leastUseStat -n 20
What's next
-
Use
-archiveTableand-unarchiveTablein SDK mode to archive and unarchive OSS data without the Jindo Namespace Service component. See archiveTable and unarchiveTable. -
Use
-moveToto migrate table or partition data to a new location. See moveTo.