E-MapReduce:The JindoTable MoveTo command

Limits

The MoveTo command is supported on clusters that run EMR V3.36.0 or later, or EMR V5.2.0 or later.

Use the MoveTo command

1. Log on to the cluster using Secure Shell (SSH). For more information, see Log on to a cluster.
2. Run the following command to view the help information.

   jindo table -help moveTo

   The help information is similar to the following:

   <dbName.tableName>      The table to move.
   <destination path>      The destination base directory which is always at the
                             same level of a 'table location', where the moved
                             partitions or un-partitioned data would located in.
   <condition>/-fullTable  A filter condition to determine which partitions should
                             be moved, supporting common operators (like '>') and
                             built-in UDFs (like to_date) (UDFs not supported
                             yet...), while -fullTable means that all partitions (or
                             a whole un-partitioned table) should be moved. One but
                             only one option must be specified among -c
                             "<condition>" and -fullTable.
   <before days>           Optional, saying that table/partitions should be moved
                             only when they are created (not updated or modified)
                             more than some days before from now.
   <parallelism>           The maximum concurrency when copying partitions, 1 by
                             default.
           <OSS storage policy>: Storage policy for OSS destination, which can be Standard
     (by default), IA, Archive, or ColdArchive. Not applicable for destinations other
     than OSS. NOTE: if you are willing to use ColdArchive storage policy, please
     make sure that Cold Archive has been enabled for your OSS bucket.
   
   -o/-overWrite     Overwriting the final paths where the data would be moved.
                       For partitioned tables this overwrites partitions' locations
                       which are subdirectories of <destination path>; for
                       un-partitioned table this overwrites the <destination path>
                       itself.
   -r/-removeSource  Let the source data be removed when the corresponding
                       table/partition is successfully moved to the new destination.
                       Otherwise (by default), the source data would be left as it
                       was.
   -skipTrash        Applicable only when [-r/-removeSource] is enabled. If
                       present, source data would be immediately deleted from the
                       file system, bypassing the trash.
   -e/-explain       If present, the command would not really move data, but only
                       prints the table/partitions that would be moved for given
                       conditions.
   <log directory>   A directory to locate log files, '/tmp/<current user>/' by
                       default.

   The syntax of the MoveTo command is as follows:

   jindo table -moveTo \
     -t <dbName.tableName> \
     -d <destination path> \
     [-c "<condition>" | -fullTable] \
     [-b/-before <before days>] \
     [-p/-parallel <parallelism>] \
     [-s/-storagePolicy <OSS storage policy>] \
     [-o/-overWrite] \
     [-r/-removeSource] \
     [-skipTrash] \
     [-e/-explain] \
     [-l/-logDir <log directory>]

   Parameter	Description	Required
   -t <dbName.tableName>	The name of the table to move. Use the format database_name.table_name. The database name and table name are separated by a period (.). The table can be partitioned or non-partitioned.	Yes
   -d <destination path>	The destination path. Whether you move partitions or an entire non-partitioned table, this path corresponds to the table-level location. If you move a partition, the full path of the partition is the destination path plus the partition name, such as <destination path>/p1=v1/p2=v2/.	Yes
   -c "<condition>" | -fullTable	You must specify either -c "<condition>" or -fullTable. If you specify -fullTable, the entire table is moved. The table can be partitioned or non-partitioned. If you specify -c "<condition>", a filter condition is provided to select the partitions to move. Common operators, such as the greater-than sign (>), are supported. For example, for a partition key column `ds` of the String data type, if you want to select partitions where the partition name is greater than 'd', use -c " ds > 'd' ".	No
   -b/before <before days>	Moves only tables or partitions that were created more than a specified number of days ago.	No
   -p/-parallel <parallelism>	The degree of parallelism for the migration operation.	No
   -s/-storagePolicy <OSS storage policy>	The storage policy for data that is copied to OSS. The following policies are available: Standard: (Archive Storage). IA: Infrequent Access (IA) storage. Storage class: Standard ColdArchive: Cold Archive storage. Note Make sure that this feature is enabled for your OSS bucket before you use this policy.	No
   -o/-overWrite	Specifies whether to overwrite the destination path. For a partitioned table, this option clears only the paths of the partitions to be moved, not the entire table path.	No
   -r/-removeSource	Specifies whether to remove the source data after the migration is complete and the metadata is updated. For a partitioned table, this option removes only the source paths of the partitions that were successfully moved.	No
   -skipTrash	Specifies whether to bypass the Trash when the source path is removed. Note This parameter is applicable only when -r/-removeSource is specified.	No
   -e/-explain	If this option is specified, the command runs in explain mode. It lists the partitions that would be moved but does not actually move any data.	No
   -l/-logDir <log directory>	Specifies the directory for log files.	No

Configure the lock directory

1. Go to the HDFS service page.
   1. Log on to the Alibaba Cloud E-MapReduce console.
      
   2. In the top menu bar, select the region and resource group where your cluster resides.
      
   3. Click the Clusters tab.
      
   4. On the Clusters page, click Details in the row of the desired cluster.
      
   5. In the navigation pane on the left, choose Services > HDFS.
2. Modify the configuration.
   1. On the HDFS service page, click the Configure tab. Then, click the hdfs-site or core-site tab.
   2. In the upper-right corner, click Custom Configuration.
   3. In the Add Configuration Item dialog box, add the jindotable.moveto.tablelock.base.dir configuration item and set its value to an existing HDFS path.
      Important Before you configure a custom lock directory, make sure that no MoveTo process is running on any node in the cluster. Otherwise, the MoveTo operation may fail or cause data corruption.
3. Save the configuration.
   1. In the upper-right corner, click Save.
   2. In the Confirm dialog box, enter a reason for the change and click OK.