Object Storage Service:Migrate Hive tables and partitions to the OSS-HDFS service

Procedure

1. Log on to the cluster using Secure Shell (SSH). For more information, see Log on to a cluster.

2. Run the following command to obtain help information.

   sudo jindo table -help moveTo

   The following help information is returned.

   <dbName.tableName>      The table to move.
   <destination path>      The destination base directory which is always at the
                             same level of a 'table location', where the moved
                             partitions or un-partitioned data would located in.
   <condition>/-fullTable  A filter condition to determine which partitions should
                             be moved, supporting common operators (like '>') and
                             built-in UDFs (like to_date) (UDFs not supported
                             yet...), while -fullTable means that all partitions (or
                             a whole un-partitioned table) should be moved. One but
                             only one option must be specified among -c
                             "<condition>" and -fullTable.
   <before days>           Optional, saying that table/partitions should be moved
                             only when they are created (not updated or modified)
                             more than some days before from now.
   <parallelism>           The maximum concurrency when copying partitions, 1 by
                             default.
   <OSS storage policy>    Storage policy for OSS destination, which can be Standard
                             (by default), IA, Archive, or ColdArchive. Not applicable for destinations other
                             than OSS. NOTE: if you are willing to use ColdArchive storage policy, please
                             make sure that Cold Archive has been enabled for your OSS bucket.
   
   -o/-overWrite           Overwriting the final paths where the data would be moved.
                             For partitioned tables this overwrites partitions locations
                             which are subdirectories of <destination path>; for
                             un-partitioned table this overwrites the <destination path>
                             itself.
   -r/-removeSource        Let the source data be removed when the corresponding
                             table/partition is successfully moved to the new destination.
                             Otherwise (by default), the source data would be left as it
                             was.
   -skipTrash              Applicable only when [-r/-removeSource] is enabled. If
                             present, source data would be immediately deleted from the
                             file system, bypassing the trash.
   -e/-explain             If present, the command would not really move data, but only
                             prints the table/partitions that would be moved for given
                             conditions.
   <log directory>         A directory to locate log files, '/tmp/<current user>/' by
                             default.

   • Command format

     sudo jindo table -moveTo \
       -t <dbName.tableName> \
       -d <destination path> \
       [-c "<condition>" | -fullTable] \
       [-b/-before <before days>] \
       [-p/-parallel <parallelism>] \
       [-s/-storagePolicy <OSS storage policy>] \
       [-o/-overWrite] \
       [-r/-removeSource] \
       [-skipTrash] \
       [-e/-explain] \
       [-l/-logDir <log directory>]

   • Description

     Parameter	Required	Description
     -t <dbName.tableName>	Yes	The name of the table to migrate. Use the Database name.Table name format. Separate the database name and table name with a period (.). The table can be a partitioned or non-partitioned table.
     -d <destination path>	Yes	The destination path. This path is at the table level, whether you move a partition or an entire non-partitioned table. If you move a partition, the full path of the partition is the destination path plus the partition name. For example: <destination path>/p1=v1/p2=v2/.
     -c "<condition>" | -fullTable	No	You must specify either -c "<condition>" or -fullTable. If you specify -fullTable, the entire table is moved. The table can be partitioned or non-partitioned. If you specify -c "<condition>", a filter condition is provided to select the partitions to move. Common operators, such as the greater-than sign (>), are supported. For example, for a partition key column `ds` of the String data type, if you want to select partitions where the partition name is greater than 'd', use -c " ds > 'd' ".
     -b/before <before days>	No	Moves only tables or partitions created more than a specified number of days ago.
     -p/-parallel <parallelism>	No	The degree of parallelism for the migration.
     -s/-storagePolicy <OSS storage policy>	No	This option is not supported by the OSS-HDFS service.
     -o/-overWrite	No	Specifies whether to overwrite the destination path. For a partitioned table, only the path of the partition to be moved is cleared, not the entire table path.
     -r/-removeSource	No	Specifies whether to clear the source path after the migration is complete and the metadata is updated. For a partitioned table, only the source path of the successfully moved partition is cleared.
     -skipTrash	No	Specifies whether to bypass the trash when the source path is cleared. Note This option must be used with the -r/-removeSource option.
     -e/-explain	No	If this option is present, the command runs in explain mode. It only displays the list of partitions to be moved and does not move any data.
     -l/-logDir <log directory>	No	Specifies the directory for log files. Default value: /tmp/<current user>/

3. Migrate partition data to the OSS-HDFS service.

   1. Verify the partitions to be migrated.

      Use the -e option to list the partitions for migration without actually moving the data.

      sudo jindotable -moveTo -t tdb.test_table -d oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/data/tdb.test_table -c " dt > 'v' " -e

      The following result is returned:

      Found 1 partitions to move:
            dt=value-2
      MoveTo finished for table tdb.test_table to destination oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/data/tdb.test_table with condition " dt > 'v' " (explain only).

   2. Migrate the partition to the OSS-HDFS service.

      sudo jindotable -moveTo -t tdb.test_table -d oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/data/tdb.test_table  -c " dt > 'v' " 

      The following result is returned:

      Found 1 partitions in total, and all are successfully moved.
      Successfully moved partitions:
          dt=value-2
      No failed partition.
      MoveTo finished for table tdb.test_table to destination oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/data/tdb.test_table with condition " dt > 'v' ".

   3. Verify that the partition was migrated successfully by checking the Location property.

      sudo hive> desc formatted test_table partition (dt='value-2');

      The following result is returned:

      OK
      # col_name              data_type               comment
      id                      int
      content                 string
      
      # Partition Information
      # col_name              data_type               comment
      dt                      string
      
      # Detailed Partition Information
      Partition Value:        [value-2]
      Database:               tdb
      Table:                  test_table
      CreateTime:             UNKNOWN
      LastAccessTime:         UNKNOWN
      Location:               oss://examplebucket.cn-hangzhou.oss-dls.aliyuncs.com/data/tdb.test_table/dt=value-2

   4. Optional: Migrate the partition from OSS-HDFS to HDFS.

      sudo jindotable -moveTo -t tdb.test_table -d hdfs://<hdfs-path>/user/hive/warehouse/tdb.db/test_table  -c " dt > 'v' "

      The following result is returned:

      No successfully moved partition.
      Failed partitions:
          dt=value-2    New location is not empty but -overWrite is not enabled.
      MoveTo finished for table tdb.test_table to destination hdfs://<hdfs-path>/user/hive/warehouse/tdb.db/test_table with condition -c " dt > 'v' ".

      The No successfully moved partition. message is returned because the destination directory in HDFS is not empty. If you can discard the data in the destination directory, you can use the -overWrite option to force an overwrite. This ensures that the partition is migrated from OSS-HDFS to HDFS.

      sudo jindotable -moveTo -t tdb.test_table -d hdfs://<hdfs-path>/user/hive/warehouse/tdb.db/test_table  -c " dt > 'v' " -o

      After the migration is successful, the following result is returned:

      Found 1 partitions in total, and all are successfully moved.
      Successfully moved partitions:
          dt=value-2
      No failed partition.
      MoveTo finished for table tdb.test_table to destination hdfs:///user/hive/warehouse/tdb.db/test_table with condition " dt > 'v' ", overwriting new locations.

Troubleshooting

If the migration fails with a Conflicts found error, perform the following steps to resolve the issue.

• Ensure that no other commands, such as DistCp or JindoDistCp, are migrating data to the same destination path.

• Delete the destination directory. For a non-partitioned table, delete the table-level directory. For a partitioned table, delete the conflicting partition-level directory.

• Do not delete the source directory.