Holo Shipper is a migration tool in Hologres. It allows you to migrate Hologres tables to realize offline backup of the Hologres ecosystem. You can use Holo Shipper to migrate tables between Hologres instances or to dump and restore tables.

Limits

  • Holo Shipper is supported only by Hologres V1.1 and later. If the version of your Hologres instance is earlier than V1.1, submit a ticket or join the Hologres DingTalk group for technical support.
  • Holo Shipper can be used only by superusers of Hologres instances. If you are not a superuser, the system prompts that you do not have the required permissions.
  • When you use Holo Shipper, make sure that you can access the source and destination tables and you have set up a Java environment for Hologres.
  • When you use Holo Shipper, make sure that no data is being written to source tables.
Note Holo Shipper is available on GitHub. For more information, see Holo Shipper.

Syntax and parameters

  • Specify the source from which tables are migrated.

    The source can be a Hologres instance, an Object Storage Service (OSS) bucket, or the local storage. The following syntax is for your reference:

    • Hologres instance
      -s holo -h endpoint -p port_number -u username -w password
      Parameter Description
      endpoint The endpoint of the Hologres instance. You can log on to the Hologres console and obtain the endpoint on the instance details page.
      port_number The port number of the Hologres instance. You can log on to the Hologres console and obtain the port number on the instance details page.
      username The account used to log on to the Hologres instance.
      password The password of the account used to log on to the Hologres instance.
    • OSS bucket
      -s oss -h endpoint -u accessKeyId -w accessKeySecret -b bucketName -p folder_path
      Parameter Description
      endpoint The OSS endpoint of the region in which the OSS bucket resides. For more information about how to obtain the OSS endpoint, see Obtain the internal endpoint of OSS.
      accessKeyId The AccessKey ID of the account used to access the OSS bucket. You can obtain the AccessKey ID on the Security Management page.
      accessKeySecret The AccessKey secret of the account used to access the OSS bucket. You can obtain the AccessKey secret on the Security Management page.
      bucketName The name of the OSS bucket.
      folder_path The OSS path from which the tables are migrated.
    • Local storage
      -s local_path
      Parameter Description
      local_path The local path from which the tables are migrated.
  • Specify the destination to which tables are migrated.

    The destination can be a Hologres instance, an OSS bucket, or the local storage. The following syntax is for your reference:

    • Hologres instance
      -d holo -h endpoint -p port_number -u username -w password
      Parameter Description
      endpoint The endpoint of the Hologres instance. You can log on to the Hologres console and obtain the endpoint on the instance details page.
      port_number The port number of the Hologres instance. You can log on to the Hologres console and obtain the port number on the instance details page.
      username The account used to log on to the Hologres instance.
      password The password of the account used to log on to the Hologres instance.
    • OSS bucket
      -d oss -h endpoint -u accessKeyId -w accessKeySecret -b bucketName -p folder_path
      Parameter Description
      endpoint The OSS endpoint of the region in which the OSS bucket resides. For more information about how to obtain the OSS endpoint, see Obtain the internal endpoint of OSS.
      accessKeyId The AccessKey ID of the account used to access the OSS bucket. You can obtain the AccessKey ID on the Security Management page.
      accessKeySecret The AccessKey secret of the account used to access the OSS bucket. You can obtain the AccessKey secret on the Security Management page.
      bucketName The name of the OSS bucket.
      folder_path The OSS path to which the tables are migrated.
    • Local storage
      -d local_path
      Parameter Description
      local_path The local path to which the tables are migrated.
  • Specify the path of the JSON file that stores the information about the tables and the source database of the tables.
    • Syntax
      -l ship_list_json_path --no-owner --no-all-roles --no-guc --no-ext --no-priv --no-data --incl-foreign
      Parameter or option Description
      ship_list_json_path The path of the JSON file that stores the information about the tables and the source database of the tables.
      --no-owner Optional. Specifies not to synchronize the table owner settings of the source database. If you do not set this option, Holo Shipper synchronizes the table owner settings by default. If you set this option and enable neither the simple permission model (SPM) nor the schema-level permission model (SLPM), the owner of the tables is the account that you use to migrate the tables.
      --no-all-roles Optional. Specifies to migrate only the table owner and users who have permissions on the tables.
      --no-guc Optional. Specifies not to synchronize the values of the Grand Unified Configuration (GUC) parameters. If you do not set this option, the values of the GUC parameters are synchronized by default.
      --no-ext Optional. Specifies not to synchronize the extension of the source database. If you do not set this option, the extension is synchronized by default.
      --no-priv Optional. Specifies not to synchronize permissions on the tables. If you do not set this option, the permissions are synchronized by default.
      --no-data Optional. Specifies not to synchronize data in the tables but only create a database and table schemas. If you do not set this option, all data in the tables is synchronized.
      --incl-foreign Optional. Specifies to synchronize the DDL statements of foreign tables that meet the condition specified by the shipList parameter. If you do not set this option, the DDL statements of foreign tables are not synchronized by default.
    • Format of the JSON file
      The JSON file contains an array of JSON objects. Each JSON object represents a database. The following table describes the parameters contained in each JSON object.
      Parameter Description
      dbName The name of the database.
      shipList The tables to be migrated. The value of this parameter is a JSON object that contains key-value pairs. The keys represent the names of the schemas, whereas the values represent the names of the tables to be migrated.
      blackList Optional. The tables not to be migrated. The value of this parameter is a JSON object that contains key-value pairs. The keys represent the names of the schemas, whereas the values represent the names of the tables not to be migrated.
      sinkDB Optional. The name of the destination database. If you do not set this parameter, the system uses the value of the dbName parameter as the default value of this parameter.
      Note If the source and destination databases reside in the same Hologres instance, the tables are migrated from the database specified by the dbName parameter to that specified by the sinkDB parameter.
      schemaMapping Optional. The mappings between source schemas and destination schemas. The value of this parameter is a JSON object that contains key-value pairs. The keys represent the names of the source schemas, whereas the values represent the names of the destination tables. Set this parameter only if you want to change the schemas.
    • Sample code of the JSON file

      In this example, tables in the Database DB1, DB2, and DB3 are migrated. To be specific, tables other than Table table3 are migrated from Schema schema1 of Database DB1. Table table1 and Table table2 are migrated from Schema schema2 of Database DB1. All tables in Schema schema3 of Database DB2 are migrated. Table table4 and Table table5 are migrated from Schema schema4 of Database DB2. The name of Schema schema4 is changed to schema5. All tables in the public schemas of Database DB3 are migrated to the public schemas of Database DB3_backup.

      [
       {
           "dbName": "DB1",
           "shipList": {
               "schema1": ["*"],
               "schema2": ["table1", "table2"]
           },
           "blackList": {
               "schema1": ["table3"]
           }
       },
       {
           "dbName": "DB2",
           "shipList": {
               "schema3": ["*"],
               "schema4": ["table4", "table5"]
           },
           "schemaMapping": {
               "schema4": "schema5"
           }
       },
       {
           "dbName": "DB3",
           "shipList": {
               "public": ["*"]
           },
           "sinkDB" : "DB3_backup"
       }
      ]

Examples

The following commands provide examples of using Holo Shipper to migrate tables in different scenarios.

  • Export a table from a Hologres instance to another Hologres instance and do not synchronize the information about the table owner.
    $ java -jar holo-shipper.jar -s holo -h xx.xx.xx.xx -p xxxx -u username -w password -d holo -h xx.xx.xx.xx -p xxxx -u username2 -w password2  -l ship_list_json_path --no-owner
  • Export some tables from a Hologres instance to another Hologres instance, and synchronize the information about table owners and permissions on the tables. Do not synchronize the information about users that do not have permissions on the tables. Do not synchronize the information about GUC parameters and the extension of the source database.
    $ java -jar holo-shipper.jar -s holo -h xx.xx.xx.xx -p xxxx -u username -w password -d holo -h xx.xx.xx.xx -p xxxx -u username2 -w password2  -l ship_list_json_path --no-all-roles --no-guc --no-ext
  • Dump a table in a Hologres instance to the local storage and do not synchronize the information about GUC parameters of the source database.
    $ java -jar holo-shipper.jar -s holo -h xx.xx.xx.xx -p xxxx -u username -w password -d local_storage_path -l ship_list_json_path --no-guc
  • Restore a backup table from the local storage to another Hologres instance.
    $ java -jar holo-shipper.jar -s local_storage_path -d holo -h xx.xx.xx.xx -p xxxx -u username -w password -l ship_list_json_path
  • Back up a table in a Hologres instance to an OSS bucket and synchronize all information. The name of the OSS bucket is testBucket, and the OSS path is testDump/.
    $ java -jar holo-shipper.jar -s holo -h xx.xx.xx.xx -p xxxx -u username -w password -d oss -h endpoint -u accessKeyId -w accessKeySecret -b testBucket -p testDump/ -l ship_list_json_path
  • Restore a backup table from an OSS bucket to another Hologres instance.
    $ java -jar holo-shipper.jar -s oss -h endpoint -u accessKeyId -w accessKeySecret -b testBucket -p testDump/ -d holo -h xx.xx.xx.xx -p xxxx -u username -w password -l ship_list_json_path

JAR

To obtain the JAR file of Holo Shipper, click Holo Shipper.