OpenLake:Migrate a Paimon FileSystem catalog to DLF

Step 1: Create a JAR job

1. Log on to the Realtime Compute for Apache Flink console.

2. In the list of fully managed Flink workspaces, click the name of your workspace.

3. In the left navigation pane, choose Operation Center > Deployments.

4. Click Deploy Job, select JAR Job, and set the following parameters.

   For a full description of deployment parameters, see Deploy a JAR job.

   Parameter	Description	Example
   Deployment Mode	Fixed to Batch Mode.	Batch Mode
   Deployment Name	A name for the JAR job.	migrate_paimon
   Engine Version	The Flink engine version.	vvr-8.0.11-flink-1.17
   JAR URI	Upload the paimon-flink-action-1.3-SNAPSHOT-for-clone-20250909.jar package. If you have uploaded it before, select it from the drop-down list.	—
   Entry Point Class	The entry point class of the JAR.	Leave blank
   Entry Point Main Arguments	Parameters passed to the main method. Set in step 2.	Leave blank for now
   Additional Dependencies	Upload the paimon-ali-vvr-8.0-vvp-1.3-ali-SNAPSHOT-for-clone-20250909.jar package. If you have uploaded it before, select it from the drop-down list.	—

5. Click Deploy.

Step 2: Configure parameters and start the job

The Entry Point Main Arguments field controls which source catalog to read from, which target catalog to write to, and which scope to migrate.

1. On the Job O&M page, find the JAR job that you created and click Details.

2. On the Deployment Details page, click Edit in the upper-right corner.

3. Set the Entry Point Main Arguments field using the command template below, replacing placeholders with your values.

   Command template

   clone
   --parallelism '<parallelism>'
   --database '<database-name>'
   --table '<table-name>'
   --catalog_conf 'metastore=filesystem'
   --catalog_conf "warehouse=<warehouse>"
   --catalog_conf 'fs.oss.endpoint=<fs.oss.endpoint>'
   --catalog_conf 'fs.oss.accessKeyId=<fs.oss.accessKeyId>'
   --catalog_conf 'fs.oss.accessKeySecret=<fs.oss.accessKeySecret>'
   --target_database '<target-database-name>'
   --target_table '<target-table-name>'
   --target_catalog_conf 'metastore=rest'
   --target_catalog_conf 'warehouse=<target-warehouse>'
   --target_catalog_conf 'uri=<dlf.next.endpoint>'
   --target_catalog_conf 'token.provider=dlf'
   --target_catalog_conf 'dlf.access-key-id=<dlf.access-key-id>'
   --target_catalog_conf 'dlf.access-key-secret=<dlf.access-key-secret>'
   --clone_from 'paimon'
   --where '<filter-spec>'

   Parameter reference

   Configuration item	Required	Description	Example
   parallelism	No	Concurrency of the job.	16
   database-name	No	Name of the FileSystem catalog database to clone. Omit when migrating an entire catalog.	my_database
   table-name	No	Name of the FileSystem catalog table to clone. Omit when migrating an entire database or catalog.	my_table
   warehouse	Yes	OSS path of the FileSystem catalog. Format: oss://<bucket>/<object>. Find your bucket and object names in the OSS console.	oss://my-bucket/paimon-data
   fs.oss.endpoint	Yes	Endpoint of the OSS service. For OSS: oss-cn-hangzhou-internal.aliyuncs.com. For OSS-HDFS: cn-hangzhou.oss-dls.aliyuncs.com. See Regions and endpoints.	oss-cn-hangzhou-internal.aliyuncs.com
   fs.oss.accessKeyId	Yes	AccessKey ID of the account or RAM user with read and write permissions on OSS. See Create an AccessKey.	—
   fs.oss.accessKeySecret	Yes	AccessKey secret corresponding to fs.oss.accessKeyId.
   target-database-name	No	Name of the target DLF database. Omit when migrating an entire catalog.	target_database
   target-table-name	No	Name of the target DLF table. Omit when migrating an entire database or catalog.	target_table
   target-warehouse	Yes	Name of the DLF data catalog to migrate into. View catalog names in the DLF console. See Data catalogs.	—
   dlf.next.endpoint	Yes	Endpoint of the DLF service. See Endpoints.	cn-hangzhou-vpc.dlf.aliyuncs.com
   dlf.access-key-id	Yes	AccessKey ID to access DLF.	—
   dlf.access-key-secret	Yes	AccessKey secret to access DLF.
   clone_from	Yes	Type of source catalog. Always set to 'paimon' for this migration.	paimon
   filter-spec	No	Partition filter condition. Omit to migrate all partitions.	dt = '2024-10-01'
   excluded_tables	No	Tables to exclude from catalog or database migrations. Not applicable for single-table migrations. Format: <db>.<table>,<db2>.<table2>.	my_db.my_tbl,my_db2.my_tbl2

   Important

   • To migrate a single table, include --database, --table, --target_database, and --target_table. All four parameters are required.

   • To migrate an entire database, include --database and --target_database. Do not set --table or --target_table. Optionally use --excluded_tables to skip specific tables.

   • To migrate an entire catalog, omit --database, --table, --target_database, and --target_table. Optionally use --excluded_tables to skip specific tables.

4. Click Save on the Deployment Details page.

5. On the Job O&M page, click Start next to the JAR job and start the job with the default parameters.

Step 3: Verify the result

After the job finishes, confirm that the migration completed successfully.

1. On the Job O&M page, wait until the job status changes to Finished.

2. Log on to the DLF console and navigate to your target data catalog.

3. Verify the migrated data based on your migration scope:

   Migration scope	What to verify
   Full catalog migration	The catalog structure, databases, and tables in DLF match those in the source FileSystem catalog.
   Full database migration	The database and table structures in DLF match those in the source FileSystem catalog.
   Single table migration	The table structure in DLF matches that in the source FileSystem catalog.