Migrate from DLF-Legacy (Old Version) to DLF - Data Lake Formation

Migrate your data catalogs from DLF-Legacy to DLF by running a Paimon clone action as a JAR job in Realtime Compute for Apache Flink. The job copies Hive tables (Parquet, ORC, or Avro format), Iceberg tables, Hudi tables, or Paimon tables from DLF-Legacy into DLF as Paimon append-only tables.

Prerequisites

Before you begin, make sure you have:

A fully managed Flink workspace. See Activate Realtime Compute for Apache Flink
An OSS bucket. See Create a bucket
A DLF-Legacy data catalog. See Metadata management
A DLF data catalog. See Create a data catalog

Step 1: Create a JAR job

Log on to the Realtime Compute for Apache Flink console.
In the list of fully managed Flink workspaces, click the name of your workspace.
In the left navigation pane, choose Operation Center > Deployments.

Click Deploy Job, select JAR Job, and set the following parameters.

For a full description of deployment parameters, see Deploy a JAR job.

Parameter	Description	Example
Deployment Mode	Fixed to Batch Mode.	Batch Mode
Deployment Name	A name for the JAR job.	migrate_paimon
Engine Version	The Flink engine version.	vvr-8.0.11-flink-1.17
JAR URI	Upload the paimon-flink-action-1.3-SNAPSHOT-for-clone-20250909.jar package. If you have uploaded it before, select it from the drop-down list.	—
Entry Point Class	The entry point class of the JAR.	Leave blank
Entry Point Main Arguments	Parameters passed to the main method. Set in step 2.	Leave blank for now
Additional Dependencies	Upload the paimon-ali-vvr-8.0-vvp-1.3-ali-SNAPSHOT-for-clone-20250909.jar package. If you have uploaded it before, select it from the drop-down list.	—

Click Deploy.

Step 2: Configure and start the job

A single Flink job can migrate an entire catalog, an entire database, or a single table. The scope depends on which parameters you set. The table below covers all parameters; see Migration scope for which ones to omit.

In the Deployments list, find the JAR job and click Details.

On the Deployment Details page, click Edit and fill in the Entry Point Main Arguments field using the template below. Use the same template for both Hive and Paimon source tables. The only difference is the value of --clone_from.

Important

Keep your AccessKey secret confidential. It is only displayed once at creation and cannot be retrieved later.

Parameter reference

Parameter	Description	Required	Example
`parallelism`	Parallelism of the Flink job.	No	`16`
`database-name`	Name of the source DLF-Legacy database.	No	`my_database`
`table-name`	Name of the source DLF-Legacy table.	No	`my_table`
`warehouse`	OSS path of the source DLF-Legacy data catalog. Format: `oss://<bucket>/<object>`. Find your bucket and object in the OSS console.	Yes	`oss://my-bucket/my-path`
`dlf.catalog.id`	ID of the DLF-Legacy data catalog. Find it in the DLF-Legacy console. See Data Catalog.	Yes	—
`dlf.catalog.accessKeyId`	AccessKey ID for DLF-Legacy. See Create an AccessKey.	Yes	—
`dlf.catalog.accessKeySecret`	AccessKey secret for DLF-Legacy.	Yes	—
`dlf.catalog.endpoint`	DLF-Legacy service endpoint. See Regions and endpoints.	Yes	`dlf-vpc.cn-hangzhou.aliyuncs.com`
`dlf.catalog.region`	Region of DLF-Legacy. Must match `dlf.catalog.endpoint`. See Regions and endpoints.	Yes	`cn-hangzhou`
`fs.oss.endpoint`	OSS service endpoint. See Regions and endpoints.	Yes	OSS: `oss-cn-hangzhou-internal.aliyuncs.com`; OSS-HDFS: `cn-hangzhou.oss-dls.aliyuncs.com`
`fs.oss.accessKeyId`	AccessKey ID with read and write permissions on OSS. See Create an AccessKey.	Yes	—
`fs.oss.accessKeySecret`	AccessKey secret with read and write permissions on OSS.	Yes	—
`target-database-name`	Name of the target database in DLF.	No	`target_database`
`target-table-name`	Name of the target table in DLF.	No	`target_table`
`target-warehouse`	Name of the target data catalog in DLF. Find it in the DLF console. See Data Catalog.	Yes	—
`dlf.next.endpoint`	DLF service endpoint. See Endpoints.	Yes	`cn-hangzhou-vpc.dlf.aliyuncs.com`
`dlf.access-key-id`	AccessKey ID for DLF. See Create an AccessKey.	Yes	—
`dlf.access-key-secret`	AccessKey secret for DLF.	Yes	—
`clone_from`	Source table type.	Yes	`'hive'` or `'paimon'`
`filter-spec`	Partition filter for selective cloning.	No	`dt = '2024-10-01'`

Clone Hive tables

The source tables can be Hive tables (such as Parquet, ORC, and Avro), Iceberg tables, or Hudi tables. After migration, they are converted into Paimon append-only tables.

clone
--parallelism '<parallelism>'
--database '<database-name>'
--table '<table-name>'
--catalog_conf 'metastore=dlf'
--catalog_conf "warehouse=<warehouse>"
--catalog_conf 'dlf.catalog.id=<dlf.catalog.id>' 
--catalog_conf 'dlf.catalog.accessKeyId=<dlf.catalog.accessKeyId>'
--catalog_conf 'dlf.catalog.accessKeySecret=<dlf.catalog.accessKeySecret>'
--catalog_conf 'dlf.catalog.endpoint=<dlf.catalog.endpoint>'
--catalog_conf 'dlf.catalog.region=<dlf.catalog.region>'
--catalog_conf 'fs.oss.endpoint=<fs.oss.endpoint>'
--catalog_conf 'fs.oss.accessKeyId=<fs.oss.accessKeyId>'
--catalog_conf 'fs.oss.accessKeySecret=<fs.oss.accessKeySecret>'
--target_database '<target-database-name>'
--target_table '<target-table-name>'
--target_catalog_conf 'metastore=rest'
--target_catalog_conf 'warehouse=<target-warehouse>'
--target_catalog_conf 'uri=<dlf.next.endpoint>'
--target_catalog_conf 'token.provider=dlf'
--target_catalog_conf 'dlf.access-key-id=<dlf.access-key-id>'
--target_catalog_conf 'dlf.access-key-secret=<dlf.access-key-secret>'
--clone_from 'hive'
--where '<filter-spec>'

Clone Paimon tables

clone
--parallelism '<parallelism>'
--database '<database-name>'
--table '<table-name>'
--catalog_conf 'metastore=dlf'
--catalog_conf "warehouse=<warehouse>"
--catalog_conf 'dlf.catalog.id=<dlf.catalog.id>' 
--catalog_conf 'dlf.catalog.accessKeyId=<dlf.catalog.accessKeyId>'
--catalog_conf 'dlf.catalog.accessKeySecret=<dlf.catalog.accessKeySecret>'
--catalog_conf 'dlf.catalog.endpoint=<dlf.catalog.endpoint>'
--catalog_conf 'dlf.catalog.region=<dlf.catalog.region>'
--catalog_conf 'fs.oss.endpoint=<fs.oss.endpoint>'
--catalog_conf 'fs.oss.accessKeyId=<fs.oss.accessKeyId>'
--catalog_conf 'fs.oss.accessKeySecret=<fs.oss.accessKeySecret>'
--target_database '<target-database-name>'
--target_table '<target-table-name>'
--target_catalog_conf 'metastore=rest'
--target_catalog_conf 'warehouse=<target-warehouse>'
--target_catalog_conf 'uri=<dlf.next.endpoint>'
--target_catalog_conf 'token.provider=dlf'
--target_catalog_conf 'dlf.access-key-id=<dlf.access-key-id>'
--target_catalog_conf 'dlf.access-key-secret=<dlf.access-key-secret>'
--clone_from 'paimon'
--where '<filter-spec>'

Migration scope

Control the migration scope by including or omitting specific parameters.

Goal	Omit	Optional
Migrate a single table	—	Set `--where` to clone specific partitions
Migrate an entire database	`--table` and `--target_table`	Use `--excluded_tables` to skip specific tables
Migrate an entire catalog	`--database`, `--target_database`, `--table`, and `--target_table`	Use `--excluded_tables` to skip specific tables

--excluded_tables accepts a comma-separated list of database.table pairs (example: my_db.my_tbl,my_db2.my_tbl2). Do not set this parameter for single-table migrations.

Click Save.
On the Deployments page, find the JAR job and click Start. Accept the default start parameters.

Step 3: Verify the migration

When the job status changes to Finished, verify the result in both consoles.

Open the DLF-Legacy console and note the structure of the source catalog, databases, and tables you migrated.

Open the DLF console and confirm the following based on your migration scope:

Migration scope	What to check in DLF
Entire catalog	All databases and tables exist; table count matches DLF-Legacy
Entire database	All tables in the database exist; table count and schemas match DLF-Legacy
Single table	The table exists; the schema (column names, data types, and partition fields) matches DLF-Legacy

Open a representative table in the DLF console and compare its schema and partition list with the corresponding table in DLF-Legacy to confirm the data was cloned correctly.