Build Object Tables with PVFS for Seamless Metadata Access - Data Lake Formation

Object tables store metadata that maps to files in object storage. Use an Object table when you need to register existing files in OSS as a queryable table in Data Lake Formation (DLF) and access them through a unified virtual path via Paimon Virtual File System (PVFS).

Prerequisites

Before you begin, make sure you have:

A DLF catalog and a database created in that catalog
The permissions required to create, view, and delete tables in the database

Create a table

Create a table in the console

Log on to the DLF console.
In the left navigation menu, select Catalogs. Click your catalog name to open the catalog details page.
In the Database section, click your database name.
Click Create Table.

Configure the following parameters and click OK.

Parameter	Description
Table Format	Select Object Table.
Table Name	Enter a table name. The name must be unique within the database.
Table Description	Enter a description.

Create a table using SQL

If you have registered your DLF catalog in other systems like EMR Serverless Spark or Realtime Compute for Apache Flink, you can create databases and tables directly on those platforms using SQL. The metadata is written to DLF. For more information, see Engine integration.

Both examples use 'type'='object-table' to specify the table format.

Flink SQL — run in the Realtime Compute for Apache Flink console:

CREATE TABLE object_table WITH (
    'type'='object-table'
);

Spark SQL — run in the EMR Serverless Spark console:

CREATE TABLE object_table OPTIONS (
    'type'='object-table'
)

View a table

In the Database section, click your database name.
In the Tables list, click the Object table name to view its details.
On the Table Details tab, review the table's basic information and columns.
On the Permissions tab, click Grant Permissions to grant table permissions to users or roles. For more information, see Data authorization management.

Access a table via PVFS

Paimon provides a file system interface that lets you read data from an Object table using a virtual path:

pvfs://catalog_name/database_name/table_name/file

Replace catalog_name, database_name, and table_name with your actual catalog, database, and table names. For more information, see PVFS.

Delete a table

Warning

Deleted table data is retained for 1 day to prevent accidental deletion. Data is permanently deleted afterward.

In the Database section, click your database name.
On the Tables tab, click Delete in the Actions column of the target table.
In the dialog box, click OK to confirm.