Data Lake Formation:PVFS

Path structure

All PVFS paths follow a three-level namespace:

pvfs://<catalog_name>/<database_name>/<table_name>/[path]

Listing behavior

When you list a PVFS path, the response reflects the virtual namespace hierarchy:

• pvfs://<catalog_name>/ — returns all databases in the catalog as virtual paths

• pvfs://<catalog_name>/<database_name>/ — returns all tables in the database as virtual paths

• pvfs://<catalog_name>/<database_name>/<table_name>/ — returns the table's actual files and directories

All listed paths are virtual paths. Read and write operations resolve the virtual path to the table's real storage location automatically.

Java SDK

The Java SDK implements the HDFS interface, making PVFS accessible from any Hadoop-compatible framework.

Prerequisites

Before you begin, ensure that you have:

• Java 8 or later

• A DLF catalog with at least one Paimon table

• A valid DLF access token

Configure using the Hadoop FileSystem API

Set the following properties in your Configuration object or core-site.xml:

Example: read a file using the Hadoop FileSystem API

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

Configuration conf = new Configuration();
conf.set("fs.pvfs.impl", "org.apache.paimon.vfs.hadoop.PaimonVirtualFileSystem");
// Replace <dlf-endpoint> with the actual DLF service endpoint
conf.set("fs.pvfs.uri", "<dlf-endpoint>");

Path path = new Path("pvfs://<catalog_name>/<database_name>/<table_name>/");
FileSystem fs = path.getFileSystem(conf);
// List files in the table
fs.listStatus(path);

Example: configure in core-site.xml for cluster-wide access

<configuration>
  <property>
    <name>fs.pvfs.impl</name>
    <value>org.apache.paimon.vfs.hadoop.PaimonVirtualFileSystem</value>
  </property>
  <property>
    <name>fs.pvfs.uri</name>
    <!-- Replace with the actual DLF service endpoint -->
    <value><dlf-endpoint></value>
  </property>
</configuration>

After adding these properties, frameworks such as Hive, Spark, and Presto can access PVFS paths without additional configuration.

Example: query a table with Spark SQL

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .config("spark.hadoop.fs.pvfs.impl",
          "org.apache.paimon.vfs.hadoop.PaimonVirtualFileSystem")
  // Replace <dlf-endpoint> with the actual DLF service endpoint
  .config("spark.hadoop.fs.pvfs.uri", "<dlf-endpoint>")
  .getOrCreate()

spark.sql("""
  CREATE TEMPORARY VIEW my_table
  USING parquet
  OPTIONS (path 'pvfs://<catalog_name>/<database_name>/<table_name>/')
""")
spark.sql("SELECT * FROM my_table LIMIT 10").show()

Python SDK

The Python SDK implements the fsspec interface, letting you use PVFS with any fsspec-compatible Python library.

Prerequisites

Before you begin, ensure that you have:

• Python 3.7 or later

• pypaimon installed

• A DLF catalog with at least one Paimon table

• A valid DLF access token

Initialize the filesystem

Pass the following options when creating a PaimonVirtualFileSystem instance:

import pypaimon

options = {
    "uri": "<dlf-endpoint>",          # Replace with the DLF service endpoint
    "token.provider": "dlf",
    "token": "<your-dlf-token>",      # Replace with a valid DLF access token
}
fs = pypaimon.PaimonVirtualFileSystem(options)

List and explore tables

# List all tables in a database
fs.ls("pvfs://<catalog_name>/<database_name>/")

# List files in a specific table
fs.ls("pvfs://<catalog_name>/<database_name>/<table_name>/")

Read with PyArrow

import pypaimon
import pyarrow.parquet as pq

fs = pypaimon.PaimonVirtualFileSystem(options)

# Read a Parquet file from a Paimon table
dataset = pq.ParquetDataset(
    "pvfs://<catalog_name>/<database_name>/<table_name>/",
    filesystem=fs
)
table = dataset.read()
df = table.to_pandas()
print(df.head())

Read with Pandas

import pypaimon
import pandas as pd

fs = pypaimon.PaimonVirtualFileSystem(options)

# Read a CSV file from a Paimon table
with fs.open("pvfs://<catalog_name>/<database_name>/<table_name>/data.csv") as f:
    df = pd.read_csv(f)
print(df.head())