MaxCompute:FileSystem Catalog external projects

When to use this approach

Use a Filesystem Catalog external project when:

• You need fine-grained, self-managed permission control over the Paimon catalog directory in OSS.

• Your workload involves stream-batch integration scenarios.

• You want to maintain the Paimon file system hierarchy yourself.

Limitations

• Only Paimon format tables are supported.

• Writing to Dynamic Bucket tables is not supported.

• Writing to Cross Partition tables is not supported.

• For data type compatibility, see Data type mapping.

Overview

Setting up a Filesystem Catalog external project involves four steps:

1. Grant permissions to a Resource Access Management (RAM) user.

2. Create a Filesystem Catalog external data source.

3. Create an external project.

4. Use SQL to access data in the external project.

Prerequisites

Before you begin, make sure you have:

• An OSS bucket to store Paimon data. For setup instructions, see Quick Start.

• A RAM role with permissions to access OSS.

Step 1: Grant permissions to a RAM user

If you are a RAM user, attach the AliyunMaxComputeFullAccess policy to your account. This policy grants the permissions required to create external data sources and external projects.

For instructions on attaching a policy, see Manage RAM user permissions.

Step 2: Create a Filesystem Catalog external data source

1. Log on to the MaxCompute console and select a region in the upper-left corner.

2. In the navigation pane on the left, choose Manage Configurations > External Data Source.

3. On the External Data Source page, click Create External Data Source.

4. In the Create External Data Source dialog box, configure the following parameters.

   Parameter	Required	Description
   External Data Source Type	Required	Select Filesystem Catalog.
   External Data Source Name	Required	A custom name. Must start with a letter and contain only lowercase letters, underscores (_), and digits. Cannot exceed 128 characters. Example: external_fs.
   Description	Optional	Enter information as needed.
   Region	Required	Defaults to the current region.
   Authentication And Authorization	Required	Defaults to Alibaba Cloud RAM role.
   RoleARN	Required	The Alibaba Cloud Resource Name (ARN) of a RAM role that has permissions to access OSS. To find the ARN: log on to the RAM console, choose Identities > Roles in the navigation pane, and locate the ARN in the Basic Information section. Example: acs:ram::124****:role/aliyunodpsdefaultrole.
   Storage Type	Required	Select OSS or OSS-HDFS.
   Endpoint	Required	Auto-generated based on the selected region. For example, for China (Hangzhou): oss-cn-hangzhou-internal.aliyuncs.com.
   Foreign Server Supplemental Properties	Optional	Additional properties for the external data source. Tasks that use this data source access the source system based on the defined behavior. Supported parameters will be added gradually as the product evolves.

5. Click OK.

6. On the External Data Source page, locate the data source you created and click Details in the Actions column to verify the configuration.

Step 3: Create an external project

1. Log on to the MaxCompute console and select a region in the upper-left corner.

2. In the navigation pane on the left, choose Manage Configurations > Projects.

3. On the External Project tab, click Create Project.

4. Configure the following parameters and click OK.

   Parameter	Required	Description
   Project Type	Required	Defaults to external project.
   Region	Required	Defaults to the current region and cannot be changed.
   Project Name (Globally Unique)	Required	Must start with a letter and contain only letters, digits, and underscores (_). Length must be 3–28 characters.
   MaxCompute Foreign Server Type	Optional	Select Filesystem Catalog.
   MaxCompute Foreign Server	Optional	Select Use Existing to use a created external data source, or Create Foreign Server to create a new one.
   MaxCompute Foreign Server Name	Required	If using an existing data source, select it from the drop-down list. If creating a new one, the new external data source name is used.
   Bucket Catalog	Required	The complete path from the OSS bucket to the catalog-level directory. Example: oss://paimon-fs/paimon-test/.
   Table Format	Required	Defaults to Paimon.
   Billing Method	Required	Subscription or Pay-as-you-go.
   Default Quota	Required	Select an existing quota.
   Description	Optional	A custom description of the project.

Step 4: Use SQL to access the external project

Select a connection tool to log on to the external project, then use the following SQL commands.

List schemas

By default, only database paths that store Paimon tables are listed.

-- Enable schema syntax at the session level.
SET odps.namespace.schema=true;
SHOW schemas;

-- Example output:
-- ID = 20250922********wbh2u7
-- default

List tables in a schema

-- Replace <schema_name> with the name of the schema shown in the external project.
USE SCHEMA <schema_name>;
SHOW tables;

Create a schema

CREATE schema <schema_name>;

-- Example:
CREATE schema schema_test;

Switch to a schema

use schema <schema_name>;

-- Example:
use schema schema_test;

Create a table and insert data

Syntax:

-- Create a table.
CREATE TABLE [IF NOT EXISTS] <table_name>
(
  <col_name> <data_type>,
  ...
)
[COMMENT <table_comment>]
[PARTITIONED BY (<col_name> <data_type>, ...)];

-- Insert data.
INSERT {INTO|OVERWRITE} TABLE <table_name> [PARTITION (<pt_spec>)] [(<col_name> [,<col_name> ...])]
<select_statement>
FROM <from_statement>

Example:

CREATE TABLE new_table(id INT, name STRING);

INSERT INTO new_table VALUES (101, 'Zhang San'), (102, 'Li Si');

-- Query the table.
SELECT * FROM new_table;

-- Result:
-- +------------+------------+
-- | id         | name       |
-- +------------+------------+
-- | 101        | Zhang San  |
-- | 102        | Li Si      |
-- +------------+------------+

Data type mapping

The following table shows how Paimon data types map to MaxCompute 2.0 data types. For details on MaxCompute data types, see Data types (Version 1.0) and Data types (Version 2.0).