OpenLake:Paimon_DLF external projects (invitational preview)

Step 1: Grant permissions

Step 2: Create a Paimon_DLF external data source

1. Log on to the MaxCompute console and select a region in the top-left corner.

2. In the navigation pane on the left, choose Manage Configurations > Foreign Server.

3. On the Foreign Server page, click Create Foreign Server.

4. In the Add Foreign Server dialog box, configure the parameters. The following tables describe the parameters.

   Parameter	Required	Description
   Foreign Server Type	Required	Select Paimon_DLF.
   Foreign Server Name	Required	Enter a custom name. The name must follow these rules: Starts with a letter and contains only lowercase letters, underscores (_), and digits. Cannot exceed 128 characters. For example, paimon_dlf.
   Foreign Server Description	Optional	Enter a description as needed.
   Region	Required	The current region is selected by default.
   Authentication And Authorization	Required	The default value is Alibaba Cloud RAM role.
   Service-linked Role	Required	This is generated by default.
   Endpoint	Required	This is automatically generated. For the China (Hangzhou) region, the Endpoint is cn-hangzhou-intranet.dlf.aliyuncs.com.
   Foreign Server Supplemental Properties	Optional	Specifies additional attributes for the external data source. After you specify these attributes, tasks that use this data source can access the source system based on the defined behavior. Note For information about supported parameters, see future updates to the official documentation. More parameters will be made available as the product evolves.

5. Click OK to create the external data source.

6. On the Foreign Server page, find the target data source and click Details in the Actions column.

Step 3: Create an external project

1. Log on to the MaxCompute console and select a region in the top-left corner.

2. In the navigation pane on the left, choose Manage Configurations > Projects.

3. On the External Project tab, click Create Project.

4. In the Create Project dialog box, configure the project information as prompted and click OK.

   Parameter	Required	Description
   Project Type	Required	The default value is External Project.
   Region	Required	The current region is selected by default and cannot be changed.
   Project Name (Globally Unique)	Required	The name must start with a letter, contain only letters, digits, and underscores (_), and be 3 to 28 characters in length.
   MaxCompute Foreign Server Type	Optional	The default value is Paimon_DLF.
   MaxCompute Foreign Server	Optional	Use Existing: Lists the external data sources that you have created. Create Foreign Server: Creates and uses a new external data source.
   MaxCompute Foreign Server Name	Required	If you select **Select Existing**, select the name of an existing external data source from the drop-down list. If you select **Create New External Data Source**, the name of the new external data source is used.
   Data Catalog	Required	The DLF data catalog.
   Billing Method	Required	Subscription or Pay-as-you-go.
   Default Quota	Required	Select an existing quota.
   Description	Optional	Enter a custom description for the project.

Step 4: Use SQL to access the data source

1. Select a connection tool to log on to the external project.

2. List the schemas in the external project.

   -- Enable the schema syntax at the session level.
   SET odps.namespace.schema=true;
   SHOW schemas;
   
   -- Sample result. 
   ID = 20250919****am4qb
   default
   system
   
   
   OK

3. List the tables in a schema within the external project.

   -- schema_name is the name of a schema displayed in the external project.
   USE schema <schema_name>;
   SHOW tables;
   
   -- Result.
   ID = 20250919****am4qb
   acs:ram::<uid>:root  emp
   
   
   OK

4. Create a schema in the external project.

   -- In this example, schema_name is schema_demo_test.
   CREATE schema <schema_name>;

5. Use the new schema.

   USE schema <schema_name>;

6. Create a table in the schema and insert data.

   If you log on as a RAM user, this operation requires permissions on the DLF Catalog. For more information, see Data authorization management.

   • Command format:

     -- Create a table.
     CREATE TABLE [IF NOT EXISTS] <table_name> 
     (
       <col_name> <data_type>,
       ...
     )
     [COMMENT <table_comment>]
     [PARTITIONED BY (<col_name> <data_type>, ...)] 
     ;
     
     -- Insert data.
     INSERT {INTO|OVERWRITE} TABLE <table_name> [PARTITION (<pt_spec>)] [(<col_name> [,<col_name> ...)]]
     <select_statement>
     FROM <from_statement>

   • Example:

     Note

     When you write TIMESTAMP data to a Paimon source table, the data is truncated based on its precision. Data with a precision of 0 to 3 is truncated to 3 decimal places, data with a precision of 4 to 6 is truncated to 6 decimal places, and data with a precision of 7 to 9 is truncated to 9 decimal places.

     CREATE TABLE schema_table(id int,name string);
     
     INSERT INTO schema_table VALUES (101,'Zhang San'),(102,'Li Si');
     
     -- Query the schema_table table.
     SELECT * FROM schema_table;
     
     -- Result.
     +------------+------------+
     | id         | name       | 
     +------------+------------+
     | 101        | Zhang San  | 
     | 102        | Li Si      | 
     +------------+------------+

7. Switch to the existing 'default' schema.

   use schema default;
   SHOW tables;
   
   -- Sample result.
   ID = 20250919*******yg5
   acs:ram::<uid>:root	emp
   acs:ram::<uid>:root	emp_detail
   acs:ram::<uid>:root	test_table
   
   
   OK
   
   -- Read the table.
   SELECT * FROM test_table;
   
   -- Result.
   +------------+------------+
   | id         | name       | 
   +------------+------------+
   | 101        | Zhang San  | 
   | 102        | Li Si      | 
   +------------+------------+
   
   -- Write to the table and query it to check if the write operation was successful.
   INSERT INTO test_table VALUES (103,'Wang Wu');
   
   SELECT * FROM test_table;
   
   -- Result.
   +------------+------------+
   | id         | name       | 
   +------------+------------+
   | 101        | Zhang San  | 
   | 102        | Li Si      | 
   | 103        | Wang Wu    | 
   +------------+------------+

Problem 1: An error occurs when you click to create an external project

Problem description: When you create an external project, the error message "You are not authorized to perform this action." appears.

Solution:

1. If you are a RAM user, first ensure that the AliyunMaxComputeFullAccess permission is granted to the RAM user. For more information, see Step 1.

2. This operation requires you to select a DLF catalog. Ensure that the required permissions for DLF are granted. For more information, see Step 1.

Problem 2: An error occurs when you run `show tables` in the default schema of an external project

Problem description: When you run `show tables` in the default schema of an external project, the error message "Forbidden:User acs: ram: :<uid>:user/** doesn't have privilege LIST on DATABASE default" appears.

Forbidden:User acs: ram: :<uid>:user/** doesn't have privilege LIST on DATABASE

Default

Solution:

1. Log on to the DLF console. In the navigation pane on the left, click System And Security. On the Access Control tab, refresh the page and check whether the current RAM user exists.

2. If the user exists, click Roles and grant permissions to the RAM user.