All Products
Search

This Product

    E-MapReduce:Read from and write to MaxCompute

    Document Center

    E-MapReduce:Read from and write to MaxCompute

    Last Updated:Jun 21, 2026

    EMR Serverless Spark provides a built-in MaxCompute data source connector based on Spark DataSource V2. You can connect to MaxCompute by adding the required configurations during development. This topic describes how to read data from and write data to MaxCompute by using EMR Serverless Spark.

    Background information

    MaxCompute (formerly known as ODPS) is a fast, fully managed, exabyte-scale data warehouse solution. It is designed for storing and computing batch structured data and provides large-scale data warehouse solutions and analytics and modeling services. For more information about MaxCompute, see What is MaxCompute?.

    Prerequisites

    • You have created a workspace in EMR Serverless Spark.

    • You have created a MaxCompute project and enabled Open Storage for it.

      The examples in this topic use the pay-as-you-go billing method for Open Storage.

    Limits

    • This topic applies only to the following engine versions:

      • esr-4.x: esr-4.6.0 and later

      • esr-3.x: esr-3.5.0 and later

      • esr-2.x: esr-2.9.0 and later

    • The operations in this topic require you to enable the Open Storage feature for MaxCompute. For more information, see Open Storage.

    • The MaxCompute endpoint you use must support the Storage API. If not, switch to a supported endpoint. For more information, see Data transmission resources.

    Notes

    When you use Open Storage (pay-as-you-go), data transfers that exceed 1 TB are billed based on the logical data size.

    Procedure

    Step 1: Create a MaxCompute session

    You can create an SQL session or a notebook session to connect to MaxCompute. For more information about sessions, see Session management.

    SQL session

    1. Go to the Sessions page.

      1. Log on to the EMR console.

      2. In the left-side navigation pane, choose EMR Serverless > Spark.

      3. On the Spark page, click the name of the target workspace.

      4. On the EMR Serverless Spark page, click Sessions in the left-side navigation pane.

    2. On the SQL Session tab, click Create SQL Session.

    3. On the Create SQL Session page, configure the following parameters and click Create.

      Parameter

      Description

      Name

      Enter a custom name for the SQL session. For example, mc_sql_compute.

      Spark Configuration

      Enter the Spark configuration parameters to connect to MaxCompute.

      Important

      If you need to access a MaxCompute project that uses the three-level model, you must also set the spark.sql.catalog.odps.enableNamespaceSchema parameter to true in the Spark configuration. For more information about parameters, see Spark Connector. For more information about schemas, see Schema operations.

      spark.sql.catalog.odps                        org.apache.spark.sql.execution.datasources.v2.odps.OdpsTableCatalog
spark.sql.extensions                          org.apache.spark.sql.execution.datasources.v2.odps.extension.OdpsExtensions
spark.sql.sources.partitionOverwriteMode      dynamic
spark.hadoop.odps.tunnel.quota.name           pay-as-you-go
spark.hadoop.odps.project.name                <project_name>
spark.hadoop.odps.end.point                   https://service.cn-hangzhou-vpc.maxcompute.aliyun-inc.com/api
spark.hadoop.odps.access.id                   <accessId>
spark.hadoop.odps.access.key                  <accessKey>

      Replace the following placeholders with your actual values:

      • <project_name>: the name of your MaxCompute project.

      • https://service.cn-hangzhou-vpc.maxcompute.aliyun-inc.com/api: the endpoint of your MaxCompute service. For more information, see Endpoints.

      • <accessId>: the AccessKey ID of the Alibaba Cloud account used to access MaxCompute.

      • <accessKey>: the AccessKey Secret of the Alibaba Cloud account used to access MaxCompute.

    Notebook session

    1. Go to the Notebook Sessions page.

      1. Log on to the EMR console.

      2. In the navigation pane on the left, choose EMR Serverless > Spark.

      3. On the Spark page, click the name of the target workspace.

      4. On the EMR Serverless Spark page, choose Sessions in the navigation pane on the left.

      5. Click the Notebook Session tab.

    2. Click Create Notebook Session.

    3. On the Create Notebook Session page, configure the following parameters and click Create.

      Parameter

      Description

      Name

      Enter a custom name for the notebook session. For example, mc_notebook_compute.

      Spark Configuration

      Enter the Spark configuration parameters to connect to MaxCompute.

      Important

      If you need to access a MaxCompute project that uses the three-level model, you must also set the spark.sql.catalog.odps.enableNamespaceSchema parameter to true in the Spark configuration. For more information about parameters, see Spark Connector. For more information about schemas, see Schema operations.

      spark.sql.catalog.odps                        org.apache.spark.sql.execution.datasources.v2.odps.OdpsTableCatalog
spark.sql.extensions                          org.apache.spark.sql.execution.datasources.v2.odps.extension.OdpsExtensions
spark.sql.sources.partitionOverwriteMode      dynamic
spark.hadoop.odps.tunnel.quota.name           pay-as-you-go
spark.hadoop.odps.project.name                <project_name>
spark.hadoop.odps.end.point                    https://service.cn-hangzhou-vpc.maxcompute.aliyun-inc.com/api
spark.hadoop.odps.access.id                   <accessId>
spark.hadoop.odps.access.key                  <accessKey>

      Replace the following placeholders with your actual values:

      • <project_name>: the name of your MaxCompute project.

      • https://service.cn-hangzhou-vpc.maxcompute.aliyun-inc.com/api: the endpoint of your MaxCompute service. For more information, see Endpoints.

      • <accessId>: the AccessKey ID of the Alibaba Cloud account used to access MaxCompute.

      • <accessKey>: the AccessKey Secret of the Alibaba Cloud account used to access MaxCompute.

    Step 2: Query and write MaxCompute data

    Spark SQL

    1. In the left-side navigation pane of the EMR Serverless Spark page, click Development.

    2. On the Development tab, click the image icon.

    3. Create a Spark SQL job.

      1. In the dialog box that appears, enter a name such as mc_load_task, select SparkSQL as the type, and then click OK.

      2. Copy the following code into the new Spark SQL tab (mc_load_task).

        CREATE TABLE odps.default.mc_table (name STRING, num BIGINT);
INSERT INTO odps.default.mc_table (name, num) VALUES ('Alice', 100),('Bob', 200);
SELECT * FROM odps.default.mc_table;

      3. Select a database from the Database drop-down list. Then, from the Compute drop-down list, select the SQL session that you created in Step 1: Create a MaxCompute session (mc_sql_compute).

      4. Click Run to execute the Spark SQL job.

        After the query is successfully executed, the query result is displayed on the Execution Results tab.

        Result #3 shows the result table with two columns, name and num. The table shows that the value for 'Alice' is 100 and the value for 'Bob' is 200.

    4. View the created table in the MaxCompute console.

      1. Log on to the MaxCompute console and select a region in the upper-left corner.

      2. On the Projects page, find the project that you created and click Manage in the Actions column.

      3. Click the Tables tab.

        You can see the new table named mc_table in the MaxCompute console.

    Notebook

    1. In the left-side navigation pane of the EMR Serverless Spark page, click Development.

    2. On the Development tab, click the image icon.

    3. Create a notebook.

      1. In the dialog box that appears, enter a name such as mc_load_task. For the type, select Interactive Development > Notebook, and then click OK.

      2. From the session drop-down list, select the running notebook session that you created in Step 1: Create a MaxCompute session (mc_notebook_compute).

      3. Write and run the code.

        1. In a Python cell, enter the following command to create a table.

          spark.sql("""
CREATE TABLE odps.default.mc_table (name STRING, num BIGINT);
""")

        2. In a new Python cell, enter the following command to insert data.

          spark.sql("INSERT INTO odps.default.mc_table (name, num) VALUES ('Alice', 100),('Bob', 200);")

        3. In a new Python cell, enter the following command to query data.

          spark.sql("SELECT * FROM odps.default.mc_table;").show()

          After the query is successfully executed, the query result is displayed in the Execution Results.

          +-----+---+
| name|num|
+-----+---+
|Alice|100|
|  Bob|200|
+-----+---+

    4. View the created table in the MaxCompute console.

      1. Log on to the MaxCompute console and select a region in the upper-left corner.

      2. On the Projects page, find the project that you created and click Manage in the Actions column.

      3. Click the Tables tab.

        You can see the new table named mc_table in the MaxCompute console.

    FAQ

    Why do I receive an "Access Denied" error when I query a MaxCompute table?

    • Symptom: When you query a MaxCompute table, the following error message is returned.

      Access Denied - Not allowed to use storage api service on current endpoint

    • Cause: This error indicates that the current user is not authorized to use the MaxCompute Storage API service, or the specified endpoint does not support the Storage API.

    • Solution:

      • Check whether the Open Storage feature is enabled.

        In the MaxCompute console, go to Tenants > Tenant Property to check the status. If the feature is not enabled, see Use Open Storage to enable it and complete the configuration.

      • Check whether the current endpoint supports the Storage API. If not, switch to a supported endpoint. For more information, see Supported regions.

    Related topics

    This topic uses Spark SQL and notebooks as examples. To learn about other methods for reading from and writing to MaxCompute, see Develop a batch or streaming task.