All Products
Search

This Product

    E-MapReduce:Manage runtime environments

    Document Center

    E-MapReduce:Manage runtime environments

    Last Updated:Mar 26, 2026

    EMR Serverless Spark includes matplotlib, NumPy, and pandas by default. To use other third-party Python libraries in your Spark jobs, create a runtime environment that packages the libraries you need.

    A runtime environment applies across all job types in a workspace — PySpark batch jobs, Notebook sessions, Job orchestration, Spark Submit, and jobs submitted through Livy Gateway or Apache Airflow — so you define dependencies once and reuse them everywhere.

    Important

    Changes to a runtime environment do not take effect in active Notebook sessions automatically. After editing an environment, restart the session resources to apply the updated environment.

    Prerequisites

    Before you begin, make sure you have:

    Create a runtime environment

    1. Go to the Runtime Environment Management page.

      1. Log on to the E-MapReduce console.

      2. In the navigation pane on the left, choose EMR Serverless > Spark.

      3. On the Spark page, click the name of the target workspace.

      4. On the EMR Serverless Spark page, choose Environment in the navigation pane on the left.

    2. Click Create Environment.

    3. On the Create Environment page, configure the following parameters.

      Parameter Required Description
      Name Yes Enter a name for the runtime environment.
      Description No Enter a description for the environment.
      Resource Queue Yes Select a queue to use for environment initialization. Initialization consumes 1 Core and 4 GB of resources from this queue. The resources are released automatically after initialization completes.
      Network connection No Required only when adding PyPI libraries from sources other than the Alibaba Cloud source. The network connection is used to access the custom source address during environment creation. For more information, see Establish network connectivity between EMR Serverless Spark and other VPCs.
      Python version Yes Python 3.8 is used by default. Select another version if needed. Make sure the selected version is compatible with the libraries you plan to install — version mismatches can cause packaging failures or runtime errors.

    4. Click Add Library and add one or more libraries.

      In the New Library dialog box, select a Type and configure the related parameters.

      PyPI

      Parameter Description
      PyPI package Enter the library name and, optionally, the version. If you omit the version, the latest version is installed. The Alibaba Cloud source is used by default. For example: Plotly or Plotly==4.9.0.
      Package source Enter a custom PyPI source URL. Leave blank to use the Alibaba Cloud source. If you use a custom source, select the appropriate network connection in the previous step.

      Workspace

      Select a file resource from the current workspace. If no resources are listed, upload a file on the Artifacts page first.

      Supported file types: .zip, .tar, .whl, .tar.gz, .jar, .txt

      Note

      If the file type is .txt, the system reads it as a requirements file and installs the Python libraries and versions listed in it, similar to a requirements.txt file.

      OSS

      Enter the path of the file stored in Alibaba Cloud Object Storage Service (OSS).

      Supported file types: .zip, .tar, .whl, .tar.gz, .jar, .txt

      Note

      If the file type is .txt, the system reads it as a requirements file and installs the Python libraries and versions listed in it, similar to a requirements.txt file.

    5. Click Create. Environment initialization starts. The environment is ready to use once its status changes to Ready.

    Edit a runtime environment

    Edit a runtime environment to update the libraries it contains.

    Important

    After you save changes, the environment is re-initialized. The updated environment does not take effect in active Notebook sessions automatically — restart the session resources to apply the new environment.

    1. On the Environment page, find the target runtime environment and click Edit in the Actions column.

    2. On the Edit Environment page, update the environment configuration.

    3. Click Save Changes.

    Use a runtime environment

    A runtime environment must be in the Ready state before you can use it.

    Job type How to use the runtime environment
    PySpark batch jobs When a job starts, the system pre-installs the libraries from the selected runtime environment.
    Job orchestration When adding a Notebook node to a workflow, select the corresponding runtime environment in the node configuration.
    Notebook sessions When a Notebook session starts, libraries are pre-installed based on the selected environment.
    Livy Gateway When you submit a job through Livy Gateway, resources are pre-configured based on the selected environment.
    Spark Submit, Apache Airflow, and Livy Specify the runtime environment by passing the environment ID as a Spark configuration parameter (see below).

    For Spark Submit, Apache Airflow, and Livy, pass the environment ID as a configuration parameter:

    --conf spark.emr.serverless.environmentId=<runtime_environment_id>

    Replace <runtime_environment_id> with the ID of the target runtime environment.