E-MapReduce:Manage notebook sessions

Create a notebook session

After a notebook session is created, you can select it when developing notebooks.

1. Go to the Notebook Sessions page.

   1. Log on to the EMR console.

   2. In the navigation pane on the left, choose EMR Serverless > Spark.

   3. On the Spark page, click the name of the target workspace.

   4. On the EMR Serverless Spark page, choose Sessions in the navigation pane on the left.

   5. Click the Notebook Session tab.

2. On the Notebook Session page, click Create Notebook Session.

3. On the Create Notebook Session page, configure the following parameters and click Create.

   Note

   We recommend that you set the maximum concurrency of the selected resource queue to at least the amount of resources required by the notebook session. This value is displayed on the console.

   Parameter	Description
   Name	The name of the new notebook session. The name must be 1 to 64 characters in length and can contain letters, digits, hyphens (-), underscores (_), and spaces.
   Engine Version	The engine version for the session. For more information about engine versions, see Engine version overview.
   Use Fusion Acceleration	Fusion can accelerate Spark workloads and reduce the total cost of jobs. For billing information, see Product Billing. For more information about the Fusion engine, see Fusion engine.
   Resource Queue	Select a resource queue for the session. You can only choose queues designated for development or for shared use. For more information, see Manage resource queues.
   Automatic Stop	Enabled by default. You can set how long the session can be idle before it automatically stops.
   spark.driver.cores	The number of cores for the driver process. The default value is 1.
   spark.driver.memory	The amount of memory for the driver process. The default value is 3.5 GB.
   spark.executor.cores	The number of cores for each executor process. The default value is 1.
   spark.executor.memory	The amount of memory for each executor process. The default value is 3.5 GB.
   spark.executor.instances	The number of executors allocated by Spark. The default value is 2.
   Dynamic Resource Allocation	Disabled by default. When enabled, configure the following parameters: Minimum Number of Executors: The default value is 2. Maximum Number of Executors: If spark.executor.instances is not set, the default value is 10.
   More Memory Configurations	spark.driver.memoryOverhead: The non-heap memory available for the driver. If this parameter is not set, Spark automatically allocates a value based on the default, which is max(384 MB, 10% * spark.driver.memory). spark.executor.memoryOverhead: The non-heap memory available for each executor. If this parameter is not set, Spark automatically allocates a value based on the default, which is max(384 MB, 10% * spark.executor.memory). spark.memory.offHeap.size: The amount of off-heap memory available to Spark. The default value is 1 GB. This parameter takes effect only when spark.memory.offHeap.enabled is set to true. When the Fusion engine is used, this feature is enabled by default with 1 GB of off-heap memory.
   Environment	You can select a custom environment created on the Environment page. When the notebook session starts, the system pre-installs libraries from the selected environment. Note You can select only runtime environments that are in the Ready state.
   Network Connection	Select an existing network connection to access data sources in a VPC or external services. For more information about how to create a network connection, see Network connectivity between EMR Serverless Spark and other VPCs.
   Mount Integrated File Directory	This feature is disabled by default. To use this feature, add a file directory on the Artifacts page, on the Integrated File Directory tab. For more information, see Manage the integrated file directory. When enabled, the system mounts the integrated file directory to the session, allowing you to directly access its files. The mount operation consumes driver compute resources. The amount consumed is the greater of the following two values: Fixed resources: 0.3 vCPUs + 1 GB memory. Dynamic resources: 10% of the spark.driver resources (that is, 10% of the cores and memory of spark.driver). For example, if spark.driver is configured with 4 cores and 8 GB of memory, the dynamic resources are 0.4 vCPUs + 0.8 GB of memory. In this case, the actual consumed resources are max(0.3 vCPUs + 1 GB, 0.4 vCPUs + 0.8 GB), which is 0.4 vCPUs + 1 GB of memory. Note Mount scope: By default, the file directory is mounted only to the driver. To mount it to executors as well, enable Mount to Executor. Multiple directories: You can mount multiple integrated file directories. However, CPFS directories cannot be used together with other types. For example, you can mount multiple OSS and NAS directories together, but you cannot mount CPFS with OSS or NAS directories. Network requirements: When you mount a NAS or CPFS file directory, you must configure a network connection. The VPC of the network connection must be the same as the VPC of the NAS or CPFS mount target.
   Mount to Executor	When enabled, the system mounts the integrated file directory to the session executors, allowing them to access files directly. Mounting consumes executor resources. The amount of resources consumed depends on file usage.
   Spark Configuration	Enter Spark configuration parameters, separated by spaces. Example: spark.sql.catalog.paimon.metastore dlf.

View execution records

After a job completes, you can view its execution records.

1. On the SQL Sessions page, click the name of the desired session.

2. Click the Execution Records tab.

   On this tab, you can view details for each job execution, such as the run ID, start time, and a link to the Spark UI.

   

Related documents

• For information about resource queue operations, see Manage resource queues.

• For information about session roles and permissions, see Manage users and roles.

• For an end-to-end example of notebook development, see Get started with notebook development.