E-MapReduce:Create a cluster

Parameter

Description

Region

A region is the geographical area where a data center is located. Selecting a region closer to your location can reduce network latency. You cannot change the region after the cluster is created.

Business Scenario

Select a scenario based on your business needs:

• Data Lake (DataLake): Provides a flexible, reliable, and efficient managed cluster for running big data computing engines and performing data analytics.

  • Supports building a data lake architecture and uses JindoFS to accelerate data lake operations.

  • Supports OSS-HDFS (fully managed HDFS) for storage, reducing your O&M costs with a usage-based billing model.

  For more information, see DataLake cluster.

• Data Analytics (OLAP): Imports massive datasets into an OLAP analytics engine, such as ClickHouse or StarRocks, using methods such as data import or external tables. It provides efficient, real-time, and flexible data analytics for business scenarios such as user profiling, audience segmentation, BI reporting, and business analysis.

• Real-time Data Streaming (DataFlow): An all-in-one real-time computing solution on the EMR platform. It features two main components: Kafka, a distributed, high-throughput, and highly scalable messaging system, and the Flink commercial kernel from Ververica (based on Apache Flink). This solution focuses on addressing end-to-end real-time computing challenges and is widely used for real-time data ETL and log collection and analysis. You can also use either component individually.

• Data Service (DataServing):

  • Provides flexible, reliable, and efficient data service clusters.

  • Provides semi-managed HBase clusters and decouples compute clusters from data storage by using OSS-HDFS (the JindoFS service).

  • Supports JindoData local caching to further improve the read and write performance of the data service cluster.

  For more information, see DataServing cluster.

• Custom Cluster (Custom): Provides a rich selection of services. You can choose the services you need based on your requirements.

  Note

  In a production environment, we recommend that you avoid deploying multiple storage services on the same node group.

Product Version

The release version of the EMR product. For more information, see Release versions.

High Service Availability

This feature is disabled by default. When you enable high availability, EMR creates multiple Master nodes to support high availability for ResourceManager and NameNode. These nodes are distributed across different underlying hardware to reduce the risk of failure.

Optional Services (Select One At Least)

Select other services based on your business requirements. The selected services automatically start their related processes.

Collect Service Operational Logs

This setting, enabled by default, controls log collection for all services. These logs are used only for cluster diagnostics.

After the cluster is created, you can go to the Basic Information page to change the Collection Status of Service Operational Logs.

Metadata

The following methods are supported for storing and managing metadata:

• DLF Unified Metadata (Recommended): Stores metadata in Data Lake Formation (DLF).

  After activating DLF, the system selects a default DLF Catalog for you (typically your UID). To use different catalogs for different clusters, create a new one:

  1. Click Create Catalog. In the dialog box that appears, enter a catalog ID and click OK.

  2. From the DLF Catalog drop-down list, select the catalog that you created.

• Self-managed RDS: Uses your own or an Alibaba Cloud RDS instance as metadata storage.

  If you select this method, you must configure RDS-related parameters. For more information, see Configure a self-managed RDS database.

• Built-in MySQL (Not Recommended): Stores metadata in a MySQL database on the cluster's Master node.

  Note

  • For testing scenarios: We recommend using DLF Unified Metadata.

  • For production scenarios: You can use DLF Unified Metadata or Self-managed RDS.

Root Storage Directory of Cluster

This parameter is required only when you select the OSS-HDFS service. It is not needed if you select the HDFS service.

When you use the OSS-HDFS service for the first time, your Alibaba Cloud account (main account) must click here and follow the prompts to complete the authorization. If you are a RAM user, your Alibaba Cloud account must authorize you to activate the service and grant the AliyunEMRDlsFullAccess policy and the AliyunOSSDlsDefaultRole and AliyunEMRDlsDefaultRole roles. For more information, see Grant permissions to a RAM user. Select a bucket in the same region where the OSS-HDFS service is already enabled, or click Create OSS-HDFS Bucket to create a new OSS-HDFS instance as the root storage path for the cluster.

Parameter

Description

Billing Method

Subscription is selected by default. The following billing methods are supported:

• Pay-as-you-go: A post-paid model where you are billed hourly for actual usage. This method is suitable for short-term testing or flexible, dynamic tasks.

• Subscription: A pre-paid model where you pay before use.

  Note

  • We recommend using the Pay-as-you-go method for testing scenarios. After testing is successful, create a Subscription cluster for production use.

  • For subscription instances, you must also select a Subscription Duration and decide whether to enable Auto-renewal. The default renewal period is six months, with auto-renewal enabled. When auto-renewal is enabled, the instance is automatically renewed seven days before it expires. For more information, see Renewal.

Zone

An availability zone is a distinct physical location within a region, interconnected by a low-latency private network. You can typically use the default availability zone.

VPC

A VPC is a logically isolated network environment on Alibaba Cloud that gives you full control over your network configuration.

Select an existing VPC, or click Create VPC to create one instantly in the VPC console. For more information, see Create and manage a VPC.

vSwitch

A vSwitch is a basic network subdivision of a VPC used to connect different cloud resources.

Select an existing vSwitch, or click . Create vSwitch to create one instantly in the VPC console. For more information, see Create and manage a vSwitch.

The CIDR block size of the vSwitch determines the number of available IPv4 addresses. For example, a /24 CIDR block provides about 250 available IP addresses, a /20 block provides about 4,090, and a /17 block provides about 32,764. If your cluster requires a large number of nodes, select a vSwitch with a larger CIDR block to ensure enough IP addresses can be allocated to all cluster nodes.

You cannot change the CIDR block of a vSwitch after it is created. If you need a larger IP address range, you must create a new vSwitch with a larger CIDR block and then create a new cluster based on that vSwitch. When you create a vSwitch, the console displays the number of available IP addresses for the selected CIDR block, which helps you assess whether it meets your cluster size requirements.

Default Security Group

A security group is a virtual firewall that controls inbound and outbound traffic for instances within the group. For more information, see Security group overview.

Select an existing security group, or click create a new security group. to create one in the ECS console. For more information, see Create a security group.

Node Group

You can select instance types based on your requirements. For more information, see Instance families.

• Master: Runs control plane services such as ResourceManager and NameNode.

• Core: Stores data in the cluster's distributed file system (HDFS). You can also scale out Core nodes on demand after the cluster is created.

• Task: Provides additional compute capacity and does not store data. This node group is optional and can be added to scale computing resources.

  Important

  Task node groups support pay-as-you-go, preemptible instance, and subscription billing methods.

• Add to Deployment Set: When high availability is enabled, Master nodes are added to a deployment set by default. A deployment set is a policy that controls the distribution of instances. For more information, see Deployment sets.

• System Disk: Select a standard SSD, enhanced SSD (ESSD), or ultra disk as needed. Adjust the system disk size as required.

• Data Disk: Select a standard SSD, enhanced SSD (ESSD), or ultra disk as needed. Adjust the data disk size as required.

  Note

  When you select an enhanced SSD (ESSD), you can set different performance levels (PLs) based on the disk capacity to meet various cluster performance needs. The default performance level is PL1. Depending on the disk capacity, system disks support PL0, PL1, and PL2, while data disks support PL0, PL1, PL2, and PL3. For more information about cloud disks, see Cloud disk overview.

• Instances: The Master node group has one instance by default. If high availability is enabled, you can have multiple Master instances.

  The Core node group has two instances by default. You can adjust this number as needed.

• Additional Security Group: Additional security groups allow flexible access control between different external resources or applications. You can associate up to two additional security groups with this node group.

• Assign Public Network IP: Specifies whether to assign an Elastic IP (EIP) address to the cluster. This is disabled by default. Only DataLake clusters support assigning public IPs at the node group level.

  Note

  If you do not enable this feature but want to use a public IP address to access the cluster after it is created, you must apply for one on the ECS console. For more information, see Apply for an EIP.

Cluster Scaling

You can select an auto scaling rule as needed:

• Do Not Use Auto Scaling Rules (Default).

• Custom Auto Scaling Rule: You can define custom rules for time-based or load-based auto scaling. For more information, see Create a custom auto scaling rule.

• Managed Auto Scaling Rule: When you start the cluster, EMR pre-provisions the number of Task nodes based on the managed scaling rule. For more information, see Create a managed auto scaling rule.

Parameter

Description

Cluster Name

The name of the cluster. The name must be 1 to 64 characters long and can contain only Chinese characters, letters, digits, hyphens (-), and underscores (_).

Identity Credentials

Identity Credentials provide secure access to the cluster's Master node. For logon instructions, see Log on to a cluster. The following credential types are supported:

• Key Pair (Default): Select an existing key pair or click Create Key Pair to create one instantly.

  A key pair, which consists of a public key and a private key, is a secure and convenient authentication method for logging on to Linux instances. For more information about using key pairs, see SSH key pairs.

• Password: Set a logon password for the Master node and confirm it. The default username is root.

  • Password requirements: 8 to 30 characters in length and must contain uppercase letters, lowercase letters, digits, and special characters.

  • Supported special characters include: exclamation mark (!), at sign (@), hash sign (#), dollar sign ($), percent sign (%), caret (^), ampersand (&), and asterisk (*).