All Products
Search

This Product

    E-MapReduce:Create a Dataflow Kafka cluster

    Document Center

    E-MapReduce:Create a Dataflow Kafka cluster

    Last Updated:Jul 17, 2026

    A Dataflow Kafka cluster is an E-MapReduce (EMR) cluster that includes the Kafka service.

    Limitations

    The Kafka service is not supported in E-MapReduce (EMR) versions 5.18.0 and later or 3.52.0 and later. We recommend that you use ApsaraMQ for Kafka or install Kafka manually.

    Usage notes

    Before you create a Dataflow Kafka cluster, select an ECS instance type and the number of brokers based on your estimated workload. No universal sizing plan is provided because requirements vary by scenario. Consider the following when you select an instance type:

    • For brokers, use a CPU-to-memory ratio of 1:4.

    • Use cloud disks as data disks.

    • Balance the I/O throughput of cloud disks with the NIC bandwidth.

    For deployment parameters, note the following:

    • Kafka in EMR depends on ZooKeeper for high availability. We recommend that you enable high availability when you create a cluster. This deploys a three-node ZooKeeper service.

    • If the master node group hosts only ZooKeeper, you need to configure only one data disk for the node group.

    For detailed sizing recommendations, see Cluster resource assessment.

    Procedure

    1. Go to the cluster creation page.

      1. Log on to the E-MapReduce console.

      2. Optional: In the top navigation bar, select a region and a resource group based on your business requirements.

        • Region: The cluster is created in the selected region. This setting cannot be changed after creation.

        • Resource Group: By default, all resources in your account are displayed.

      3. Click CREATE_CLUSTER.

    2. Configure cluster settings.

      You need to configure software, hardware, and basic settings for the cluster.

      Important

      Once a cluster is created, only the cluster name can be changed. Verify all configurations carefully before proceeding. For more information about the parameters, see Create a cluster.

      1. Configure the software settings.

        Parameter

        Example

        Description

        Region

        China (Hangzhou)

        The region where the cluster is created. Cannot be changed after creation.

        Business Scenario

        Real-time Data Streaming

        Select Real-time Data Streaming.

        Product Version

        EMR-3.43.1

        Select an EMR version to view the versions of the included services.

        For example, EMR-3.43.1 includes Kafka 2.12_2.4.1, where 2.12 is the Scala version and 2.4.1 is the open-source Kafka version.

        High Service Availability

        On

        Turned off by default.

        Important

        We recommend enabling high availability because the Kafka service in EMR depends on ZooKeeper. Enabling this option deploys a three-node ZooKeeper service on the master node group.

        Optional Services

        Kafka

        Select the Kafka service.

        You can also select other components. The services for selected components start automatically.

        Collect Service Operational Logs

        On

        This setting, enabled by default, controls log collection for all services. These logs are used only for cluster diagnostics.

        After the cluster is created, you can go to the Basic Information page to change the Collection Status of Service Operational Logs.

        Important

        If you disable log collection, EMR health checks and technical support will be limited, but other features will continue to function normally. For details on how to disable this feature and its impact, see How do I stop collecting service logs?.

      2. Configure the hardware settings.

        Parameter

        Example

        Description

        Billing Method

        Pay-as-you-go

        The default billing method is subscription. The following billing methods are supported:

        • Pay-as-you-go: A post-paid billing method that charges you hourly. Suitable for short-term testing or dynamic workloads.

        • Subscription: A pre-paid billing method where you pay for resources upfront.

          Note

          We recommend that you use the Pay-as-you-go option for testing. After testing is successful, create a new Subscription cluster for production use.

        Zone

        China (Hangzhou) Zone I

        Zones are distinct physical locations within the same region, interconnected over an internal network. You can typically use the default zone.

        VPC

        emr_test/vpc-bp1f4epmkvncimpgs****

        An existing VPC is selected by default.

        To create a new VPC, go to the VPC console. For more information, see Create and manage a VPC.

        vSwitch

        vsw_test/vsw-bp1e2f5fhaplp0g6p****

        Select a vSwitch in the specified zone of the selected VPC. If no vSwitch is available in the zone, create one in the VPC console. For more information, see VPCs and vSwitches.

        Default Security Group

        sg-bp1ddw7sm2risw****/sg-bp1ddw7sm2risw****

        An existing security group is selected by default. For more information about security groups, see Security group overview.

        You can also click create a new security group. to create a security group in the ECS console. For more information, see Create a security group.

        Important

        Enterprise security groups created in the ECS console are not supported.

        Node Group

        Configure as required.

        • Instance Type: Select an instance type based on your needs. For sizing recommendations, see Cluster resource assessment.

        • Add to Deployment Set: When high availability is enabled, master nodes are added to a deployment set by default. For more information, see Enable a deployment set.

        • System Disk: Select a type for the system disk.

        • System disk size: The value can range from 80 GiB to 500 GiB, but a minimum of 120 GiB is recommended.

        • Data Disk: Select a type for the data disk.

          Note

          We recommend that you use cloud disks.

        • Data disk size: The value can range from 40 GiB to 32,768 GiB, but a minimum of 80 GiB is recommended.

        • Instances: By default, three master instances and three core instances are created.

        • Additional Security Group: Associate up to two additional security groups with the node group to control access between different external resources or applications.

        • Assign Public Network IP: Whether to associate an elastic IP address (EIP) with the cluster. Disabled by default.

          Note

          To access the cluster over the public network after creation, apply for an EIP in the ECS console. For more information, see the documentation about applying for an EIP.

      3. Configure the basic settings.

        In the Basic Information section, configure the following parameters.

        Important

        The parameters in the Advanced Settings section are not supported. Do not configure them.

        Parameter

        Example

        Description

        Cluster Name

        Emr-Kafka

        The name must be 1 to 64 characters long and can contain letters, digits, hyphens (-), and underscores (_).

        Identity Credentials

        Custom password

        Key Pair (default): Use an SSH key pair to log on to the Linux instances.

        For more information about how to use a key pair, see SSH key pairs.

        Password: Set a password to log on to the master node instances.

        The password must be 8 to 30 characters in length and contain uppercase letters, lowercase letters, digits, and special characters.

        Allowed special characters are: !, @, #, $, %, ^, &, and *.

    3. On the Confirm page, select the E-MapReduce Terms of Service checkbox.

    4. Click create.

      The cluster is successfully created when its Status changes to Running.

    Next steps

    After the cluster is created, modify the default parameters to meet your production requirements. For example: