Learn how to create a Kafka cluster in E-MapReduce (EMR) and configure key parameters for clusters that use local disks.
Create a Kafka cluster
To create a cluster that includes Kafka, log on to the new E-MapReduce console and create a Dataflow cluster. For more information, see Create a Dataflow Kafka cluster.
The Kafka service is not supported in E-MapReduce (EMR) versions 5.18.0 and later or 3.52.0 and later. We recommend that you use ApsaraMQ for Kafka or install Kafka manually.
Kafka clusters with local disks
If the Kafka service runs on instances with local disks, configure the following parameters on the Configure page in the E-MapReduce console.
|
Parameter |
Description |
|
default.replication.factor |
Fixed at 3. Each topic has three replicas. |
|
min.insync.replicas |
Fixed at 2. At least two replicas must be in sync at all times. A write succeeds only if the producer sets request.required.acks to all or -1 and at least two replicas acknowledge the write. |
Parameters
View the Kafka service parameters on the Configure page of the E-MapReduce console.
|
Parameter |
Description |
|
zookeeper.connect |
The ZooKeeper connection string of the cluster. |
|
kafka.heap.opts |
The heap memory size of each Kafka broker. |
|
num.io.threads |
The number of I/O threads per Kafka broker. Defaults to twice the number of CPU cores on the master node. |
|
num.network.threads |
The number of network threads per Kafka broker. Defaults to the number of CPU cores on the master node. |