This topic answers common operational questions about Kafka on E-MapReduce (EMR), covering log management, component control, error diagnosis, and capacity planning.
How do I clean up the output logs of the Kafka component?
If output logs from the Kafka component are consuming too much disk space, delete the log files directly from the log directory.
Kafka-related components store logs under $LOG_DIR_ROOT (default: /mnt/disk1/log). Each component has its own subdirectory:
| Component | Log directory |
|---|---|
| Kafka | $LOG_DIR_ROOT/kafka |
| Cruise Control | $LOG_DIR_ROOT/cruise-control |
| Schema Registry | $LOG_DIR_ROOT/kafka-schema-registry |
| REST Proxy | $LOG_DIR_ROOT/kafka-rest-proxy |
Go to the relevant directory and delete the log files you no longer need.
How do I clean up the output logs of the Kafka Manager component?
Go to $LOG_DIR_ROOT/kafka-manager (default: /mnt/disk1/log/kafka-manager) and delete the log files.
Can I stop the Kafka Manager component?
Yes. Kafka does not depend on Kafka Manager for read and write operations, so stopping it does not affect your Kafka service.
If you haven't integrated another Kafka management platform, keep Kafka Manager running — it's your only management interface. If you have an alternative and no longer need Kafka Manager, stop it from the Services tab of your cluster in the EMR console.
How do I fix the error "Replication factor: 1 larger than available brokers: 0"?
This error has two possible causes:
-
The broker process has exited. Check the broker logs to identify the failure and restart the broker process.
-
The ZooKeeper hosts configuration is incorrect. In the EMR console, update the Cluster Zookeeper Hosts parameter to match the value of
kafka.manager.zookeeper.hosts.
How do I fix the error "java.net.BindException: Address already in use (Bind failed)"?
The Java Management Extensions (JMX) port is already in use. Specify a different JMX port before running the command:
JMX_PORT=10101 kafka-topics.sh --bootstrap-server core-1-1:9092 --listHow do I fix the error "current leader's lastest offset xxxx is less than replica's lastest offset xxxxxx"?
This error means the current leader is behind a replica, which blocks a clean leader election.
Before proceeding, confirm that either all data has been consumed or that data loss is acceptable. Then:
-
Set
unclean.leader.election.enabletotrueon the Kafka broker component. -
Restart the Kafka broker component.
-
After the restart, set
unclean.leader.election.enableback tofalse.
What do I do if the disk that stores Kafka data in the log directory is full?
When the log directory disk is full, the log directory goes offline. Follow the steps in Perform O\&M operations when the disk space of an EMR Kafka cluster is full to recover.
How do I fix the error "Too many open files"?
This error occurs when the number of open file descriptors exceeds the system limit, typically due to a large number of partitions or network connections.
Increase the file descriptor limit in /etc/security/limits.conf. Add or update the following lines at the end of the file:
* soft nofile <new-limit>
* hard nofile <new-limit>Replace <new-limit> with a value appropriate for your workload. After saving the file, restart the affected components.
How do I estimate the number of partitions required for a Kafka topic?
Use the following approach:
-
Measure producer throughput per partition. Run a stress test to get the throughput (in MB/s) a single partition can sustain within your target latency.
-
Estimate your expected traffic. Determine the peak ingestion rate your topic needs to handle.
-
Calculate the required partition count. Based on your measured throughput per partition and expected business traffic, calculate the number of partitions needed for the topic.
-
Adjust for consumer parallelism. More partitions allow more consumers to read in parallel. If consumers are falling behind, increase the partition count to meet your consumption latency target.