E-MapReduce:Kafka Manager

Usage notes

When you use Kafka Manager to reassign partitions, Kafka Manager does not support throttling. If you want to enable throttling, you can run the kafka-configs.sh command to configure throttling-related parameters. For more information, see Throttle the O&M traffic of Kafka brokers.

Procedure

1. Log on to the Kafka Manager web user interface (UI) in SSH mode. For more information, see Create an SSH tunnel to access web UIs of open source components.

   Note

   • We recommend that you change the password if this is the first time you use Kafka Manager.

   • To prevent port 8085 from being exposed, we recommend that you log on to the web UI in SSH mode. If you want to log on to the web UI by visiting http://localhost:8085, you must configure an IP address whitelist to prevent data leak.

2. On the logon page, enter the username and password that are configured for Kafka Manager.

   You can perform the following steps to obtain the username, password, and ZooKeeper address:

   1. Log on to the EMR on ECS console.

   2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.

   3. On the EMR on ECS page, find the cluster that you want to manage and click Services in the Actions column.

   4. Obtain the following information:

      • Obtain the username and password:

        1. On the Services tab, click Configure next to Kafka-Manager.

        2. On the Configure tab, obtain the values of the following configuration items:

           • kafka.manager.authentication.username: the username that is used to log on to the Kafka Manager web UI.

           • kafka.manager.authentication.password: the password that is used to log on to the Kafka Manager web UI.

      • Obtain the Zookeeper address of the Kafka cluster:

        1. On the Services tab, click Configure next to Kafka.

        2. On the Configure tab, click the server.properties tab, and obtain the value of the zookeeper.connect configuration item. The obtained value is the Zookeeper address of the Kafka cluster.

3. On the Kafka Manager page, choose Cluster > Add Cluster to add the created Kafka cluster of the Dataflow type.

4. On the Add Cluster page, set the parameters that are described in the following table and click Save.

   

   Parameter	Description
   Cluster Name	The name of the Kafka cluster.
   Cluster Zookeeper Hosts	The Zookeeper address of the Kafka cluster. Enter the value of the zookeeper.connect configuration item obtained in Step 2 in the field.
   Kafka Version	The version of the Kafka cluster. Note Kafka Manager may not support Kafka clusters of later versions. Therefore, you can select a latest version that is supported by Kafka Manager.
   Enable JMX Polling (Set JMX_PORT env variable before starting kafka server)	Specifies whether to enable the Java Management Extensions (JMX) feature. In this example, the JMX feature is enabled.
   brokerViewThreadPoolSize	The size of the thread pool for obtaining data. Important Set the parameter to a value greater than 2.
   offsetCacheThreadPoolSize	The size of the thread pool for caching the offsets. Important Set the parameter to a value greater than 2.
   kafkaAdminClientThreadPoolSize	The thread pool size of the administrative client for Kafka. Important Set the parameter to a value greater than 2.

   After you add the Kafka cluster, you can use the common Kafka features.