All Products
Search
Document Center

E-MapReduce:Kafka Manager

Last Updated:Feb 27, 2024

Kafka Manager is a cluster management tool designed for Kafka. Kafka Manager provides a web UI that allows you to manage Dataflow Kafka clusters in a convenient manner. This topic describes how to access the web UI of Kafka Manager and add a cluster to Kafka Manager.

Prerequisites

A Dataflow cluster is created in the E-MapReduce (EMR) console, and Kafka is selected when you create the cluster. For more information, see Create a cluster.

Note

If you select Kafka for a Dataflow cluster, Kafka Manager is installed, and the authentication feature of Kafka Manager is enabled for the cluster by default.

Precautions

When you use Kafka Manager to reassign partitions, Kafka Manager of the current version does not support throttling. If you want to enable throttling, you can run the kafka-configs.sh command to configure throttling-related parameters. For more information, see Throttle the O&M traffic of Kafka brokers.

Procedure

  1. Access the web UI of Kafka Manager in SSH mode. For more information, see Create an SSH tunnel to access web UIs of open source components.

    Note
    • We recommend that you change the password if this is the first time you use Kafka Manager.

    • To prevent port 8085 from being exposed, we recommend that you access the web UI of Kafka Manager in SSH mode. If you want to access the web UI by visiting http://localhost:8085, you must configure an IP address whitelist to prevent data leak.

  2. On the logon page, enter the username and password configured for Kafka Manager.

    You can perform the following steps to obtain the username, password, and the address of ZooKeeper:

    1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.

    2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.

    3. On the EMR on ECS page, find the desired cluster and click Services in the Actions column.

    4. Obtain the required information.

      • Username and password

        1. On the Services tab, find Kafka Manager and click Configure.

        2. On the Configure tab, view the values of the following parameters:

          • kafka.manager.authentication.username: the username that is used to access the web UI of Kafka Manager.

          • kafka.manager.authentication.password: the password that is used to access the web UI of Kafka Manager.

      • Address of ZooKeeper

        1. On the Services tab, find Kafka and click Configure.

        2. On the Configure tab, click the server.properties tab and view the value of the zookeeper.connect parameter.

  3. On the web UI of Kafka Manager, choose Cluster > Add Cluster.

  4. On the Add Cluster page, configure the parameters that are described in the following table and click Save.

    Add Kafka

    Parameter

    Description

    Cluster Name

    The name of the Dataflow cluster.

    Cluster Zookeeper Hosts

    The address of ZooKeeper.

    Enter the value of the zookeeper.connect parameter obtained in Step 2 in the field.

    Kafka Version

    The version of Kafka that is deployed in the Dataflow cluster.

    Note

    Kafka Manager may not support Dataflow clusters that contains Kafka of later versions. You can select the latest version that is supported by Kafka Manager.

    Enable JMX Polling (Set JMX_PORT env variable before starting kafka server)

    Specifies whether to enable the Java Management Extensions (JMX) feature.

    In this example, the JMX feature is enabled.

    brokerViewThreadPoolSize

    The size of the thread pool for obtaining data.

    Important

    Set the parameter to a value that is greater than 2.

    After you add the Dataflow cluster, you can use the common features of Kafka.Kafka