You can use Telegraf to collect metric data from Kafka servers, and then use Logtail to send the metric data to a Metricstore in Log Service. This way, you can monitor metric data of Kafka servers in a visualized manner. This topic describes how to use Log Service to collect metric data from Kafka servers and visualize the data.

Prerequisites

  • Logtail V0.16.48 or later is installed on a Linux server. For more information, see Install Logtail on a Linux server.
  • Java 1.6 or later is installed on the server.

Step 1: Create a Logtail configuration

  1. Log on to the Log Service console.
  2. In the Import Data section, click Kafka Monitoring Data.
  3. In the Specify Logstore step, select the project and the Metricstore that you want to use. Then, click Next.
    You can also click Create Now to create a project or a Metricstore. For more information, see Create a project and Create a Metricstore.
  4. In the Create Machine Group step, create a machine group.
    • If a machine group is available, click Using Existing Machine Groups.
    • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
      1. Select the ECS instance on which you want to install Logtail. For more information, see Install Logtail on ECS instances.
        If Logtail is installed on the ECS instance, click Complete Installation.
        Note If you want to collect data from servers in self-managed clusters or servers on third-party clouds, you must manually install Logtail. For more information, see Install Logtail on a Linux server.
      2. After Logtail is installed, click Complete Installation.
      3. Create a machine group.

        For more information about how to create a machine group, see Create an IP address-based machine group or Create a custom ID-based machine group.

  5. Select the newly created machine group and move it from the Source Server Groups section to the Applied Server Groups section. Then, click Next.
    Notice If you apply a machine group immediately after it is created, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. In this case, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
  6. In the Specify Data Source step, configure the parameters.
    Parameter Description
    Configuration Name Enter a name for the Logtail configuration.
    Cluster Name Enter the name of the Kafka cluster. After you configure this parameter, Log Service adds a cluster=<Cluster name> tag to the data that is collected.
    Note Make sure that the cluster name is unique. Otherwise, data conflicts may occur.
    Server List Click the + icon to add a Kafka server.
    • Address: the address of the server.
    • Port: the port number of the server. You can specify a custom port number. The port number must be the same as the port number that you specify in Step 2: Configure JavaAgent.

    You can add multiple servers based on your business requirements.

    Custom Tags Create a custom tag in Custom Tags. The created tag is added to the data that is collected based on the Logtail configuration. You can use custom tags to identify the data that is collected based on different Logtail configurations in a Metricstore.

    You can click the + icon to create a custom tag. You can create multiple custom tags. The custom tags are added to all data that is collected based on Logtail configurations.

Step 2: Configure JavaAgent

After the Logtail configuration is created, you must enable access to JMX data over HTTP. Log Service allows you to use Jolokia to access JMX data over HTTP. You can download and load Jolokia based on the official documentation of Jolokia. You can also use Jolokia JavaAgent that is provided together with Logtail in Log Service. Jolokia JavaAgent is stored in /etc/ilogtail/telegraf/javaagent/jolokia-jvm.jar.

You must configure the KAFKA_JVM_PERFORMANCE_OPTS environment variable on your Kafka server. For example, specify export KAFKA_JVM_PERFORMANCE_OPTS=-javaagent:/etc/ilogtail/telegraf/javaagent/jolokia-jvm.jar=port=7777. In this example, 7777 indicates the port number of your Kafka server, and the port number is the same as the port number that you specify in Step 1: Create a Logtail configuration.
Note By default, Jolokia JavaAgent listens only on the IP address 127.0.0.1 and allows requests only from the local host. If Logtail and your Java application are installed on different servers, you can add the host= field to the added script. This way, Jolokia JavaAgent can listen on other IP addresses. If you add host=0.0.0.0, Jolokia JavaAgent listens on all IP addresses. Example:
-javaagent:/tmp/jolokia-jvm.jar=port=7777,host=0.0.0.0
After the configuration is complete, you must restart your Java application. If your Java application fails to restart, run the following command to connect Jolokia JavaAgent to a specified Java process. This way, the configuration immediately takes effect. Replace PID with the actual value.
Note This command is used only for testing. You must complete the configuration based on preceding descriptions. Otherwise, the configuration becomes invalid after your application restarts. 
java -jar /etc/ilogtail/telegraf/javaagent/jolokia-jvm.jar --port 7777 start PID
If the following information is returned, the connection is successful. 
Jolokia is already attached to PID 752
http://127.0.0.1:7777/jolokia/
After the connection is established, you can access the following URL to verify the connection. 
curl http://127.0.0.1:7777/jolokia/
# Sample response
{"request":{"type":"version"},"value":{"agent":"1.6.2","protocol":"7.2","config":{"listenForHttpService":"true","maxCollectionSize":"0","authIgnoreCerts":"false","agentId":"30.43.124.186-752-5b091b5d-jvm","debug":"false","agentType":"jvm","policyLocation":"classpath:\/jolokia-access.xml","agentContext":"\/jolokia","serializeException":"false","mimeType":"text\/plain","maxDepth":"15","authMode":"basic","authMatch":"any","discoveryEnabled":"true","streaming":"true","canonicalNaming":"true","historyMaxEntries":"10","allowErrorDetails":"true","allowDnsReverseLookup":"true","realm":"jolokia","includeStackTrace":"true","maxObjects":"0","useRestrictorService":"false","debugMaxEntries":"100"},"info":{"product":"tomcat","vendor":"Apache","version":"8.5.57"}},"timestamp":1602663330,"status":200}⏎

FAQ

How can I check whether Telegraf collects data as expected?

You can check the logs of the /etc/ilogtail/telegraf/telegraf.log file on your server. You can also collect this log file to Log Service and search for the required information.

What to do next

  • Search and analyze

    After you complete the settings, Telegraf uses the Logtail to upload collected metrics to the Metricstore of Log Service. You can query the data on the Search & Analysis page of the Metricstore. For more information, see Query and analyze time series data.

  • Visualization

    After the configuration for your Kafka server is complete, Log Service automatically creates a dashboard named Kafka Monitoring Data_<Cluster name> in the project that is used. In the dashboard, you can perform various operations. For example, you can view query results and configure alerts.