Collect monitoring data from Kafka servers

Step 1: Create a Logtail configuration

1. Log on to the Log Service console.
2. In the Log Application section, click Full-stack Monitoring.
3. On the Full-stack Monitoring page, click the instance.
4. On the Data Import page, enable Kafka.

   If this is your first time to create a Logtail configuration for host monitoring, turn on the switch to go to the configuration page. If you have created a Logtail configuration, click the icon to go to the configuration page.

5. In the Install Logtail step, select the machine on which you want to install Logtail and click Next.
   • If you want to install Logtail on an Elastic Compute Service (ECS) instance, select the ECS instance on the ECS Instances tab and click Execute Now. For more information, see Install Logtail on ECS instances.
   • If you want to install Logtail on a self-managed Linux server or a Linux server from a third-party cloud, you must manually install Logtail V0.16.48 or later on the server. For more information, see Install Logtail on a Linux server.

   Notice Make sure that the server on which you want to install Logtail can connect to the Kafka server whose metric data you want to collect.
6. In the Create Machine Group step, create a machine group and click Next.
   Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom ID-based machine group.
7. In the Machine Group Settings step, select the machine group that you create in the Source Server Groups section and move the machine group to the Applied Server Groups section. Then, click Next.
   Notice If you immediately apply a machine group after it is created, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. In this case, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
8. In the Specify Data Source step, configure the following parameters and click Complete.

   Parameter	Description
   Configuration Name	The name of the Logtail configuration. You can enter a custom value.
   Cluster Name	The name of the Kafka cluster. You can enter a custom value. After you configure this parameter, Log Service adds a cluster=Cluster name tag to the Kafka monitoring data that is collected by using the Logtail configuration. Notice Make sure that the cluster name is unique. Otherwise, data conflicts may occur.
   Server List	The information about the Kafka server. The information includes the following configuration items: Address: the address of the Kafka server. You can enter the IP address, hostname, or domain name of the server. Port: the port number of the Kafka server. Default value: 7777. You can add information about multiple Kafka servers based on your business requirements.
   Custom Tags	The custom tags that are added to the collected Kafka monitoring data. The tags are key-value pairs. After you configure this parameter, Log Service adds the custom tags to the Kafka monitoring data that is collected by using the Logtail configuration.

   After you complete the configurations, Log Service automatically creates assets such as Metricstores. For more information, see Assets.

Step 2: Configure JavaAgent

After you create the Logtail configuration, you must configure JavaAgent on the Kafka server. Log Service allows you to use Jolokia to configure JavaAgent. For more information, see Jolokia. You can download and use Jolokia based on the official documentation of Jolokia. You can also use Jolokia JavaAgent that is provided together with Logtail in Log Service. Jolokia JavaAgent is stored in /etc/logtail/telegraf/javaagent/jolokia-jvm.jar.

1. Configure the environment variable KAFKA_JVM_PERFORMANCE_OPTS for the Kafka server.
   For example, specify export KAFKA_JVM_PERFORMANCE_OPTS=-javaagent:/etc/logtail/telegraf/javaagent/jolokia-jvm.jar=port=7777. 7777 is the port number of the server. Make sure that the port number is the same as the port number that you specify in Step 1: Create a Logtail configuration.

   Note By default, Jolokia JavaAgent listens only on the IP address 127.0.0.1 and allows requests only from the local host. If Logtail and your Java application are installed on different servers, you can add the host= field to the added script. This way, Jolokia JavaAgent can listen on other IP addresses. If you add host=0.0.0.0, Jolokia JavaAgent listens on all IP addresses. Example:

   -javaagent:/tmp/jolokia-jvm.jar=port=7777,host=0.0.0.0

2. Restart your Java application.
   If your Java application fails to restart, run the following command to connect Jolokia JavaAgent to a specified Java process. This way, the configuration immediately takes effect. Replace Java PID with the actual value.

   Notice This operation is used only for testing. In actual scenarios, you must complete the configuration based on preceding descriptions. Otherwise, the configuration becomes invalid after your application restarts.

   java -jar /etc/ilogtail/telegraf/javaagent/jolokia-jvm.jar --port 7777 start Java PID

   If information similar to the following code is returned, the connection is successful:

   Jolokia is already attached to PID 752
   http://127.0.0.1:7777/jolokia/

3. Access the following URL to verify the connection:

   curl http://127.0.0.1:7777/jolokia/

   If information similar to the following code is returned, the connection is normal:

   {"request":{"type":"version"},"value":{"agent":"1.6.2","protocol":"7.2","config":{"listenForHttpService":"true","maxCollectionSize":"0","authIgnoreCerts":"false","agentId":"30.43.124.186-752-5b091b5d-jvm","debug":"false","agentType":"jvm","policyLocation":"classpath:\/jolokia-access.xml","agentContext":"\/jolokia","serializeException":"false","mimeType":"text\/plain","maxDepth":"15","authMode":"basic","authMatch":"any","discoveryEnabled":"true","streaming":"true","canonicalNaming":"true","historyMaxEntries":"10","allowErrorDetails":"true","allowDnsReverseLookup":"true","realm":"jolokia","includeStackTrace":"true","maxObjects":"0","useRestrictorService":"false","debugMaxEntries":"100"},"info":{"product":"tomcat","vendor":"Apache","version":"8.5.57"}},"timestamp":1602663330,"status":200}