Collect monitoring data from Kubernetes JVMs - Log Service

Data monitoring for Kubernetes Java virtual machines (JVMs) is a flexible and end-to-end metric collection solution for all JVM-based services in Kubernetes clusters. Data monitoring is supported on top of Kubernetes-adapted Logtail configurations and automatic service discovery.

Prerequisites

An instance is created. For more information, see Create an instance.
A monitoring component is installed. For more information, see Install a monitoring component.

Note You must install a Logtail image of V1.2.2.0 or later by using the full-stack monitoring Kubernetes component. The tag added to the Logtail image must be in the v1.2.2.0-monitor-aliyun format.

Data collection processes

The Java Management Extensions (JMX) technology is a standard part of the Java SE platform. JMX is mainly used to monitor and manage information such as the status of Java applications and JVM servers and resources. JMX supports Remote Method Invocation (RMI) for data ingestion, which functions in a way similar to a Prometheus exporter. Metric read is more secure and data integration is more cost-effective by calling RMI than by using a Java agent.

In Kubernetes clusters, the DaemonSet deployment model is used for both JVM metric collection and log collection components. When you collect monitoring data from JVMs, you can filter the data by using environment variables, Kubernetes labels, namespaces, pod names, and container names. Data filtering is implemented by using log collection components that support multi-dimensional selection of Kubernetes resources. RMI involves communication between Java processes, and Kubernetes resource selection may hit a large number of unrelated processes. In this case, Logtail introduces a special environment variable ILOGTAIL_JMX_PORT to further narrow the scope of data.

Preparations

If you want to use Logtail to automatically collect monitoring data from JVMs for Java applications deployed in Kubernetes clusters, you must perform the following operations:

Add the POD_IP declaration to your environment variables.

Configuration example:

            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

Configure JVM startup parameters.
Configure the following com.sun.management.jmxremote parameters to enable JMX remote access:
- Set the com.sun.management.jmxremote.local.only parameter to false.
- Set both the com.sun.management.jmxremote.ssl and com.sun.management.jmxremote.authenticate parameters to false. Otherwise, you must configure the JMX Password parameter in your Logtail configuration.
- Set the com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port parameters to the declared RMI port.
- Set the java.rmi.server.hostname parameter to the value of the POD_IP environment variable.
```
   -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.authenticate=false
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.local.only=false
    -Dcom.sun.management.jmxremote.port=9999
    -Dcom.sun.management.jmxremote.rmi.port=9999
    -Djava.rmi.server.hostname=$(POD_IP)
```
Add the ILOGTAIL_JMX_PORT declaration to your environment variables. The declaration specifies the exposed metric port and is indicated by the value of the com.sun.management.jmxremote.port parameter.

After you complete the preceding operations, you can connect to the required pod and run the jps command to view JVM startup parameters. The JAVA_TOOL_OPTIONS="" declaration in the jps command is used to avoid port conflicts. If the environment variable JAVA_TOOL_OPTIONS is introduced when you run the jps command, port conflicts may occur. JVM monitoring

Procedure

Log on to the Log Service console.
In the Log Application section, click Full-stack Monitoring.
On the Full-stack Monitoring page, click the instance.
On the Data Import page, select Kubernetes JVM Monitoring.

If this is your first time to create a Logtail configuration for host monitoring, turn on the switch to go to the configuration page. If you have created a Logtail configuration, click the icon to go to the configuration page.
Click Use Existing Machine Groups.
After the monitoring component is installed, Log Service automatically creates a machine group whose name is in the {Instance ID}-{Cluster ID}-k8s-nodes format. You can directly use this machine group.
Select the machine group, move the machine group from the Source Server Groups section to the Applied Server Groups section, and then click Next.

Important If the heartbeat status of the machine group is FAIL, you can click Automatic Retry. If the issue persists, see What do I do if a Logtail machine group has no heartbeats?

In the Specify Data Source step, configure the following parameters and click Complete.

After you complete the configurations, Log Service automatically creates assets such as Metricstores. For more information, see Assets.


Parameter	Description
General configurations
Config Name	The name of the Logtail configuration. You can enter a custom name.
Cluster	The name of the Kubernetes cluster. You can enter a custom name. After you configure this parameter, Log Service adds a `cluster=Cluster name` tag to the Kubernetes JVM monitoring data that is collected by using the Logtail configuration. Important Make sure that the cluster name is unique. Otherwise, data conflicts may occur.
JMX Username	The username that is used to access the JMX server. We recommend that you do not configure this parameter in Kubernetes clusters.
JMX Password	The password that is used to access the JMX server. We recommend that you do not configure this parameter in Kubernetes clusters.
Global Tags	The custom tags that are added to the collected Kubernetes JVM monitoring data. The tags are key-value pairs. After you configure this parameter, Log Service adds the custom tags to the JVM monitoring data that is collected by using the Logtail configuration.
Configurations for K8s Selector
Namespace	The regular expression used to match the names of namespaces from which data is collected.
Pod Name	The regular expression used to match the names of pods from which data is collected.
Container Name	The regular expression used to match the names of containers from which data is collected.
Environment Variable Whitelist	The environment variable whitelist. The whitelist specifies the containers from which data is collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the environment variable whitelist, the EnvKey parameter is required, and the EnvValue parameter is optional. If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are matched. If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are matched. By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to `NGINX_SERVICE_PORT` and set the EnvValue parameter to `^(80\|6379)$`, containers whose port number is 80 and containers whose port number is 6379 are matched. Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is matched.
Environment Variable Blacklist	The environment variable blacklist. The blacklist specifies the containers from which data is not collected. This parameter is empty by default, which indicates that data is collected from all containers. When you configure the environment variable blacklist, the EnvKey parameter is required, and the EnvValue parameter is optional. If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are filtered out. If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are filtered out. By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to `NGINX_SERVICE_PORT` and set the EnvValue parameter to `^(80\|6379)$`, containers whose port number is 80 and containers whose port number is 6379 are matched. Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is filtered out.
K8s Label Whitelist	The Kubernetes label whitelist. The whitelist specifies the containers from which data is collected. When you configure the Kubernetes label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are matched. If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are matched. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`), regular expression matching is performed. For example, if you set the LabelKey parameter to `app` and set the LabelValue parameter to `^(test1\|test2)$`, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched. Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is matched.
K8s Label Blacklist	The Kubernetes label blacklist. The blacklist specifies the containers from which data is not collected. When you configure the Kubernetes label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are filtered out. If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`), regular expression matching is performed. For example, if you set the LabelKey parameter to `app` and set the LabelValue parameter to `^(test1\|test2)$`, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched. Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is filtered out.
Container Label Whitelist	The container label whitelist. The whitelist specifies the containers from which data is collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the container label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are matched. If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are matched. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to `io.kubernetes.container.name` and set the LabelValue parameter to `^(nginx\|cube)$`, a container named nginx and a container named cube are matched. Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is matched.
Container Label Blacklist	The container label blacklist. The blacklist specifies the containers from which data is not collected. This parameter is empty by default, which indicates that data is collected from all containers. When you configure the container label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are filtered out. If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to `io.kubernetes.container.name` and set the LabelValue parameter to `^(nginx\|cube)$`, a container named nginx and a container named cube are matched. Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is filtered out.
Advanced configurations
Collect Kubernetes Labels	After Kubernetes labels are configured, Log Service adds Kubernetes label-related fields to the collected JVM monitoring data. For example, set the name of the label to `app` and the alias of the name to `k8s_label_app`. If a Kubernetes cluster contains a label named `app`, such as `app=serviceA`, Log Service adds the field `k8s_label_app: serviceA` to the JVM monitoring data.
Collect Environment Variable	After environment variable labels are configured, Log Service adds environment variable label-related fields to the collected JVM monitoring data. For example, set the name of the environment variable to `VERSION` and the alias of the name to `env_version`. If a container contains an environment variable named `VERSION`, such as `VERSION=v1.0.0`, Log Service adds the field `env_version: v1.0.0` to the JVM monitoring data.
JMX Garbage Collector of New Version	After you turn on the switch, JMX garbage collectors of the new versions such as G1 are supported.
Default JVM Metrics	After you turn on the switch, Logtail collects predefined core JVM metrics.
JMX Metric Collection Filter	The filtering and collection by using JMX bean ranges, JMX bean regular expressions, JMX bean types, and JMX attribute ranges are supported.

Configuration example

For example, you can start the SpringBoot Demo service in a namespace named default. Then, you can use the ILOGTAIL_JMX_PORT declaration to expose port 9999. In the environment variable JAVA_TOOL_OPTIONS, you can add JVM startup parameters for JMX metrics that correspond to the exposed port 9999.

Note Before you use the environment variable JAVA_TOOL_OPTIONS to reference the environment variable POD_IP, you must define the environment variable POD_IP.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jvm-new-1
  namespace: default
spec:
...
  template:
    metadata:
      labels:
        app: jvm1
    spec:
      containers:
        - image: evanljp/full-stack:jvm-new-0.1
          name: jvm
          env:
            - name: "ILOGTAIL_JMX_PORT"
              value: "9999"
            - name: "ILOGTAIL_JMX_TAGS"
              value: "a=b,c=d"
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: JAVA_TOOL_OPTIONS
              value: >-
                -Dcom.sun.management.jmxremote
                -Dcom.sun.management.jmxremote.authenticate=false
                -Dcom.sun.management.jmxremote.ssl=false
                -Dcom.sun.management.jmxremote.local.only=false
                -Dcom.sun.management.jmxremote.port=9999
                -Dcom.sun.management.jmxremote.rmi.port=9999
                -Djava.rmi.server.hostname=$(POD_IP)
                -Dserver.port=8081
          ...

After you specify the preceding declarations, you need to only complete the following settings in your Logtail configuration to collect monitoring data from Kubernetes JVMs to Log Service.

What to do next

After Kubernetes JVM monitoring data is collected and sent to the Full-stack Monitoring application, the application automatically creates dedicated dashboards for the monitoring data. You can use the dashboards to analyze the monitoring data. For more information, see View dashboards.