Troubleshooting guide for no data from Java applications connected to ARMS in an ACK environment

Scenario

This document provides a comprehensive guide to troubleshooting the issue of a Java application not reporting data to ARMS in a Container Service for Kubernetes (ACK) environment.

Troubleshooting overview

• Basic checks:

  • Whether data collection is disabled.

  • Whether the license is a trial version.

  • Whether the application is receiving traffic.

• ACK connection checks:

  • Whether the initContainer is injected correctly.

  • Whether the pod YAML labels are configured correctly.

  • Whether the application was started after ack-onepilot was installed.

  • Whether an add-on is configured in the Kubernetes cluster.

  • The application's JVM heap size configuration.

• Special scenario troubleshooting:

  • Whether the application startup command includes a `sudo -u` user switch.

• Network connectivity checks:

  • Whether there are network connectivity issues.

Troubleshooting steps

1. Basic checks (perform first)

1.1 Check basic console configurations

1. Check the data collection switch

   • In the ARMS console, choose Application List in the navigation pane. In the upper-right corner, click Settings and then select Application On/Off Settings.

   • Verify that data collection is stopped.

2. Check the main agent switch

   • Check if the main agent switch is turned off.

3. Check the trial period

   • If you are using a trial version, confirm that it has not expired.

1.2 Check JVM monitoring data

• On the application details page, click the JVM Monitoring tab.

• If no JVM monitoring data is displayed: The node might be disconnected from the console. Check the network connectivity.

• If JVM monitoring data is displayed but interface monitoring data is not:

  • Confirm that the application is receiving traffic.

  • Confirm that the component versions used in the application are supported by ARMS. For more information, see Supported Java components and frameworks.

  • If the agent version is earlier than 2.7.3.5, upgrade the agent.

2. ACK connection checks

2.1 Check if the initContainer is injected

In the Kubernetes cluster, run the following command:

kubectl get {podName} -n {namespace} -oyaml

Check the pod's YAML configuration for an init container named one-pilot-initcontainer.

• If the init container exists: The initContainer is injected correctly. Proceed to step 2.2.

• Otherwise, go to step 2.3.

2.2 Check the initContainer logs

View the initContainer logs to check for abnormal exits:

kubectl logs {podName} -n {namespace} -c one-pilot-initcontainer

Normal case: The logs should contain the following message, which indicates that the initContainer finished correctly and the agent was mounted successfully:

[init] ack-onepilot k8s init ended. cost: 12.90225787s, error: <nil>

Abnormal cases:

• If the logs contain an error message, troubleshoot the issue based on the error.

• If the error message Error: container has runAsNonRoot and image will run as root is displayed, the pod is configured to run as a non-root user. Remove the securityContext.runAsNonRoot: true configuration.

2.3 Check application label configurations

Check the application's YAML configuration to confirm that the required labels are added:

labels: armsPilotAutoEnable: "on" armsPilotCreateAppName: "your-app-name"

Alternatively, you can configure them in the annotations:

annotations: armsPilotAutoEnable: "on" armsPilotCreateAppName: "your-app-name"

• If the labels are missing: Add the required labels. For more information, see Install the ARMS agent for a Java application in an ACK cluster.

• If the labels exist: Proceed to step 2.4.

2.4 Check the application restart time

Confirm that the application was restarted after ack-onepilot was installed:

• If the application was restarted before ack-onepilot was installed: Restart the application to allow ack-onepilot to reinject the agent.

• If the application was restarted after ack-onepilot was installed: Proceed to step 2.5.

2.5 Check the ack-onepilot logs

View the ack-onepilot pod logs to check for error messages:

# View the ack-onepilot pod
kubectl get pod -n ack-onepilot
# View the logs of a specific pod
kubectl logs {podName} -n ack-onepilot

Common errors and solutions:

1. SDK error: No corresponding permission

   • For an ACK cluster: Check if an add-on is configured in the cluster. For more information, see What do I do if the agent fails to be installed for an application in a Container Service for Kubernetes cluster?.

   • If a point of presence (POP) call fails, check the network connectivity of the relevant POP domain name.

2. Taints are set on nodes in the cluster

   • If taints are set on nodes in the cluster, the Pilot may fail to be scheduled and started. This also causes the connection to fail.

   • You can adjust the node's taint configuration or the pilot's toleration configuration.

3. Network access is blocked

   • If the logs contain the message Public network access is not permitted, the POP request to create the application was blocked for security reasons.

   • Connect using the component center. Do not install the agent using Helm and manually entering an AccessKey pair.

2.6 Check the agent log directory

Enter the application container and check if the agent log directory exists:

# Enter the pod container
kubectl exec -it {podName} -n {namespace} -- /bin/bash
# Check the agent directory
ls -la /home/admin/.opt/AliyunJavaAgent/logs
# Or
ls -la /home/admin/.opt/ArmsAgent/logs

If the directory does not exist, the possible causes are as follows:

1. Out of memory

   • Check the first few lines of the application container logs for errors related to heap memory.

   • The JVM parameter cannot be set to a value that is too small. The heap memory must be greater than 500 MB to use ARMS for monitoring.

2. The JAVA_TOOL_OPTIONS environment variable is too long

   • The JAVA_TOOL_OPTIONS environment variable exceeds the maximum length of 1,024 bytes specified by Java.

   • Check if multiple pilots are injected at the same time, which causes the environment variable to be too long.

3. Troubleshoot special scenarios

3.1 Check for user switching with sudo

Symptom: The init-container is injected successfully and the JAVA_TOOL_OPTIONS environment variable is injected into the pod, but no data is reported.

Troubleshooting method:

1. You can enter the Pod container and check the probe directory for a log directory.

2. If no log directory exists, check if the script used to start the application contains an action that switches users, such as sudo -u xxx.

Solution:

• Add the USER xxx:xxx command to the Dockerfile to switch the pod's default user.

• Reason: If you run the sudo -u command after the container starts, the environment variables that are injected into the application pod by ack-onepilot are lost after the user switch.

3.2 Check if arms-pilot is also installed

If both arms-pilot and ack-onepilot (version 3.1.0 or later) are installed, conflicts occur.

Check method:

kubectl get ns | grep -E "arms-pilot|mse-pilot"

Solution:

• Uninstall arms-pilot or mse-pilot.

• For more information, see How do I install ack-onepilot and uninstall arms-pilot?.

3.3 Check for both manual injection and ACK connection

If you connect to ARMS manually and also use the ACK connection method, two agents are injected.

Check method: In the application container, run java -version. Check the JAVA_TOOL_OPTIONS in the standard output. If two -javaagent parameters appear, a duplicate injection has occurred.

Solution:

• Choose either the manual connection method or the ACK connection method. We recommend that you use the ACK connection method.

• If you choose the ACK connection method, remove the manual connection configuration.

3.4 Check the agent version

Check method:

# Enter the pod container
kubectl exec -it {podName} -n {namespace} -- /bin/bash
# Check the agent directory
ls -la /home/admin/.opt/AliyunJavaAgent/logs
# Or
ls -la /home/admin/.opt/ArmsAgent/logs

1. If a version file exists, run cat version to view the agent version.

2. If a version file does not exist, view the first few lines of the application container logs. Search for Load Aliyun Java Agent version to find the agent version that is loaded by the application.

Version requirements:

• For agent versions earlier than 2.7.3.5, you may need to provide agent logs for troubleshooting.

• We recommend upgrading to the latest version.

4. Network and permission checks

4.1 Check network connectivity

1. Check network connectivity between the node and the ARMS server-side

   • Log on to the node where the application is located.

   • Test the network connectivity with the ARMS server-side. For more information, see Automatically check network connectivity.

2. Check the network connectivity of the POP call domain name

   • If the ack-onepilot logs contain errors related to POP calls, check the network connectivity of the relevant domain names.

4.2 Check AccessKey pair permissions

1. Confirm the AccessKey pair type

   • Self-managed Kubernetes or registered clusters: You must use the AccessKey pair of an Alibaba Cloud account. The AccessKey pair of a RAM user cannot be used to report data.

2. Check AccessKey pair permissions

   • Confirm that the AccessKey pair has the permission to create applications.

   • If you are connecting using a registered cluster, check whether the correct accessKey and accessSecretKey are entered in the helm value.yaml file that corresponds to ack-onepilot.

4.3 Check the license key

If the agent logs show a license key is invalid error:

1. First-time connection: The server-side role creation and the application creation are parallel processes. You may need to wait for a moment and then restart the application.

2. Manual connection: Check if the license key is entered correctly.

3. Automatic connection: Check if you have overwritten the license key that is automatically populated by ARMS. You may have mistakenly used a license key from another region.

4. Open source Kubernetes connection: Confirm that the AccessKey pair entered in the values.yaml file of the one-pilot installation package belongs to the specified user ID (UID).

5. Analyze agent logs

If the issue persists after you perform the preceding checks, you must view the agent logs for a detailed analysis:

5.1 Obtain agent logs

Probe logs:

• /home/admin/.opt/AliyunJavaAgent/logs

• /home/admin/.opt/ArmsAgent/logs (for earlier versions)

5.2 Common log errors

1. license key is invalid: For more information, see step 4.3.

2. responsebody_StoreFailed: Data reporting may have failed. Check the network and permissions.

3. agent library failed to init: The agent failed to initialize. The agent package may be corrupted or the path may be incorrect.

6. Other possible causes

6.1 No application traffic

• Confirm that the application is receiving HTTP request traffic.

• You can use the Arthas tool to monitor whether the interface has traffic.

6.2 Unsupported component version

• Confirm that the component versions used in the application are supported by ARMS. For more information, see Supported Java components and frameworks.

6.3 Outdated agent version

• If the agent version is earlier than 2.7.3.5, upgrade the agent.

• For upgrade instructions, see Update the ARMS agent for Java applications.

7. Debugging tips

7.1 Enable debug logs

Add the following environment variable to the ack-onepilot configuration:

ONEPILOT_LOG_LEVEL: "debug"

7.2 Check JAVA_TOOL_OPTIONS

Run the following command in the application container:

echo $JAVA_TOOL_OPTIONS

Confirm that the environment variable is injected correctly and includes the -javaagent parameter.

7.3 Check the probe file

In the application container, check if the agent file exists by running the following command:

ls -la /home/admin/.opt/AliyunJavaAgent/
# Or
ls -la /home/admin/.opt/ArmsAgent/