Troubleshoot common issues - Application Real-Time Monitoring Service

When you connect Grafana to Application Real-Time Monitoring Service (ARMS) through the Tempo-compatible API, you may encounter connection or authentication errors. This page explains each error, its root cause, and how to fix it.

Error summary

Error message	Root cause	Section
`Unable to connect with Tempo`	Incorrect endpoint URL or unsupported region	Unable to connect with Tempo
`Authentication failed`	Mismatched AccessKey credentials or missing permissions	Authentication failed
`Trace Logstore Not Found`	Incorrect UserId or RegionId in HTTP headers	Trace Logstore not found

Before you begin

To troubleshoot effectively, have the following information ready:

The endpoint URL configured in your Grafana Tempo data source
The AccessKey ID and AccessKey secret used for authentication
The Alibaba Cloud account ID (UserId) set in the HTTP headers

For the complete list of supported regions and endpoints, see Compatibility between ARMS and Grafana Tempo.

Unable to connect with Tempo

Full error:

Unable to connect with Tempo. Please check the server logs for more details.

Cause: The endpoint URL is incorrect or the Tempo-compatible API is not available in the specified region.

Diagnosis:

Open the Grafana Tempo data source settings and check the URL field.
Confirm that the region in your URL is listed in the supported regions.

Fix:

Verify that the endpoint is in the correct format and that the Tempo-compatible API is supported in the region. Click Save & Test in Grafana to verify the connection.

Verification: A successful connection displays a confirmation message in the Grafana data source settings page.

Authentication failed

Full error:

Unable to connect with Tempo ([Authentication failed] The userId in HTTP header does not match the credentials (username/password). TraceId=xxxx). Please check the server logs for more details.

Cause: The AccessKey ID and AccessKey secret do not match, or the AccessKey ID lacks the required permissions.

Diagnosis:

Open the Grafana Tempo data source settings.
Under Basic auth, check that the User field contains your AccessKey ID and the Password field contains your AccessKey secret.
Verify that the AccessKey pair belongs to the same Alibaba Cloud account specified in the userId HTTP header.

Fix:

Correct the User and Password fields in the Grafana Tempo data source:
- User: Your AccessKey ID
- Password: Your AccessKey secret
If the credentials are correct, confirm that the AccessKey ID has the required permissions.
Click Save & Test in Grafana to verify the connection.

Verification: If authentication succeeds, Grafana displays a success message.

Trace Logstore not found

Full error:

Unable to connect with Tempo ([Trace Logstore Not Found] Please verify the UserId and RegionId in HTTP headers are correct, and ensure the trace logstore exists. TraceId=xxxxx). Please check the server logs for more details.

Cause: The UserId in the HTTP header does not match a valid Alibaba Cloud account, or the trace Logstore does not exist in the specified region.

Diagnosis:

Open the Grafana Tempo data source settings and locate the Custom HTTP Headers section.
Check the userId header value. This must be your Alibaba Cloud account ID.
Check the regionId header value. This must match the region where your ARMS trace data is stored.

Fix:

Verify that the Alibaba Cloud account is correct by setting the userId header to the Alibaba Cloud account ID that owns the ARMS resource.
Set the regionId header to the region where your trace Logstore is located, such as cn-hangzhou.
Click Save & Test in Grafana to verify the connection.

Verification: A successful connection confirms that the UserId and RegionId values are correct.