ApsaraMQ for Kafka:Troubleshoot ApsaraMQ for Kafka client errors

Error message

Client language

Cause

Solution

TimeoutException

Java

• Network issue

• Client authentication failure (sasl.mechanisms).

• Ensure that the servers parameter is configured correctly.

• Run the telnet command to rule out network issues.

• If the network is working, check your authentication configuration by referring to the following documents:

  • Grant permissions to RAM users

  • ApsaraMQ for Kafka demos

  Note

  This solution applies only to public instances of ApsaraMQ for Kafka.

run out of brokers

Go

Authentication failed for user

Python

Leader is not available

All

This error may occur temporarily when a topic is being initialized. If the error persists, the topic may not have been created.

1. Log on to the ApsaraMQ for Kafka console.

2. Check whether the topic has been created.

3. If the topic does not exist, create it. For more information, see Step 1: Create a topic.

leader is in election

array index out of bound exception

Java

Spring Cloud parses message content in its own format.

Use one of the following solutions:

• Recommended: Use Spring Cloud to both produce and consume messages.

• If you send messages by using other methods, such as by calling the native Java client, you must set headerMode to raw when you consume the messages by using Spring Cloud. This setting disables the parsing of message content. For more information, see the Spring Cloud official website.

No such configuration property: "sasl.mechanisms"

• C++

• Clients that wrap C++, such as PHP and Node.js.

The SASL and SSL modules are not installed or are installed incorrectly.

Run the following commands to install the SASL and SSL modules:

• Install SSL: sudo yum install openssl openssl-devel

• Install SASL: sudo yum install cyrus-sasl{,-plain}

No worthy mechs found

No KafkaClient Entry

Java

Could not find the kafka_client_jaas.conf configuration file.

Create the kafka_client_jaas.conf file and save it to an accessible directory, such as /home/admin. Java security login settings are system-wide and can be configured in one of the following ways:

• Set a system property:

  • Set a JVM parameter: -Djava.security.auth.login.config=/home/admin/kafka_client_jaas.conf

  • Set the property in your code: System.setProperty("java.security.auth.login.config","/home/admin/kafka_client_jaas.conf")

    Note

    If you set this property in your code, ensure it executes before the client starts.

• Modify the Java security properties file: In the ${JAVA_HOME}/jre/lib/java.security file, add the following line: login.config.url.1=file:/home/admin/kafka_client_jaas.conf

For more information, see the Oracle JAAS Login Configuration File.

Error sending fetch request

Java

The consumer failed to fetch messages. Possible causes include:

• Network issue

• Fetch timeout

1. Ensure that the servers parameter is configured correctly.

2. Run the telnet command to rule out network issues.

3. If the network is not the issue, a fetch timeout may be the cause. To resolve this, tune the following parameters to limit the data returned per fetch request:

   • fetch.max.bytes: The maximum number of bytes returned by the server for a single fetch request.

   • max.partition.fetch.bytes: The maximum number of bytes that the server returns for a single partition in a single fetch operation.

4. Check for server-side traffic throttling. In the ApsaraMQ for Kafka console, go to the Instance Details page and check the relevant metrics.

   • For VPC access, check the peak traffic.

   • For public access, check the Public Traffic.

DisconnectException

CORRUPT_MESSAGE

All

• For the cloud storage engine: The cloud storage engine does not support idempotence, which is enabled by default in Kafka clients version 3.0 and later.

• For the local storage engine: A compacted message was sent without a key.

• If you use a cloud storage engine, set enable.idempotence=false on the client.

• For the local storage engine: Add a key to the message.

read tcp i/o timeout

Go (or other)

The network connection between the consumer and the broker timed out, the consumer group offset was not correctly initialized, or a Rebalance issue occurred.

1. Verify network connectivity between the client and the Kafka instance.

2. Check whether the client version is earlier than 0.10.2. If yes, upgrade the client.

3. Adjust the max.poll.interval.ms and session.timeout.ms parameters.

4. Log on to the console to check the consumer status and verify whether the message accumulation and offsets are normal.

Connection terminated during authentication

All

Authentication was interrupted when an ACK cluster or other environment connected to Kafka Serverless. This is typically caused by incorrect protocol configuration, insufficient permissions, or network policy restrictions.

1. Verify that the RAM user is properly authorized.

2. Check whether the client IP address is in the allowlist of the Kafka instance.

3. Ensure that the ACK cluster and the Kafka instance are in the same VPC.

4. Check whether the security group allows traffic on port 9094.

5. Confirm that the ACL policy grants the SASL user access to the relevant topics and consumer groups.

6. Try switching endpoints or ports to test connectivity, and check whether the client protocol configuration is correct.

Connection to node -1 could not be established

All

The client cannot establish a connection to the Kafka node. This is typically caused by network issues, the allowlist not being configured, or incorrect public network access configuration.

1. Check network connectivity: Verify that the client machine (such as an ECS instance) and the Kafka instance are in the same VPC. If they are not in the same VPC or if you are using local access, ensure that the network is connected.

2. Check the allowlist configuration: Verify that the egress IP address of the client has been added to the allowlist of the Kafka instance. You can run a telnet command to test connectivity to the endpoint domain name and port (such as port 9092).

3. Configure public network access: If you need to use public network access, the network type of the instance must be Internet/VPC (VPC instances must be upgraded to support Internet access). You must use the SSL endpoint (port 9093). In the client code, configure the public network endpoint and SSL certificate, set security.protocol to SASL_SSL, and keep the username and password unchanged.