All Products
Search
Document Center

E-MapReduce:Vulnerability notice | RCE vulnerability in Apache Kafka Connect

Last Updated:Mar 25, 2024

On February 8, 2023, Apache released a security notice about fixing a deserialization vulnerability in Apache Kafka. The Common Vulnerabilities and Exposures (CVE) ID of the vulnerability is CVE-2023-25194. In the case where an attacker can control the Apache Kafka Connect client, the attacker can use SASL JAAS configurations and SASL-based security protocols to create or modify connectors on the client. This will trigger the Java Naming and Directory Interface (JNDI) code execution vulnerability.

Impacts

  • Affected Apache Kafka versions: 2.3.0 to 3.3.2

  • Affected E-MapReduce (EMR) versions:

    • EMR 5.X series: versions earlier than EMR V5.16.0

    • EMR 3.X series: versions earlier than EMR V3.50.0

    Note

    The vulnerability is fixed in EMR V3.50.0, EMR V5.16.0, or a minor version later than EMR V3.50.0 or EMR V5.16.0. In these versions, you do not need to manually fix the vulnerability.

Fixing procedure

The method of fixing the vulnerability varies based on the type of the cluster in which Kafka is deployed.

Dataflow cluster

  1. Download a Kafka software package with the vulnerability fixed.

  2. Log on to the master node of your cluster and upload the Kafka software package that is downloaded in the previous step to the /opt/apps/KAFKA/ directory of the master node.

    For information about how to log on to a cluster, see Log on to a cluster.

  3. Run the following commands to copy the Kafka software package to the /opt/apps/KAFKA directory of the core-1-1 node.

    In a minor version earlier than EMR V5.16.0

    sudo su - emr-user
    ssh core-1-1 'sudo chmod a+w /opt/apps/KAFKA'
    scp /opt/apps/KAFKA/kafka-2.13-3.6.1-1.0.0.tar.gz core-1-1:/opt/apps/KAFKA/

    In a minor version earlier than EMR V3.50.0

    sudo su - emr-user
    ssh core-1-1 'sudo chmod a+w /opt/apps/KAFKA'
    scp /opt/apps/KAFKA/kafka-2.12-2.4.1-1.1.5.tar.gz core-1-1:/opt/apps/KAFKA/
  4. Run the following commands to log on to the core-1-1 node and decompress the Kafka software package.

    In a minor version earlier than EMR V5.16.0

    ssh core-1-1
    sudo su - kafka
    cd /opt/apps/KAFKA/
    
    # Create a directory and decompress the Kafka software package. 
    sudo mkdir kafka-2.13-3.6.1-1.0.0
    
    sudo chown kafka:hadoop kafka-2.13-3.6.1-1.0.0
    tar xvf kafka-2.13-3.6.1-1.0.0.tar.gz

    In a minor version earlier than EMR V3.50.0

    ssh core-1-1
    sudo su - kafka
    cd /opt/apps/KAFKA/
    
    # Create a directory and decompress the Kafka software package. 
    sudo mkdir kafka-2.12-2.4.1-1.1.5
    
    sudo chown kafka:hadoop kafka-2.12-2.4.1-1.1.5
    tar xvf kafka-2.12-2.4.1-1.1.5.tar.gz
  5. Run the following commands to create a link file:

    In a minor version earlier than EMR V5.16.0

    sudo rm -f kafka-current
    sudo ln -s /opt/apps/KAFKA/kafka-2.13-3.6.1-1.0.0 kafka-current

    In a minor version earlier than EMR V3.50.0

    sudo rm -f kafka-current
    sudo ln -s /opt/apps/KAFKA/kafka-2.12-2.4.1-1.1.5 kafka-current
  6. Go to the Status tab of the Kafka service page in the EMR console, and restart the KafkaBroker component that is deployed on the current core node.

    image

    Important

    You do not need to restart the entire Kafka service.

  7. Wait until the restart operation is complete, make sure that the Kafka service is run as expected on the core-1-1 node, and then proceed to the next step.

  8. Repeat Step 3 to Step 7 to replace the Kafka software package in each of the other core nodes in the current cluster.

Kafka cluster (available only to existing users)

  1. Check whether the Kafka version in the current cluster is 2.4.1. You can check whether the /usr/lib/kafka-current symbolic link points to the directory in which the kafka-2.12.-2.4.1-xxx package is stored to determine the Kafka version in the current cluster.

    • If the Kafka version is 2.4.1, click kafka-2.12-2.4.1-1.1.5.tar.gz to download the Kafka software package with the vulnerability fixed.

    • If the Kafka version is not 2.4.1, the vulnerability does not need to be fixed.

  2. Log on to the master node of the cluster and store the Kafka software package that is downloaded in the previous step in the /usr/lib/ directory of the hadoop user.

    For information about how to log on to a cluster, see Log on to a cluster.

  3. Run the following commands to copy the Kafka software package to the /usr/lib/ directory of the worker-1-1 node on which Kafka is deployed:

    su - hadoop
    scp /usr/lib/kafka-2.12-2.4.1-1.1.5.tar.gz work-1-1:/tmp/
    ssh worker-1-1
    sudo cp /tmp/kafka-2.12-2.4.1-1.1.5.tar.gz /usr/lib/
  4. Run the following commands to log on to the worker-1-1 node and decompress the Kafka software package:

    su - hadoop
    ssh work-1-1
    sudo su - kafka
    sudo mkdir kafka-2.12-2.4.1-1.1.5
    sudo chown kafka:hadoop kafka-2.12-2.4.1-1.1.5
    tar xvf kafka-2.12-2.4.1-1.1.5.tar.gz
  5. Run the following commands to create a link file:

    cd /usr/lib/
    sudo rm -f kafka-current
    sudo ln -s /usr/lib/kafka-2.12-2.4.1-1.1.5 kafka-current
  6. Go to the Status tab of the Kafka service page in the EMR console, and restart the KafkaBroker component that is deployed on the current core node.

    Important

    You do not need to restart the entire Kafka service.

  7. Wait until the restart operation is complete, make sure that the Kafka service is run as expected on the worker-1-1 node, and then proceed to the next step.

  8. Repeat Step 3 to Step 7 to replace the Kafka software package in each of the other core nodes in the current cluster.

(Optional) Subsequent O&M

If Kafka brokers need to be added to your cluster, you can perform the preceding steps to fix the vulnerability on new core nodes on which Kafka brokers are deployed. You can also use the EMR cluster script feature to fix the vulnerability on new nodes during the scale-out process.

  1. Upload the Kafka software package to the Object Storage Service (OSS) console. For more information, see Simple upload.

    For example, upload the kafka-2.12-2.4.1-1.1.5.tar.gz package to the oss://emr-kafka-software/ directory.

  2. Create a script file named upgrade_kafka_software.sh and upload the script file to OSS for subsequent addition of a bootstrap action.

    #!/bin/bash
    sudo su - kafka
    
    #upgrade_kafka_software.sh
    #pls update the vars before use:
    #KAFKA_HOME,OSS_HOST,OSS_AK,OSS_SK,OSS_DIR,OSS_FILE
    KAFKA_HOME="/opt/apps/KAFKA"
    OSS_HOST=<yourEndPoint>
    OSS_AK=<yourAccessKeyId>
    OSS_SK=<yourAccessKeySecret>
    #yourOssDir, e.g.:emr-kafka-software
    OSS_DIR=<yourOssDir>
    #change to your kafka version
    OSS_FILE=<yourOssFile>
    
    if [ -f ${KAFKA_HOME}/${OSS_FILE} ]; then
     echo "file exist, no action to do"
     exit 0
    fi
    
    sudo /usr/local/bin/ossutil64 -i $OSS_AK -k $OSS_SK -e $OSS_HOST cp oss://${OSS_DIR}/${OSS_FILE} ${KAFKA_HOME}/${OSS_FILE}
    
    cd ${KAFKA_HOME}
    #change to your kafka version
    sudo mkdir kafka-2.12-2.4.1-1.1.5
    #change to your kafka version
    sudo chown kafka:hadoop kafka-2.12-2.4.1-1.1.5
    tar xvf kafka-2.12-2.4.1-1.1.5.tar.gz
    
    sudo rm -f kafka-current
    sudo ln -s /opt/apps/KAFKA/kafka-2.12-2.4.1-1.1.5 kafka-current
    
    exit 0

    The following table describes the parameters in the code. Replace the values of the parameters with actual values.

    Note

    The name of the newly created directory must be the same as the name of the Kafka software package. In this example, the name of the directory is kafka-2.12-2.4.1-1.1.5. You can replace the name based on your business requirements.

    Parameter

    Description

    KAFKA_HOME

    The directory where the Kafka software package is stored.

    OSS_HOST

    The internal endpoint of Alibaba Cloud OSS. Example: oss-cn-hangzhou-internal.aliyuncs.com.

    OSS_AK

    The AccessKey ID and AccessKey secret that are used to access Alibaba Cloud OSS.

    For information about how to create and view an AccessKey pair, see Create an AccessKey pair.

    OSS_SK

    OSS_DIR

    The directory in the OSS bucket. Replace the value of this parameter with the directory where the Kafka software package is stored in Alibaba Cloud OSS. Example: oss://emr-kafka-software/.

    OSS_FILE

    The name of the Kafka software package in Alibaba Cloud OSS. Replace the value of this parameter with the name of the Kafka software package that you downloaded. Example: kafka-2.12-2.4.1-1.1.5.tar.gz.

  3. Add a bootstrap action.

    Add a bootstrap action in the EMR console. For more information, see Manage bootstrap actions.

    Parameter

    Description

    Script Address

    The OSS path where the script file upgrade_kafka_software.sh is located.

    Execution Scope

    Select Node Group from the Execution Scope drop-down list and then select emr-core.

    Execution Time

    Select Before Component Installation.

References

For information about how to scale out a cluster, see Scale out an EMR cluster.