On February 8, 2023, Apache released a security notice about fixing a deserialization vulnerability in Apache Kafka. The Common Vulnerabilities and Exposures (CVE) ID of the vulnerability is CVE-2023-25194. In the case where an attacker can control the Apache Kafka Connect client, the attacker can use SASL JAAS configurations and SASL-based security protocols to create or modify connectors on the client. This will trigger the Java Naming and Directory Interface (JNDI) code execution vulnerability.
Impacts
Affected Apache Kafka versions: 2.3.0 to 3.3.2
Affected E-MapReduce (EMR) versions:
EMR 5.X series: versions earlier than EMR V5.16.0
EMR 3.X series: versions earlier than EMR V3.50.0
NoteThe vulnerability is fixed in EMR V3.50.0, EMR V5.16.0, or a minor version later than EMR V3.50.0 or EMR V5.16.0. In these versions, you do not need to manually fix the vulnerability.
Fixing procedure
The method of fixing the vulnerability varies based on the type of the cluster in which Kafka is deployed.
Dataflow cluster
Download a Kafka software package with the vulnerability fixed.
In a minor version earlier than EMR V5.16.0, download kafka-2.13-3.6.1-1.0.0.tar.gz.
In a minor version earlier than EMR V3.50.0, download kafka-2.12-2.4.1-1.1.5.tar.gz.
Log on to the master node of your cluster and upload the Kafka software package that is downloaded in the previous step to the
/opt/apps/KAFKA/directory of the master node.For information about how to log on to a cluster, see Log on to a cluster.
Run the following commands to copy the Kafka software package to the /opt/apps/KAFKA directory of the core-1-1 node.
In a minor version earlier than EMR V5.16.0
sudo su - emr-user ssh core-1-1 'sudo chmod a+w /opt/apps/KAFKA' scp /opt/apps/KAFKA/kafka-2.13-3.6.1-1.0.0.tar.gz core-1-1:/opt/apps/KAFKA/In a minor version earlier than EMR V3.50.0
sudo su - emr-user ssh core-1-1 'sudo chmod a+w /opt/apps/KAFKA' scp /opt/apps/KAFKA/kafka-2.12-2.4.1-1.1.5.tar.gz core-1-1:/opt/apps/KAFKA/Run the following commands to log on to the core-1-1 node and decompress the Kafka software package.
In a minor version earlier than EMR V5.16.0
ssh core-1-1 sudo su - kafka cd /opt/apps/KAFKA/ # Create a directory and decompress the Kafka software package. sudo mkdir kafka-2.13-3.6.1-1.0.0 sudo chown kafka:hadoop kafka-2.13-3.6.1-1.0.0 tar xvf kafka-2.13-3.6.1-1.0.0.tar.gzIn a minor version earlier than EMR V3.50.0
ssh core-1-1 sudo su - kafka cd /opt/apps/KAFKA/ # Create a directory and decompress the Kafka software package. sudo mkdir kafka-2.12-2.4.1-1.1.5 sudo chown kafka:hadoop kafka-2.12-2.4.1-1.1.5 tar xvf kafka-2.12-2.4.1-1.1.5.tar.gzRun the following commands to create a link file:
In a minor version earlier than EMR V5.16.0
sudo rm -f kafka-current sudo ln -s /opt/apps/KAFKA/kafka-2.13-3.6.1-1.0.0 kafka-currentIn a minor version earlier than EMR V3.50.0
sudo rm -f kafka-current sudo ln -s /opt/apps/KAFKA/kafka-2.12-2.4.1-1.1.5 kafka-currentGo to the Status tab of the Kafka service page in the EMR console, and restart the KafkaBroker component that is deployed on the current core node.
ImportantYou do not need to restart the entire Kafka service.
Wait until the restart operation is complete, make sure that the Kafka service is run as expected on the core-1-1 node, and then proceed to the next step.
Repeat Step 3 to Step 7 to replace the Kafka software package in each of the other core nodes in the current cluster.
Kafka cluster (available only to existing users)
Check whether the Kafka version in the current cluster is 2.4.1. You can check whether the
/usr/lib/kafka-currentsymbolic link points to the directory in which thekafka-2.12.-2.4.1-xxxpackage is stored to determine the Kafka version in the current cluster.If the Kafka version is 2.4.1, click kafka-2.12-2.4.1-1.1.5.tar.gz to download the Kafka software package with the vulnerability fixed.
If the Kafka version is not 2.4.1, the vulnerability does not need to be fixed.
Log on to the master node of the cluster and store the Kafka software package that is downloaded in the previous step in the
/usr/lib/directory of the hadoop user.For information about how to log on to a cluster, see Log on to a cluster.
Run the following commands to copy the Kafka software package to the /usr/lib/ directory of the worker-1-1 node on which Kafka is deployed:
su - hadoop scp /usr/lib/kafka-2.12-2.4.1-1.1.5.tar.gz work-1-1:/tmp/ ssh worker-1-1 sudo cp /tmp/kafka-2.12-2.4.1-1.1.5.tar.gz /usr/lib/Run the following commands to log on to the worker-1-1 node and decompress the Kafka software package:
su - hadoop ssh work-1-1 sudo su - kafka sudo mkdir kafka-2.12-2.4.1-1.1.5 sudo chown kafka:hadoop kafka-2.12-2.4.1-1.1.5 tar xvf kafka-2.12-2.4.1-1.1.5.tar.gzRun the following commands to create a link file:
cd /usr/lib/ sudo rm -f kafka-current sudo ln -s /usr/lib/kafka-2.12-2.4.1-1.1.5 kafka-currentGo to the Status tab of the Kafka service page in the EMR console, and restart the KafkaBroker component that is deployed on the current core node.
ImportantYou do not need to restart the entire Kafka service.
Wait until the restart operation is complete, make sure that the Kafka service is run as expected on the worker-1-1 node, and then proceed to the next step.
Repeat Step 3 to Step 7 to replace the Kafka software package in each of the other core nodes in the current cluster.
(Optional) Subsequent O&M
If Kafka brokers need to be added to your cluster, you can perform the preceding steps to fix the vulnerability on new core nodes on which Kafka brokers are deployed. You can also use the EMR cluster script feature to fix the vulnerability on new nodes during the scale-out process.
Upload the Kafka software package to the Object Storage Service (OSS) console. For more information, see Simple upload.
For example, upload the kafka-2.12-2.4.1-1.1.5.tar.gz package to the
oss://emr-kafka-software/directory.Create a script file named upgrade_kafka_software.sh and upload the script file to OSS for subsequent addition of a bootstrap action.
#!/bin/bash sudo su - kafka #upgrade_kafka_software.sh #pls update the vars before use: #KAFKA_HOME,OSS_HOST,OSS_AK,OSS_SK,OSS_DIR,OSS_FILE KAFKA_HOME="/opt/apps/KAFKA" OSS_HOST=<yourEndPoint> OSS_AK=<yourAccessKeyId> OSS_SK=<yourAccessKeySecret> #yourOssDir, e.g.:emr-kafka-software OSS_DIR=<yourOssDir> #change to your kafka version OSS_FILE=<yourOssFile> if [ -f ${KAFKA_HOME}/${OSS_FILE} ]; then echo "file exist, no action to do" exit 0 fi sudo /usr/local/bin/ossutil64 -i $OSS_AK -k $OSS_SK -e $OSS_HOST cp oss://${OSS_DIR}/${OSS_FILE} ${KAFKA_HOME}/${OSS_FILE} cd ${KAFKA_HOME} #change to your kafka version sudo mkdir kafka-2.12-2.4.1-1.1.5 #change to your kafka version sudo chown kafka:hadoop kafka-2.12-2.4.1-1.1.5 tar xvf kafka-2.12-2.4.1-1.1.5.tar.gz sudo rm -f kafka-current sudo ln -s /opt/apps/KAFKA/kafka-2.12-2.4.1-1.1.5 kafka-current exit 0The following table describes the parameters in the code. Replace the values of the parameters with actual values.
NoteThe name of the newly created directory must be the same as the name of the Kafka software package. In this example, the name of the directory is
kafka-2.12-2.4.1-1.1.5. You can replace the name based on your business requirements.Parameter
Description
KAFKA_HOME
The directory where the Kafka software package is stored.
OSS_HOST
The internal endpoint of Alibaba Cloud OSS. Example: oss-cn-hangzhou-internal.aliyuncs.com.
OSS_AK
The AccessKey ID and AccessKey secret that are used to access Alibaba Cloud OSS.
For information about how to create and view an AccessKey pair, see Create an AccessKey pair.
OSS_SK
OSS_DIR
The directory in the OSS bucket. Replace the value of this parameter with the directory where the Kafka software package is stored in Alibaba Cloud OSS. Example:
oss://emr-kafka-software/.OSS_FILE
The name of the Kafka software package in Alibaba Cloud OSS. Replace the value of this parameter with the name of the Kafka software package that you downloaded. Example: kafka-2.12-2.4.1-1.1.5.tar.gz.
Add a bootstrap action.
Add a bootstrap action in the EMR console. For more information, see Manage bootstrap actions.
Parameter
Description
Script Address
The OSS path where the script file upgrade_kafka_software.sh is located.
Execution Scope
Select Node Group from the Execution Scope drop-down list and then select emr-core.
Execution Time
Select Before Component Installation.
References
For information about how to scale out a cluster, see Scale out an EMR cluster.