Configure SSL to encrypt Kafka data connections - E-MapReduce

Secure Sockets Layer (SSL) is a network protocol for establishing a secure connection between clients and servers. When SSL is enabled, all data transfers in a Dataflow cluster, including message exchanges between producers, consumers, and brokers, are encrypted. This ensures that sensitive information is not intercepted or tampered with during network transmission. This topic describes how to enable SSL and connect to Kafka over SSL.

Prerequisites

A Dataflow cluster (which is a Kafka cluster) with the Kafka service selected must be created in the EMR console. For more information, see Create a Dataflow Kafka cluster.

Procedure

Step 1: Enable SSL

E-MapReduce provides two methods to configure SSL for a Kafka cluster:

Enable SSL with the default certificate: Use the default E-MapReduce certificate and configuration method to quickly enable SSL.
Enable SSL with custom settings: Use your own certificate and configuration values to enable SSL.

The kafka.ssl.config.type parameter in the server.properties configuration file manages SSL configuration policies in E-MapReduce.

Method 1: Default

Note

By default, SSL is disabled for Kafka clusters. Follow these steps to quickly enable SSL.

Go to the service configuration page.
1. Log on to the E-MapReduce console.
2. In the top navigation bar, select a region and a resource group as needed.
3. Click Services in the Actions column of the target cluster.
4. On the Services page, click Configure in the Kafka service area.
Modify the configuration parameter.
1. On the Configure page, click the server.properties tab.
2. Set the kafka.ssl.config.type parameter to DEFAULT.
Save the configuration.
1. Click Save.
2. In the dialog box that appears, enter an Execution Reason, and click Save.
Restart the Kafka service.
1. On the Configure page of the Kafka service, select More > Restart.
2. In the dialog box that appears, enter the execution reason, and click OK.
3. In the Confirm dialog box, click OK.

Method 2: Custom

Note

By default, SSL is disabled for Kafka clusters. You can enable SSL with a custom configuration.

Go to the service configuration page.
1. Log on to the E-MapReduce console.
2. In the top navigation bar, select a region and a resource group as needed.
3. Click Services in the Actions column of the target cluster.
4. On the Services page, click Configure in the Kafka service area.
Modify the configuration parameter.
1. On the Configure page, click the server.properties tab.
2. Change the parameter value of kafka.ssl.config.type to CUSTOM.
Save the configuration.
1. Click Save.
2. In the pop-up dialog box, enter an Execution Reason and click Save.
Configure other SSL parameters.

Based on your business requirements, configure the SSL-related parameters other than listeners. Examples include ssl.keystore.location, ssl.keystore.password, ssl.truststore.location, ssl.truststore.password, ssl.key.password, ssl.keystore.type, and ssl.truststore.type.
Restart the Kafka service.
1. On the Configure page of the Kafka service, select More > Restart.
2. In the dialog box that appears, enter the execution reason, and click OK.
3. In the Confirm dialog box, click OK.

Step 2: Connect to Kafka over SSL

To connect to Kafka over SSL, you must configure the following client parameters: security.protocol, ssl.truststore.password, and ssl.truststore.location.

For example, to run a job using Kafka's built-in producer and consumer on an SSL-enabled Kafka cluster, follow these steps:

Connect to the master node of the cluster over SSH. For more information, see Log on to a cluster.
Create a configuration file.
1. Run the following command to create the ssl.properties configuration file.
```
vim ssl.properties
```
2. Add the following content to the ssl.properties file.
```
security.protocol=SSL
ssl.truststore.location=/var/taihao-security/ssl/ssl/truststore
ssl.truststore.password=${password}
ssl.keystore.location=/var/taihao-security/ssl/ssl/keystore
ssl.keystore.password=${password}
ssl.endpoint.identification.algorithm=
```
  You can find the values for these parameters on the Configure page for the Kafka service in the EMR console. If you run your job in an environment outside the Kafka cluster, copy the truststore and keystore files from the corresponding directory on any cluster node to your runtime environment and configure the file paths accordingly.

Run the following command to create a topic.

kafka-topics.sh --partitions 10 --replication-factor 2 --bootstrap-server core-1-1:9092 --topic test --create --command-config ssl.properties

Run the following commands to produce data using the SSL configuration file.

export IP=<your_InnerIP>
kafka-producer-perf-test.sh --topic test --num-records 123456 --throughput 10000 --record-size 1024 --producer-props bootstrap.servers=${IP}:9092 --producer.config ssl.properties

Note

In this code example, <your_InnerIP> is the internal IP address of the master-1-1 node.

Run the following commands to consume data using the SSL configuration file.

export IP=<your_InnerIP>
kafka-consumer-perf-test.sh --broker-list ${IP}:9092 --messages 100000000 --topic test --consumer.config ssl.properties

E-MapReduce:Use SSL to encrypt Kafka connections

Prerequisites

Procedure

Step 1: Enable SSL

Method 1: Default

Method 2: Custom

Step 2: Connect to Kafka over SSL

Related topics