E-MapReduce:EMR Kafka quick start

Planning considerations

Detailed sizing guidance is available in Cluster Resource Assessment Recommendations.

Create an EMR Kafka cluster

The following steps cover the basics. For the complete procedure, see Create a cluster.

1. Go to the Create Cluster page.

   1. Log on to the E-MapReduce console.

   2. Click CREATE_CLUSTER.

2. In the Software Configuration step, select an EMR version based on your required Kafka version.

   Turn on the High Service Availability switch to deploy a three-node ZooKeeper cluster. For the business scenario, select Real-time Data Stream and verify that Kafka (2.12_2.4.1) and Kafka-Manager (2.0.0.2) are included.

   Important

   Enabling high availability deploys a three-node ZooKeeper service on the master node group. Kafka availability depends on ZooKeeper.

3. In the Hardware Configuration step, select an appropriate ECS instance type and the number of nodes.

   • Instance type: Select a core node instance type with a 1:4 CPU-to-memory ratio.

   • Number of nodes: Set the core node count to one more than the replication factor. For example, use 4 nodes for a replication factor of 3.

   In this example, the master group uses three 4 vCPU/16 GiB instances, each with one 80 GiB ESSD system disk and one 80 GiB ESSD data disk. The core group uses four 16 vCPU/64 GiB instances, each with one 80 GiB ESSD system disk and four 80 GiB ESSD data disks.

4. Configure the remaining parameters and create the cluster.

Use Kafka topics

Produce and consume data with Kafka topics. In production, you can also use Kafka Manager or Cruise Control for cluster management.

1. Log on to the master node of the Kafka cluster over SSH. Log on to a cluster.

2. Run the following command to create a Kafka topic.

   sudo su - kafka
   kafka-topics.sh --partitions 10 --replication-factor 2 --bootstrap-server core-1-1:9092 --topic test --create

3. Run the following command to view the details of the Kafka topic.

   kafka-topics.sh --bootstrap-server core-1-1:9092 --topic test --describe

4. Run the following command to produce data.

    kafka-console-producer.sh  --broker-list core-1-1:9092 --topic test

   Type a message and press Enter to send it.

5. Open a new terminal window and run the following command to consume data.

   kafka-console-consumer.sh --bootstrap-server core-1-1:9092 --topic test --from-beginning --group test-consumer-group

Use Kafka Connect

Kafka Connect is available in EMR 3.41.0 and later and EMR 5.7.0 and later.

1. Go to the node management page.

   1. Log on to the EMR console.

   2. In the top navigation bar, select a region and a resource group as needed.

   3. In the Actions column of the target cluster, click Nodes.

2. Create a Kafka Connect node group.

   Kafka Connect runs on a task node group. When you add a task node group to an EMR Kafka cluster, EMR automatically deploys Kafka Connect on it.

   1. Create an EMR task node group

      On the Nodes page, click Add Node Group. Add a node group.

   2. Scale out the task node group

      Add instances to the task node group as needed. Scale out a cluster.

3. Verify that the Kafka Connect cluster is running.

   On the Status tab for the Kafka service, go to the Components section and expand KafkaConnect. In the Topology list, confirm that a node such as task-1-1 shows Good health and Running status.

4. Check the Kafka Connect REST service status.

   1. Log on to the master node of the Kafka cluster over SSH. Log on to a cluster.

   2. Run the following command to check the Kafka Connect REST API status.

      curl -X GET http://task-1-1:8083| jq .

      Expected output:

        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100    91  100    91    0     0  13407      0 --:--:-- --:--:-- --:--:-- 15166
      {
        "version": "2.4.1",
        "commit": "42ce056344c5625a",
        "kafka_cluster_id": "6Z7IdHW4SVO1Pbql4c****"
      }

5. Migrate data using Kafka Connect.

   Start a MirrorMaker task to replicate data across clusters. Use MirrorMaker 2 (on Connect) to synchronize data across clusters.

Related topics

• Enable SSL encryption for Kafka connections. Use SSL to encrypt Kafka connections.

• Enable SASL authentication for Kafka connections. Use SASL to authenticate Kafka connections.