E-MapReduce:Access Kafka over the Internet

Enable Internet-based access when you create a cluster

Turn on Assign Public Network IP for the core node group when you create a Dataflow cluster in the EMR console.

After you enable port 9093 for a security group, you can directly access the Kafka service by using port 9093 of the public IP address. For more information about how to enable a port, see Manage security groups.

Enable Internet-based access for an existing cluster

A Dataflow cluster is deployed in a virtual private cloud (VPC). You can access the Kafka service over the Internet by using one of the following methods: associate elastic IP addresses (EIPs) or use Express Connect.

Method 1: Associate EIPs

1. Go to the Basic Information tab.

   1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.

   2. In the top navigation bar, select the region in which your cluster resides and select a resource group based on your business requirements.

   3. On the EMR on ECS page, find the desired cluster and click the cluster name.

2. On the Basic Information tab, click the link next to Cluster Security Group in the Security section to enable port 9093 for the security group. For more information, see Manage security groups.

3. Associate EIPs with all Elastic Compute Service (ECS) instances on which Kafka brokers are deployed.

   1. Click the Nodes tab.

   2. On the Nodes tab, find the node with which you want to associate an EIP and click the ID in the Node Name/ID column.

   3. In the ECS console, associate an EIP with the ECS instance. For more information, see Associate an EIP with an instance.

   4. Repeat Step b and Step c to associate EIPs with all ECS instances on which Kafka brokers are deployed.

4. In the upper-right corner of the Nodes tab, choose All Operation > Synchronize Host Information to synchronize the information about the public IP addresses to the Dataflow cluster.

5. On the Configure tab of the Kafka service page, change the value of the kafka.public-access.enable parameter to true.

6. Deploy the client configuration.

   1. On the Configure tab of the Kafka service page, click Deploy Client Configuration.

   2. In the dialog box that appears, configure the Execution Reason parameter and click OK.

   3. In the Confirm message, click OK.

7. Restart the Kafka service.

   1. On the Status tab of the Kafka service page, find KafkaBroker in the Components section and click Restart in the Actions column.

   2. In the dialog box that appears, configure the Execution Reason parameter and click OK.

   3. In the Confirm message, click OK.

Method 2: Deploy Express Connect

You can deploy Express Connect to establish a connection between the Internet and a VPC. For more information, see What is Express Connect?

Access Kafka

If you want to access the KafkaBroker component for which a public IP address is configured, the URL is in the format of <Public IP address of the broker>:9093.

For example, if the public IP address of a broker is 211.90.**.**, the address to access the Kafka service is 211.90.**.**:9093.

References

For more information about how to create a Dataflow Kafka cluster, see Create a Dataflow Kafka cluster.