This topic describes how to deploy Apache Kafka in a cluster by creating a stack in the Resource Orchestration Service (ROS) console.
Background information
Apache Kafka is an open source stream processing platform that is written in the Scala and Java programming languages. Apache Kafka is a high-throughput distributed publish-subscribe messaging system for processing streaming data of customer behaviors on websites.
You can use the Existing_Vpc_Cluster_Kafka sample template to create multiple Elastic Compute Service (ECS) instances based on existing resources, such as a virtual private cloud (VPC), vSwitch, and security group. Among the ECS instances, one is associated with an elastic IP address (EIP) to serve as the management node and the others are managed by using Auto Scaling. The following software versions are used in the sample template:
JDK 1.8.0: the Java Development Kit (JDK)
Scala 2.12: the programming language
Apache Kafka 0.10.2.2: the compute engine
The Apache Kafka data in the sample template is stored in a data disk. The default location of the storage directory /home/software/. The default location of the Apache Kafka bin directory is /home/software/kafka/bin.
After you create a stack by using the sample template, you can obtain the value of KafkaManagerUrl and log on to the Apache Kafka management console. If you want to access the URL specified by KafkaManagerUrl over the Internet, you must configure an inbound rule for the security group to allow traffic on port 9000. For more information, see Add a security group rule.
Step 1: Create a stack
Log on to the ROS console.
In the left-side navigation pane, choose Templates > Public Templates.
Search for the Existing_Vpc_Cluster_Kafka sample template.
Click Create Stack.
In the Configure Parameters step, configure the Stack Name parameter and the following parameters.
Parameter
Description
Example
Existing VPC Instance ID
The ID of the VPC.
For more information about how to create and query a VPC, see Create and manage a VPC.
vpc-bp1m6fww66xbntjyc****
VSwitch Zone ID
The zone ID of the vSwitch that resides in the VPC.
Qingdao Zone C
VSwitch ID
The ID of the vSwitch that resides in the VPC.
For more information about how to create and query a vSwitch, see Create and manage a vSwitch.
vsw-bp183p93qs667muql****
Business Security Group ID
The ID of the ECS security group.
For more information about how to query the ID of a security group, see Search for security groups.
sg-bp15ed6xe1yxeycg7o****
Instance Type
The instance type of the ECS instances.
Select a valid instance type. For more information, see Overview of instance families.
ecs.c5.large
Instance Password
The password of the ECS instances.
Test_12****
Public IP Bandwidth
The bandwidth of the public IP address.
Unit: Mbit/s.
5
Disk Type
The disk category. Valid values:
cloud_efficiency: ultra disk.
cloud_ssd: standard SSD.
cloud_essd: Enterprise SSD (ESSD).
cloud: basic disk.
ephemeral_ssd: local SSD.
For more information, see Disks.
cloud_efficiency
Data Disk Space
The data disk size of an ECS instance.
Valid values: 20 to 32786.
Unit: GB.
20
Kafka Listeners Port
The listener port. The port number must be greater than 1000.
Default value: 9092.
9092
Message Retention Time
The maximum retention period of a message.
Default value: 24.
Unit: hour.
24
Topic Specifications
The number of topics.
Default value: 50.
50
Instance Amount
The number of ECS instances in the cluster in which you want to deploy Apache Kafka.
Valid values: 3 to 10.
3
Click Next:Check and Confirm. Then, click Create.
On the Stack Information tab, view the stack status. Wait until the stack is created. Then, click Outputs to obtain the value of KafkaManagerUrl.
Access the URL specified by KafkaManagerUrl and log on to the Apache Kafka management console.
Step 2: View resources
In the left-side navigation pane, choose Deployment > Stacks.
On the Stacks page, click the ID of the desired stack.
Click the Resources tab to view information about the resources in the stack.
The following table describes the resources in this example.
Resource
Quantity
Resource description
Specification description
ALIYUN::ECS::InstanceGroup
1
Creates an ECS instance to deploy the Apache Kafka primary service.
An ECS instance that has the following specifications is created:
Instance type: ecs.c5.large.
Disk category: ultra disk.
System disk size: 40 GB.
Public IP address: A public IP address is allocated.
ALIYUN::ESS::ScalingGroup
2
Creates two scaling groups to deploy Apache Kafka secondary services.
Scaling groups automatically scale elastic computing resources based on the scaling rules that you configure to meet your business requirements.
Two ECS instances are created. Each ECS instance has the following specifications:
Instance type: ecs.c5.large.
Disk category: ultra disk.
System disk size: 40 GB.
Public IP address: A public IP address is allocated.
ALIYUN::RAM::Role
1
Creates a Resource Access Management (RAM) role to issue a Security Token Service (STS) token that is valid within a short period of time. This way, you can grant access permissions in a secure manner.
None.
ALIYUN::VPC::EIP
1
Creates an EIP to associate the EIP with an ECS instance. This way, the ECS instance can be accessed over the Internet.
None.
ALIYUN::OOS::Template
1
Creates a CloudOps Orchestration Service (OOS) template to create lifecycle hooks.
For more information, see Lifecycle hooks.
None.
NoteFor more information about the pricing details of resources, go to the relevant console or refer to the pricing documentation of each resource.