Create a Kafka data source - ApsaraDB for OceanBase - Alibaba Cloud Documentation Center

You must configure data sources before you create a data migration or synchronization task. This topic describes how to create a Kafka data source by using the data transmission service.

Security risks

You can choose to automatically or manually add public CIDR blocks for the data transmission service, but security risks exist. Your use of the data transmission service indicates that you have understood and acknowledged the possible security risks. You must take basic security protection measures, such as setting a complex password for your account, limiting the open port numbers of CIDR blocks, enabling authentication for communication among internal APIs, and regularly checking for and limiting CIDR blocks that are not required.

The data transmission service will add or delete the CIDR blocks automatically added to a whitelist or security group based on business needs and security risks. Do not use the CIDR blocks in a whitelist or security group for business needs other than the data transmission service. If you use such CIDR blocks for other business needs, the issues thus incurred are beyond the guarantee scope of the SLA of the data transmission service. For more information about how to add an IP address to a whitelistor security group for the data transmission service, see Add a whitelist.

Limitations

The data transmission service allows you to add a Kafka instance only as the target for data synchronization.

Background

Data security is a major concern throughout service connectivity verification, link creation, and data transmission. Based on the security system provided by the Kafka service, data transmission can meet most security requirements in data encryption and user authentication.

Data transmission supports the following Kafka authentication methods:

GSSAPI
Generic Security Services Application Program Interface (GSSAPI) is a framework that provides generic security services. It supports the Kerberos protocol.
PLAIN
PLAIN authentication is simple and does not support dynamic changes of users. In this authentication mode, usernames and passwords are configured in plaintext, resulting in low security.
SCRAM-SHA-256
Salted Challenge Response Authentication Mechanism (SCRAM) authenticates users by username and password. SCRAM-SHA-256 can be used together with Transport Layer Security (TLS) for security authentication.
In this authentication method, users can be dynamically changed, and the user data is stored in Zookeeper. Before a broker is started, a user for communication between the broker and ZooKeeper must be created. However, usernames and passwords are configured in plaintext in this authentication method.
SCRAM-SHA-512
SCRAM-SHA-512 can be used together with TLS for security authentication.

Procedure

Log on to the ApsaraDB for OceanBase console.
In the left-side navigation pane, choose Data Transmission > Data Source Management.
On the Data Sources page, click New Data Source in the upper-right corner.
In the New Data Source dialog box, select Kafka for Data Source Type.

Select a value for Instance Type and configure the parameters.

If you select Alibaba Cloud Kafka Instance, configure the following parameters.

Parameter	Description
Data Source Identifier	We recommend that you set it to a combination of digits and letters. It must not contain any spaces and cannot exceed 32 characters in length.
Cross Alibaba Cloud Account	The data transmission service allows you to configure a task to migrate or synchronize data between instances under different Alibaba Cloud accounts. You can choose whether to select this option based on business needs. If you select this option, enter the Alibaba Cloud account of the target instance. If you do not have permissions on this Alibaba Cloud account, request for authorization first. For more information, see Apply for cross-account authorization.
Kafka Instance ID	The unique ID of the Kafka instance that you applied for.
Access Point	The IP address and port list of the Kafka server. The system automatically imports this list.
Username	The logon username of the Kafka instance.
Password	The logon password of the Kafka instance.
Remarks (Optional)	The additional information about the data source.

If you select Self-Managed Kafka Instance in VPC or Kafka Instance in Public Network, configure the following parameters.

Parameter	Description
Data Source Identifier	We recommend that you set it to a combination of digits and letters. It must not contain any spaces and cannot exceed 32 characters in length.
Cross Alibaba Cloud Account	The data transmission service allows you to configure a task to migrate or synchronize data between instances under different Alibaba Cloud accounts. You can choose whether to select this option based on business needs. If you select this option, enter the Alibaba Cloud account of the target instance. If you do not have permissions on this Alibaba Cloud account, request for authorization first. For more information, see Apply for cross-account authorization. Important This parameter is not displayed when Kafka Instance in Public Network is selected as the instance type.
VPC	Select the unique ID of a VPC you have applied for from the drop-down list, or enter a VPC name to perform a fuzzy search. Important This parameter is displayed only when you set the instance type to Self-Managed Kafka Instance in VPC.
Deploy in VPC/Cross-ISP Deployment	In cross-ISP deployment, the source and target data sources are in different networks, including different VPCs or cloud service providers. Choose Deploy in VPC or Cross-ISP Deployment based on your business needs. Select all vSwitches to which all bootstrap server and `broker server` instances of the Kafka service belong from the `vSwitch` list. Add the CIDR blocks of the vSwitch to the security group whitelist of the current VPC. A vSwitch is a basic network module of a VPC and is used to connect cloud resource instances. For more information, see Overview. Important If the instance type is set to Self-Managed Kafka Instance in VPC, you can select the deployment mode and vSwitches. In the case of cross-ISP deployment, static routing addresses (addresses or CIDR blocks in VPCs on other clouds or on-premises IDCs) are automatically associated with the first selected vSwitch.
Access Point	The IP address and port list of the Kafka server.
Enable SSL	Choose whether to enable Secure Sockets Layer (SSL) based on the business needs. To enable SSL, click Upload File and upload an SSL certificate suffixed with `.jks`.
Enable Authentication	Choose whether to enable authentication based on business needs. Kafka provides data encryption and multiple identity authentication mechanisms to ensure the security of user data and services.
Authentication Method	If authentication is enabled, you must specify the authentication method. Data transmission supports the following authentication methods: GSS-API, PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
KDC Server Address	The IP address or domain name of the Kerberos Key Distribution Center (KDC) server. Notice: This parameter is displayed only when you set Authentication Method to GSSAPI.
User Entity	Enter the username. Notice: This parameter is displayed only when you set Authentication Method to GSSAPI.
Keytab File	Click Upload File and upload a key file suffixed with `.keytab`. Notice: This parameter is displayed only when you set Authentication Method to GSSAPI.
Username	The username of the account used for data migration or synchronization. Notice: This parameter is not displayed when you set Authentication Method to GSSAPI.
Password	The password of the account used for data migration or synchronization. Notice: This parameter is not displayed when you set Authentication Method to GSSAPI.
Remarks (Optional)	The additional information about the data source.

Click Test Connection to verify the network connection between the data transmission service and the data source, as well as the validity of the username and password.
After the test is passed, click OK.