Moving data between ApsaraMQ for Kafka and other Alibaba Cloud services typically requires custom integration code. Connector Ecosystem Integration eliminates that overhead. This low-code module connects your Kafka topics to Simple Log Service, Tablestore, Function Compute, and more.
With Connector Ecosystem Integration, you can:
Ingest data into Kafka from logs, databases, self-managed applications, and message-oriented middleware across regions and clouds.
Export data from Kafka to downstream services with UI-based ETL capability to cleanse, filter, and convert the format of data during distribution.
Transform and process data using built-in ETL capabilities or write custom business logic in Function Compute, a serverless computing service provided by Alibaba Cloud.
Connector Ecosystem Integration also provides features to help you operate, manage, and monitor the integrated data.
Source connectors pull data into ApsaraMQ for Kafka. Sink connectors push data out. Each connector task combines a source, a sink, a filtering rule, and a transformation rule. Tasks run on EventBridge event streams.
How it works
Message Inflow (Source)
Message Inflow synchronizes messages, logs, relational data, and non-relational data from external sources into ApsaraMQ for Kafka. It supports integration across data centers and clouds, and provides tools to manage and filter incoming data.
Message Outflow (Sink)
Message Outflow exports data from ApsaraMQ for Kafka to destination services. During export, it reliably distributes messages and cleanses and filters data in transit.
ETL (Transformation)
ETL is an optional data processing layer available when you create a source or sink connector task. It combines the data processing capabilities of EventBridge with custom code execution in Function Compute.
Matching modes (7 types): fixed-value, prefix, suffix, exclusion-based, numeric-value, array, and combined-condition matching.
Transformers (5 types): complete event, partial event, constant, template, and Function Compute template.
Tasks
A task is the resource entity that runs a connector integration. It consists of a source, a sink, a filtering rule, and a transformation rule. The underlying resources are EventBridge event streams.
After you create a task, the resource type cannot be changed.
Before you begin
Connector Ecosystem Integration depends on EventBridge and Function Compute. Before you create a connector task, complete the following steps:
Activate the destination service. For example, if your sink is Tablestore, activate Tablestore.
If the destination service is Simple Log Service, you do not need to activate Function Compute.
Required permissions
Alibaba Cloud account
Follow the on-screen instructions in the ApsaraMQ for Kafka console to configure the permission policy.
RAM user
Attach the following permission policies to the RAM user:
| Policy | Purpose |
|---|---|
| AliyunKafkaFullAccess | Manage ApsaraMQ for Kafka resources |
| AliyunFCFullAccess | Manage Function Compute resources |
| AliyunEventBridgeFullAccess | Manage EventBridge resources |
| Policy for the destination service | Manage the target service |
For extended capabilities, attach additional policies:
| Policy | Purpose |
|---|---|
| AliyunVPCFullAccess | Access the destination service in a virtual private cloud (VPC) |
| AliyunLogFullAccess | View running logs of Function Compute |
Billing
For billing details of the underlying services, see:
Connector versions
ApsaraMQ for Kafka provides two connector versions: new version (recommended) and old version. If you have never created an old version connector, the console shows only the new version entry. If you have old version connectors, both entries appear.
New version source connectors provide the same features as old version source connectors. The following table compares new and old version sink connectors.
| Item | New version (recommended) | Old version |
|---|---|---|
| Underlying dependency | EventBridge and Function Compute. Note The Simple Log Service sink connector depends only on EventBridge. | Varies by connector type: EventBridge, Function Compute, or no dependency. See the documentation for each connector. |
| Message filtering | Multiple filtering modes. See Event patterns. | Not supported. |
| Dead-letter queue | Simple Message Queue (formerly MNS) and ApsaraMQ for RocketMQ. | ApsaraMQ for Kafka. |
| Retry policy | All sink connector types. | Specific connector types only. |
| Depends on ApsaraMQ for Kafka resources | No. | Yes. Topics for offsets, configurations, status, dead-letter messages, and error data are created automatically or manually. |
| Cross-account transmission | Not supported. To transmit data across accounts, submit a ticket. | Supported. |
| Concurrent consumption threads | Custom value up to 32. | Fixed values: 1, 2, 3, 6, or 12. |
| Consumer group | Create new or use existing consumer groups. | Must create new consumer groups. |
Limits
| Item | Limit |
|---|---|
| Connector tasks per region | 20 (including source and sink tasks). To increase the quota, submit a ticket. |
| Task name | Letters, digits, and hyphens (-). Must start with a letter or digit. Maximum 127 characters. Names longer than 127 characters are truncated. |
| Event pattern (stringExpression) | Up to 5 expressions per field in the map data structure. |
| Event transformation: template | Up to 10,240 characters. |
| Event transformation: value | Up to 1,024 characters. |
| Data loss in extreme cases | See Retry policies and dead-letter queues. |