Kafka is a distributed message queue service with high throughput and scalability. It is widely used in scenarios such as log collection, monitoring data aggregation, stream data processing, and online and batch analytics. Hologres integrates with the big data ecosystem and supports real-time ingestion from Kafka. After data lands in Hologres, you can run high-concurrency, low-latency queries on it. and
Two methods are available:
| Method | Description |
|---|---|
| DataWorks data integration | Managed sync using DataWorks |
| Hologres connector | Direct Kafka-to-Hologres writes without DataWorks |
Synchronize data from Kafka in real time using DataWorks
DataWorks data integration reads data from Kafka using the Java SDK for Kafka and synchronizes it to Hologres in real time. For an overview of how DataWorks works, see DataWorks overview. For the underlying read and write mechanics, see Kafka Reader and Hologres Writer.
Step 1: Configure data sources
Register Kafka as the source and Hologres as the destination in DataWorks:
Step 2: Configure a sync task
Create a real-time sync task that maps the Kafka source to the Hologres destination:
Step 3: Query data
After the data is synchronized, you can query the data in Hologres.
Write data from Kafka using the Hologres connector
The Hologres connector writes data directly from Kafka to Hologres without going through DataWorks.
For setup instructions and configuration reference, see Write to Hologres from Kafka.