A Kafka reader reads data from Kafka in real time by using the Kafka SDK.

Create a Kafka reader

  1. Log on to the DataWorks console. In the left-side navigation pane, click Workspaces. On the Workspaces page, find the target workspace and click Data Analytics in the Actions column.
  2. On the Data Analytics tab, move the pointer over the Create a sync node icon and choose Data Integration > Real-Time Sync.

    You can also find the target workflow, right-click Data Integration, and choose Create > Real-Time Sync.

  3. In the Create Node dialog box that appears, set Node Name and Location, and then click Commit.
  4. On the configuration tab of the real-time sync node, drag Kafka under Reader to the editing panel.
  5. Click the Kafka reader node and set parameters in the Node Settings section.
    Parameter Description
    server The broker server address of Kafka in the format of ip:port.
    topic The name of the topic from which data is read in Kafka. Kafka maintains feeds of messages in categories called topics.

    Each message published to the Kafka cluster is assigned to a topic. Each topic contains a group of messages.

    keyType The type of the Kafka key.
    valueType The type of the Kafka value.
    Startup Mode The start time of data synchronization.
    Configuration Parameters The extended parameters specified when KafkaConsumer is created, such as bootstrap.servers, auto.commit.interval.ms, and session.timeout.ms. By setting parameters in kafkaConfig, you can control the data consumption behaviors of KafkaConsumer.
    Start Offset The start time of the sync node.
    Time Zone The time zone where the Kafka cluster resides.
    Output Fields The output fields, which can be customized.
  6. Click Save the settings in the toolbar.