DataHub Reader reads data from DataHub in real time by using the DataHub SDK.

Background information

DataHub Reader keeps running after it is started and reads data from DataHub when new data is stored to DataHub. DataHub Reader provides the following features:
  • Reads data in real time.
  • Reads data concurrently based on the number of shards in DataHub.

Procedure

  1. Go to the DataStudio page.
    1. Log on to the DataWorks console.
    2. In the left-side navigation pane, click Workspaces.
    3. In the top navigation bar, select the region in which the workspace that you want to manage resides. On the Workspaces page, find the workspace and click DataStudio in the Actions column.
  2. In the Scheduled Workflow pane, move the pointer over the Create a table icon and choose Data Integration > Real-time synchronization.
    Alternatively, you can click the required workflow, right-click Data Integration, and then choose Create > Real-time synchronization.
  3. In the Create Node dialog box, set the Sync Method parameter to End-to-end ETL and configure the Name and Path parameters.
    Important The node name cannot exceed 128 characters in length and can contain letters, digits, underscores (_), and periods (.).
  4. Click Commit.
  5. On the configuration tab of the real-time sync node, drag DataHub under Input to the canvas on the right.
  6. Click the new DataHub node. In the configuration pane that appears, set the required parameters in the Node configuration section.
    DataHub
    ParameterDescription
    Data sourceThe connection to the DataHub data store. In this example, you can select only a DataHub connection.

    If no connection is available, click New data source on the right to create one on the Data Source page. For more information, see Add a DataHub data source.

    TopicThe name of the topic from which data is read in DataHub. You can click Data preview on the right to preview the selected topic.
    Output fieldThe fields from which data is read.
  7. Click the Save icon in the toolbar.