All Products
Search
Document Center

DataWorks:Configure the data filtering node

Last Updated:Dec 29, 2023

The data filtering plug-in is used to filter data based on specific rules, such as the field size. Only data that meets the rules is retained.

Prerequisites

An input node is configured. For more information, see Data source types that support real-time synchronization.

Procedure

  1. Go to the DataStudio page.

    1. Log on to the DataWorks console.

    2. In the left-side navigation pane, click Workspaces.

    3. In the top navigation bar, select the region in which the workspace that you want to manage resides. On the Workspaces page, find the workspace and click Shortcuts > Data Development in the Actions column.

  2. In the Scheduled Workflow pane, move the pointer over the Create a table icon and choose Create Node > Data Integration > Real-time synchronization.

    Alternatively, right-click the required workflow, and then choose Create Node > Data Integration > Real-time synchronizationReal-time synchronization.

  3. In the Create Node dialog box, set the Sync Method parameter to End-to-end ETL and configure the Name and Path parameters.

    Important

    The node name cannot exceed 128 characters in length and can contain letters, digits, underscores (_), and periods (.).

  4. Click Confirm.

  5. On the configuration tab of the real-time synchronization node, drag Data Filtering in the Conversion section to the canvas on the right. Draw a line to connect the data filtering node to the input node.

  6. Click the data filtering node. In the configuration panel that appears, configure the parameters.

    数据过滤

    • Node configuration

      Rules: the rules based on which you want to filter data in a data source. Only data that meets the rules is retained. For example, in the preceding figure, a rule is configured to retain data for which the values of the "hello" field are less than the values of the "this" field.

      Note

      When you configure a rule, you can configure either the Customize parameter or the second Field parameter.

    • Output field

      The names and types of output fields after filtering.

  7. Click the 图标 in the top toolbar.