DataWorks:Data filtering

Step 1: Configure the data integration task

1. Create a data source. For more information, see Data source management.

2. Create a Data Integration task. For more information, see Configure a single-table real-time synchronization task.

   Note

   For single-table real-time Data Integration tasks, you can add a data processing component between the source and destination components. For more information, see Supported data sources and synchronization solutions.

Step 2: Add the data filtering component

1. In the real-time single-table synchronization task, enable the Data Processing toggle, click +Add Node, and then select the Data Filter component.

2. Configure the filtering rules.

   • Field Name: Select an input field. This field's value is the left operand of the conditional expression.

   • Judgement: Select a comparison operator. The following table describes the available operators.

     Comparison operators

     Type	Description
     Equals	Checks if numeric or string values are equal.
     Does not equal	Checks if numeric or string values are not equal.
     Less than	Checks if a numeric value is less than another, or if a string value comes before another in lexicographical order.
     Greater than	Checks if a numeric value is greater than another, or if a string value comes after another in lexicographical order.
     Less than or equal to	Checks if a numeric value is less than or equal to another, or if a string value is less than or equal to another in lexicographical order.
     Greater than or equal to	Checks if a numeric value is greater than or equal to another, or if a string value is greater than or equal to another in lexicographical order.
     Contains	Checks if a string contains a specified substring.
     Does not contain	Checks if a string does not contain a specified substring.
     Starts with	Checks if a string starts with a specified substring.
     Ends with	Checks if a string ends with a specified substring.
     Is null	Checks if a field of any type is null.
     Is not null	Checks if a field of any type is not null.

   • Value: The right operand of the conditional expression. You can select an input field from the upstream node or enter a constant value. Regular expressions are not supported for constant values. Java escape sequences are supported, such as \t (tab character), \n (line feed), and \u0001 (Unicode character).

3. Add rule: You can add multiple filtering rules. For example, enter Filter in the Name field and set a condition where the Field Name is _key_, the comparison operator is equals, the value type is string, and the Value is key1. Use the + and − buttons to add or remove rules, or click + Add condition group to create a group.

Next steps

After you configure the data source and Data filtering components, click Preview Data Output to review the output from the current node and verify that it meets your requirements.