Dataphin:Configure the filter component

Procedure

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

2. In the top navigation bar of the Data Integration page, select a project. If you are in Dev-Prod mode, you also need to select an environment.

3. In the left-side navigation pane, click Batch Integration. In the Batch Integration list, click the batch pipeline you want to develop to open its configuration page.

4. In the upper-right corner of the page, click Component Library to open the Component Library panel.

5. In the left-side navigation pane of the Component Library panel, select Transform. In the list of components on the right, find the filter component and drag it onto the canvas.

6. Drag the icon from an upstream component or a downstream component to the filter component to connect them.

7. Click the icon on the filter component card to open the Transform-Filter Configuration dialog box.

8. In the Transform-Filter Configuration dialog box, configure the following parameters.

Parameter	Description
Step name	The name of the filter component. Dataphin automatically generates a step name, which you can modify based on your business needs. The naming rules are as follows: Can only contain Chinese characters, letters, underscores (_), and digits. Cannot exceed 64 characters in length.
Input fields	Displays the input fields based on the output of the upstream component. If a field name includes a table name, the table name is also displayed.
Filter condition	You can set the condition in script mode or UI mode. Script mode: Set filter conditions by using a script. You do not need to enter WHERE. Directly enter the content that follows the WHERE clause. For example, user_id != nil && string.indexOf(title,'order') > -1. Script mode Aviator expressions support only the following syntax: Logical operators: Use && for AND and || for OR. The keywords and and or are not supported. Null values: Use col == nil for null and col != nil for not null. Equality: Use == for comparison. The = operator is not supported. Text matching (The LIKE function is not available) Contains: string.indexOf(col,'xx') > -1 Does not contain: string.indexOf(col,'xx') <= -1 Starts with: string.startsWith(col,'xx') Ends with: string.endsWith(col,'xx') Enclose strings and dates in single quotation marks ('). UI mode: Use the graphical interface to set the filter condition. Configure the field, operator, and value in sequence. If a field name includes a table name, the table name is also displayed. Supported filter condition operators include =, !=, >, >=, <, <=, LIKE, NOT LIKE, IS NULL, IS NOT NULL, START WITH, and END WITH. You can combine multiple conditions by using the AND and OR logical operators. For example, if your input fields are id, name, and ds, and you need to filter for data where id<123 and name does not equal abc, enter id < 123 AND name != 'abc' in script mode. If you use the UI, set the filter conditions as shown in the following figure. Important The LIKE operator does not support using the percent sign (%) as a wildcard. It only matches data that contains the specified characters. To filter based on the start or end of a string, use the START WITH and END WITH operators. For example, name START WITH 'Zhang' AND name END WITH 'San' filters for records where the name field starts with Zhang and ends with San.

9. Click OK to finish configuring the filter component.