The Filtering and Mapping component lets you select and rename columns, and filter rows by condition during data cleaning and feature engineering. Use it to control which columns and rows pass to downstream modeling components.
Configure the component
You can configure the component from the pipeline page (recommended) or by calling a PAI command in the SQL Script component.
Configure on the pipeline page
Add the Filtering and Mapping component to your pipeline and set the following parameters:
| Parameter | Description |
|---|---|
| Mapping Rules | The columns to include in the output. By default, all columns are selected. To rename a column, edit its name in this field. |
| Filter Criteria | A row filter expression, equivalent to a SQL WHERE clause. Specify a column name, an operator, and a value — for example, age>40. |
Supported operators for Filter Criteria:
| Operator |
|---|
= |
!= |
> |
< |
>= |
<= |
like |
rlike |
Use PAI commands
Use the SQL Script component to call PAI commands. For setup instructions, see SQL Script.
PAI -name Filter
-project algo_public
-DoutTableName="test_9"
-DinputPartitions="pt=20150501"
-DinputTableName="bank_data_partition"
-Dfilter="age>=40";| Parameter | Required | Description |
|---|---|---|
outputTableName | Yes | The name of the output table. |
inputTableName | Yes | The name of the input table. |
inputPartitions | No | The partitions to read from the input table. Set to None to read the full table. |
filter | No | A row filter expression, equivalent to a SQL WHERE clause. Example: age>40. |