This topic describes the Stop Word Filter component available in Designer.
Use this component to remove stop words—common, high-frequency words with little semantic weight, such as "the", "is", or "a"—from word tokenization results.
The Stop Word Filter component requires two inputs: an input table containing the text to be filtered, and a single-column stop word table where each row is a stop word.
You can configure the Stop Word Filter component's parameters using the Designer visual interface or a PAI command.
Component configuration
Configure the Stop Word Filter component using one of the following methods.
Method 1: Visual interface
Configure the component parameters on the Designer pipeline page.
|
Tab |
Parameter |
Description |
|
field settings |
columns to filter |
Specifies the columns to filter. To specify multiple columns, separate the column names with a comma (,). |
|
tuning |
cores |
By default, the system automatically allocates resources. |
|
memory size |
By default, the system automatically allocates resources. |
Method 2: PAI command
Configure the component's parameters with a PAI command. You can run PAI commands using the SQL Script component. For more information, see SQL Script.
PAI -name FilterNoise -project algo_public \
-DinputTableName=”test_input” -DnoiseTableName=”noise_input” \
-DoutputTableName=”test_output” \
-DselectedColNames=”words_seg1,words_seg2” \
-Dlifecycle=30|
Parameter |
Required |
Description |
Default |
|
inputTableName |
Yes |
The name of the input table that contains the tokenized text. |
None |
|
inputTablePartitions |
No |
The input table partitions to use. |
All partitions |
|
noiseTableName |
Yes |
The name of the stop word table. |
None |
|
noiseTablePartitions |
No |
The stop word table partitions to use. |
All partitions |
|
outputTableName |
Yes |
The name of the output table. |
None |
|
selectedColNames |
Yes |
The columns to filter. To specify multiple columns, separate the column names with a comma (,). |
None |
|
lifecycle |
No |
The lifecycle of the output table in days. The value must be a positive integer. |
None |
|
coreNum |
No |
The number of cores for the job. |
System-allocated |
|
memSizePerCore |
No |
The memory size per core. |
System-allocated |