The LLM-Special Character Ratio Filter (MaxCompute) component filters text samples by the ratio of special characters to text length. Add it to your large language model (LLM) preprocessing pipeline to remove text that contains too many or too few special characters — improving training data quality before it reaches your model.
Limitations
This component supports MaxCompute resources only.
How it works
The component scans every character in the text and calculates:
special character ratio = count of special characters / total text lengthSpecial characters fall into five categories:
Punctuation (
string.punctuation)Digits (
string.digits)Whitespace (
string.whitespace)Emojis
Other special characters
A text sample is filtered out if its ratio falls below the minimum threshold or above the maximum threshold.
Configure the component
Configure the following parameters in Machine Learning Designer.
Fields setting
| Parameter | Required | Description | Default |
|---|---|---|---|
| Select Target Column | Yes | Columns to process. Select one or more columns. | — |
| Minimum Ratio | No | Filters out samples whose special character ratio is below this value. Set to 0 to keep all samples regardless of how few special characters they contain. | 0 |
| Maximum Ratio | Yes | Filters out samples whose special character ratio exceeds this value. | — |
| Output table lifecycle | No | Retention period for temporary tables generated by the component, in days. The value must be a positive integer. After this period, the tables are recycled. | 28 |
Tuning
| Parameter | Required | Valid values | Default |
|---|---|---|---|
| Number of CPUs per instance of map task | No | 50–800 | 100 |
| The memory size per instance of map task | No | 256–12288 MB | 1024 |
| The maximum size of input data for a map | No | 1–Integer.MAX_VALUE MB | 256 |
What's next
For an overview of Machine Learning Designer and how to build preprocessing pipelines, see Overview of Machine Learning Designer.