LLM pretraining corpora often contain texts with excessive repeated character sequences or word phrases — a signal of low-quality or garbled content. The LLM-N-Gram Repetition Filter (DLC) component detects and removes such texts by calculating their N-gram repetition ratio. Add it to a Machine Learning Designer pipeline to clean your LLM pretraining data before model training.
Input data must be in JSON Lines format: each line is a valid JSON object, but the file as a whole is not.
Supported computing resources
How it works
The component moves an N-character window across a text to generate sequences of N characters or words. Each sequence is called an N-gram. It then calculates the repetition ratio using this formula:
Repetition ratio = Cumulative frequency of N-grams that appear more than once
÷ Total frequency of all N-gramsTexts whose repetition ratio falls outside the configured min–max range are filtered out.
For word-level N-grams, all words are converted to lowercase before the ratio is calculated.
Configure the component
On the Pipeline page of Machine Learning Designer, configure the LLM-N-Gram Repetition Filter (DLC) component.
Filter types
The component supports two independent filter types. Enable one or both depending on your data quality requirements.
| Filter type | Detects | Key parameters |
|---|---|---|
| Character-level N-gram | Repeated character sequences (e.g., repeated substrings, garbled text) | Length N, Minimum Ratio Value, Maximum Ratio Value |
| Word-level N-gram | Repeated word sequences (e.g., duplicated phrases, boilerplate text) | Length N, Minimum Ratio, Maximum Ratio |
Fields Setting tab
| Parameter | Required | Description | Default |
|---|---|---|---|
| Target Process Field | Yes | The JSON field name containing the text to filter | — |
| Whether to Filter with Character-level N-Gram Repetition Ratio | No | Enable character-level N-gram filtering. When enabled, configure Length N, Minimum Ratio Value, and Maximum Ratio Value. | Unselected |
| Whether to Filter with Word-level N-Gram Repetition Ratio | No | Enable word-level N-gram filtering. When enabled, configure Length N, Minimum Ratio, and Maximum Ratio. | Unselected |
| OSS Directory for Saving Output Data | No | The Object Storage Service (OSS) bucket path for storing output data. If left blank, the default workspace path is used. | — |
Character-level N-gram sub-parameters (visible when the character-level filter is enabled):
| Parameter | Valid values | Description |
|---|---|---|
| Length N | Positive integer | The length of each N-gram sequence |
| Minimum Ratio Value | 0.0–1.0 | Texts with a repetition ratio below this threshold are filtered out |
| Maximum Ratio Value | 0.0–1.0 | Texts with a repetition ratio above this threshold are filtered out |
Word-level N-gram sub-parameters (visible when the word-level filter is enabled):
| Parameter | Valid values | Description |
|---|---|---|
| Length N | Positive integer | The length of each N-gram sequence |
| Minimum Ratio | 0.0–1.0 | Texts with a repetition ratio below this threshold are filtered out |
| Maximum Ratio | 0.0–1.0 | Texts with a repetition ratio above this threshold are filtered out |
Tuning tab
| Parameter | Required | Description | Default |
|---|---|---|---|
| Number of Processes | No | Number of parallel processes for data processing | 8 |
Select Resource Group tab
| Resource group type | Required | Description |
|---|---|---|
| Public Resource Group | No | Specify the instance type (CPU or GPU), number of instances, and virtual private cloud (VPC) |
| Dedicated resource group | No | Specify the number of vCPUs, memory, shared memory, GPUs, and instances |
| Maximum Running Duration | No | If the job exceeds this duration, it is terminated |