You can use the LLM-Text Normalizer (MaxCompute) component of Platform for AI (PAI) to perform operations such as Unicode text normalization or language switching from traditional to simplified Chinese. You can use the component during text preprocessing of the large language models (LLMs).
Limits
The LLM-Text Normalizer (MaxCompute) component supports only MaxCompute resources.
Algorithm
The LLM-Text Normalizer (MaxCompute) component supports the following features:
Unicode text normalization by using the Normalization Form Compatibility Composition (NFKC) method.
ftfy.fix_text(text, normalization='NFKC')Switching from traditional to simplified Chinese by using the opencc package.
opencc
The following figures show the result.
Before processing:
After processing:
Configure the component
You can configure the parameters of the LLM-Text Normalizer (MaxCompute) component in Machine Learning Designer. The following table describes the parameters.
Tab | Parameter | Required | Description | Default value |
Fields Setting | Select Target Column | Yes | The columns that you want to process. You can select multiple columns. | No default value |
Output table lifecycle | No | The value is a positive integer. Unit: days. Default value: 28. After the default lifecycle of the table elapses, the temporary tables generated by the component are recycled. | 28 | |
Tuning | Number of CPUs per instance of map task | No | The number of CPUs for each instance of a map task. Valid values: 50 to 800. | 100 |
The memory size per instance of map task | No | The memory size of each instance of a map task. Unit: MB. Valid values: 256 to 12288. | 1024 | |
The maximum size of input data for a map | No | The maximum amount of data that each instance of a map task can process. You can use this parameter to manage the input of a map. Unit: MB. Valid values: 1 to Integer.MAX_VALUE. | 256 |
References
For more information about Machine Learning Designer, see Overview of Machine Learning Designer.