The LLM-MD5 Deduplicator (MaxCompute) component of Platform for AI (PAI) is used to calculate the MD5 hash value of a text and deduplicate the text based on the hash value. You can use the component during text preprocessing of large language models (LLMs).
Limits
The LLM-MD5 Deduplicator (MaxCompute) component supports only MaxCompute resources.
Algorithm
The algorithm calculates the hash value by using the hashlib.md5 hash method of the input text. If multiple text entries have the same hash value, only one text entry is retained.
After the algorithm removes the blank characters at the beginning and end of a text entry, the algorithm recalculates the hash value for the text entry. The characters are case-sensitive.
Configure the component
You can configure the parameters of the LLM-MD5 Deduplicator (MaxCompute) component in Machine Learning Designer. The following table describes the parameters.
Tab | Parameter | Required | Description | Default value |
Fields Setting | Select Target Column | Yes | The columns that you want to process. | No default value |
Output table lifecycle | No | The value is a positive integer. Unit: days. Default value: 28. After the default lifecycle of the table elapses, the temporary tables generated by the component are recycled. | 28 | |
Tuning | Number of CPUs per instance of map task | No | The number of CPUs for each instance of a map task. Valid values: 50 to 800. | 100 |
The memory size per instance of map task | No | The memory size of each instance of a map task. Unit: MB. Valid values: 256 to 12288. | 1024 | |
The maximum size of input data for a map | No | The maximum amount of data that each instance of a map task can process. You can use this parameter to manage the input of a map. Unit: MB. Valid values: 1 to Integer.MAX_VALUE. | 256 |
References
For more information about Machine Learning Designer, see Overview of Machine Learning Designer.