Platform for AI (PAI) - Machine Learning Designer provides data preprocessing operators for LLM and common templates
Feb 28 2024
Platform for AI (PAI)Content
Intended customers: customers who require Large Language Model (LLM) pre-training and supervised fine-tuning (SFT). New features: High-quality data preprocessing is an important step in LLM application. Machine Learning Designer of Platform for AI (PAI) provides commonly used high-performance operators for data preprocessing, such as deduplication, standardization, and sensitive information masking. You can use large-scale distributed computing capabilities based on MaxCompute when you preprocess data for LLMs to improve efficiency and improve the reliability and performance of LLMs.