Process LLM data with MaxFrame

As artificial intelligence (AI) advances, large language models (LLMs) are being widely adopted for various business and data analysis tasks. Data processing is a critical step in LLM development, as data quality directly impacts the performance of model training and inference. Compared to expensive GPU resources, MaxCompute's massive and elastic CPU resources provide a cost-effective foundation for large-scale LLM data processing. The distributed computing capabilities of MaxFrame help you work more efficiently and easily.

Prerequisites

MaxFrame is installed. For more information, see Preparations.

Prepare the data

This tutorial uses a small subset (10,000+ records) of GitHub data from the open-source RedPajama project to show you how to use MaxFrame to clean and process GitHub code data.

For your convenience, the data is stored in the llm_redpajama_github_demo_data table within the data_science schema of the BIGDATA_PUBLIC_DATASET public project in MaxCompute. You can use the data directly.

The table contains 11 columns: repo_name, id, size, content, binary, copies, ref, path, mode, license, and language. These columns record metadata for code files in GitHub repositories, including the repository name, a content snippet, a binary flag, the number of copies, the branch reference, the file path, file permissions, the license, and the programming language.

For more information about the public dataset, see Overview of public datasets.

Remove copyright information using MaxFrame

If the source data contains copyright information, such as the word "Copyright", you need to remove this sensitive information.

For example, when repo_name is "menuka94/cdnjs", the corresponding content field contains copyright information.

Create a MaxCompute entry object.
```
import os
import time
import numpy as np
import maxframe.dataframe as md
from odps import ODPS
from maxframe import new_session
# from maxframe.udf import with_resource_libraries
from maxframe.config import options
from maxframe import config
o = ODPS(
    # Ensure that the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable is set to your AccessKey ID,
    # and the ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variable is set to your AccessKey Secret.
    # We recommend that you do not hardcode your AccessKey ID and AccessKey Secret.
    os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'),
    os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'),
    project='your-default-project',
    endpoint='your-end-point',
)
```
Parameters:
- ALIBABA_CLOUD_ACCESS_KEY_ID: Set this environment variable to an AccessKey ID with the required MaxCompute permissions on the objects to be processed in the target MaxCompute project. You can obtain an AccessKey ID on the AccessKey Management page.
- ALIBABA_CLOUD_ACCESS_KEY_SECRET: Set this environment variable to the AccessKey Secret for the AccessKey ID.
- your-default-project : Specify the name of the MaxCompute project. Log on to the MaxCompute console and, in the left-side navigation pane, choose Workspace > Projects to view the MaxCompute project name.
- your-end-point: Specify the endpoint of the region where your MaxCompute project is located. You can select an endpoint based on your network connection type. Example: http://service.cn-chengdu.maxcompute.aliyun.com/api. For more information, see Endpoints.
Reference the MaxCompute built-in image common, which contains a Python environment and third-party packages, such as regex, required for this data processing.
```
config.options.sql.settings = {
    "odps.session.image": "common"
}
```

Use a user-defined function (UDF) to define the data processing logic.

def clean_copyright(row):
    import re
    pat = re.compile('/\\*[^*]*\\*+(?:[^/*][^*]*\\*+)*/')
    cpat = re.compile('copyright', re.IGNORECASE)
    text = row['content']
    if not text:
        return row
    r = pat.search(text)
    if r:
        span = r.span()
        sub = text[span[0]:span[1]]
        if cpat.search(sub):
            # Remove the copyright block.
            text = text[:span[0]] + text[span[1]:]
        row['content'] = text
        return row
    lines = text.split('\n')
    skip = 0
    for k in range(len(lines)):
        if (lines[k].startswith('//') or lines[k].startswith('#')
                or lines[k].startswith('--') or not lines[k]):
            skip = skip + 1
        else:
            break
    if skip:
        text = '\n'.join(lines[skip:])
    row['content'] = text
    return row

Create a MaxFrame session and submit the job to MaxCompute.

def maxframe_job():
    s_time = time.time()
    table_name = 'bigdata_public_dataset.data_science.llm_redpajama_github_demo_data'
    session = new_session(o)
    print('session id: ', session.session_id)
    df = md.read_odps_table(table_name, index_col='id')
    df = df.apply(
                clean_copyright,
                axis=1,  # row
                output_type="dataframe",
            )
    out_table = 'tmp_mf_clean_copyright'
    md.to_odps_table(df, out_table).execute()
    session.destroy()
maxframe_job()

Data processing results

Query the tmp_mf_clean_copyright table to view the data that previously contained copyright information. The sensitive information is now removed.

SELECT * FROM tmp_mf_clean_copyright;

The result includes columns such as id, repo_name, size, content, binary, copies, ref, path, mode, license, and language. For example, in the approximately 10 records from cdnjs-related repositories, the content column shows the code snippets without copyright information. For these records, the value of mode is 33188, and the value of license is 'mit'.

Next steps

MaxCompute is integrated with Alibaba Cloud Platform for AI (PAI). You can use PAI Designer to develop and use more LLM operators. For more information about LLM operators, see Component reference: Data processing for foundation model.