Platform For AI:Enable memory caching for a local directory

How it works

EAS mounts model files to a local directory from OSS, NAS, or a Docker image. For more information, see Mount storage. Reading models, switching models, and scaling containers all require network bandwidth. In Stable Diffusion scenarios, inference requests frequently switch between base models and LoRA models, reading files from OSS or NAS and increasing network latency.

Memory caching solves this problem. The following diagram shows the architecture.

• Model files in a local directory are cached to memory.

• Cache uses an LRU eviction policy and supports file sharing among instances. Cached files appear as a standard file system directory.

• Service reads cached files directly from memory. No changes to business code are required.

• Instances in a service share a P2P network. When scaling out the service, new instances read cached files from nearby instances over P2P, accelerating cluster scale-out.

Limitations

• Cache directory is read-only to ensure data consistency.

• To add a model file, place it in the source directory. The file becomes available in the cache directory automatically.

• Do not modify or delete model files in the source directory. Doing so may cause dirty data in the cache.

Prerequisites

Ensure the following requirements are met:

• PAI workspace with EAS access

• Model files in OSS or NAS

• Instance type with sufficient memory for caching (for example, ml.gu7i.c16m60.1-gu30)

Procedure

This section uses Stable Diffusion as an example with the following configuration:

• Image startup command: ./webui.sh --listen --port 8000 --skip-version-check --no-hashing --no-download-sd-model --skip-prepare-environment --api --filebrowser

• OSS directory for model files: oss://path/to/models/

• Container directory for model files (source directory): /code/stable-diffusion-webui/data-slow

Source directory /code/stable-diffusion-webui/data-slow stores model files, which are then cached to /code/stable-diffusion-webui/data-fast. The service reads model files from the cache directory instead of the source directory.

{
    "cloud": {
        "computing": {
            "instances": [
                {
                    "type": "ml.gu7i.c16m60.1-gu30"
                }
            ]
        }
    },
    "containers": [
        {
            "image": "eas-registry-vpc.cn-hangzhou.cr.aliyuncs.com/pai-eas/stable-diffusion-webui:4.2",
            "port": 8000,
            "script": "./webui.sh --listen --port 8000 --skip-version-check --no-hashing --no-download-sd-model --skip-prepare-environment --api --filebrowser --ckpt-dir /code/stable-diffusion-webui/data-fast"
        }
    ],
    "metadata": {
        "cpu": 16,
        "enable_webservice": true,
        "gpu": 1,
        "instance": 1,
        "memory": 60000,
        "name": "sdwebui_test"
    },
    "options": {
        "enable_cache": true
    },
    "storage": [
        {
            "cache": {
                "capacity": "20G",
                "path": "/code/stable-diffusion-webui/data-slow"
            },
            "mount_path": "/code/stable-diffusion-webui/data-fast"
        },
        {
            "mount_path": "/code/stable-diffusion-webui/data-slow",
            "oss": {
                "path": "oss://path/to/models/",
                "readOnly": false
            },
            "properties": {
                "resource_type": "model"
            }
        }
    ]
}