OSS Connector for AI/ML in Kubernetes - Object Storage Service

Installez et configurez le connecteur OSS pour l'IA et le ML dans Kubernetes afin de charger les données de modèle depuis OSS pour les services d'inférence.

Installer le composant Connecteur

Installez le composant Connecteur dans le conteneur de votre application en suivant l'une des méthodes ci-dessous.

Méthode 1 : Conteneur d'initialisation

Cette méthode télécharge et extrait le connecteur vers un volume partagé avant le démarrage du conteneur d'application.

initContainers:
  # Init Container: runs before the application container to prepare dependencies
  - name: install-connector
    image: busybox
    command:
      - sh
      - -c
      - |
        # Download the Connector DEB package
        wget -q https://gosspublic.alicdn.com/oss-connector/oss-connector-lib-1.2.0.x86_64.deb -O /tmp/connector.deb
        # Create a temp directory and extract the DEB package
        mkdir -p /tmp/extract && cd /tmp/extract
        ar x /tmp/connector.deb
        # Extract only the required .so file to the shared directory
        # The /shared directory is mounted as a volume visible to the main container
        mkdir -p /shared/usr/local/lib
        tar -xf data.tar.gz -O ./usr/local/lib/libossc_preload.so > /shared/usr/local/lib/libossc_preload.so
    volumeMounts:
      # Mount the connector-lib volume at /shared
      # The main container also mounts this volume for file sharing
      - name: connector-lib
        mountPath: /shared

containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    volumeMounts:
      # Mount the connector-lib volume at /usr/local/lib in the container
      # This makes the .so file extracted by the Init Container available
      - name: connector-lib
        mountPath: /usr/local/lib
        subPath: usr/local/lib

volumes:
  # Shared volume for passing files between the Init Container and the main container
  - name: connector-lib
    emptyDir: {}

Méthode 2 : Installation au démarrage

Cette méthode télécharge et installe directement le connecteur via la commande de démarrage du conteneur.

containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    command: ["/bin/bash", "-c"]
    args:
      - |
        # Download and install the Connector DEB package
        wget https://gosspublic.alicdn.com/oss-connector/oss-connector-lib-1.2.0.x86_64.deb
        dpkg -i oss-connector-lib-1.2.0.x86_64.deb
        # Start the model serving process
        ENABLE_CONNECTOR=1 python3 -m vllm.entrypoints.openai.api_server --model ${MODEL_DIR} ...

Méthode 3 : Dockerfile personnalisé

Préinstallez le connecteur dans une image personnalisée basée sur vllm/vllm-openai.

FROM vllm/vllm-openai:latest
RUN wget https://gosspublic.alicdn.com/oss-connector/oss-connector-lib-1.2.0.x86_64.deb && \
    dpkg -i oss-connector-lib-1.2.0.x86_64.deb

Construisez et publiez l'image :

docker build -t myregistry/vllm-with-connector:latest .
docker push myregistry/vllm-with-connector:latest

Utilisez l'image personnalisée :

containers:
  - name: vllm
    image: myregistry/vllm-with-connector:latest

Comparaison des méthodes d'installation

Méthode

Cas d'utilisation

Avantages

Inconvénients

Conteneur d'initialisation / Installation au démarrage

Tests, validation ou développement sans modifier l'image de base

Aucune image personnalisée requise, déploiement flexible, configuration simple

Téléchargement et extraction à chaque démarrage, démarrage à froid plus lent, nécessite un accès au réseau externe

Dockerfile personnalisé

Environnements de production, services de longue durée, clusters à grande échelle

Démarrage le plus rapide, image autonome, stabilité élevée, réutilisable

Nécessite la maintenance d'une image personnalisée, verrouillage de version

Déployer un service d'inférence de modèle

Cet exemple déploie un service d'inférence qui charge les données de modèle depuis OSS, en utilisant la méthode du conteneur d'initialisation avec vllm/vllm-openai.

# ConfigMap: stores the Connector configuration file
# Purpose: mounts the Connector config as a file inside the container
# Mount path: /etc/oss-connector/config.json
apiVersion: v1
kind: ConfigMap
metadata:
  name: connector-config
data:
  config.json: |
    {
        "logLevel": 1,
        "logPath": "/var/log/oss-connector/connector.log",
        "auditPath": "/var/log/oss-connector/audit.log",
        "expireTimeSec": 120,
        "prefetch": {
            "vcpus": 16,
            "workers": 16
        }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-connector-deployment
spec:
  selector:
    matchLabels:
      app: model-connector
  template:
    metadata:
      labels:
        app: model-connector
    spec:
      # Init Container: runs before the application container to prepare dependencies
      initContainers:
        - name: install-connector
          image: busybox
          command: ["/bin/sh", "-c"]
          args:
            - |
              # Download the Connector DEB package
              wget -q https://gosspublic.alicdn.com/oss-connector/oss-connector-lib-1.2.0.x86_64.deb -O /tmp/connector.deb

              # Create a temp directory and extract the DEB package
              mkdir -p /tmp/extract && cd /tmp/extract
              ar x /tmp/connector.deb

              # Extract only the required .so file to the shared directory
              # The /shared directory is mounted as a volume visible to the main container
              mkdir -p /shared/usr/local/lib
              tar -xf data.tar.gz -O ./usr/local/lib/libossc_preload.so > /shared/usr/local/lib/libossc_preload.so

          volumeMounts:
            # Mount the connector-lib volume at /shared
            # The main container also mounts this volume for file sharing
            - name: connector-lib
              mountPath: /shared
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: "16"
            memory: "70Gi"
          limits:
            cpu: "20"
            memory: "80Gi"
        command: ["/bin/bash", "-c"]
        args:
          - |
            # Add the Connector library to LD_PRELOAD to intercept file system calls
            export LD_PRELOAD="/usr/local/lib/libossc_preload.so${LD_PRELOAD:+:$LD_PRELOAD}"
            # Start the vLLM server
            # ENABLE_CONNECTOR=1 enables OSS Connector for accelerated model loading
            # The model path ${MODEL_DIR}/qwen/Qwen3-8B/ points to an OSS path,
            # intercepted and redirected by the Connector
            ENABLE_CONNECTOR=1 python3 -m vllm.entrypoints.openai.api_server \
              --model ${MODEL_DIR}/qwen/Qwen3-8B/ \
              --trust-remote-code \
              --tensor-parallel-size 1 \
              --disable-custom-all-reduce
        env:
        # OSS access configuration: specify the internal endpoint and region
        - name: OSS_ENDPOINT
          value: "oss-cn-beijing-internal.aliyuncs.com"
        - name: OSS_REGION
          value: "cn-beijing"

        # Root path on OSS. The Connector maps local paths to this OSS path
        - name: OSS_PATH
          value: "oss://examplebucket/"

        # Local mount directory (intercepted by the Connector, data is loaded from OSS)
        - name: MODEL_DIR
          value: "/var/model"

        # Read OSS access credentials from the Secret (oss-access-key-connector)
        # Create the Secret in advance:
        #   kubectl create secret generic oss-access-key-connector \
        #     --from-literal=key=<OSS_ACCESS_KEY_ID> \
        #     --from-literal=secret=<OSS_ACCESS_KEY_SECRET>
        - name: OSS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: oss-access-key-connector
              key: key
        - name: OSS_ACCESS_KEY_SECRET
          valueFrom:
            secretKeyRef:
              name: oss-access-key-connector
              key: secret
        volumeMounts:
          # Mount the Connector configuration file
          - name: connector-config
            mountPath:  /etc/oss-connector/
          # Mount the connector-lib volume at /usr/local/lib in the container
          # This makes the .so file extracted by the Init Container available
          - name: connector-lib
            mountPath: /usr/local/lib
            subPath: usr/local/lib
      terminationGracePeriodSeconds: 10
      volumes:
      # Connector configuration ConfigMap
      - name: connector-config
        configMap:
          name: connector-config
      # Shared volume for passing files between the Init Container and the main container
      - name: connector-lib
        emptyDir: {}

Déployer un service de diffusion de modèle multi-instances

Cet exemple utilise une image préconstruite (myregistry/vllm-with-connector) pour diffuser le modèle sur plusieurs réplicas. Pour plus d'informations sur la diffusion de modèles, consultez Diffusion de modèles.

# ConfigMap: stores the Connector configuration template
# Purpose: provides a template (config.json.tmpl) that the init container renders
#          into the final config file for the main container
# Template mount path (init container): /tmpl/config.json.tmpl
# Rendered output path (main container): /etc/oss-connector/config.json
apiVersion: v1
kind: ConfigMap
metadata:
  name: connector-config
data:
  config.json.tmpl: |
    {
        "logLevel": 1,
        "logPath": "/var/log/oss-connector/connector.log",
        "auditPath": "/var/log/oss-connector/audit.log",
        "expireTimeSec": 120,
        "prefetch": {
            "vcpus": 16,
            "workers": 16
        },
        "broadcast": {
            "enableBroadcast": true,
            "tenant": "${REDIS_TENANT}",
            "db": {
                "host": "${REDIS_HOST}",
                "port": 6379,
                "username": "${REDIS_USERNAME}",
                "password": "${REDIS_PASSWORD}"
            }
        },
        "bindPort": 19989
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-connector-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: model-connector
  template:
    metadata:
      labels:
        app: model-connector
    spec:
      # Init container: renders the config template into the final config.json
      #                  before the main container starts
      initContainers:
      - name: render-config
        image: busybox
        command: ["/bin/sh", "-c"]
        args:
          # Use sed to replace template placeholders with actual environment variable values
          - |
            sed -e "s|\${REDIS_HOST}|$REDIS_HOST|g" \
                -e "s|\${REDIS_USERNAME}|$REDIS_USERNAME|g" \
                -e "s|\${REDIS_PASSWORD}|$REDIS_PASSWORD|g" \
                -e "s|\${REDIS_TENANT}|$REDIS_TENANT|g" \
                /tmpl/config.json.tmpl > /etc/oss-connector/config.json
        env:
        # Read Redis connection details from the Secret (redis-secret)
        # Create the Secret in advance:
        #   kubectl create secret generic redis-secret \
        #     --from-literal=host=<host> \
        #     --from-literal=username=<username> \
        #     --from-literal=password=<password>
        - name: REDIS_HOST
          valueFrom:
            secretKeyRef:
              name: redis-secret
              key: host
        - name: REDIS_USERNAME
          valueFrom:
            secretKeyRef:
              name: redis-secret
              key: username
        - name: REDIS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: redis-secret
              key: password
        - name: REDIS_TENANT
          value: "broadcast-demo"
        volumeMounts:
          # Mount the ConfigMap template as the input for sed
          - name: connector-config
            mountPath: /tmpl
          # Mount the shared emptyDir to write the rendered config for the main container
          - name: rendered-config
            mountPath: /etc/oss-connector
      containers:
      - name: vllm
        image: myregistry/vllm-with-connector:latest
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: "16"
            memory: "70Gi"
          limits:
            cpu: "20"
            memory: "80Gi"
        command: ["/bin/bash", "-c"]
        args:
          - |
            # Add the Connector library to LD_PRELOAD to intercept file system calls
            export LD_PRELOAD="/usr/local/lib/libossc_preload.so${LD_PRELOAD:+:$LD_PRELOAD}"
            # Start the vLLM server
            # ENABLE_CONNECTOR=1 enables OSS Connector for accelerated model loading
            ENABLE_CONNECTOR=1 python3 -m vllm.entrypoints.openai.api_server \
              --model ${MODEL_DIR}/qwen/Qwen3-8B/ \
              --trust-remote-code \
              --tensor-parallel-size 1 \
              --disable-custom-all-reduce
        env:
        # OSS access configuration: specify the internal endpoint and region
        - name: OSS_ENDPOINT
          value: "oss-cn-beijing-internal.aliyuncs.com"
        - name: OSS_REGION
          value: "cn-beijing"

        # Root path on OSS. The Connector maps local paths to this OSS path
        - name: OSS_PATH
          value: "oss://examplebucket/"

        # Local mount directory (intercepted by the Connector, data is loaded from OSS)
        - name: MODEL_DIR
          value: "/var/model"

        # Read OSS access credentials from the Secret (oss-access-key-connector)
        - name: OSS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: oss-access-key-connector
              key: key
        - name: OSS_ACCESS_KEY_SECRET
          valueFrom:
            secretKeyRef:
              name: oss-access-key-connector
              key: secret
        volumeMounts:
          # Connector configuration (rendered by init container)
          - name: rendered-config
            mountPath: /etc/oss-connector/
      terminationGracePeriodSeconds: 10
      volumes:
      # Connector config template ConfigMap (mounted in init container)
      - name: connector-config
        configMap:
          name: connector-config
      # Rendered config (shared from init container to main container)
      - name: rendered-config
        emptyDir: {}