Object Storage Service:Model broadcasting

How it works

When multiple nodes running AI inference services pull model files from OSS simultaneously, the download activity can saturate the source's egress bandwidth. This creates a performance bottleneck that can cause startup delays or failures. This issue is particularly pronounced in regions with lower OSS egress bandwidth, where concurrent back-to-source traffic can severely impact deployment efficiency.

OSS Connector model broadcasting optimizes large-scale AI inference deployments. When multiple inference instances for the same model are launched, only one or a few nodes load the model data directly from OSS. This data is then distributed to other nodes through a chain-based topology. Model broadcasting leverages node storage and network resources to reduce back-to-source traffic, lessen the source's load, and improve distribution efficiency.

OSS Connector model broadcasting uses a chain-based transport method where model files are passed serially from one node to the next. Each node receives and forwards the data only once. For model file transfers, a single data stream is often sufficient to saturate the network bandwidth of most mainstream instance types. The chain-based method avoids the bandwidth bottlenecks that can occur in tree-based transport, where a node must send data to multiple downstream nodes simultaneously.

OSS Connector preloads model files from OSS into a memory buffer using a high-concurrency strategy. This lets the inference engine load the model into GPU memory as needed. The buffered memory is then released after a delay once inference is complete. The model broadcasting feature builds on this by enabling the buffer to be shared across nodes. It integrates DADI P2P functionalities and requires only a Redis or Tair service for node discovery and metadata management. This setup allows buffered data to be distributed to other nodes. Compared to a single-node deployment, this solution adds only lightweight buffer-sharing logic while fully using idle node egress bandwidth during model loading. This provides a cost-effective and efficient method for distributed model loading.

Prerequisites

• OSS Connector for AI/ML v1.2.0 or later is installed. For installation instructions, see Improve model deployment efficiency with OSS Connector for AI/ML.

• You have a Redis or Tair database available for node discovery and metadata management.

Configure the database

The model broadcasting feature requires a Redis or Tair service for node discovery and metadata management. You must configure this database to use the feature.

Enable model broadcasting

Add the model broadcasting configuration to the OSS Connector configuration file at /etc/oss-connector/config.json.

{
  ...
  "broadcast": {
    "enableBroadcast": true,
    "tenant": "${P2P_KEY_PREFIX}",
    "db": {
      "host": "${P2P_REDIS_HOST}",
      "port": 6379,
      "username": "${P2P_REDIS_USERNAME}",
      "password": "${P2P_REDIS_PASSWD}"
    }
  },
  "bindPort": 19898
  ...
}

The following table describes the configuration parameters.

For a complete example of how to deploy a model broadcasting service with multiple instances in a Kubernetes cluster, see Deploy a model broadcasting service with multiple instances.

Limit the cache size

During model broadcasting, nodes cache model data in memory for retrieval by other nodes. You can limit this cache memory in the following ways.

• Method 1: Set an environment variable

  export CONNECTOR_MAX_CACHE_ADVISE_GB=100

• Method 2: Set in the configuration file

  Set prefetch.maxCacheAdviseGB in /etc/oss-connector/config.json:

  {
    ...
    "prefetch": {
      "vcpus": 16,
      "workers": 24,
      "maxCacheAdviseGB": 100
    },
    ...
  }

Performance report

The following are the performance test results for the model broadcasting feature with the Qwen2.5-72B model (135.437 GB) in different regions.