Platform For AI:Deploy an asynchronous inference service

Access an asynchronous inference service

The system creates a same-named service group. Because the queue sub-service handles incoming traffic, access it directly through the following endpoints. Access a queue service.

Manage an asynchronous inference service

Manage an asynchronous inference service like a regular service. The system manages sub-services automatically: deleting the service removes both sub-services, and updating the inference sub-service leaves the queue sub-service unchanged.

Even with one configured replica, the instance list shows an additional queue sub-service instance.

The replica count refers to inference sub-service replicas. Queue sub-service replicas scale automatically. For example, scaling inference replicas to 3 increases queue replicas to 2.

Replica scaling rules:

• When the service is stopped, both sub-services scale to 0 replicas.

• With one inference replica, the queue sub-service also has one replica (unless configured otherwise).

• With two or more inference replicas, the queue sub-service maintains two replicas (unless configured otherwise).

• If auto scaling allows a minimum of 0 replicas, the queue sub-service retains one standby replica when inference replicas scale to 0.

Queue sub-service parameters

The queue sub-service works with the default configuration in most cases. Customize it in the top-level queue field of the JSON file:

{  
  "queue": {
     "sink": {
        "memory_ratio": 0.3
     },
     "source": {
        "auto_evict": true,
     }
 }

The following sections describe the configuration options.

Queue sub-service resources

By default, queue sub-service resources inherit from metadata. Configure them separately if needed.

• Declare the resource group for the queue sub-service using queue.resource.

  {
    "queue": {
      "resource": "eas-r-slzkbq4tw0p6xd****"  // By default, it uses the resource group of the inference sub-service.
    }
  }

  • Defaults to the inference sub-service resource group.

  • To deploy the queue sub-service in a public resource group, set resource to an empty string (""). This is useful when your dedicated resource group lacks CPU or memory.

    Note

    Deploy the queue sub-service in a public resource group when possible.

• Declare the CPU (in cores) and memory (in MB) for each queue sub-service replica using queue.cpu and queue.memory.

  {
    "queue": {
       "cpu": 2,  // Default: 1.
       "memory": 8000  // Default: 4000.
    }
  }

  The default (1 CPU core, 4 GB memory) is sufficient for most scenarios.

  Important

  • For more than 200 subscribers (inference sub-service replicas), configure 2 or more CPU cores.

  • Do not reduce queue sub-service memory in production.

• Configure the minimum number of replicas for the queue sub-service using queue.min_replica.

  {
    "queue": {
       "min_replica": 3  // Default: 1.
    }
  }

  Queue sub-service replicas scale automatically with running inference replicas. The default range is [1, min{2, the number of inference sub-service replicas}]. If auto scaling allows scaling to 0, one queue replica is retained. Use queue.min_replica to adjust this minimum.

  Note

  More queue replicas improve availability, not performance.

Queue sub-service features

The queue sub-service supports the following feature configurations.

• Configure automatic data eviction for the sink and input queues using queue.sink.auto_evict or queue.source.auto_evict, respectively.

  {
    "queue": {
       "sink": {
          "auto_evict": true  // Enables automatic eviction for the sink queue. Default: false.
        },
        "source": {
           "auto_evict": true  // Enables automatic eviction for the input queue. Default: false.
        }
    }
  }

  Automatic eviction is disabled by default—a full queue rejects new data. Enable eviction to drop the oldest data and make room for new entries.

• Configure the maximum number of delivery attempts using queue.max_delivery.

  {
     "queue": {
        "max_delivery": 10  // The maximum number of delivery attempts is 10. Default: 5. If set to 0, this feature is disabled, and data can be delivered an unlimited number of times.
     }
  }

  When delivery attempts exceed the threshold, the message is marked as a dead letter. Dead-letter policy.

• Configure the maximum processing time for a message using queue.max_idle.

  {
      "queue": {
        "max_idle": "1m"  // Configures the maximum processing time for a single message to 1 minute. If this time is exceeded, the message is delivered to another subscriber, and the delivery count is incremented. The default value is 0, which means no maximum processing time.
      }
  }

  Supported time units: h (hour), m (minute), and s (second). If processing exceeds the configured duration:

  • If the queue.max_delivery threshold is not exceeded, the message is redelivered to other subscribers.

  • If the queue.max_delivery threshold is exceeded, the dead-letter policy applies.

• Configure the dead-letter policy using queue.dead_message_policy.

  {
      "queue": {
        "dead_message_policy":  "Rear"  // The value can be Rear (default) or Drop. Rear moves the message to the end of the queue. Drop deletes the message. 																 
      }
  }

Queue length or maximum payload size

Queue replica memory is fixed: increasing the maximum payload size per message reduces the maximum queue length.

• Configure the maximum length of the sink and input queues using queue.sink.max_length or queue.source.max_length, respectively.

  {
      "queue": {
         "sink": {
            "max_length": 8000  // Configures the maximum length of the sink queue to 8,000 messages.
         },
         "source": {
            "max_length": 2000  // Configures the maximum length of the input queue to 2,000 messages.
         }
      }
  }

• Configure the maximum payload size per message for the sink and input queues using queue.sink.max_payload_size_kb or queue.source.max_payload_size_kb, respectively.

  {
      "queue": {
         "sink": {
            "max_payload_size_kb": 10  // Configures the maximum payload size per message for the sink queue to 10 KB. Default: 8 KB.
         },
         "source": {
            "max_payload_size_kb": 1024  // Configures the maximum payload size per message for the input queue to 1024 KB (1 MB). Default: 8 KB.
         }
      }
  }

Memory allocation ratio

• Adjust the memory allocation between the input and sink queues using queue.sink.memory_ratio.

  {
      "queue": {
         "sink": {
            "memory_ratio": 0.9  // Configures the memory ratio for the sink queue. Default: 0.5.
         }
      }
  }

  Note

  By default, the input and sink queues share memory equally. Increase queue.sink.memory_ratio if the sink queue needs more space (for example, text input and image output), or decrease it for the reverse.

Horizontal auto scaling

How it works

The system dynamically scales inference replicas based on queue state, including scaling to zero when the queue is empty. The following diagram illustrates the mechanism.