Platform For AI:Deploy an asynchronous inference service

Use cases

Architecture

An asynchronous inference service consists of two sub-services that communicate through message queues:

• Inference sub-service: Processes requests by calling inference interfaces.

• Queue sub-service: Manages two message queues: input queue and sink queue.

Request flow

1. Client sends request to input queue.

2. EAS framework in inference sub-service subscribes to queue and fetches request data in streaming fashion.

3. Inference sub-service processes data.

4. Inference sub-service writes response to sink queue.

5. Client retrieves result from sink queue by polling or subscription.

Prerequisites

• PAI workspace with EAS activated. See Custom Deployment.

• (For EASCMD deployment) EASCMD client installed and authenticated. See Download and authenticate the client.

Create an asynchronous inference service

Creating an asynchronous inference service automatically creates a service group with the same name. The queue sub-service is created automatically with the following defaults:

Access the service

The queue sub-service serves as traffic ingress. Send requests directly to the queue sub-service using the following endpoints. For more information, see Access the queue service.

Manage the service

Manage an asynchronous inference service like a regular service. Sub-services are managed automatically:

• Deleting the asynchronous inference service deletes both sub-services.

• Updating the inference sub-service does not affect the queue sub-service, ensuring availability.

Due to the sub-service architecture, the instance list shows an additional instance for the queue sub-service even if one instance is configured.

Instance count refers to inference sub-service instances only. Queue sub-service instances adjust automatically based on inference sub-service instance count.

Configure horizontal auto scaling

The system dynamically scales inference sub-service instances based on queue status. It supports scaling to zero instances when the queue is empty.

1. In the service list, click the service name to open service details.

2. On the Auto Scaling tab, in the Auto Scaling area, click Enable Auto Scaling.

3. In the Auto Scaling Settings dialog box, configure the following parameters. For more information about parameters and EASCMD client configuration, see Horizontal auto scaling.

   Parameter	Description	Default
   Minimum number of instances	Minimum instance count. Minimum value is 0.	-
   Maximum number of instances	Maximum instance count. Maximum value is 1000.	-
   Standard scaling metrics	Metric to trigger scaling. Select Asynchronous Queue Length, which represents average task count per instance in the queue.	-
   Scale-out duration	Observation window in seconds for scale-out decisions. If metric falls below threshold during this window, scale-out is canceled. Set to 0 to execute immediately.	0
   Scale-in effective time	Observation window in seconds for scale-in decisions. Scale-in occurs only after metric remains below threshold for this duration. Avoid setting too low to prevent instability.	300
   Effective duration of scale-in to 0	When Minimum number of instances is 0, wait time in seconds before instance count is reduced to 0.	-
   Number of instances scaled out from zero	Number of instances to add when the service scales from 0 instances.	-

Customize queue sub-service

In most cases, default configuration is sufficient. To customize the queue sub-service, add a queue block to the JSON configuration file.

Queue parameters reference

Parameter	Default	Description
queue.cpu	1 core	CPU allocation per queue sub-service instance. Set to 2 or more cores for more than 200 subscribers.
queue.memory	4000 MB (4 GB)	Memory allocation per queue sub-service instance. Avoid reducing in production.
queue.min_replica	1	Minimum queue sub-service instances.
queue.resource	Same as inference sub-service	Resource group for queue sub-service. Set to empty string ("") to deploy to public resource group.
queue.sink.memory_ratio	0.5	Memory ratio allocated to the sink queue. Increase for large outputs; decrease for large inputs.
queue.sink.max_payload_size_kb / queue.source.max_payload_size_kb	8 KB	Maximum payload size per message.
queue.sink.max_length / queue.source.max_length	Not set (capacity depends on memory and payload size)	Maximum queue length. With 4 GB memory and 8 KB max payload, each queue holds up to 230,399 messages. Cannot be set with max_payload_size_kb for same queue.
queue.max_delivery	5	Maximum delivery attempts per message. Set to 0 for unlimited. When exceeded, message is marked as dead-letter.
queue.max_idle	0 (no limit)	Maximum processing time per message. Supports h, m, s units.
queue.dead_message_policy	Rear	Dead-letter policy. Rear moves the message to the end of the queue. Drop deletes it.
queue.sink.auto_evict / queue.source.auto_evict	false	Automatic data eviction. When enabled, queue evicts oldest data to accommodate new data. When disabled and queue is full, new data cannot be written.

Important

Max queue length and max payload size are inversely related. Increasing max payload size per message decreases max queue length. The system reserves 10% of total memory. Both max_length and max_payload_size_kb cannot be configured for same queue simultaneously.

{
  "queue": {
    "resource": "eas-r-slzkbq4tw0p6xd****"
  }
}

{
  "queue": {
    "cpu": 2,
    "memory": 8000
  }
}

{
  "queue": {
    "min_replica": 3
  }
}

{
  "queue": {
    "sink": {
      "memory_ratio": 0.9
    }
  }
}

{
  "queue": {
    "sink": {
      "max_length": 8000
    },
    "source": {
      "max_length": 2000
    }
  }
}

{
  "queue": {
    "sink": {
      "max_payload_size_kb": 10
    },
    "source": {
      "max_payload_size_kb": 1024
    }
  }
}

{
  "queue": {
    "sink": {
      "auto_evict": true
    },
    "source": {
      "auto_evict": true
    }
  }
}

{
  "queue": {
    "max_delivery": 10
  }
}

{
  "queue": {
    "max_idle": "1m"
  }
}

{
  "queue": {
    "dead_message_policy": "Rear"
  }
}