Platform For AI:Deploy an asynchronous inference service

Access an asynchronous inference service

As described previously, a service group with the same name as your asynchronous inference service is created by default. Because the queue sub-service within this group serves as the traffic ingress, access the queue sub-service directly by using the following paths. For more information, see Access the queue service.

Manage the asynchronous inference service

You can manage an asynchronous inference service like a regular service. Its sub-services are managed automatically by the system. For example, when you delete the asynchronous inference service, both the queue sub-service and the inference sub-service are deleted. When you update the Inference sub-service, the queue sub-service remains unchanged to ensure maximum availability.

Due to the sub-service architecture, even if you configure one instance for your asynchronous inference service, the instance list shows an additional instance for the queue sub-service.

The number of instances for an asynchronous inference service refers to the number of inference sub-service instances. The number of queue sub-service instances adjusts automatically based on the number of Inference sub-service instances. For example, when you scale out the Inference sub-service to three instances, the queue sub-service scales out to two instances.

The instance ratio between the two sub-services follows these rules:

• When the asynchronous inference service is stopped, the instance count for both the queue sub-service and the inference sub-service scales down to 0. The instance list is empty.

• When the inference sub-service has 1 instance, the queue sub-service also has 1 instance, unless you customized the queue sub-service parameters.

• When the number of inference sub-service instances exceeds 2, the number of queue sub-service instances remains at 2, unless you customized its parameters.

• If you configure auto scaling for the asynchronous inference service with a minimum instance count of 0, one instance of the queue sub-service remains active even when the inference sub-service scales down to 0.

Queue sub-service parameters and descriptions

In most cases, the default configuration for the queue sub-service is sufficient. If you have specific requirements, you can configure the queue instances through the queue block in the top level of your JSON file. The following code provides an example:

{  
  "queue": {
     "sink": {
        "memory_ratio": 0.3
     },
     "source": {
        "auto_evict": true,
     }
 }

The following sections describe the configuration options in detail.

Configure queue sub-service resources

By default, the resources for the queue sub-service are configured based on the fields in the metadata section. However, in some use cases, you may need to configure its resources separately as described in this section.

• Use queue.resource to specify the resource group for the queue sub-service.

  {
    "queue": {
      "resource": eas-r-slzkbq4tw0p6xd****  # By default, the resource group of the inference subservice is used.
    }
  }

  • The queue sub-service uses the same resource group as the inference sub-service by default.

  • If you want to deploy the queue sub-service to a public resource group, you can declare resource as an empty string (""). This is useful when your dedicated resource group has insufficient CPU or memory.

    Note

    Deploy the queue sub-service to a public resource group.

• Use queue.cpu and queue.memory to specify the CPU (in cores) and memory (in MB) for each queue sub-service instance.

  {
    "queue": {
       "cpu": 2,  # Default value: 1.
       "memory": 8000  # Default value: 4000.
    }
  }

  If you do not configure these resources, the queue sub-service defaults to 1 CPU core and 4 GB of memory, which is adequate for most scenarios.

  Important

  • If you have more than 200 subscribers, such as inference sub-service instances, configure 2 or more CPU cores.

  • Do not reduce the memory allocation for the queue sub-service in a production environment.

• Use queue.min_replica to configure the minimum number of queue sub-service instances.

  {
    "queue": {
       "min_replica": 3  # Default value: 1.
    }
  }

  When you use an asynchronous inference service, the number of queue sub-service instances automatically adjusts based on the runtime number of inference sub-service instances. The default adjustment range is [1, min(2, number of Inference Sub-service instances)]. In the special case where you configure an auto scaling rule that allows scaling down to 0 instances, one queue sub-service instance is automatically retained. You can use queue.min_replica to adjust this minimum number of retained instances.

  Note

  Increasing the number of queue sub-service instances improves availability but does not improve performance.

Configure queue sub-service features

The queue sub-service has several configurable features that you can adjust.

• Use queue.sink.auto_evict or queue.source.auto_evict to enable automatic data eviction for the sink queue and input queue, respectively.

  {
    "queue": {
       "sink": {
          "auto_evict": true  # Enable automatic eviction for the output queue. Default value: false.
        },
        "source": {
           "auto_evict": true  # Enable automatic eviction for the input queue. Default value: false.
        }
    }
  }

  By default, automatic data eviction is disabled. If a queue is full, you cannot write new data to it. In scenarios where data overflow is acceptable, enable this feature. The queue automatically evicts the oldest data to make room for new data.

• Use queue.max_delivery to configure the maximum number of delivery attempts.

  {
     "queue": {
        "max_delivery": 10  # Set the maximum number of data deliveries to 10. Default value: 5. If you set the value to 0, data can be delivered for an unlimited number of times.
     }
  }

  When the number of delivery attempts for a single message exceeds this threshold, the message is considered unprocessable and is marked as a dead-letter message. For more information, see dead-letter policies.

• Use queue.max_idle to configure the maximum processing time for a message.

  {
      "queue": {
        "max_idle": "1m"  # Set the maximum processing time for a single data entry to 1 minute. If the processing time exceeds this limit, the data entry is delivered to other subscribers and the delivery count is incremented by 1. The default value is 0, which means no maximum processing time limit.
      }
  }

  The time value in the example is 1 minute. Multiple time units are supported, such as h (hour), m (minute), and s (second). If a message's processing time exceeds the configured duration, one of two things happens:

  • If the delivery count has not reached the queue.max_delivery threshold, the message is redelivered to another subscriber.

  • If the queue.max_delivery threshold has been reached, the dead-letter policy is executed for the message.

• Use queue.dead_message_policy to configure the dead-letter policy.

  {
      "queue": {
        "dead_message_policy":  "Rear"  # The valid values are Rear (default) and Drop. Rear moves the message to the end of the queue, and Drop deletes the message. 																 
      }
  }

Configure maximum queue length or maximum data volume

The maximum length and maximum payload size of a queue sub-service are inversely related. The relationship is calculated as follows:

The memory of a queue sub-service instance is fixed. Therefore, increasing the maximum payload size per message decreases the maximum length of the queue.

• Use queue.sink.max_length or queue.source.max_length to configure the maximum length of the sink queue or input queue, respectively.

  {
      "queue": {
         "sink": {
            "max_length": 8000  # Set the maximum length of the output queue to 8000 entries.
         },
         "source": {
            "max_length": 2000  # Set the maximum length of the input queue to 2000 entries.
         }
      }
  }

• Use queue.sink.max_payload_size_kb or queue.source.max_payload_size_kb to configure the maximum payload size for a single message in the sink queue or input queue, respectively.

  {
      "queue": {
         "sink": {
            "max_payload_size_kb": 10  # Set the maximum size of a single data entry in the output queue to 10 KB. Default value: 8 KB.
         },
         "source": {
            "max_payload_size_kb": 1024  # Set the maximum size of a single data entry in the input queue to 1024 KB (1 MB). Default value: 8 KB.
         }
      }
  }

Configure memory allocation skew

• Use queue.sink.memory_ratio to adjust the memory allocation between the input and sink queues.

  {
      "queue": {
         "sink": {
            "memory_ratio": 0.9  # Specify the memory ratio of the output queue. Default value: 0.5.
         }
      }
  }

  Note

  By default, the input queue and sink queue evenly share the memory of the queue sub-service instance. If your service takes text as input and produces images as output, and you want to store more data in the sink queue, you can increase queue.sink.memory_ratio. Conversely, if your service takes images as input and produces text as output, you can decrease queue.sink.memory_ratio.

Configure horizontal auto scaling

How it works

In an asynchronous inference scenario, the system can dynamically scale the number of inference service instances based on the queue status. It also supports scaling the number of instances down to zero when the queue is empty to reduce costs. The following diagram illustrates how auto scaling for an asynchronous inference service works: