This topic describes near-real-time inference scenarios and explains how to use on-demand GPU instances to build a cost-effective service.
Use cases
In near-real-time inference scenarios, workloads have one or more of the following characteristics.
Intermittent traffic
Daily invocations can range from a few to tens of thousands, but the total daily GPU usage is often well below 8 to 12 hours, resulting in significant idle time.
Long processing time per request
Processing time for near-real-time inference typically ranges from seconds to minutes. For example, common computer vision (CV) tasks take seconds, while typical video processing and AIGC workloads often take minutes.
Tolerance for cold starts
The application can tolerate GPU cold start latency, or the traffic patterns result in a low probability of a cold start.
Function Compute offers the following advantages for near-real-time inference workloads.
Native serverless experience
Function Compute uses on-demand GPU instances by default to automatically manage GPU resources. It performs auto scaling in response to traffic changes, scaling down to zero during idle periods and elastically scaling up GPU capacity in seconds during peak periods. You can focus on your application code while Function Compute manages the underlying infrastructure.
Optimal specifications
Function Compute provides a variety of GPU instance specifications, allowing you to independently configure CPU, GPU, memory, and disk resources to best match your workload. The smallest available GPU specification offers 1 GB of GPU memory and proportional computing power, which helps you right-size your instances.
Cost-effective
With a pay-as-you-go model and per-second billing, Function Compute helps you optimize costs after right-sizing your GPU instances. For workloads with low GPU resource utilization, you can reduce costs by over 70%.
Handles traffic bursts
Function Compute provides an ample supply of GPU resources. When your application experiences a sudden traffic burst, the platform can provision massive GPU computing power in seconds. This prevents performance degradation caused by insufficient GPU capacity or slow scaling.
How it works
After you deploy a GPU function, Function Compute serves it by using on-demand GPU instances by default. This is different from provisioned instances. For more information, see Instance types and specifications. This approach provides the necessary infrastructure for near-real-time inference scenarios.
You can send an inference request to a function's trigger, such as an HTTP trigger. The GPU function runs in a GPU container, performs the inference, and returns the result in the response. Function Compute automatically orchestrates and scales GPU resources to match your traffic, and you pay only for the GPU resources you use during request processing.
Container support
For GPU workloads, Function Compute currently supports deployment only by using a custom image. For more information about custom images, see Introduction to Custom Container.
Functions based on a custom image require a web server within the image to handle different code paths and respond to events or HTTP requests. This is suitable for scenarios with multiple execution paths, such as AI learning and inference.
Specifications for GPU-accelerated instances
In inference scenarios, you can select different GPU types and configure specifications of GPU-accelerated instances based on the computing power required by your business. The specifications of GPU-accelerated instances include CPU, GPU memory, memory, and disk capacity. For more information about specifications of GPU-accelerated instances, see Instance specifications
Deployment methods
You can deploy your models in Function Compute by using one of the following methods:
Use the Function Compute console. For more information, see Create a function in the console.
Call SDKs. For more information, see API overview.
Use Serverless Devs. For more information, see Common Serverless Devs commands.
For more deployment examples, see start-fc-gpu.
Concurrent invocations
Your GPU function's maximum concurrency in a region, such as China (Hangzhou), depends on the function's instance concurrency and your physical GPU card quota.
Instance concurrency
By default, a GPU function instance has a concurrency of 1, meaning it processes only one request at a time. You can adjust the instance concurrency in the Function Compute console or by using the Serverless Devs tool. For instructions, see Configure instance concurrency. Set the concurrency based on your application's requirements.
For compute-intensive inference applications, keep the default concurrency of 1.
For applications that support request batching, set the concurrency to match your desired batch size.
Physical GPU card quota
By default, an Alibaba Cloud account has a quota of 30 physical GPU cards per region. The actual quota is displayed in Quota Center. If you need a higher quota, request an increase in Quota Center.
Cold start
When your GPU function is idle for an extended period, the platform releases all its on-demand GPU instances. The next request triggers a cold start, adding latency while the platform provisions a new instance. This process includes preparing GPU resources, pulling the container image, starting the GPU container, loading and initializing the model, and launching the inference application. For more information, see Best practices for optimizing cold starts in Function Compute.
The cold start duration for an AI application depends on your image size, model size, and initialization time. You can use monitoring metrics to observe cold start latency and estimate its probability.
Cold start performance
On the Function Compute GPU platform, the end-to-end cold start performance for common models is as follows.
The end-to-end cold start duration, which includes instance startup and the first invocation processing time, is typically 10 to 30 seconds.
Cold start probability
In Function Compute serverless GPU, a cold start takes seconds, whereas on Kubernetes platforms, it can take minutes. The probability of a cold start in Function Compute decreases as concurrency increases, reducing its impact on your application.
Cost assessment
The following unit prices and examples are for reference only. Your actual costs are determined by the official pricing.
The lower your current average daily GPU utilization, the greater your cost savings after migrating to Function Compute.
The following examples compare the cost of using a T4 GPU-accelerated ECS instance with a Function Compute GPU instance of equivalent specifications. The unit price for the GPU-accelerated ECS instance is approximately USD 2/hour. For more billing details, see Billing of GPU-accelerated ECS instances.
Example 1
Assume a GPU function is invoked 3,600 times a day, with each invocation lasting 1 second. The function uses a GPU instance with 4 GB of GPU memory to run a model of approximately 3 GB.
Average daily resource utilization (time-based only) = 3,600 seconds / 86,400 seconds = 0.041, or 4.1%.
Daily GPU resource cost for the ECS instance = USD 2/hour × 24 hours = USD 48.
Daily GPU resource cost for Function Compute = 3,600 seconds × 4 GB × USD 0.000018/GB-second = USD 0.259.
Using Function Compute reduces costs by over 99%.
Example 2
Assume a GPU function is invoked 50,000 times a day, with each invocation lasting 1 second. The function uses a GPU instance with 4 GB of GPU memory to run a model of approximately 3 GB.
Average daily resource utilization (time-based only) = 50,000 seconds / 86,400 seconds = 0.57, or 57%.
Daily GPU resource cost for the ECS instance = USD 2/hour × 24 hours = USD 48.
Daily GPU resource cost for Function Compute = 50,000 seconds × 4 GB × USD 0.000018/GB-second = USD 3.6.
Using Function Compute reduces costs by over 90%.