By Guichu
AI applications are evolving from single-turn text Q&A to long-context, multimodal, and Agent calls. A single request may carry an entire conversation, retrieved materials, or multiple images, and access different models and tools. Larger requests and longer call chains place greater demands on the processing efficiency and operational stability of AI Gateway.
This article compares Alibaba Cloud AI Gateway Serverless Enterprise Edition with a self-hosted open-source Higress deployment, hereinafter referred to as "Serverless Enterprise Edition" and "open-source Higress," respectively. Two experiments examine gateway processing latency for 1 MB and 10 MB requests, followed by a discussion of costs, maintenance effort, and production assurance for the two deployment approaches.
When users ask an AI assistant a question, they generally care only about how long it takes to receive an answer. For the system, however, a single call may pass through the gateway with a long context and a large request payload before reaching the model backend. Waiting in the call path has already begun before the model starts inference.
AI Gateway sits between applications and model services, handling authentication, routing, and traffic governance. Text Q&A, knowledge assistants, and Agents differ in request size and call frequency. When gateway plugins need to read and parse request bodies to perform model-protocol conversion or content inspection, they incur data-copying and object-allocation overhead. Long contexts, images embedded directly in request bodies, and similar content increase this overhead. When multiple large requests arrive simultaneously, processing efficiency and memory usage warrant even closer attention.
Open-source Higress already provides extensive model proxy and traffic-governance capabilities, making it a strong choice for a self-hosted AI Gateway. Common requirements for model access, authentication, routing, and rate limiting can generally be met with existing capabilities, and teams can use it within their own deployment and operations systems.
Moving from model access to reliably carrying production traffic requires teams to address additional challenges. In particular, for workloads that continuously parse large request bodies, plugin processing efficiency, memory consumption, and concurrency performance can affect application responsiveness. Open-source Higress can be optimized by adjusting plugins, versions, and resource configurations, while Serverless Enterprise Edition provides dedicated processing optimizations for such workloads. Latency tests reveal the differences between the two in large-request-body scenarios such as long context and multimodality.
In concurrency tests ranging from 10 to 100, Serverless Enterprise Edition delivered lower average gateway processing latency than open-source Higress: reductions of up to 97.1% for 1 MB requests and up to 90.2% for 10 MB requests.
Both sides used the same load-testing source code and synthetic JSON requests, accessing the same Mock backend with no artificial delay to eliminate the impact of real model inference time. The two experiments fixed request-body sizes at 1 MB and 10 MB, respectively, and gradually increased concurrency. The figures show average gateway processing latency at each level. Because the deployment resources and access endpoints differed between the two sides, the results reflect performance under the respective deployments.
In concurrency tests ranging from 10 to 100, Serverless Enterprise Edition maintained average processing latency of 13–18 ms, 68.8%–97.1% lower than open-source Higress. The largest reduction occurred at concurrency 90: average latency fell from 446.2 ms to 12.7 ms, a reduction of 97.1%.
At 1 MB, the request body was the smaller of the two tested sizes. As concurrency increased, the average latency of open-source Higress rose progressively, while Serverless Enterprise Edition remained within a narrow millisecond-level range. The difference between the two was already apparent at this request size.
In concurrency tests ranging from 10 to 100, Serverless Enterprise Edition delivered average processing latency 82.7%–90.2% lower than open-source Higress. The largest reduction occurred at concurrency 30: average latency fell from 1,688.1 ms to 166.1 ms, a reduction of 90.2%.
Long contexts and embedded media increase the amount of data transferred per request. If plugins also need to parse this content, the copying and processing overhead of large payloads becomes more pronounced. At concurrency 100, the average latency of Serverless Enterprise Edition was still 384.3 ms, 84.8% lower than open-source Higress at 2,527.4 ms, reducing the wait by approximately 2.14 seconds.
The performance difference must be understood in the context of specific plugin processing paths. When plugins read, parse, or rewrite an entire request body, the gateway must bear the overhead of buffering, copying, and object allocation in addition to forwarding data. For open-source Wasm-plugin-based Higress, if multiple plugins separately process the same large payload, the burden of repeated copying and allocation may grow with request size and concurrency. Such workloads require dedicated tuning; common model calls do not necessarily encounter the same degree of pressure.
Traditional Wasm linear memory can grow but cannot shrink directly. After a plugin releases objects, the space can generally be reused within the instance, but it may not be returned to the host system immediately. If a plugin continues to hold the request body while waiting for the model response, the data remains resident for even longer. Under sustained concurrent access, data that is not promptly released can combine with new allocations, potentially keeping memory usage high or causing it to continue climbing. Without corresponding optimizations, tightening resources may affect processing efficiency and queuing latency.
Serverless Enterprise Edition uses Envoy's native plugin mechanism and optimizes AI request-processing flows and memory usage for these workloads, reducing repeated data copying and unnecessary memory consumption. These dedicated optimizations are continuously maintained by the platform, providing a more efficient processing approach for production workloads that need to parse large request bodies.
Beyond performance, the two deployment approaches also entail different resource preparation and maintenance responsibilities. Open-source Higress is deployed and maintained by the team and can be incorporated into its existing infrastructure system. With Serverless Enterprise Edition, the platform manages the underlying resources, while the team primarily configures models, routes, and business policies.
The performance comparison focuses on workloads that need to process large request bodies; the reduction shown in the figure corresponds to the 10 MB group in this article.
Self-hosted open-source Higress primarily incurs costs for compute, networking, and redundant replicas, with capacity also reserved for traffic peaks. The labor required for image maintenance, plugin compatibility checks, upgrade regression testing, and incident troubleshooting constitutes an ongoing investment. Serverless Enterprise Edition has the platform manage the underlying capacity and nodes, reducing the team's work in provisioning resources and maintaining infrastructure.
Production operations also require high availability and failure-recovery capabilities. Open-source Higress can rely on the team's existing monitoring and incident-response systems. AI Gateway Serverless Enterprise Edition provides a 99.95% service level, with the platform responsible for underlying operational assurance and version maintenance, helping reduce infrastructure incident-handling work and the risk of losses caused by service interruptions. For teams seeking to reduce investment in node operations and underlying troubleshooting, the managed service also allows engineers to devote more time to application development and business iteration.
Alibaba Cloud AI Gateway provides a unified access point for large models, Agents, and MCP services. It centrally handles authentication, intelligent routing, traffic governance, and security protection, while providing call observability and Token usage statistics. Applications access services through the gateway, allowing teams to centrally configure call policies and view operating conditions while reducing duplicated access and governance development.
The product offers three main instance types for business requirements at different stages:
References:
[1] AI Gateway Types and Edition Capabilities: https://www.alibabacloud.com/help/api-gateway/ai-gateway/product-overview/ai-gateway-capacity-description
[2] Billing Description for the New Serverless Edition: https://www.alibabacloud.com/help/api-gateway/ai-gateway/product-overview/overview-of-billing-during-the-serverless-public-preview-of-the
[3] AI Gateway Engine Version Description: https://www.alibabacloud.com/help/api-gateway/ai-gateway/product-overview/engine-version-description
Alibaba Cloud's Open-Source AI Agent Handbook, 3K+ GitHub Stars
829 posts | 61 followers
FollowAlibaba Cloud Native Community - November 11, 2026
Alibaba Cloud Native Community - January 7, 2026
Alibaba Cloud Native Community - September 9, 2025
Alibaba Cloud Native Community - July 24, 2025
Alibaba Cloud Native Community - July 4, 2025
Alibaba Cloud Native Community - June 4, 2025
829 posts | 61 followers
Follow
API Gateway
API Gateway provides you with high-performance and high-availability API hosting services to deploy and release your APIs on Alibaba Cloud products.
Learn More
Managed Service for Prometheus
Multi-source metrics are aggregated to monitor the status of your business and services in real time.
Learn More
AgentBay
Multimodal cloud-based operating environment and expert agent platform, supporting automation and remote control across browsers, desktops, mobile devices, and code.
Learn More
Function Compute
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn MoreMore Posts by Alibaba Cloud Native Community
