All Products
Search

This Product

    API Gateway:Proxy Dify traffic with AI Gateway

    Document Center

    API Gateway:Proxy Dify traffic with AI Gateway

    Last Updated:Jun 20, 2026

    To enhance the high availability and security of your Dify applications, you can use AI Gateway to proxy their inbound and outbound traffic. This topic describes the procedure and demonstrates some of the key features.

    AI Gateway capabilities

    AI Gateway acts as a bridge between external environments and your enterprise AI applications, and between those applications and large language model (LLM) or MCP services. It solves challenges such as complex model integration, strict security and compliance, and inefficient management by providing a unified entry point for traffic governance. Its core features include:

    1. Protocol standardization: Converts heterogeneous APIs from different models into a standard, OpenAI-compatible interface format.

    2. Observability: Supports token-level metrics monitoring (such as QPS, success rate, and response latency) and full-link request tracing.

    3. Security layer: Provides automatic API key rotation, JWT authentication, and real-time detection and blocking of sensitive content.

    4. Stability engine: Integrates traffic governance features such as multi-level fallback strategies, AI-powered cache acceleration, and token-level rate limiting.

    For high availability, AI Gateway provides comprehensive mechanisms optimized for AI applications and models to ensure service continuity, stability, and reliability.

    1. Multi-dimensional request rate limiting: Provides fine-grained control over request traffic based on dimensions like application or model, with time windows such as per second, per minute, or per hour. This helps manage traffic spikes and high-concurrency scenarios, preventing service overload and ensuring system stability.

    2. Token-level resource flow control: In addition to request volume control, AI Gateway offers flow control based on token consumption. This enables precise management of LLM resource usage, prevents individual high-consumption requests from depleting the resource pool, and improves resource allocation fairness.

    3. Model fallback mechanism: When a primary model service fails or responds with an error, the gateway automatically and transparently routes requests to a preconfigured backup model. This ensures business continuity with sub-second failover and disaster recovery.

    4. Model load balancing: For self-managed model clusters, AI Gateway supports intelligent load balancing strategies such as GPU-aware and prefix-matching. These strategies increase system throughput, reduce response latency, and maximize GPU resource utilization without additional hardware investment.

    5. AI cache mechanism: Caches responses for frequent and repetitive requests. The gateway returns cached content directly. This significantly reduces calls to the underlying LLM, improving response speed and lowering invocation costs.

    Proxy Dify traffic with AI Gateway

    To improve the high availability, security, and observability of your Dify application, you need to integrate it with AI Gateway. The integration architectures are as follows:

    • In the original architecture, Dify uses a built-in Nginx as a reverse proxy to handle incoming traffic, and Dify directly calls backend services such as LLMs, RAG services, and Mcp Servers.

    • In the new architecture, AI Gateway replaces the built-in Nginx to proxy all inbound and outbound traffic for the Dify application.

    For inbound traffic, we recommend replacing Nginx directly with AI Gateway instead of routing traffic from AI Gateway to Nginx for the following reasons:

    1. Full capability coverage: AI Gateway fully covers Nginx's reverse proxy capabilities and provides over 20 governance policies designed for AI services. Nginx's default buffering mechanism breaks Server-Sent Events (SSE) streaming responses. Supporting streaming requires manually tuning multiple parameters, making the configuration complex and precluding deep observability.

    2. Simplified architecture: Using AI Gateway to directly proxy traffic to Dify services eliminates an extra proxy layer. A dual-gateway architecture (AI Gateway > Nginx > Dify) introduces unnecessary network hops, increasing latency and impacting performance. Furthermore, troubleshooting must cover the Nginx layer, which increases problem resolution time and reduces operational efficiency.

    3. Optimized operational costs: Nginx requires a separate deployment, consuming additional compute and memory resources, and its scaling relies on manual intervention. Routing changes must be synchronized across both systems, increasing the risk of configuration inconsistencies. In contrast, AI Gateway is a fully managed service with an enterprise-grade SLA and native integration for monitoring, logging, and alerting, significantly reducing operational complexity and Total Cost of Ownership (TCO).

    Procedure

    Proxy inbound Dify traffic

    AI Gateway allows you to create an Agent API to proxy requests to your AI application, enabling observability, security, and high availability governance for the traffic.

    Step 1: Create a service source

    When creating a service source in AI Gateway for Dify's API component, use the following service discovery configuration if Dify is deployed in an SAE or ACK environment:

    SAE

    1. Log on to the AI Gateway console.

    2. In the navigation pane on the left, choose Instance. In the top menu bar, select a region.

    3. On the Instance page, click the target instance ID.

    4. In the left-side navigation pane, choose Service and click Create Service. Set Service Source to SAE Kubernetes Service. For Namespace, select the namespace where your Dify application is located. From the Services, select the dify-api-{namespace} application.

    5. Click OK to complete the service configuration.

    ACK

    Create a container source by navigating to Services > Sources > Create Source. Then, navigate to Services > Services > Create Service to add the ack-dify-api service from the dify-system namespace.

    1. Log on to the AI Gateway console.

    2. In the navigation pane on the left, choose Instance. In the top menu bar, select a region.

    3. On the Instance page, click the target instance ID.

    4. In the left-side navigation pane, choose Service and click Create Service. Set Service Source to ACK. For Namespace, select the namespace where your Dify application is located. From the Services, select the following service:

      • ack-dify-api

    5. Click OK to complete the service configuration.

    Step 2: Configure a route

    Follow these steps to configure a route for the Dify service using an Agent API in AI Gateway:

    1. In the left-side navigation pane of the instance details page, click Agent API.

    2. Click Create Agent API and configure the following settings:

      • Configure the Domain Name and Base path based on your requirements. This allows you to access Dify through a domain name and avoid path conflicts with other services.

      • Select the Remove during backend forwarding checkbox.

      • Set Protocol to Dify.

    3. Click the newly created Agent API, and then click Create Route. Configure the Path based on your use case, and for Agent Service, select the service created in Step 1.

      Note

      To map a route to a single Dify application, you can configure request header or request parameter matching conditions in More Match Rules. For example, set header-key=app-id. Note that when you access a Dify application through AI Gateway, you must include the corresponding matching field in the request.

      • If Dify has a workflow application, set the path to /v1/workflows/run.

      • If Dify has an agent application, set the path to /v1/chat-messages.

    After the route is published, access the Dify application using the configured domain and path to verify the setup. If the request is successful, the inbound traffic proxy is configured correctly.

    Proxy outbound Dify traffic

    AI Gateway allows you to create an LLM API to proxy access to large language models, an MCP Server for MCP proxying, and a RAG plugin for an AI RAG retrieval proxy. It provides observability, security controls, and high availability governance for these different traffic types.

    In Dify applications, the primary use cases involve accessing large language models and external knowledge bases. This section explains how to configure AI Gateway to proxy traffic for these two scenarios.

    Proxy model traffic

    1. On the AI Gateway console, create a Model API to access your self-managed or third-party large language models. For more information, see Manage Model API.

    2. Go to the Dify App Market and install the Higress plugin.

    3. Go to the Dify console and choose Settings > Model providers. Add a model for Higress. The following example uses an LLM model:

      • For Model Type, select LLM.

      • Enter a custom Model Name and Credential Name.

      • Select the Usage Scenario and Model Protocol based on your AI Gateway Model API's scenario and protocol, such as text generation or image generation, and OpenAI-compatible or Model Studio protocols.

      • For Higress AI Gateway Route, enter the domain and prefix of the Model API you created in AI Gateway.

      • Configure other parameters such as Consumer Authentication Policy, Function Call Type, and Thinking Mode Support as needed.

    4. In the application node where you need to select a model, choose the model you just created.

    5. Run a workflow or agent in Dify to verify that requests to the LLM are successfully proxied through AI Gateway and that the model returns a valid result. This confirms that the outbound model traffic proxy is working.

    Proxy external knowledge bases

    Some Dify users want to connect to external knowledge bases such as Model Studio or RAGFlow to improve RAG performance while leveraging Dify's efficient building and orchestration capabilities.

    AI Gateway provides a RAG proxy plugin designed specifically for Dify, allowing it to connect to external knowledge bases easily and efficiently. This reduces the technical difficulty and development cost of implementing this integration manually.

    For a detailed introduction and procedure for the RAG proxy plugin, see AI RAG retrieval proxy.

    High availability capabilities

    After you complete the preceding steps to configure AI Gateway to proxy inbound and outbound traffic for your Dify application, you can enable various AI Gateway capabilities to observe and govern the AI traffic. This section demonstrates some of the high availability governance features. For more information about AI Gateway capabilities, see AI Gateway.

    Govern inbound Dify traffic

    By using AI Gateway as the traffic entry point for your Dify application, you can configure cluster-wide rate limiting policies to achieve multi-dimensional traffic control at global and application levels. This section uses application-level rate limiting as an example.

    To support global and application-level rate limiting, you need a Redis instance for counting. Because Redis is a required component in the Dify system's storage architecture, you can reuse the existing Redis instance from your Dify system.

    After connecting the Redis instance in AI Gateway, you can enable the Key-based Cluster Rate Limiting plugin from the plugin marketplace. Configure rules to implement independent rate limiting policies for different Dify applications. For detailed configuration instructions for this plugin, see the cluster-key-rate-limit plugin documentation.

    The following configuration is an example that limits a Dify application to one request per minute:

    rule_name: dify_higress_demo_rule
rule_items:
- Limit_by_header: x-app-id
  limit_keys:
  - key: 9a342******************3e1250b5
    query_per_minute: 1
redis:
  service_name: dify-redis.dns
show_limit_quota_header: true

    After the plugin is enabled, if a second request is sent to this application within one minute, it triggers the rate limiting rule in AI Gateway, causing the request to fail. Other applications without this rate limiting rule can still be called without restriction.

    Govern outbound Dify traffic

    Request and token rate limiting

    By using AI Gateway to proxy calls from Dify applications to model services, you can implement request rate limiting with different time granularities. The configuration method and effect are the same as described previously, with the only difference being where the policy is applied, so this section does not repeat those details.

    Additionally, when a Dify application calls a model service, you can also control traffic based on token consumption. For detailed instructions, see Rate Limiting.

    The following configuration is an example that limits a Model API to 500 tokens per minute:

    Turn on the Rate limiting switch. In the rate limiting policy, set the condition to By Consumer, set the rule to Any Match, and set the Token limit per minute to 500.

    After the configuration takes effect, if the total token consumption for the model service exceeds 500 tokens within a one-minute window, subsequent requests to the model will be rejected by AI Gateway.

    Model fallback

    Configuring a fallback mechanism for model access allows the system to automatically switch to a backup model service when the default model service responds with an error. This ensures the continuity and high availability of model calls, improving the overall service availability of the Dify application. For specific configuration steps and usage instructions, see AI Fallback.

    To verify the effect of the fallback mechanism, this example configures the primary model service of the Model API as an unreachable address, and a Model Studio service as an accessible backup model. When you run the Dify application, the LLM node still returns results correctly, and the workflow runs normally.

    In the Large Model Service configuration panel, set Service Type to Single Service and the primary service name to wrong-service. Set Use Specified Model Name to Yes and the model name to Passthrough. Turn on the Fallback switch, select Fallback Only on Backend Service Errors, set the fallback service name to model-studio, and set Use Specified Model Name to No.

    After the configuration takes effect, run the Dify application. You can observe that the LLM node still returns results correctly, and the workflow runs normally.

    The gateway logs show that AI Gateway encountered a 503 error when trying to access the primary model service. It then automatically fell back to the Model Studio service, which returned a normal result, ensuring the Dify application continued to operate correctly.

    Advanced load balancing

    For scenarios where Dify accesses self-managed models (for example, models deployed on Alibaba Cloud PAI), AI Gateway offers several load balancing strategies for LLM services. These include global least-request, prefix-matching, and GPU-aware load balancing. These strategies can increase system throughput, reduce response latency, and achieve fairer and more efficient task scheduling without additional hardware costs.

    Using prefix-matching load balancing as an example, a performance test was conducted with NVIDIA GenAI-Perf. The test was configured with an average of 200 input tokens and 800 output tokens per round, a concurrency of 20, and 60 total sessions with 5 rounds of dialogue each. The results show that enabling this load balancing strategy significantly reduces the Time to First Token (TTFT), indicating that a proper load balancing strategy can markedly improve response performance.

    Metric

    No load balancing

    Prefix-matching load balancing

    Time to First Token (TTFT)

    240 ms

    120 ms

    Average RT

    14934.85 ms

    14402.36 ms

    P99 RT

    35345.65 ms

    30215.01 ms

    Token throughput

    367.48 (token/s)

    418.96 (token/s)

    Prefix cache hit rate

    40% +

    80% +

    Using prefix-matching load balancing as an example, you can easily configure the load balancing policy for the Model API via a plugin, as shown in the sample configuration below. After the plugin is enabled, when the Dify application calls the LLM service through AI Gateway, requests are load-balanced across your self-managed LLM service instances.

    lb_policy: prefix_cache
Ib_config:
  serviceFQDN: redis.dns
  servicePort: 6379
  username: default
  password: xxxxxxxxxxxx
  redisKeyTTL: 60