Agentic Cloud Governance Center:Supported check items

Check item

Description

Manual remediation

AgentIdentity workload identity doesn't have OIDC user identity passthrough configured

Agent Identity is an Alibaba Cloud service that manages identities and permissions for AI Agents. Workload Identity represents an Agent workload when it calls Alibaba Cloud services. When an Agent handles user requests, the end user's identity (OIDC ID token) must be passed through to the Agent so that downstream resources are accessed under the user's identity. Without this passthrough, all requests run under the Agent's own service account, making it impossible to distinguish end users, enforce least-privilege access, or trace actions back to individual users.

In the Agent Identity console, associate the workload identity with an identity provider (IdP) so that the end user's identity (OIDC ID token) is passed through to the Agent before it calls downstream resources.

Path: Agent Identity console > Workload Identity > Associate Identity Provider. Associate the workload identity with the IdP. For detailed steps, see Alibaba Cloud Agent Identity Workload Identity Management

[Billing] Agent Identity is currently free of charge.

[Impact] After you enable this feature, permission checks on downstream resources are performed against the end user's identity. Make sure the required permissions are granted to the corresponding users or roles. Otherwise, calls that previously succeeded under the Agent's service account may fail.

AI gateway doesn't have global authentication configured

AI Gateway provides two layers of authentication: global authentication (instance-level) and consumer authentication (API-level). At least one of these must be enabled. Otherwise, anyone who knows the endpoint address can access Model API, Agent API, or MCP Server without credentials. This check item only verifies whether global authentication is configured for the gateway instance. If you have already enabled API-level consumer authentication, you can ignore this item.

Enable one of the two authentication methods: (1) Global authentication: AI Gateway console > Instance > Security Management > Global Authentication (JWT/OIDC/Custom); or (2) Consumer authentication for each API: AI Gateway console > Model API / Agent API / MCP Server > Consumer Authentication.

For detailed steps, see Configure consumer authentication

[Billing] Authentication is included in the AI Gateway instance fee at no extra cost.

[Impact] After you enable authentication, requests without valid credentials are rejected. Distribute consumer credentials or JWT/OIDC tokens to all callers before enabling authentication to avoid service disruptions.

AgentRuntime doesn't have access credentials configured (anonymous access is enabled)

AgentRuntime is a runtime instance that deploys Agents in AgentRun. AgentRuntime supports inbound credentials that control how external users or systems access the Agent. When CredentialName is not configured, anonymous access is effectively enabled, which means anyone who knows the endpoint address can access the Agent without authentication. This poses serious risks of unauthorized access, resource abuse, and data leakage.

Configure inbound access credentials (CredentialName) for the AgentRuntime to disable anonymous access.

Path: Function Compute console > AgentRun > Agent Runtime > Configuration > Credentials / Access Control. Bind an Agent Identity workload identity or an API Key credential. For detailed steps, see Credential management

[Billing] Agent Identity is free of charge.

[Impact] After you enable inbound credentials, callers that don't carry valid credentials are denied access. Notify all callers to update their credentials before making this change to avoid service disruptions.

ModelService uses a plaintext API key instead of credential management

The AgentRun model proxy layer (ModelService) supports centralized API key management through Credentials, which avoids hardcoding keys in configurations or code. Specifying a plaintext API key in providerSettings.apiKey is non-compliant. Use a credential reference instead.

In the AgentRun console, create a credential to manage the model API key, and then update the ModelService API key field to reference that credential.

Path: Function Compute console > AgentRun > Credential Management > Create Credential > Enter the model API key. Then edit the ModelService and change the apiKey field to reference the credential instead of using plaintext. For detailed steps, see Credential management

[Billing] AgentRun credential management is free. For KMS pricing, see Billing rules .

[Impact] Switching to credential references has no impact on your workloads. After you verify that calls work correctly with the credential reference, rotate the original plaintext API key on the model platform to eliminate the leakage risk.

Check item

Description

Quick fix

AgentIdentity policy set doesn't have Cedar policies configured to restrict tool invocations

Agent Identity uses the Cedar policy language to provide fine-grained authorization for Agent MCP tool invocations, defining who can perform what actions on which resources under what conditions. A policy set is a logical collection of policies that, when bound to an AI Gateway, intercepts and evaluates all requests. If no Cedar policies are configured or the policy is set to permit all, Agent tool invocations have no permission restrictions.

In the Agent Identity console, create Cedar policies in a policy set and bind the policy set to the corresponding AI Gateway.

Path: Agent Identity console > Policy Sets > Create Policy Set > Write Cedar Policies > Bind to AI Gateway Instance. For detailed steps, see Agent Identity permission policies, Policy set management, and Fine-grained Agent permission control

[Billing] Agent Identity is currently free of charge.

[Impact] After you enable the policies, Agent tool invocations are evaluated against the policy set, and unauthorized invocations are rejected. Start in observation mode to verify that the policy set meets your business requirements before switching to enforcement mode to avoid disrupting your online Agent workloads.

PAI workspace has an excessive number of administrator role members (violates least privilege)

PAI workspaces authorize members through roles such as Resource Administrator, Workspace Administrator, Algorithm Developer, and O&M Engineer. When the number of PAI.WorkspaceAdmin members exceeds 3 and accounts for more than 30% of total members, it violates the principle of least privilege, increasing the risk of privilege abuse and accidental changes.

Remove redundant Workspace Administrator role assignments and reassign daily users to business roles such as Algorithm Developer or O&M Engineer.

Path: PAI console > Workspace > Member and Role Configuration > Edit Member Role > Remove excess WorkspaceAdmin assignments. For detailed steps, see Create and manage workspaces

[Billing] No extra cost.

[Impact] Members who lose the administrator role can no longer add or remove workspace members or modify workspace configurations, but their regular algorithm development work is unaffected. Communicate with the affected members before making the change.

Check item

Description

Quick fix

PAI EAS inference service doesn't have SLS access log delivery enabled

PAI EAS services support delivering inference access logs to Simple Log Service (SLS). Logs are stored in the pai-eas-service-log logstore under the aliyun-product-data-{uid}-{regionId} project. Without log delivery enabled, you can't perform security audits, trace attacks, or analyze traffic for inference requests.

In the PAI console, enable log collection for the public resource group or dedicated resource group.

Path: PAI console > EAS > Resource Group Management > Edit > Enable Log Collection. SLS automatically creates the project and logstore. For detailed steps, see Configure log collection for a resource group.

[Billing] SLS billing rules

[Impact] No impact on your workloads. Log delivery increases SLS costs. Enable TTL-based automatic log cleanup to control storage costs.

AI gateway doesn't have AI request logging policy enabled

AI Gateway log delivery has two layers: instance-level log delivery (basic) and API-level AI request logging policy (AiStatistics/AccessLog). The API-level layer provides a complete audit trail for model invocations. Without it, you can't trace anomalous calls, prompts, or responses.

(1) Enable instance-level log delivery to SLS. (2) Enable the AiStatistics policy for each API.

Path: (1) AI Gateway console > Instance > Observability > Log Center > Enable Log Delivery. (2) Model API / Agent API / MCP Server > Policies & Plugins > AI Request Logging > Enable. For detailed steps, see Enable gateway log delivery

[Billing] SLS billing rules

[Impact] No impact on your workloads. Logs contain prompt and response content, which may include sensitive information. Configure access control and set an appropriate TTL.

Check item

Description

Quick fix

AI gateway doesn't have network protection configured (IP allowlist/denylist or Web Application Firewall (WAF))

AI Gateway supports unified IP allowlist/denylist and WAF protection at the gateway layer. Without IP ACLs and WAF, malicious IPs and OWASP Top 10 attacks such as injection, scraping, and HTTP flood (CC) attacks reach your backend model and Agent services directly.

(1) AI Gateway console > Instance > Security Management > Allowlist/Denylist. Configure trusted IP ranges. (2) Under the same instance, enable WAF integration or add your domain to WAF 3.0. For detailed steps, see Set gateway IP allowlist and blocklist and Enable WAF protection

[Billing] The IP allowlist/denylist feature is included in the AI Gateway fee. WAF 3.0 billing rules .

[Impact] A misconfigured IP allowlist may block legitimate callers. Start in observation mode or use a denylist-only approach first, and then switch to strict allowlist mode after you verify the configuration.

AgentRuntime has public network access enabled (VPC private network isolation is not configured)

AgentRuntime uses VPC networks and vSwitches for network isolation. When disablePublicNetworkAccess is set to false, the AgentRuntime is exposed to the public internet, creating risks of unauthorized scanning and attacks. For production environments, access AgentRuntime through a VPC private network and use PrivateLink or VPN for access from trusted networks.

In the AgentRun console, disable public network access and access AgentRuntime only through the VPC internal network.

Path: Function Compute console > AgentRun > Select AgentRuntime > Network Configuration > Disable the Public Access toggle and configure the VPC and vSwitch.

[Billing] VPC internal access incurs no extra cost. If you need controlled access from the public internet, use the AI Gateway + WAF approach instead, which incurs gateway and WAF instance fees.

[Impact] After you disable public access, all callers over the public internet lose access. Make sure all callers are deployed in the same VPC or can reach the AgentRuntime through internal network or PrivateLink.

Check item

Description

Quick fix

AI gateway doesn't have the content safety check policy (AiSecurityGuard) enabled

The AI Gateway AiSecurityGuard policy integrates Alibaba Cloud AI safety guardrails to perform unified sensitive-word and compliance checks on prompts and model responses at the gateway entry point. Without this policy, content safety is scattered across individual backend services, which makes unified governance difficult and increases the risk of review gaps.

In the AI Gateway console, enable the AiSecurityGuard policy for the HttpApi and select the content safety configuration.

Path: AI Gateway console > Instance > Model API / Agent API / MCP Server > Policies & Plugins > AI Safety Protection > Enable AI Safety Protection > Select AI Safety Guardrails / Content Safety.

[Billing] Content Moderation pricing and AI Safety Guardrails pricing

[Impact] After you enable this policy, requests or responses that contain risky keywords or non-compliant content are blocked. Evaluate the dictionaries and policy thresholds to avoid incorrectly blocking legitimate requests.

AgentRun model proxy layer doesn't have a unified content safety policy configured

The AgentRun model proxy layer (ModelProxy) supports a unified aiGuardrailConfig that performs request and response content safety checks for all model calls routed through the proxy. Without this configuration, each AgentRuntime manages its own safety policy, leading to inconsistent standards and higher management overhead.

In the AgentRun console, enable aiGuardrailConfig for the model proxy layer.

Path: Function Compute console > AgentRun > Model Proxy > Edit > Advanced Policies > Enable AI Safety Guardrails.

[Billing] Built on Alibaba Cloud safety guardrails, this feature provides comprehensive security protection for AI applications. For details, see Activation and billing overview.

[Impact] After you enable this feature, requests or responses that contain non-compliant content are blocked. Test the dictionaries and thresholds in a staging environment before enabling them in production to avoid incorrectly blocking legitimate Agent workloads.

Check item

Description

Quick fix

AgentRun custom domain doesn't have HTTPS enabled

AgentRuntime provides an HTTPS endpoint by default, but custom domains (CustomDomain) bound by users require manual certificate management. When the CustomDomain protocol isn't set to HTTPS, data transmitted between external clients and the Agent is sent in plaintext, creating risks of request and response interception and tampering.

In the AgentRun console, upload an SSL certificate for the custom domain and set the protocol to HTTPS.

Path: Function Compute console > AgentRun > Custom Domain > Edit > Set Protocol to HTTPS > Upload a certificate or reference a certificate from Certificate Management Service.

[Billing] HTTPS is free. Certificates must be purchased through Certificate Management Service or obtained as free certificates.

[Impact] After you switch to HTTPS, callers must update their access protocol to https://. Notify all Agent callers before making this change.

Check item

Description

Quick fix

PAI EAS service is deployed on a public resource group

PAI EAS offers three resource types: public resources, EAS resource groups, and resource quotas. Public resources use pay-as-you-go billing and share compute capacity with other users, which may cause resource contention and lower SLA guarantees. This resource type isn't suitable for production disaster recovery architectures. Deploy production services on dedicated EAS resource groups instead.

Purchase a dedicated EAS resource group and migrate the service to it.

Path: (1) PAI console > EAS > Resource Group Management > Create Dedicated Resource Group (subscription or pay-as-you-go). (2) Edit Service > Deployment Resources > Select the dedicated resource group > Redeploy. For detailed steps, see EAS deployment resource overview

[Billing] EAS billing rules

[Impact] The migration involves redeploying the service, which causes a brief interruption. Create a new version of the service on the new resource group and use traffic splitting to gradually shift traffic.

PAI EAS service uses a shared gateway (no dedicated gateway configured)

PAI EAS uses a shared inference gateway by default, where multiple tenants share the same entry point, which may cause bandwidth contention and traffic isolation issues. For production services, use a dedicated gateway (ServiceConfig.networking.gateway is not empty) to get dedicated bandwidth and a stable SLA.

In the PAI console, bind a dedicated gateway to the EAS service.

Path: PAI console > EAS > Inference Gateway > Create Dedicated Gateway. Edit Service > Deployment Configuration > Network > Select the dedicated gateway.

[Billing] A dedicated gateway is billed based on its specification and duration, which incurs an additional gateway instance fee

[Impact] After you bind a dedicated gateway, the service access endpoint changes. Update the endpoint address used by all callers accordingly.

AgentRuntime isn't deployed across multiple zones

When all vSwitches in networkConfiguration.vswitchIds of the AgentRuntime belong to the same zone, a single-zone failure takes down the entire Agent service. Associate vSwitches from at least two different zones to enable cross-zone disaster recovery.

In the AgentRun console, edit the AgentRuntime network configuration and associate at least two vSwitches from different zones.

Path: Function Compute console > AgentRun > AgentRuntime > Network Configuration > vSwitch > Add vSwitches from different zones.

[Billing] Using vSwitches in different zones incurs no extra network fees. However, cross-zone traffic is billed as VPC cross-zone traffic.

[Impact] Configuration changes may trigger instance rescheduling. Perform this operation during off-peak hours and test in a staging environment first.

Check item

Description

Quick fix

PAI EAS inference service has only one replica configured

The TotalInstance field of an EAS service specifies the total number of replicas. A single replica creates a single point of failure that can cause service interruptions, violating disaster recovery principles. For production services, configure at least two replicas and distribute them across multiple zones.

In the PAI console, edit the service and set the instance count (TotalInstance / replicas) to 2 or more.

Path: PAI console > EAS > Service > Update > Replicas > Click Scale and set the value to 2 or more.

[Billing] Doubling the number of replicas proportionally increases EAS costs.

[Impact] Scaling out uses rolling updates with no service interruption. Make sure your resource group has sufficient quota.

Check item

Description

Quick fix

PAI EAS service doesn't have auto scaling enabled

PAI EAS ServiceConfig.autoscaler configures automatic scaling based on metrics such as QPS and GPU utilization. Without auto scaling, the service can't automatically handle traffic fluctuations.

In the PAI console, enable autoscaler for the EAS service and set the minimum and maximum instance counts along with scaling metric thresholds.

Path: PAI console > EAS > Service > Update > Auto Scaling > Elastic Scaling > Enable > Configure metrics and thresholds. For detailed steps, see Elastic resource pool configuration

[Billing] Instances added during scale-out are billed at standard EAS rates

[Impact] Scaling events may cause brief fluctuations. Set appropriate cooldown periods to prevent frequent scaling.

AgentRuntime doesn't have an auto scaling policy configured

AgentRuntime uses scalingConfig.minInstances to configure the minimum number of instances. For latency-sensitive workloads, avoid scaling down to zero (set minInstances to 1 or more). This provides the following benefits:
- Warm start: Achieves millisecond-level warm starts with no service interruptions.
- Guaranteed delivery: Locks elastic resources to ensure consistent capacity.
- Smart idle billing: When no requests or background tasks are being processed, instances automatically enter idle billing mode (CPU is free, GPU is billed at only 10%-20% depending on the card type), further reducing costs.

In the AgentRun console, configure an auto scaling policy (minimum/maximum instance counts and scaling metrics) for the AgentRuntime.

Path: Function Compute console > AgentRun > AgentRuntime > Elasticity & Instances > Scaling Configuration > Create Scaling Policy > Set minInstances to 1 or more. For detailed steps, see Instance scaling restrictions and rules

[Billing] Instances added during scale-out are billed at standard AgentRun rates. See billing overview

[Impact] Setting minInstances too high increases standing costs. Setting it too low leads to more noticeable cold-start latency. Evaluate based on your traffic patterns.

Check item

Description

Quick fix

PAI EAS service doesn't use canary release

PAI EAS uses service groups (DescribeGroup) to distribute traffic by weight across service versions, enabling canary releases. When you update a service directly without a service group, the update takes effect for all traffic immediately, which increases the blast radius of any failure.

Add multiple versions of the service to a service group and configure traffic weights.

Path: PAI console > EAS > Service Groups > Create Group > Add the old and new service versions to the group > Set weights (for example, 5%/95%) > Validate the canary release and then gradually shift traffic. For detailed steps, see Manage service groups

[Billing] The canary release feature is free. Purchasing a dedicated resource group incurs additional costs.

[Impact] During the canary release, multiple versions run simultaneously. Make sure the versions are compatible with each other.

Check item

Description

Quick fix

PAI EAS service doesn't have CloudMonitor alert rules configured

CloudMonitor (CMS) uses DescribeMetricRuleList to query alert rules for PAI EAS services. Without any alert rules, service anomalies such as unavailability, rising 5xx error rates, or latency spikes can't be automatically reported to the operations team.

In the CloudMonitor console, create alert rules for PAI EAS (for example, 5xx ratio, average latency, and unhealthy instance count).

Path: CloudMonitor console > Alert Service > Alert Rules > Create Alert Rule > Select the PAI-EAS namespace > Configure metrics and thresholds > Associate a notification group. For detailed steps, see Create alert rules

[Billing] Basic alerts are free. SMS and phone notifications incur charges. See billing overview

[Impact] No impact on your workloads. Set appropriate silence periods and notification frequencies to prevent alert storms.

PAI EAS service doesn't have Application Real-Time Monitoring Service (ARMS) tracing enabled

Setting ServiceConfig.monitoring.enable_tracing to true for a PAI EAS service enables ARMS tracing. Without tracing, the internal execution of inference requests (preprocessing, model invocation, and external dependencies) isn't observable.

In the PAI console, enable tracing for the EAS service.

Path: PAI console > EAS > Service > Update > Monitoring > Enable Tracing > Select or create an ARMS application. For detailed steps, see EAS tracing solution

[Billing] ARMS billing rules

[Impact] Enabling the ARMS agent introduces a minor performance overhead (typically less than 5%).

AI gateway doesn't have CloudMonitor alert rules configured

CloudMonitor uses DescribeMetricRuleList under the acs_cnapigateway namespace to query AI Gateway alert rules. Without alert rules, anomalies in key metrics such as 5xx error rates, latency, and connection counts can't be automatically reported.

In the CloudMonitor console, create alert rules for the AI Gateway.

Path: CloudMonitor console > Alert Service > Alert Rules > Create Alert Rule > Configure metrics (such as 5xx ratio, token rate, and latency) and thresholds > Associate an alert contact group.

[Billing] Basic alerts are free. SMS and phone notification billing See billing overview

[Impact] No impact on your workloads. Set appropriate thresholds and silence periods.

AgentRuntime doesn't have ARMS full-stack monitoring enabled

Enabling armsConfiguration.enableArms for an AgentRuntime connects requests to ARMS application monitoring, which provides call chain tracing, SQL analysis, and memory/thread profiling. Without it, the full request chain isn't observable, making it difficult to diagnose failures.

In the AgentRun console, enable ARMS monitoring for the AgentRuntime.

Path: Function Compute console > AgentRun > AgentRuntime > Observability > Application Monitoring > Enable with One Click. For detailed steps, see Agent observability

[Billing] Enabling application monitoring may incur charges. For details, see Application monitoring billing

[Impact] Enabling the ARMS agent introduces a minor performance overhead (typically less than 5%).

AgentRuntime doesn't have health checks configured

The healthCheckConfiguration.httpGetUrl field of an AgentRuntime configures HTTP GET health checks. Without health checks, abnormal service instances can't be automatically detected and replaced. Health checks can only be configured when artifactType is Code. Container mode may not support this feature.

In the AgentRun console, configure a health check URL (such as /health) for the AgentRuntime.

Path: Function Compute console > AgentRun > AgentRuntime > Advanced Configuration > Health Check > Set httpGetUrl and the check interval.

[Billing] No extra cost.

[Impact] An incorrect health check implementation (such as returning non-2xx responses) causes instances to restart repeatedly. Test the health check endpoint in a staging environment first.

AgentRuntime doesn't have log monitoring configured

The logConfiguration.logstore field of an AgentRuntime configures application log delivery to SLS. Without this configuration, Agent runtime logs can't be collected, making it difficult to troubleshoot failures and perform audits.

In the AgentRun console, configure log delivery for the AgentRuntime.

Path: Function Compute console > AgentRun > AgentRuntime > Observability > Logs > Enable with One Click or manually configure.

[Billing] SLS billing rules

[Impact] No impact on your workloads. Set an appropriate TTL and use compressed storage to control costs.

Check item

Description

Quick fix

AI gateway Agent API / Model API doesn't have inbound protection plugins configured

The inbound direction (direction=InBound) of AI Gateway Agent API and Model API should have rate limiting, timeout, circuit breaking, and security protection plugins configured. Without these plugins, malicious traffic and abnormal requests reach backend Agent and model services directly.

In the AI Gateway console, enable inbound rate limiting, circuit breaking, content safety, and other plugins for Agent API and Model API.

Path: AI Gateway console > Agent API / Model API > Policies & Plugins > Inbound Processing > Enable policies/plugins (rate limiting, circuit breaking, security, logging, and others).

[Billing] Policies are included in the AI Gateway instance fee. Some advanced security protections may depend on additional products such as Content Moderation or WAF, which are billed separately.

[Impact] After policies take effect, requests that don't meet the rules are rejected. Evaluate your traffic patterns before setting thresholds.

Check item

Description

Quick fix

AgentRun model proxy layer doesn't have a timeout configured

The AgentRun model proxy layer (ModelProxy) uses proxyConfig.policies.requestTimeout to configure request timeouts. Without a timeout, an unresponsive model blocks the entire request chain, and long-tail requests can exhaust AgentRuntime resources.

In the AgentRun console, configure a request timeout for the model proxy layer (30-60 seconds recommended).

Path: Function Compute console > AgentRun > Model Proxy > Edit > Advanced Policies > Request Timeout.

[Billing] No extra cost.

[Impact] A timeout that's too short may truncate long model responses. Set the value based on the maximum acceptable response time for your workloads.

AgentRun model proxy layer doesn't have a fallback policy configured

The AgentRun model proxy layer supports multi-level fallback (fallbacks array). When the primary model is throttled, times out, or fails, traffic automatically switches to a backup model to prevent the AgentRuntime from becoming completely unavailable.

In the AgentRun console, configure a fallback list for the model proxy layer with backup models listed in priority order.

Path: Function Compute console > AgentRun > Model Proxy > Edit > Advanced Policies > Add Fallback Model.

[Billing] When a fallback is triggered, the backup model is billed based on its own token pricing.

[Impact] Output style may differ significantly from the primary model when a fallback is triggered. Configure compatible backup models for critical workflows.

AI gateway Model API doesn't have multi-model load balancing configured

AI Gateway Model API serviceConfigs supports multiple backend model services with weight-based load balancing. With only one model service configured, all traffic routes to a single backend, creating risks of overload and cascading failures.

In the AI Gateway console, configure multiple backend model services for the Model API and set traffic weights to enable load balancing.

Path: AI Gateway console > Model API > Backend Services > Add multiple model services and configure weights. For detailed steps, see Configure Model API

[Billing] The load balancing feature is included in the AI Gateway instance fee at no extra cost. Backend models are billed based on their own token or invocation pricing.

[Impact] Improper weight distribution may cause traffic skew. Start with equal weights for canary validation, and then adjust to match your actual traffic proportions.

AI gateway doesn't have AI Fallback multi-model failover enabled

AI Fallback is the multi-model failover capability of AI Gateway. When the primary model service is unavailable, traffic automatically switches to a backup model, improving overall AI application availability. Without aiFallbackConfig enabled in policyConfigs, failures on the primary model return errors directly to callers.

In the AI Gateway console, enable the AI Fallback policy for the Model API and configure primary and backup models in priority order.

Path: AI Gateway console > Model API > Policies & Plugins > AI Fallback > Enable > Add backup model services. For detailed steps, see AI Fallback

[Billing] The policy itself incurs no extra cost. When a fallback is triggered, the backup model is billed based on its own token pricing.

[Impact] Output style may differ from the primary model when a fallback is triggered. Configure compatible backup models for critical workflows and test them thoroughly.

AI gateway Model API doesn't have a response caching policy configured

AI Gateway AI Cache uses Redis for exact-match caching or DashVector for semantic caching to match repeated prompts. Cached responses are returned directly from the gateway, significantly reducing backend model invocations, improving response speed, and providing a fallback when the model is unavailable.

In the AI Gateway console, enable the AI Cache policy for the Model API. Choose between Redis exact-match caching and DashVector semantic caching.

Path: AI Gateway console > Model API > Policies & Plugins > AI Cache > Enable > Select a cache type and configure the Redis or DashVector instance. For detailed steps, see AI Cache

[Billing] The policy itself incurs no extra cost. Redis and DashVector instances are billed based on the selected specifications. Cache hits significantly reduce backend model token consumption.

[Impact] Semantic caching may return incorrect responses if the similarity threshold is set too low. Start with a high threshold and adjust based on results.

AI gateway doesn't have multi-dimensional rate limiting and circuit breaking configured

The AI Gateway AI Token rate limiting policy supports multi-dimensional traffic control based on token consumption, request count, and concurrency. Without traffic control plugins, traffic spikes can overwhelm backend model services and cause cascading failures.

In the AI Gateway console, enable AI Token rate limiting and circuit breaking policies for the Model API. Configure thresholds across multiple dimensions including TPM, TPS, and concurrency.

Path: AI Gateway console > Model API > Policies & Plugins > AI Token Rate Limiting / Circuit Breaking > Enable > Set thresholds for TPM, TPS, concurrency, and other dimensions. For detailed steps, see AI Token rate limiting

[Billing] Policies are included in the AI Gateway instance fee at no extra cost.

[Impact] Requests that exceed the thresholds are rejected. Set thresholds based on your peak traffic and inform callers to implement retry logic.

Check item

Description

Quick fix

AgentRun model proxy layer doesn't have concurrency control or token rate limiting configured

The AgentRun model proxy layer uses proxyConfig.policies with concurrencyLimit and tokenRateLimiter (TPM/TPS/TPH/TPD) to control concurrency and token rate. When neither is configured, traffic spikes pass directly to downstream models, triggering model-side throttling or incurring uncontrolled costs.

In the AgentRun console, configure concurrency control or token rate limiting for the model proxy layer.

Path: Function Compute console > AgentRun > Model Proxy > Edit > Advanced Policies > Concurrency Control / Token Rate Limiting.

[Billing] No extra cost.

[Impact] Requests that exceed the thresholds are rejected. Set thresholds based on your peak traffic and inform callers to implement retry logic.

Check item

Description

Quick fix

PAI EAS service GPU utilization has been consistently below 10% (resource waste)

CloudMonitor DescribeMetricData is used to query the GPU utilization of a PAI EAS service. If GPU utilization stays below 10% over time, the service has too many replicas or uses oversized instance specifications, resulting in wasted GPU resources.

Reduce the number of replicas, downsize the instance specification, or enable auto scaling to scale on demand.

Path: PAI console > EAS > Service > Update > Reduce replicas or switch to a smaller specification. Also enable autoscaler with a low minInstances value. For detailed steps, see Elastic resource pool configuration

[Billing] Reducing replicas, downsizing specifications, and scaling in reduce EAS costs

[Impact] Scaling in may reduce peak-traffic handling capacity. Evaluate based on peak traffic patterns and use auto scaling as a safety net.

Check item

Description

Quick fix

AI gateway doesn't have the tool selection plugin enabled

The AI Gateway tool selection plugin (AiToolSelection) filters relevant tool descriptions based on the prompt before passing them to the model, rather than bundling all registered tool descriptions. This reduces token consumption per inference call.

In the AI Gateway console, enable the tool selection policy for the Model API.

Path: AI Gateway console > Model API > Policies & Plugins > Tool Selection > Enable > Configure selection rules.

[Billing] The tool selection policy is included in the AI Gateway instance fee at no extra cost.

[Impact] Improperly configured selection rules may prevent the model from selecting the correct tool. Thoroughly test common scenarios before rolling out to production.

Check item

Description

Manual remediation

PAI EAS service GPU utilization too high

Queries PAI EAS service GPU utilization through CloudMonitor DescribeMetricData. Sustained utilization above the threshold (for example, 90%) indicates insufficient instances, which can cause increased response latency, request queuing, or service unavailability.

Scale out replicas, upgrade instance specifications, or enable auto scaling to handle peak traffic.

Path: PAI console > EAS > Service > Update > Increase replicas or switch to a larger specification > Enable autoscaler and set an appropriate maxInstances value. Detailed steps

[Billing] Scaling out replicas or upgrading specifications increases EAS costs.

[Impact] Scale-out uses rolling updates with no service interruption. Make sure the resource group has sufficient quota.