The configuration center provides hot-reload capabilities for InferX workstation settings, covering request routing, traffic protection, and scheduling optimization. Changes take effect immediately without restarting services.
Features
The configuration center organizes features by use case:
Use case | Feature | Description |
Cost optimization | Automatically selects L1/L2/L3 tier models based on request complexity — lightweight models for simple questions, larger models for complex ones — to balance cost and quality. | |
Service reliability | Combines local inference services with external API services (such as PAI Token Service) and supports failover to improve service availability. | |
Schedules inference resources across regions. When local resources are scarce, traffic is forwarded to other regions. | ||
Traffic protection | Limits concurrent requests to a workstation to prevent backend overload. | |
Controls request count, token usage, and concurrency on a per-user basis. | ||
Routing optimization | Routes requests to the optimal instance through multi-layer priorities: session affinity, prefix caching, and load balancing. | |
Matches the | ||
Canary validation | Copies live requests to a target address for new version validation and performance comparison. | |
Distributes traffic across inference services by weight for canary releases and A/B testing. |
When you send a request to an InferX workstation, the request is processed through the following pipeline:
Failover forwarding requires configuring failover in the split proxy so that requests automatically retry on an alternative proxy when the primary is unavailable.
Scheduling policy
The scheduling policy has three parts:
Primary scheduling policy: The core routing pipeline. Routes requests to inference instances through three priority layers — session affinity → prefix caching → load balancing — matching each layer in order and routing on the first hit.
Prefix index: A persistent prefix index tree inside the scheduler that tracks the prompt prefix cache state of each instance. Estimates the theoretical hit rate ceiling. When prefix caching is enabled, also serves as the routing match basis. Configure its memory boundary and cleanup cycle here.
PD disaggregation: Takes effect only in PD disaggregated deployment mode. Configures a separate scheduling policy for the decode phase.
Primary scheduling policy
The primary scheduling policy runs three layers in priority order. Each request passes through the layers sequentially — a match at any layer triggers routing; no match falls through to the next layer. Session affinity and prefix caching are optional; load balancing is always required.
The scope of the primary scheduling policy depends on the deployment mode. In PD integrated mode, it applies to all requests as the global scheduling pipeline. In PD disaggregated mode, it applies only to the prefill role. The decode phase scheduling policy is configured separately in the PD disaggregation section below.
Layer 1: Session affinity
Routes requests from the same session to the same inference instance, preventing context loss from instance switching. Applicable to multi-turn conversations and stateful inference.
When enabled, requests must carry the designated session header to take effect. Unmatched requests fall through to the prefix caching layer.
Parameter | Description |
Affinity mode | Select the strength of affinity routing.
|
Session header | The HTTP header field used to identify the session ID. Default: Note Known clients that include a session header by default: • Claude Code: X-Claude-Code-Session-Id (one per session) Other mainstream LLM clients (OpenAI/Anthropic SDK, LangChain, etc.) don't include any session header by default. The caller must inject it explicitly. Requests without this header are handled by the normal scheduling policy. |
Session TTL | The time-to-live (in seconds) for session binding entries. Expired entries are automatically removed. Default: 1800 seconds. |
Max entries | The maximum number of session records the affinity cache can hold. When the limit is reached, the oldest records are evicted. Default: 100000. |
Soft max load ratio | Applies only in Soft mode. When the preferred instance load exceeds average load × soft max load ratio, affinity routing is skipped and a less-loaded instance is selected. Default: 1.5. |
Require session ID | When enabled, inference requests without a session ID header are rejected at the request parsing stage (HTTP 400) — they aren't queued or scheduled. Prevents large requests without affinity identifiers from randomly hitting an instance and overwhelming its KV cache. Takes effect only when session affinity is enabled. |
Layer 2: Prefix caching
Prefers instances that have already cached the same prefix, reducing redundant computation and improving cache hit rates. Applicable to scenarios with highly repetitive input prefixes, such as fixed system prompts or batch document processing. Unmatched requests fall through to the load balancing layer.
Parameter | Description |
Load tolerance factor | When the load of a cache-hit instance exceeds average load × (1 + tolerance factor), affinity routing to that instance is skipped. Default: 0.3. |
Pending request threshold | When the number of pending requests on an instance exceeds this value, the instance is considered overloaded. Even if it has a cache hit, requests aren't routed to it to avoid single-instance overload. Default: 2. |
Hit rate threshold | When the ratio of prefix match length to total prompt length is below this value, it's treated as a miss. Default: 0.50. |
Layer 3: Load balancing
When neither session affinity nor prefix caching matches, distributes requests across inference instances using the selected algorithm.
Algorithm | Use case |
| Selects the instance currently processing the fewest tokens. Suitable for scenarios with large token length variance to balance compute pressure. |
| Selects the instance with the fewest active requests. Suitable for scenarios where request processing times are similar. |
| Distributes requests sequentially across all instances. Suitable for scenarios with uniform instance performance and stable loads. |
Prefix index
A persistent prefix index tree inside the scheduler that calculates the theoretical character cache hit rate ceiling. When prefix caching scheduling is enabled, the index data also serves as the routing match basis, prioritizing instances with the highest cache hit rate. The following parameters control the index memory boundary and cleanup cycle.
Parameter | Description |
Cache TTL | The time-to-live (in seconds) for prefix index entries. Expired entries are automatically removed. Default: 259200 seconds (3 days). If sessions typically end within 1 hour and VRAM is tight, reduce to 3600 seconds to speed up cold data reclamation. |
Maintenance interval | The execution cycle (in seconds) for background scanning and cleanup of expired index entries. Default: 60 seconds, suitable for most scenarios. Increase only when cluster CPU load is persistently high. |
Memory threshold ratio | Together with reserved memory, determines the memory eviction trigger point for the scheduler process — whichever triggers first. When scheduler process memory exceeds container memory × this ratio, the least recently used index entries are evicted by LRU. 0 means the default value of 0.70. Lower this value when the scheduler container experiences frequent OOM. Increase it when index entries are evicted too early, affecting routing decisions. |
Reserved memory | The minimum available memory (in GB) reserved for the container system and runtime. Together with the memory threshold ratio, determines the memory eviction trigger point. When scheduler process memory exceeds container memory − reserved memory, the least recently used index entries are evicted by LRU. 0 means the default value of 1.50 GB. Increase this value when the scheduler container is OOM-killed by the system. Decrease it when index entries are evicted too early. |
Block size | The byte size of prefix index blocks (default: 128). After changing this at runtime, old indexes no longer cross-match with new blocks, and the hit rate temporarily drops until the index refreshes. Prefixes shorter than one full block aren't indexed. |
PD disaggregation
In Prefill-Decode (PD) disaggregated deployment mode, the prefill and decode phases run on separate instances. You can configure an independent scheduling policy for the decode phase.
Parameter | Description |
PD disaggregation mode | Select the PD disaggregation implementation, which must match the inference engine used in deployment. For example, |
Decode scheduling policy | An independent scheduling policy for the decode role when PD disaggregation is enabled. The prefill role uses the primary scheduling policy. Available algorithms are the same as the load balancing layer of the primary scheduling policy ( |
SGLang PD independent pairing | Applies only to SGLang Mooncake PD disaggregation (vLLM PD is unaffected). When enabled, the scheduler can independently and freely pair prefill and decode instances without requiring aligned instance numbers on both sides, providing greater scheduling flexibility for cache affinity and load balancing.
|
Scheduling timeout and circuit breaker
The timeout limit and circuit breaker protection for scheduling results at the gateway. On timeout, the gateway falls back to local round-robin. When the sustained timeout ratio reaches the threshold, the circuit breaker opens. After the cooldown period, probes automatically attempt recovery. Disabling the circuit breaker only disables the fast fallback protection — the timeout limit always applies.
Parameter | Description |
Circuit breaker | When disabled, reverts to pure timeout fallback behavior (for troubleshooting). Scheduling wait timeout isn't affected by this switch. |
Scheduling wait timeout | The maximum wait time for a single scheduling attempt. On timeout, falls back to local round-robin. |
Trip failure ratio | When the ratio of scheduling failures (failed attempts / total attempts) within a 10-second rolling window reaches this value, the circuit breaker trips. No judgment is made when the sample size is below 10 to prevent false triggering under low traffic. |
Breaker cooldown | The duration the circuit breaker stays open after tripping. After expiry, probes attempt recovery. |
Probe wait limit | The maximum wait time for probe requests after cooldown expires. A successful probe closes the breaker and resumes scheduling. A failed probe reopens the breaker. |
Model routing
Routes requests to the corresponding inference service instance in the workstation based on the model field. When enabled, use the Default model setting to specify fallback behavior for unmatched requests.
Parameter | Description |
Default model |
|
Fuzzy match | When no model mapping matches, enable fuzzy matching against instance models: ① Different notations of the same model name match each other (case, separators, and Org prefixes are ignored — for example, qwen2.5_72b_instruct matches Qwen2.5-72B-Instruct). ② A shortened model name in the request matches the full deployment name (for example, qwen2.5-72b matches Qwen2.5-72B-Instruct). ③ Wildcard * is supported. When disabled, only exact model name matches are allowed (with case and separator variations), without abbreviation or wildcard matching. |
Model mapping | Associates model names (wildcard * supported) with backend services. Matching is evaluated top-to-bottom — the first match takes effect. After a match, requests route only to the listed services without falling back to instance model matching. Anchored to the full string, case-insensitive. A plain string is an exact match. |
Smart model selection
Smart model selection uses an external classification service to automatically evaluate request difficulty and routes requests to different tier models (L1/L2/L3), balancing response quality with inference cost.
For detailed configuration, see Smart model selection.
Weight-based splitting doesn't distinguish model names and is incompatible with smart model selection. To use smart model selection, switch splitting to prefix mode or disable split routing first.
Service canary
Service canary distributes traffic across inference services by weight for canary releases and A/B testing. Configure canary rules to control traffic allocation for specific models or all models.
Canary rules
Each canary rule contains model selection and service weight configuration. Rules are evaluated top-to-bottom — the first match takes effect.
Column | Description | Example |
Service | Select the model service to apply the canary rule to. Wildcard is supported — for example, "All models ()" matches all models. | All models (), qwen- |
Weight | Service: Select or enter the target inference service name. | Select a service |
Weight: The traffic weight assigned to this service. The percentage is automatically calculated and displayed. | 100, 0 |
Click Add target to add more services to the current rule for proportional traffic distribution.
Click Add rule to create a new canary rule. Rules are evaluated top-to-bottom — the first match takes effect.
Split proxy
Forwards requests received by the workstation to external upstream API services (such as PAI Token Service) based on configured rules. This enables traffic coordination between local inference services and external services to improve overall service reliability.
Use cases: Suitable for scenarios that require mixing local and external services for improved reliability, such as canary releases, A/B testing, and failover. The split proxy forwards a portion of traffic to external API services for high availability and flexible switching.
Split proxy and smart model selection generally aren't enabled at the same time. They serve different purposes: split proxy is for service reliability and canary releases, while smart model selection is for cost optimization.
After enabling, select a splitting strategy and configure proxy entries in the proxy list.
Splitting strategy
Strategy | Description |
Weight | Distributes requests by the weight ratio configured for each proxy. Suitable for canary releases, A/B testing, and other scenarios that require proportional traffic control. When this strategy is selected, model rules aren't needed in the proxy list — just set weight values. |
Prefix | Routes traffic to the corresponding proxy based on the |
Proxy list
The proxy list must contain at least one "Local service" entry, which represents the workstation's own inference service. You can add external upstream services as additional proxies.
Column | Description |
Status | Whether this proxy entry is enabled. Disabled entries don't receive forwarded requests. |
Proxy address | The access address of the upstream API service. "Local service" is a fixed entry that represents the workstation's own inference service. |
API key | The API key required to access this proxy. Not needed for local service. |
Weight | The traffic weight assigned to this proxy under the Weight strategy. Proxies receive requests proportional to their weights. Not displayed under the Prefix strategy. |
Model rules | How the |
Failover | When selected, the system switches to this proxy for retry (in list order) when other proxies are unavailable or fail. Local service doesn't support failover. |
External proxy services follow their own rate limiting rules and aren't affected by the workstation's request rate limiting configuration.
Request rate limiting
Limits concurrent requests to the workstation to prevent backend inference instances from being overloaded. When the limit is exceeded, choose to reject requests directly or queue them. Supports separate concurrency limits for the overall lifecycle, prefill phase, and decode phase.
Parameter | Description |
Overlimit handling | How to handle requests that exceed the concurrency limit: Reject (return an error immediately) or Queue (requests enter a waiting queue). |
Rate limit scope | The granularity of rate limit counting:
|
Overall concurrency limit | A request occupies a quota slot from dispatch to response completion (one concurrency seat until the request finishes). Applies only to PD integrated deployment. Not counted in PD disaggregated deployment. |
Prefill concurrency limit | Observation window: from dispatch to before the first token is received. This phase includes engine queuing + prefill computation, which can't be distinguished and are counted together as prefill phase usage. Applies to both PD integrated prefill phase and PD disaggregated prefill nodes. |
Decode concurrency limit | Observation window: from the first token received to response completion. Applies only to PD disaggregated deployment, limiting decode nodes. Not counted in PD integrated deployment. |
Traffic mirroring
Copies live requests to a target address based on the sampling ratio. The target service processes mirrored traffic independently, without affecting normal request responses. Suitable for canary validation of new versions, performance comparison, and data collection.
Parameter | Description |
Target address | The service address that receives mirrored traffic. |
Target model | The model name to substitute in mirrored requests. Leave empty to keep the original request's model. |
Sampling ratio | The proportion of total traffic that's mirrored (0%–100%). |
Auth token | The authentication token required to access the target address. |
Timeout | The timeout (in milliseconds) for mirrored requests. Expired mirror requests are automatically dropped without affecting normal request processing. |
Log recording | Whether to record detailed logs of mirrored requests for analysis and comparison. |
Global scheduling
When local inference resources are scarce, global scheduling redirects traffic to idle inference instances in other regions for efficient resource utilization across regions.
After enabling global scheduling, add InferX services from other regions in the member list:
Member name: A custom member name.
URL / Token: Enter the base URL and API key of the InferX service deployed in another region.
Data backup
Data backup periodically backs up management data. On service startup, the latest backup (automatic or manual) is restored. The backup directory is set in the service startup configuration (requires a restart to change). Changes to backup frequency and rotation take effect after the next backup cycle.
Note: This feature isn't applicable when the workstation is configured with Redis. Redis manages its own persistence and doesn't need periodic backups from this component. If you previously used built-in storage and have backup files, use Import to Redis to migrate.
Configuration
When the workstation isn't configured with Redis, configure a backup directory first. Add the following JSON configuration in the service startup configuration:
{
"llm_gateway": {
"persist_snapshot_dir": "/mnt/oss/persist-test/state_persist_llm_gateway/snapshots"
},
"storage": [
{
"mount_path": "/mnt/oss/persist-test/",
"oss": {
"path": "oss://pai-test-changjin/inferx/",
"read_only": false,
"type": "ossfs"
}
}
]
}By default, the smart router uses in-memory storage as the persistence method. This doesn't depend on external components, but has limited reliability — if all smart router instances crash in an abnormal scenario, data is lost. Mount an OSS bucket to back up data and prevent data loss:
Configure
storagewithmount_path: Mount OSS so the program can read and write to external storage.Add the
llm_gatewayconfiguration forpersist_snapshot_dir. This must be a subdirectory of themount_pathabove. The program periodically generates data backup files at this location during runtime.
Backup operations
After configuring the backup directory, perform the following operations on this page:
Operation | Description |
Export | Manually export the current data backup. Available anytime. |
Import | Import data from a backup file to restore service state. |
Import to Redis | If you previously used built-in storage and have backup files, use this to migrate to Redis. |
User rate limiting
User rate limiting controls request count and token usage on a per-user basis to prevent uncontrolled resource consumption by individual users. Unlike Request rate limiting, which protects individual instances or services from concurrency overload, user rate limiting manages per-user traffic speed, concurrency, and periodic quotas within a tenant.
Timezone: IANA timezone, used for allowed calling periods and daily/weekly/monthly quota boundaries.
Quota templates: Configure different rate limiting quotas for different user groups.
After creating quota templates in the configuration center, bind them to users in User management. You can also set a template as the default — users without a bound quota template use the default.
Quota templates support four rate limiting capabilities, which can be enabled independently or combined (disabling a capability preserves the entered configuration for re-enabling):
Allowed calling period: The local time period during which requests are allowed (inclusive on both ends, precise to the minute). Can be restricted by day of the week. Not configured or all days selected means every day. Empty means always allowed. For all day, select 00:00–23:59. Cross-midnight periods must be split into two segments with the correct day of the week for each (for example: Friday 22:00–23:59 + Saturday 00:00–02:00). Requests outside the allowed period are rejected.
Quota: Periodic quotas for request count and token count, controlled by period window.
Parameter
Description
Token weights
Weights used to convert input/output/cached tokens into quota usage.
Input: Weight for input tokens that didn't hit the cache.
Output: Weight for output tokens.
Cached: Weight for input tokens that hit the cache.
Quota window
Set periodic quotas, including the period (every 5 hours/daily/weekly/monthly), maximum request count, and maximum weighted token count. 0 means no limit.
Concurrency limit: Maximum concurrent requests per user. Excess requests are rejected or queued.
Rate limit (RPM/TPM): Token bucket algorithm-based rate limiting by requests per minute (RPM) and tokens per minute (TPM).
Parameter
Description
RPM
Maximum requests per minute. 0 means no limit.
TPM
Maximum tokens per minute. 0 means no limit.
Default output estimate
When a user request doesn't specify
max_tokens, this value is used as the estimated output token count for TPM pre-deduction. When specified in the request, the request value is used. Applies only when rate limiting is enabled.
ImportantIn dry run mode, rate limiting rules are calculated and hit counts are recorded as usual, but no requests are actually blocked. Use this to verify whether thresholds are reasonable before enforcing new rules — misconfigured rate limits can accidentally block large volumes of production traffic, and actual traffic distribution often differs from estimates. Observe first, then enable.
Advanced settings
Advanced settings provide fine-grained control over request timeouts and protocol conversion for special scenarios.
Request timeout
Configure request timeouts from the gateway to backend inference services, including both streaming and non-streaming requests. Unit: seconds. 0 means no limit.
Parameter | Description |
Streaming first-packet timeout | Timeout for the first data packet of a streaming response. Default: 300s. |
Streaming idle timeout | Maximum idle time between two data packets in a streaming response. Default: 300s. |
Streaming total timeout | Maximum total duration for the entire streaming request (in seconds). Default: 0 (no limit). |
Non-streaming total timeout | Total timeout for non-streaming responses. Default: 900s. |
Protocol and conversion
Configure API protocol passthrough and message format conversion for compatibility with different clients and models.
Parameter | Description |
Responses API passthrough |
|
Anthropic API passthrough |
|
Preserve Anthropic System message position | When enabled, keeps the Anthropic System message in its original position in the request instead of moving it to the beginning of the message list. |
Force streaming usage return | Returns streaming usage to the client even when |
Reasoning effort mapping mode |
|