API Gateway:Cache

Key concepts of semantic cache

To help you better understand the semantic caching mechanism, the following section describes its key concepts:

Features and benefits

• Dual-mode cache system: You can choose between two cache types and dynamically adjust the matching threshold as needed.

  1. Exact match: Based on the Redis key-value storage architecture, it provides millisecond-level responses to identical requests.

  2. Semantic cache: Uses the DashVector vector database (Alibaba Cloud Vector Retrieval Service) to intelligently match semantically similar requests. The similarity threshold is adjustable. This overcomes the limitations of traditional string matching.

• Reduced redundant computation: For identical AI requests, the cached response data is returned directly, which avoids redundant calls to the large language model.

• Performance improvement: By quickly retrieving results from the cache, it significantly reduces the response time and the load on backend servers. Semantic cache enables intent-level responses, which significantly improves user satisfaction and experience.

• Expanded scenario coverage: It is suitable for standard scenarios, such as customer service systems and knowledge base queries, and supports processing natural language instruction variants, such as 'tomorrow's weather' and 'weather forecast for the next 24 hours'.

• Log Monitoring: Provides analysis of cache hit ratio metrics.

Procedure

1. Log on to the AI Gateway console and choose Instance. In the top menu bar, select a region, then click the target instance ID.

2. In the navigation pane on the left, choose Model API, then click the target API name to go to the API Details page.

3. Click Policies and Plug-ins, enable the Cache switch, and configure the parameters.

   AI Gateway has upgraded its caching capabilities to support Semantic and Exact Match. Select the appropriate cache mode based on the Strategy comparison.

   Semantic cache

   Important

   If a request contains the x-higress-skip-ai-cache: on request header, the request bypasses the cache. It is forwarded directly to the backend service, and the response is not cached.

   

   • Cache Key Strategy: Select the default option, Latest Query Only, or select Integrate Historical Queries based on your requirements.

   • TextCectoriazation:

     • AI Services: Select the AI service that you have created. If you do not have an AI service, click Create Service to create one.

     • Model Name: Select the name of the model that you want to use.

     • Timeout Period: Set the timeout period. The default value is 5000 ms.

   • Vector Database Configuration:

     • Service Provider: If you have not activated DashVector (Vector Retrieval Service), click Go to console to open the service activation page. Follow these steps: Create a cluster > Create an API key > Create a collection. Record the collection name for later use.

       Important

       When you create a collection, select Cosine as the distance measure. The vector dimension must match the dimension of the text embedding model. For more information about the output vector dimensions of the text embedding models on the Alibaba Cloud Model Studio platform, see General-purpose text embeddings.

     • Endpoint: Enter your DashVector endpoint.

     • Collection Name: Enter the name of the collection that you created.

     • API Key: The access credential. For more information, see Manage API keys.

     • Vector Similarity Threshold: This value determines how strictly queries are matched to cached content. The value can range from 0 to 1. A value of 0.8 or 0.9 is recommended. A larger value indicates greater semantic similarity. For more information, see Vector similarity threshold configuration guide.

     • Timeout: Set the timeout period. The default value is 3000 ms.

   Exact match

   Important

   • In the Redis console, add the VPC CIDR block of the gateway instance to the whitelist.

   • If a request contains the x-higress-skip-ai-cache: on request header, the request bypasses the cache. It is forwarded directly to the backend service, and the response is not cached.

   

   • Cache Key Strategy: Select Latest Query Only (default) or Integrate Historical Queries as needed.

   • Redis cache configuration:

     • Redis service URL: Enter your Redis endpoint.

     • Port: Enter your port number.

     • Access Method: Select the method for accessing the Redis service. The options are Account+Password , Password-only, and Password-free.

     • Database Account: Enter the database account if you log on using an account and password.

     • Database Password: If you select password-based authentication, you must enter the database password.

     • Database No.: The database number.

     • Duration (seconds): The default cache duration is 1800 seconds. During this period, if the API receives an identical AI request, the LLM is not called, and the cached response is returned directly.

4. Confirm the configuration and click Save.

Vector similarity threshold configuration guide

Cache hit ratio

The following sample search statement shows how to query the cache hit ratio at the gateway level:

cluster_id:{your-gatewayId} and inner-ai-cache-{your-gatewayId} | SELECT 
SUM(CASE WHEN content LIKE '%cache hit for key%' OR content LIKE '%key accepted%' THEN 1 ELSE 0 END) AS hit_count,
SUM(CASE WHEN content LIKE '%cache miss for key%' OR content LIKE '%score not meet the threshold%' THEN 1 ELSE 0 END) AS miss_count,
SUM(CASE WHEN content LIKE '%cache hit for key%' OR content LIKE '%key accepted%' THEN 1 ELSE 0 END) * 100.0 / 
NULLIF(SUM(CASE WHEN content LIKE '%cache hit for key%' OR content LIKE '%key accepted%' OR content LIKE '%cache miss for key%' 
           OR content LIKE '%score not meet the threshold%' THEN 1 ELSE 0 END), 0) AS hit_rate

• Replace {your-gatewayId} with your gateway instance ID. Note that the first placeholder requires the gw- prefix, but the second does not. If you open the log query page from the cache settings, the `cluster_id` is automatically included. In this case, you only need to paste the rest of the search statement.

• The query result shows the cache hit ratio.

  

Examples

• When Precise Cache is enabled, only identical queries are matched:

• When Semantic Cache is enabled, semantically similar queries can also be matched. If the semantic similarity is lower than the threshold, a match is not found: