Get Started with Tair Semantic Caching in 4 Steps: Stop Burning Money on Repetitive AI Queries

By Alibaba Cloud Tair team

Ask an AI customer service bot "How do I return an item?" and the LLM typically takes several seconds to respond. During peak load or traffic spikes, you might wait over ten seconds.

Here's the frustrating part: different users, different phrasing, but the same underlying question — "How does annual leave work?", "What's the reimbursement policy?", "Where's my package?", "Who is this NPC?"

Every repeated query triggers a full LLM call: tokens are consumed, latency climbs, and user experience degrades.

What semantic caching does is straightforward:

It identifies "this is a question we've seen before" at the gateway layer and returns the correct answer directly, skipping the LLM call entirely.

Which AI Scenarios Benefit Most from Semantic Caching?

Semantic caching is not a silver bullet. It works best when request semantics are highly repetitive and answers remain stable over time. Here are the three most representative scenarios:

Scenario 1: Intelligent customer service / enterprise knowledge base

Employees ask "How does annual leave work?" Customers ask "What's the return policy?" Ten people ask the same thing, just phrased differently. During peak promotions, request volume spikes tenfold, yet the questions stay highly concentrated. Key characteristics: diverse phrasing, concentrated semantics, fixed answers.

Scenario 2: Multilingual real-time translation

In global game chat, "GG well played" might be translated tens of thousands of times. Similar phrases like "good game" or "nice play" follow the same pattern. High-frequency phrases plus extreme latency sensitivity make this semantic caching's ideal battlefield.

Scenario 3: Game NPC dialogue.

“How do I complete this quest?”, “Where can I find item XX?”, “Who are you?” — millions of player-to-NPC conversations generate massive volumes of semantically repeated requests daily. You save tokens while keeping NPCs responsive.

Get Started: Running Tair Semantic Caching from Scratch in 4 Steps

The entire onboarding takes just 4 steps with all default settings. You can be up and running in as little as 10 minutes.

Step 1: Create a Tair Cluster Instance

Semantic caching requires a Tair (Redis OSS-compatible) instance as the underlying storage for cached data and vector indexes.

Go to the Alibaba Cloud Free Trial page, select Tair (Redis OSS-compatible) Choose the following specifications: 2 GB, DRAM-based, cluster architecture, and start the free trial.

⚠️ Key points:

• Region and zone: Only the Zone I/L/F of the China (Beijing) region are supported.

• Architecture: Cluster architecture — proxy mode

• Give your instance a memorable name (you'll need it in the next step)

The instance is ready in 1–3 minutes.

Step 2: Create a Tair Semantic Cache Gateway Instance

In the Tair console, go to the left navigation pane → Tair Semantic Cache Gateway Instances → Create Instance.

⚠️ Key points (must match your Tair instance from Step 1 exactly):

• Region: Same as your Tair instance

• VPC + vSwitch: Same as your Tair instance

All other settings are pre-configured. Click Next.

Step 3: Configure the Semantic Caching Plugin (Fully Managed Mode)

For first-time users, we strongly recommend the OpenAI-compatible mode (fully managed) — no need to build your own LLM call chain. Just change one line: base_url.

At “Bind Tair Instance,” select the instance you created in Step 1. Leave everything else as default and click Purchase Now (free during public preview).

Step 4: Get Access Information & Verify Caching

Go to the instance details page and find two key parameters in the Access Information section:

• API Key: Format sk-****

• Public endpoint: Format tk-******.redis.rds.aliyuncs.com (In the Network Access section, click Apply to enable public network access. Remember to remove 0.0.0.0/0 from the whitelist after testing.)

Now verify caching with 10 lines of Python:

import time
from openai import OpenAI

client = OpenAI(
api_key="<Your API Key>",
base_url="http://<Your Service Address>/compatible-mode/v1",
)

# First request: cache miss, calls the LLM (slower)
start = time.time()
resp1 = client.chat.completions.create(
    model="qwen3.6-plus",
    messages=[{"role": "user", "content": "What is semantic caching?"}]
)
print(f"First request took: {time.time() - start:.2f}s")

# Second request: semantically similar but different wording, expected cache hit
start = time.time()
resp2 = client.chat.completions.create(
    model="qwen3.6-plus",
    messages=[{"role": "user", "content": "What does semantic caching mean?"}]
)
print(f"Similar question took: {time.time() - start:.2f}s")

Test results: First request: 41.67s; Similar question: 0.17s.

Under the Hood: What Happens to a Request Inside the Tair Semantic Cache Gateway?

After onboarding, you might wonder: how does it actually determine that “two questions mean the same thing”? The core idea is straightforward:

If someone has already asked essentially the same question and the LLM provided an answer, subsequent similar requests can skip the LLM entirely and return the cached response.

The key is “semantically identical” — the wording does not need to match exactly, only the meaning. Traditional caching can only do exact matching: “How do I return an item?” and “What’s the return process?” would be treated as two different questions. Semantic caching uses vector similarity to determine whether they express the same intent. Here is the full request flow inside the gateway:

User request arrives at the Tair Semantic Cache Gateway
First, an exact match is attempted. If it hits, the response returns within 1–2 ms
No exact match → the embedding model converts the question into a vector
The vector is used for approximate nearest neighbor (ANN) search against Tair Vector
Similarity exceeds the threshold → the cached answer is returned directly (ANN search: 5–20 ms + embedding: ~60 ms, total in milliseconds)
No match at all → the request is forwarded to the LLM. The result is simultaneously written back to the cache for future reuse

When a cache hit occurs, the expensive LLM call is skipped entirely, incurring only minimal embedding computation and vector search overhead. Embedding token costs are roughly 1/10 to 1/20 of LLM token costs.

Two Integration Modes — Pick What Fits

Mode	Best For	Key Features
OpenAI-compatible mode (fully managed)	Those who want the simplest setup	Just change base_url. Built-in Alibaba Cloud Model Studio models. Cache misses are auto-forwarded.
LangCache-compatible mode	Teams with existing self-managed agent architectures	Handles only the caching layer via REST API. You manage your own LLM orchestration.

Fine-Tuning: Push Your Hit Rate Even Higher

Once you're up and running, two parameters are your primary tuning levers:

1. Similarity threshold. Lower threshold → more hits but potentially less precise. Higher threshold → more precise but lower hit rate. Start with the default and adjust based on your production data.
2. TTL. For scenarios where answers may become stale (e.g. policy or promotion content), always set an appropriate expiration time to avoid serving outdated answers.

Community

Get Started with Tair Semantic Caching in 4 Steps: Stop Burning Money on Repetitive AI Queries

Which AI Scenarios Benefit Most from Semantic Caching?

Scenario 1: Intelligent customer service / enterprise knowledge base

Scenario 2: Multilingual real-time translation

Scenario 3: Game NPC dialogue.

Get Started: Running Tair Semantic Caching from Scratch in 4 Steps

Step 1: Create a Tair Cluster Instance

Step 2: Create a Tair Semantic Cache Gateway Instance

Step 3: Configure the Semantic Caching Plugin (Fully Managed Mode)

Step 4: Get Access Information & Verify Caching

Under the Hood: What Happens to a Request Inside the Tair Semantic Cache Gateway?

Two Integration Modes — Pick What Fits

Fine-Tuning: Push Your Hit Rate Even Higher

Read previous post:

Read next post:

ApsaraDB

You may also like

Comments

ApsaraDB

Related Products

Vector Retrieval Service for Milvus

API Gateway

Database Gateway

AgentBay