Vector Retrieval Service for Milvus:Build a development troubleshooting assistant with the Alibaba Cloud Milvus knowledge base

Solution overview

This tutorial builds on Build an intelligent customer service Q&A application with the Alibaba Cloud Milvus knowledge base. The end-to-end pipeline is identical: define tags in the console → batch-import source files by tag → publish a version → retrieve through the SDK (with optional tag filtering) → have the LLM generate answers with source citations → serve the Q&A page with Flask. You reuse the engineering code (kb_client.py, upload.py, app.py, templates/index.html, and start.sh) from that tutorial, with two modifications described in Step 4: Adapt the reused code. This document covers only what you must change for the development troubleshooting scenario: module tags, retrieval parameters for exact terms, handling of HTML source files, and constraints on high-risk commands and empty results.

For the first version, select only 5 to 20 source files from a single system and expand the scope after validation. The entire process takes about 20 to 30 minutes, and the document parsing time depends on the number and size of the source files.

Prerequisites

• A Milvus knowledge base is created, and you have recorded the knowledge base ID (for example, kd-803ae9b10cc31).

• A RAM user created with your Alibaba Cloud account, with Use Permanent AccessKey Access selected and the system policy AliyunMilvusFullAccess attached.

• An LLM endpoint that supports the OpenAI chat/completions protocol and its API key are ready.

• Python 3.8 or later is installed locally.

• The project directory and the five reused files (kb_client.py, upload.py, app.py, templates/index.html, and start.sh) are set up by following the tutorial referenced in Solution overview.

• The source files contain no secrets, tokens, or internal network accounts.

Step 1: Define module tags

Under Basic Information on the knowledge base details page, choose Tags > Manage and add the following three tags. Set the field type of each tag to string. The type dropdown is on the right side of the tag name input box, and the options are string, int64, list, float32, and bool.

The tag name version has the same name as the version parameter that represents the published version of the knowledge base in a retrieval request, but the two do not affect each other: the tag is used in tagFilter.field, and the retrieval version sits at the top level of the request. If you are concerned about confusion, name the tag apiVersion instead.

The module tag is the most critical design decision in this scenario. Different systems often have API paths with exactly the same name, and without module filtering they interfere with each other. In our test, user-service and order-service both own the same path GET /api/v2/orders/{orderId}. When we searched for this path:

Step 2: Prepare source files and the import manifest

In this step, you collect the source files, annotate each of them in the import manifest, and configure how the knowledge base chunks them.

• Place the source files in the documents/ directory. Markdown, HTML, PDF, DOCX, and TXT are supported. Keep complete error codes, API paths, and version numbers in the source files — in our tests, both error codes and full paths could be retrieved exactly.

• Create documents.jsonl and annotate each source file with its module, document type, and version:

  {"path": "documents/order-api.md", "metadata": {"module": "order-service", "docType": "API", "version": "v2"}}
  {"path": "documents/order-timeout-runbook.md", "metadata": {"module": "order-service", "docType": "Runbook", "version": "v2"}}
  {"path": "documents/order-error-codes.html", "metadata": {"module": "order-service", "docType": "ErrorCode", "version": "v2"}}
  {"path": "documents/order-postmortem-2026-06.md", "metadata": {"module": "order-service", "docType": "Postmortem", "version": "v2"}}

• Import the API reference, Runbook, error code table, and postmortem as a set, and distinguish them with docType. In our test, when we asked about an error code, all four types of source files were hit at the same time, and the model produced a complete answer of "meaning → troubleshooting steps → historical cases".

Step 3: Configure retrieval parameters and the prompt

Development troubleshooting queries are mostly error codes, API paths, and commands. They belong to exact term matching, so the parameters differ greatly from other scenarios.

{
  "retrieval": {
    "page_size": 6,
    "candidate_count": 64,
    "min_score": 0.05,
    "semantic_weight": 0.15,
    "enable_query_expansion": false,
    "rerank_model_name": "",
    "tag_filter": {
      "relation": "and",
      "conditions": [
        {"field": "module", "op": "=", "value": "order-service"}
      ]
    }
  },
  "scenario": {
    "title": "Development docs and troubleshooting assistant",
    "system_prompt": "You are a development documentation assistant. Answer only based on the retrieved materials. Preserve API names, error codes, commands, and code as-is. List troubleshooting steps in order and cite them as [Source N]. When write operations or high-risk commands are involved, explicitly state the risk level and require manual confirmation. When the retrieved materials are empty, reply only that the materials are insufficient and warn against performing any change operation.",
    "image_enabled": false,
    "sample_questions": [
      "What are the required parameters for the order query API?",
      "In what order should I troubleshoot a connection timeout?",
      "In which documents does this error code appear?"
    ]
  }
}

The following list describes the parameters tuned for this scenario. The remaining fields in the example carry over from the referenced tutorial.

• semantic_weight: 0.15: The weighting factor of the final score. The score is calculated as score ≈ (1-semantic_weight) × keywordScore + semantic_weight × semanticScore (a rank feature may also be added), and min_score performs post-filtering on this final score. This scenario uses 0.15 so that the keyword score of exact terms such as error codes and API paths dominates the ranking. Because semantic_weight interacts with min_score, tune the two together as described in Tune semantic_weight and min_score together.

• enable_query_expansion: false: Query expansion is disabled so that error codes and API names are not rewritten. In our test, after enabling it, the keywordScore of the query ORD-50021 dropped from 0.283 to 0.226, and the number of recalled results for the query ORD-42901 dropped from 2 to 1. Keep it disabled for exact term retrieval.

• tag_filter pins module: This prevents identically named APIs of different systems from interfering with each other. See Step 1: Define module tags for the effect. If one entry point needs to cover multiple modules, use {"field": "module", "op": "in", "value": ["order-service", "user-service"]}.

• A blank rerank_model_name means reranking is not enabled. Exact match for error codes relies on the keyword score, so reranking brings limited benefit.

• op supports only =, in, and not in: In our tests, ≠, >, ≥, <, ≤, empty, not empty, start with, and end with are silently ignored and all data is returned (no error is reported). contains and not contains are merely aliases of in/not in, not string containment; passing string fragments returns 0 results. Aliases such as eq, ==, and like return 400 Unsupported tag filter operator, and every question fails. After you configure the filter conditions, compare the total number of results with the unfiltered results to confirm that the filter actually takes effect.

score ≈ semantic_weight × semanticScore + (1 - semantic_weight) × keywordScore

Step 4: Adapt the reused code

The reused project requires two modifications before you verify the assistant: an empty-result fallback in app.py, and a return-code fix in kb_client.py.

Intercept empty retrieval results in app.py

In the reused app.py, llm_answer() still calls the LLM when the retrieval results are empty. At that point the context is an empty string, and the model answers entirely from its own knowledge. In our test, when we asked "How do I recover from a Redis cluster split-brain?", which the knowledge base does not cover, the model output a complete operations plan with parameters and gave no warning at all. In troubleshooting scenarios, users may copy the commands and run them directly against the production environment, so this must be intercepted at the code level:

def llm_answer(question: str, results: list[dict[str, Any]]) -> str:
    if not LLM.get("enabled", True):
        return "The LLM is not enabled. Review the retrieval results below."
    if not results:
        return "No relevant document was retrieved from the knowledge base, so no troubleshooting steps can be provided. Do not perform any change operation based on this. We recommend that you contact the module owner."
    ...

Constraining this in the prompt alone is not reliable — even if you state "notify explicitly when the materials do not cover the question", the model still answers.

The last acceptance criterion in Step 5: Upload, publish, and verify covers this modification.

Fix the return-code check in kb_client.py

In the reused kb_client.py, _check() evaluates getattr(body, "code", 0) != 0. The code of a successful response is None, so successful retrievals are reported as "Failed: None". Change the check to getattr(body, "code", None) not in (None, 0, "0"). After the fix, successful retrievals no longer report "Failed: None".

Step 5: Upload, publish, and verify

1. Upload the source files. MetaFields applies to the whole batch; the script first groups by tag and then submits in batches.

   python upload.py --manifest documents.jsonl

2. On the Data Management page of the console, confirm that the source file status is Completed. A successful response from the upload API only means that asynchronous parsing has been submitted.

3. On the Version Management page, click Publish Version. After you complete the wizard, confirm that the status of the new version is Published.

   Important

   At most 3 published versions can exist at the same time. After the limit is reached, the Publish Version button is grayed out, but the page still shows "There are currently N pending changes to publish". Delete old versions that are no longer needed from the version records first. Development documents change frequently (every release may update the API reference and Runbooks), so keep only "the current version + the most recent historical version".

4. Start the service:

   python app.py

5. Send a test question:

   curl -sS http://127.0.0.1:7860/api/ask \
     -H 'Content-Type: application/json' \
     -d '{"question":"In what order should I troubleshoot a connection timeout?"}'

6. Verify against the following five acceptance criteria:

   • The page opens normally, and after you submit a question, the answer and the retrieval sources are returned together.

   • Each [Source N] in the answer can be matched to the corresponding source file chunk in the sources section below.

   • When you ask with a complete error code, all source files in which the error code appears are listed. In our test, querying ORD-50021 correctly listed four source files — the error code table, the API reference, the Runbook, and the postmortem — together with the location in each.

   • When you ask about a risky operation, the answer includes the risk level and the manual confirmation requirement.

   • When you ask about something the knowledge base does not cover, the response is "materials are insufficient" with a warning not to perform any change operation, and no fabricated command appears. Test this case in practice.

Notes for development scenarios

• Entry point per module — After tag_filter pins module, the entry point answers only questions about that module. Keep the sample questions within the same module as well.

• Complete exact terms — Keep complete error codes, API paths, commands, and version numbers in the source files. These exact terms are the main retrieval entry points in this scenario.

• Risk marking — Mark risk levels and confirmation requirements in the source files. Do not rely on the model to judge which commands are dangerous.

• No secrets in uploads — Do not upload secrets, tokens, internal network accounts, or production database connection strings. Chunk content is sent to the LLM as context.

• Deprecated API descriptions — Keep the descriptions of deprecated API operations. In our test, after the API reference stated "the v1 API is deprecated and its parameter names are incompatible", the model did not mix in old parameters when answering questions about the parameters of the new API.

• The tags field in retrieval results does not echo the module and version that were written. It is a different field from the MetaFields of AddDocuments, and no parameter can enable its return. An empty value does not mean the upload failed. When you need to annotate module and version in the answer, join the retrieval results with documents.jsonl by documentId and append the information to the context.

• Production deployment — Do not keep using the Flask development server for online deployment. Switch to a production WSGI server and add secret management, authentication, auditing, and rate limiting.

FAQ

Next steps

• Expand coverage module by module: create an independent entry point for each module value, and keep the sample questions within the same module.

• Prepare for production: switch from the Flask development server to a production WSGI server, and add secret management, authentication, auditing, and rate limiting.

• Manage versions as your documents change: publish new versions when the API references or Runbooks are updated, and keep only the current version plus the most recent historical version.