All Products
Search
Document Center

Vector Retrieval Service for Milvus:Build a RAG system with Alibaba Cloud Milvus and Dify

Last Updated:Oct 19, 2026

This topic describes how to build a retrieval-augmented generation (RAG) system by using Vector Retrieval Service for Milvus (Milvus) and the Dify platform.

Background information

How RAG works

Large language models (LLMs) often produce "hallucinations" because of the limits of their knowledge. Retrieval-augmented generation (RAG) effectively addresses this pain point by connecting LLMs to external knowledge bases. A powerful vector database is essential for implementing efficient RAG.

This topic focuses on Milvus and uses Dify, a low-code AI platform, to show you how to seamlessly combine the two to quickly build an enterprise-grade RAG application. This gives you a hands-on view of the core value that vector databases provide in solving the "last mile" problem of AI.

About Dify

Dify is an open source AI application development platform that features low-code workflows and a user-friendly interface. Its core mission is to thoroughly simplify and accelerate the entire process of building AI applications by deeply integrating the concepts of Backend-as-a-Service and LLM operations (LLMOps).

As a full-stack solution, Dify provides infrastructure such as stable and reliable API services and data management at the backend layer, so developers do not need to build everything from scratch. At the LLMOps layer, Dify provides an intuitive visual interface for prompt orchestration, which makes complex prompt engineering simple and efficient. Its built-in high-quality retrieval-augmented generation (RAG) engine can easily connect to private knowledge bases such as enterprise documents and databases. This allows LLMs to answer questions based on domain-specific knowledge, which effectively reduces hallucinations and ensures that answers are accurate and traceable.

Prerequisites

Procedure

Step 1: Install Dify

  1. Run a Git command to clone the open source Dify project from GitHub to your local machine.

    git clone https://github.com/langgenius/dify.git
  2. Go to the deployment directory and back up the .env configuration file.

    cd dify/docker/
    cp .env.example .env
  3. Modify the following settings in the .env configuration file.

    # Vector store engine configuration
    VECTOR_STORE=milvus  # Specify Milvus as the vector store engine
    # Milvus connection information
    MILVUS_URI=http://YOUR_ALIYUN_MILVUS_ENDPOINT:19530
    MILVUS_USER=YOUR_ALIYUN_MILVUS_USER
    MILVUS_PASSWORD=YOUR_ALIYUN_MILVUS_PASSWORD

    The following parameters are used in this example. Replace them based on your actual environment.

    Parameter

    Description

    MILVUS_URI

    The endpoint of the Milvus instance, in the format of http://<Public endpoint>:<Port>.

    • <Public endpoint>: You can view it on the Details page of the Milvus instance.

    • <Port>: You can view it on the Details page of the Milvus instance. Default value: 19530.

    MILVUS_USER

    Set this parameter to the custom username that you specified when you created the Milvus instance.

    MILVUS_PASSWORD

    Set this parameter to the password of the custom user that you specified when you created the Milvus instance.

  4. Start Dify.

    docker compose up -d --build
    [root@xxx /docker]# docker compose up -d --build
    [+] Running 15/15
    ✔ Network docker_default              Created
    ✔ Network docker_milvus               Created
    ✔ Network docker_ssrf_proxy_network    Created
    ✔ Container docker-db-1               Healthy
    ✔ Container docker-redis-1            Started
    ✔ Container docker-sandbox-1          Started
    ✔ Container milvus-etcd               Started
    ✔ Container milvus-minio              Started
    ✔ Container docker-ssrf_proxy-1       Started
    ✔ Container docker-web-1              Started
    ✔ Container docker-plugin_daemon-1    Started
    ✔ Container docker-worker-1           Started
    ✔ Container docker-api-1              Started
    ✔ Container milvus-standalone         Started
    ✔ Container docker-nginx-1            Started
  5. In your browser, visit http://127.0.0.1/ to access the IP address of the deployed Dify service, set the administrator account and password, and then log on to the console.

    Note

    If Dify runs on a remote server, such as a cloud server or virtual machine, replace 127.0.0.1 with the public IP address or domain name of the server, and make sure that the server is accessible over the network.

    Enter the Email, Username, and Password (the password must contain both letters and digits and be at least 8 characters in length), and then click Set Up to initialize the administrator account.

Step 2: Configure models

  1. After you log on, click your profile picture in the upper-right corner and select Settings from the drop-down list.

  2. In the left-side pane, select Model Provider. On the right side, find Qwen and install it.

  3. After the model is installed, choose to set the API key, and enter the Model Studio API key that you obtained in advance.

  4. Configure the System Model Settings based on your requirements.

    In the System Model Settings panel, configure the System Reasoning Model, Embedding Model, Rerank Model, Speech-to-Text Model, and Text-to-Speech Model, and then click Save.

Step 3: Create a knowledge base

  1. Click Knowledge at the top, and then click Create Knowledge.

  2. Set the data source to Import from file. You can first click README.md to download the sample data, and then upload the sample data.

  3. Modify the relevant parameters and click Save & Process.

    Other key parameters retain their default values: the index mode is set to High Quality, the retrieval setting is set to Vector Search, the Rerank model gte-rerank is enabled, Top K is set to 3, and the score threshold is set to 0.5.

    In this example, the following parameters are modified:

    • Maximum chunk length: Set to 1024.

    • Embedding Model: Select text-embedding-v1.

    After the processing is complete, you can see that the knowledge base is created.

    The page displays Knowledge created, and the file What is Vector Retrieval Service for Milvus.md is embedded (indicated by a green check mark). Configuration summary: the chunk mode is Custom, the text preprocessing rule replaces consecutive spaces, line breaks, and tabs, the index mode is High Quality, and the retrieval setting is Vector Search. Click Go to document to view the details of the knowledge base.

Step 4: Verify that vector retrieval succeeds

Log on to the Alibaba Cloud Milvus console, select the Milvus instance that you configured, and click Attu Manager in the upper-right corner to go to the Attu page. You can see that the data of the corresponding collection has been imported. For more information, see Attu Management.

Step 5: Verify the RAG effect

  1. Click Studio at the top to return to the homepage, and select Create from Template.

  2. In the search box, search for and use the Knowledge Retreival + Chatbot template.

  3. In the dialog box that appears, click Create.

  4. Select the Knowledge Retrieval node and set the knowledge base to the knowledge base that you created in the preceding step.

    On the workflow canvas, the nodes are connected in the following order: START → KNOWLEDGE RETRIEVAL → LLM (using the qwen-max model) → ANSWER. The query variable is sys.query.

  5. Select the LLM node and set the model to qwen-max.

  6. Click Publish in the upper-right corner, and then click Publish Update.

  7. Click Run to go to the test page. Enter a question related to the content of the knowledge base to obtain the corresponding answer.