Container Service for Kubernetes:Quickly build an AI Q&A assistant with Dify

Clone the code repository. Run the Git command to clone the corresponding Dify code repository.

git clone https://github.com/langgenius/dify-docs.git

Organize the corpus. To facilitate the subsequent vectorization of the corpus, use the following Python script to merge and organize the .md files in the obtained Dify code repository.

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
import os


def merge_markdown_files_from_directory(root_dir, output_file):
    """
    The merge_markdown_files_from_directory function
      1. Function: Merges the content of all .md files in a specified directory into an output file.
      2. Parameters:
        root_dir: The path to the specified root directory.
        output_file: The path to the output file.
      3. Implementation:
        Uses os.walk to traverse the root_dir directory and its subdirectories.
        Checks whether each file ends with .md. If so, its path is added to the markdown_files list.
        Opens output_file and writes the content of each file from markdown_files into it.
    """
    markdown_files = []
    for root, dirs, files in os.walk(root_dir):
        for file in files:
            if file.endswith('.md'):
                markdown_files.append(os.path.join(root, file))

    with open(output_file, 'w', encoding='utf-8') as outfile:
        for file_path in markdown_files:
            with open(file_path, 'r', encoding='utf-8') as infile:
                outfile.write(infile.read())
                # outfile.write('\n\n')

               
def process_and_write_markdown(file_path: str, headers_to_split_on: list, chunk_size: int, chunk_overlap: int, output_file: str):
    """
    The process_and_write_markdown function
      1. Function:
        Processes the content of a single Markdown file, splits the text based on specified headers and chunking rules, and writes the result to a new file.
      2. Parameters:
        file_path: The path of the input Markdown file.
        headers_to_split_on: A list that contains header-level tuples, such as [("#", "Header 1"), ("##", "Header 2")].
        chunk_size: The maximum length of each text chunk.
        chunk_overlap: The overlap length between text chunks.
        output_file: The path of the output file.
      3. Implementation steps:
        Reads the content of the Markdown file from file_path.
        Creates a MarkdownHeaderTextSplitter object to split the Markdown content based on headers_to_split_on.
        Creates a RecursiveCharacterTextSplitter object to further split the text based on chunk_size and chunk_overlap.
        Opens output_file and writes the metadata and content of each split text chunk to the file.
    """
    try:
        # Reads the content of the Markdown file from file_path.
        with open(file_path, "r", encoding="utf-8") as doc:
            markdown_content = doc.read()

        # Creates a MarkdownHeaderTextSplitter object to split the Markdown content based on headers_to_split_on.
        splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on, strip_headers=True)
        md_splits = splitter.split_text(markdown_content)

        # Creates a RecursiveCharacterTextSplitter object to further split the text based on chunk_size and chunk_overlap.
        text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
        splits = text_splitter.split_documents(md_splits)

        # Opens output_file and writes the metadata and content of each split text chunk to the file.
        with open(output_file, "w") as f:
            for line in splits:
                f.write(str(line.metadata))
                f.write("\n")
                f.write(line.page_content)
                f.write("\n\n\n\n")

    except FileNotFoundError:
        raise FileNotFoundError(f"The file {file_path} does not exist.")


# Example
if __name__ == "__main__":
    """
     1. Set parameters:
      root_directory: The path to the specified root directory.
      merged_file_path: The path for the merged Markdown file.
      output_file: The path for the final processed output file.
      headers_to_split_on: A list of header levels.
      chunk_size and chunk_overlap: The size and overlap length for text chunks.
    2. Execution steps:
      Calls merge_markdown_files_from_directory to merge all .md files into merged_file_path.
      Calls process_and_write_markdown to process the merged file and writes the result to output_file.
      These steps complete the script's process, from merging multiple Markdown files to final processing and output.
    """
    
    # The path to the folder to be processed
    root_directory = 'path/to/dify-docs/en'
    # The path for the merged file
    merged_file_path = './merged_markdown.md'
    # The path for the cleaned file
    output_file = './dify_doc.md'
    
    merge_markdown_files_from_directory(root_directory, merged_file_path)
    headers_to_split_on = [
        ("#", "Header 1"),
        ("##", "Header 2"),
        ("###", "Header 3"),
    ]
    chunk_size = 500
    chunk_overlap = 50
    process_and_write_markdown(merged_file_path, headers_to_split_on, chunk_size, chunk_overlap, output_file)

Upload the prepared corpus file dify_doc.md to the knowledge base.