Data Transmission Service:Connect OSS to a RAGFlow knowledge base

Prerequisites

You have created a RAGFlow knowledge base in DTS and configured an IP whitelist.

Supported file types

• DOC, DOCX, PPT, PPTX, YML, XML, HTML, JSON, CSV, TXT, XLS, XLSX, WPS, RTF, MD, SQL

• JPG, JPEG, PNG

• INI

• MP3

Before you begin

1. Create an AccessKey and record the AccessKey ID and AccessKey secret.

   Note

   If you use an AccessKey created by a RAM user, ensure that the RAM user has the AliyunOSSReadOnlyAccess (read-only access to OSS) or AliyunOSSFullAccess (full management access to OSS) permission.

2. Obtain and record your OSS bucket information, including the bucket name and region ID.

   1. Log on to the OSS console.

   2. In the left-side navigation pane, click Buckets.

   3. Find the target bucket.

   4. Record the Bucket Name of the target bucket.

   5. Based on the Region of the target bucket, find and record the corresponding region ID.

Procedure

1. Obtain the KBSync file.

   Note

   You can join the DingTalk group (ID: 79690034672) and contact technical support to obtain the KBSync file.

2. Prepare the runtime environment for the KBSync program.

   Note

   The KBSync program must run in a Linux environment with network access to both OSS and RAGFlow.

3. Prepare the config file.

   1. Create a Linux system file named config.

   2. Copy the following code to the config file.

      whiteList=
      blackList=
      sinkType=RagFlow
      sourceType=OSS
      
      ragflowUrl=http://XX.XX.XX.XX
      ragflowApiKey=Bearer ragflow-Rh******
      ragflowDatasetId=******
      
      sourceOSSAccessKeyId=******
      sourceOSSAccessKeySecret=******
      sourceOSSRegion=cn-beijing
      sourceOSSBucket=kbsync

   3. Update the parameters in the config file.

      Important

      • If a parameter is not required, leave its value empty.

      • The blackList parameter has a higher priority than the whiteList parameter.

      Parameter	Required	Description	Source
      whiteList	No	The paths of files or folders to include (whitelist) or exclude (blacklist), including folder paths in DingTalk Drive and document paths within the knowledge base. Note Regular expressions are supported. Separate multiple paths with spaces.	Obtain the path from OSS.
      blackList
      sinkType	Yes	The sink type.	Set this parameter to RagFlow.
      sourceType		The source type.	Set this parameter to OSS.
      ragflowUrl		The RAGFlow endpoint (API server) address.	Obtain the RAGFlow API server address
      ragflowApiKey		The API key of the RAGFlow knowledge base. Important The value must start with Bearer . For example: Bearer ragflow-RhMjc0NjFhNTZmNTExZjBiYWY****.	Obtain the RAGFlow API key
      ragflowDatasetId		The ID of the RAGFlow knowledge base.	Obtain the RAGFlow knowledge base ID
      sourceOSSAccessKeyId		The AccessKey ID you recorded.	Before you begin
      sourceOSSAccessKeySecret		The AccessKey secret you recorded.
      sourceOSSRegion		The OSS region ID you recorded.
      sourceOSSBucket		The OSS bucket name you recorded.

4. Place the KBSync file and the config configuration file in the same directory in a Linux environment.

5. In your Linux environment, run the ./KBSync --config config command to start the KBSync program.

   The KBSync program runs correctly if the output is similar to the following.

   INFO config SourceType=OSS, SinkType=RagFlow
   INFO config whiteList=, blackList=
   INFO config ragflowUrl=http://XX.XX.XX.XX ragflowApiKey=Bearer ragflow-Rh******
   INFO config ragflowDatasetId=b2******
   INFO config sourceOssKeyId=******, sourceOssRegion=cn-beijing
   INFO Verifying RAGFlow connection...
   INFO Attempting to list datasets to validate the connection...
   INFO Successfully found matching dataset: Name='test', ID='b2******'
   INFO RAGFlow connection verified successfully.

Appendix