OpenSearch:Configuration center

Data structure

The primary table has a fixed data structure with seven fields: score, id, title, category, url, content, and timestamp.

Before you begin

1. Log in to the OpenSearch console.

2. Select the target region and switch to OpenSearch LLM-based Conversational Search Edition.

3. On the instance management page, click Manage to the right of the target instance, and in the left navigation pane, select Configuration Center > Data Configuration, where you can choose a data import method based on your business needs.

Import files

Click Import File to open the file import page.

• Unstructured data: You can upload files in DOC, DOCX, PDF, HTML, TXT, PPT, PPTX, XLS, and XLSX formats, as well as video files in MP4, AVI, MKV, and MOV formats. The maximum size for a single file is 128 MB. You can upload multiple files at a time.

  Note

  If a Word document contains many images, convert it to PDF format before uploading for faster processing.

• Structured data: You can upload UTF-8 encoded files in JSON or Excel format. The maximum size for a single file is 128 MB. You can upload multiple files at a time. For the required data format, download and review the Data Sample.

  Note

  • A table name can contain only letters, digits, and underscores (_) and must be no more than 20 characters long.

  • A field name can contain only letters and underscores (_), must not start with an underscore, and must be no more than 30 characters long.

  • A maximum of 30 fields in each Excel file can be written to and queried. Additional fields are ignored.

To push documents in batches using an API, see Push documents.

Import web pages

Click Web Page URL Import, select the Web Page Import tab, and enter the web page URLs. Enter one URL per line. Then, click Import to complete the import.

Import from a website

1. Click Web Page URL Import, select the Website Import tab, and then click Create Task. Enter the Website URL and a name for the Category. You can also configure URL filtering, an XPath selector, and a CSS selector as needed.

   Note

   • Website URL: The URL of the website to crawl.

   • Category: The category name for the content from the specified URL.

   • URL Filtering: A regular expression to filter URLs. By default, the rule matches all pages under the specified website URL. For example, if the website URL is http://www.abc.com/, the default regular expression is http://www\\.abc\\.com/.*.

   • XPath selector: An XPath selector that specifies which content to extract. For example, to extract all content within <div> tags, set this parameter to //div.

   • CSS selector: A CSS selector that specifies which content to extract. For example, to extract the content from <div class="content">Web page content</div>, set this parameter to div.content.

   • URLs that end with .png, .jpg, or .jpeg are not supported.

2. After the task completes, you can view the number of crawled web pages. Then, click OK to finish the import.

Import tables

For more information, see Table-based Q&A.

Query and manage data

After documents are uploaded, you can view the total document count and test Q&A effectiveness on the Q&A Test page. You can also look up, edit, or delete documents by primary key.

1. View data

   Enter the primary key in the id field and click Search to view the document details.

2. Delete data

   To delete a document, enter its primary key in the id field, and then click Delete and OK to confirm.

   If you exceed your storage capacity, deletion may fail. Expand your capacity before retrying.

3. Edit data

   Enter the primary key in the id field and click Edit to modify the editable fields.

Usage notes

• The primary key id must be unique. If you import data with a duplicate primary key, the new data overwrites the existing data.

• The maximum size for a single upload of structured data is 128 MB.

• The maximum size for a single unstructured data file is 128 MB.

• After data is uploaded, the time for it to become searchable depends on the current data update volume.