OpenSearch:Data Lake Formation (DLF)

Prerequisites

• You are familiar with Data Lake Formation.

• You have configured the Data Lake Formation catalog ID, database, and table, which you will use to configure data synchronization.

Add a Data Lake Formation data source

1. On the Instance Details > Table Management page, click Add Table:

2. Enter the basic table information, then click Next. In the Table Basic Information step, enter the Table Name (for example, dlf_table), set the Number of Data Shards (default: 1) and Number of Data Update Resources (default: 2; used to adjust concurrency for consuming real-time data and improve TPS), and click Next.Configuration notes:

• Table name: You can specify a custom name.

• Number of data shards: Specify a positive integer up to 256. The value should not exceed three times the number of data nodes in your instance.

• Number of data update resources: The number of resources used for data updates. By default, each table provides two free resources, each with 4 vCPUs and 8 GB of memory. Resources exceeding the free quota incur charges. For more information, see Billing overview of OpenSearch Retrieval Engine Edition.

3. Configure the data source in the data synchronization step. After validation succeeds, click Next:

   For Full data source, select Data Lake Formation (DLF), and then enter the catalog ID, database, and table.

• Full data source: Select Data Lake Formation (DLF).

• Catalog ID: The ID of the Data Lake Formation catalog that contains your data.

• Database: The database within the catalog that contains your table.

• Table: The table within the database that you want to synchronize.

  Note

  • To use DLF as a data source for an existing instance, you must first upgrade its offline version.

  • Currently, only Paimon-type catalogs are supported.

  • Paimon tables with a primary key support add, update, delete, and query operations. Paimon append-only tables support only write operations and do not allow data to be modified or deleted.

4. Configure the index schema in Form Mode or Developer Mode, then click Next:

   In the Field Settings section, define your data fields, such as an id field (Type: INT8, as primary key), a name (STRING), email field (Type: STRING). You can also set data compression and analyzers here. In the Index Settings section, define your search indexes. For example, add an index named phone (Type: STRING) based on the name).

5. After you confirm creation, the system creates the configured table automatically. Check its creation progress in the change history. When the table status is In Use, you can test queries on the query test page.

Precautions

When new data is written to a Paimon table in DLF, OpenSearch automatically triggers real-time indexing. However, manually writing data by using an API can introduce data inconsistencies, so exercise caution.