OpenSearch:MaxCompute data source

Prerequisites

• You have a basic understanding of MaxCompute (formerly ODPS). For more information, see MaxCompute overview.

• You have the required table permissions. The account used to access the MaxCompute table must have describe, select, and download permissions for the table, as well as label permissions for its fields.

Run the following statements to grant permissions:

-- Add a user.
add user ****@aliyun.com;

-- Grant the required table permissions to the user.
GRANT describe,select,download ON TABLE table_xxx TO USER ****@aliyun.com
GRANT describe,select,download ON TABLE table_xxx_done TO USER ****@aliyun.com

-- MaxCompute requires field-level permissions. Failure to grant these permissions for all fields, including high-privilege fields, will cause index builds to fail.
-- Grant permissions to all tables in the Project.
SET LABEL 3 to USER ****@aliyun.com

-- Grant permissions to a specific table.
GRANT LABEL 3 ON TABLE table_xxx(col1, col2) TO ****@aliyun.com

• OpenSearch Vector Search Edition supports the following field types for MaxCompute tables: STRING, BOOLEAN, DOUBLE, BIGINT, and DATETIME.

Configure a MaxCompute data source

1. Log on to the OpenSearch console. In the upper-left corner, switch to OpenSearch Vector Search Edition. On the Instance Management page, find your instance and click Manage in the Actions column.

2. 2. In the left-side navigation pane, choose Configuration Center > Data Source Configuration. Click Add Data Source, select MaxCompute, and configure the following parameters: Data Source Name, Project, AccessKey ID, AccessKey Secret, Table, Partition Key, and Automatic Index Rebuilding.

3. After you configure the parameters, click Validate. After the validation passes, the OK button becomes clickable.

4. After you add the data source, you must configure an index schema and add an index table. For more information, see Add an index table.

5. After the index table is configured, you must apply the configuration update and rebuild the index to make it available to the production cluster. For more information, see Update a configuration.

Note:

• Data Source Name: The name of the data source. The name must follow the format instance_name_custom_name.

• Project, AccessKey ID, AccessKey Secret, Table, and Partition Key: The parameters required to access the MaxCompute data source.

• Automatic index rebuilding: If enabled, the system automatically rebuilds the index for any index table that references the data source when the data source changes.

• If you enable automatic index rebuilding, you must create a done table. For more information, see Automatic index rebuilding.

Automatic index rebuilding

Purpose of the done table: When automatic index rebuilding is enabled, the OpenSearch Vector Search Edition instance monitors the done table and triggers an index rebuild whenever the done table is updated.

Example: Suppose you have a MaxCompute data table named mytable partitioned by ds=20220113. After the initial index build, a new partition with a full data load is generated daily. To have the OpenSearch Vector Search Edition instance detect the new partition and automatically rebuild the index, use automatic index rebuilding with a done table.

Procedure:

1. When you add the data source, enable Automatic Index Rebuilding.

2. Create a done table in MaxCompute. If the data table is named mytable and its partition key is ds, the done table must be named mytable_done and also use ds as its partition key. The two tables appear in MaxCompute as follows:

odps:sql:xxx> show tables;
InstanceId: xxx  
SQL: .                                                                                              

ALIYUN$****@aliyun.com:mytable          # Data table for full loads
ALIYUN$****@aliyun.com:mytable_done     # Done table to trigger automatic full loads

Statement to create the done table:

create table mytable_done (attribute string) partitioned by (ds string);

3. After the data for the new partition ds=20220114 is generated in the mytable table, update the done table to trigger an index rebuild in your OpenSearch Vector Search Edition instance.

-- Add the partition.
alter table mytable_done add if not exists partition (ds="20220114");

-- Insert the signal data to trigger the automatic full load.
insert into table mytable_done partition (ds="20220114") select '{"swift_start_timestamp":1642003200}';

The final content of the done table is as follows:

odps:sql:xxx> select * from mytable_done where ds=20220114 limit 1;
InstanceId: xxx  
SQL: .                                                                                              
+-----------+----+
| attribute | ds |
+-----------+----+
| {"swift_start_timestamp":1642003200} | 20220114 |
+-----------+----+

After the signal data is inserted into the done table, the OpenSearch Vector Search Edition instance detects the update and automatically triggers an index rebuild.

Important

• The done table must have at least one partition key, and its partition key must have the same name as the data table's partition key. For example, if the data table's partition key is ds, the done table's partition key must also be ds.

• The done table must contain a single STRING field named attribute.

• The partition that you add to the done table must already exist in the data table. For example, if the data table contains partitions ds="20220114", ds="20220115", and ds="20220116", a new partition added to the done table must be one of these existing partitions.

• When you insert data into the done table, the value of the attribute field must be a JSON-formatted string, such as {"swift_start_timestamp":1642003200}. The timestamp specifies the starting point for incremental data ingestion.

Edit a MaxCompute data source

1. On the Data Source Configuration page, find the data source that you want to edit and click Modify in the Actions column.

2. On the Edit Data Source page, modify parameters such as Project, AccessKey ID, AccessKey Secret, Table, and Partition Key.

After you modify the parameters, click Validate. After validation succeeds, click OK to save your changes.

3. After you modify the data source, you must apply the configuration update and rebuild the index for the changes to take effect in the production cluster. For more information, see Update a configuration.

Delete a MaxCompute data source

1. 1. On the Data Source Configuration page, find the data source that you want to delete and click Delete in the Actions column.

2. 2. After you click Delete, the system checks whether the data source is referenced by an index table.

• If the data source is not referenced by any index table, click OK to delete it.

• If the data source is referenced by an index table, an error is reported.

The system displays the message: "The data source is referenced by an index table. Delete the index table or remove the association, and then delete the data source."

Delete all index tables that reference the data source before you delete it. For more information, see Delete an index table.

Usage notes

• You cannot change the name of a data source after it is created.

• MaxCompute does not support external tables. You must use an internal table.

• The table that you configure as the MaxCompute data source must be a partitioned table.

• You can use tables generated in MaxCompute for full load operations and use an API data source to push real-time data.