Data processing plug-ins transform field values as data is synchronized from a source table to an OpenSearch table. Configure them during field mapping setup—not when defining the application schema.
Plug-ins are only available when you use a data source to synchronize data in the console. If you upload data via an API operation or SDK, process the data before uploading. Plug-ins cannot be applied after the fact.
Prerequisites
Before you begin, ensure that you have:
Configured a data source for your OpenSearch application
Started configuring field mappings between the source table and the OpenSearch table
Available plug-ins
OpenSearch provides five data processing plug-ins for common field transformation needs.
| Plug-in | What it does |
|---|---|
JsonKeyValueExtractor | Extracts a value by key from a JSON-formatted source field |
MultiValueSpliter | Splits a source field value into multiple array elements using a delimiter |
KeyValueExtractor | Extracts a value by key from a key-value pair source field |
StringCatenateExtractor | Concatenates values from multiple fields into a single string |
HTMLTagRemover | Strips HTML tags from a source field value |
Plug-in reference
JsonKeyValueExtractor
Extracts the value of a specified key from a source field that contains a JSON object. The extracted key value is used as the name of the destination table field.
Type conversion:
If the extracted value is a JSON array (as defined by OpenSearch), it is automatically converted to the corresponding array field type:
| Field type | Example source value |
|---|---|
LITERAL_ARRAY | {"tags":["a","b","c"]} |
INT_ARRAY | {"tags":[1,2,3]} |
Type mismatch behavior: If the extracted value's type does not match the destination field type, the value is silently dropped. Verify field types before configuring this plug-in.
Example:
| Value | |
|---|---|
| Source field | {"title":"the content","body":"the content"} |
| Configured key | title |
| Destination field receives | the content |
MultiValueSpliter
Splits the value of a source field into multiple values using a delimiter. Each resulting value becomes an element of the destination field array.
The destination field must be of an array type.
Non-printable delimiters: Use Unicode escape sequences, such as \u001D.
Example:
| Value | |
|---|---|
| Source field | 1,2,3 |
| Delimiter | , (comma) |
| Destination field receives | [1, 2, 3] |
For configuration steps, see Configure the MultiValueSpliter plug-in.
KeyValueExtractor
Extracts the value of a specified key from a source field that contains key-value pairs. The extracted value becomes the value of the destination table field. Delimiters are not required.
Behavior:
If you specify a delimiter to separate extracted values, they are automatically converted to an array field type.
The extracted value's type must match the destination field type. If they don't match, the value is silently dropped.
If the source field contains duplicate keys, only the value of the second key is extracted.
Example:
Source field value: key1:value1,value2;key2:value3
| Separator | Character | Separates |
|---|---|---|
; | Semicolon | Key-value pairs |
: | Colon | Key from value within a pair |
, | Comma | Multiple values for a single key |
To extract key1 with comma as the value delimiter, the destination field receives ["value1", "value2"].
StringCatenateExtractor
Concatenates the values of specified destination table fields into a single string, in the order you define.
Constraints:
This plug-in cannot concatenate fields of the
INTtype. We recommend that you use fields of theLITERALtype.Fields must be from the destination table fields.
Separate field names with commas (
,).
System variable: Use $table to include the current table name in the concatenated string. $table is only populated when a table-sharding wildcard is configured.
Example:
Concatenate field1 and field2 with an underscore (_) as the separator. The result is {field1_value}_{field2_value}.
HTMLTagRemover
Removes HTML tags from the value of a source field. The cleaned text replaces the original field value.
Example:
| Value | |
|---|---|
| Source field | <div id="copyright">OpenSearch</div> |
| Destination field receives | OpenSearch |
Constraints and limits
| Constraint | Details |
|---|---|
| Plug-in configuration timing | Configure plug-ins when setting up a data source for an application. You cannot configure them at the application schema definition stage. |
| Data upload method | Plug-ins only work with console-based data source synchronization. API and SDK uploads cannot use plug-ins. |
| MaxCompute table association | An OpenSearch table can be associated with only one MaxCompute source table. To synchronize data from multiple MaxCompute source tables, join them into a single table first, then upload. |
| ApsaraDB RDS and PolarDB table association | An OpenSearch table can be associated with multiple tables in an ApsaraDB RDS or PolarDB data source (for database and table sharding). |
| Array type requirement | MultiValueSpliter requires the destination field to be an array type. JsonKeyValueExtractor and KeyValueExtractor automatically convert to array types when the extracted value is a JSON array or when a value delimiter is specified, respectively. |
| Type mismatch | If the type of an extracted value does not match the destination field type, the value is silently dropped. Verify field types before configuring plug-ins. |