Data in OpenSearch can be imported through APIs, SDKs, or the upload console, or synchronized directly from an existing cloud database. If you choose to upload data through an API or SDK, follow the API reference to upload the data directly; the plug-in configuration described in this topic is not supported. If you choose to synchronize cloud data, configure the relevant data source information in the console. The system provides several data processing plug-ins for simple data transformation operations. You can enable them when you configure the field mappings for a data source. (Plug-ins are not supported for data uploaded through APIs; you must process the data before pushing it.)
A single OpenSearch table can have multiple RDS and PolarDB source tables (for example, in sharded database and table scenarios), but only one MaxCompute (ODPS) source can be configured. If you need multiple MaxCompute source tables, merge the data into a single table before importing.
Data processing plug-ins
Some search features or feature functions in the system require special field types. For example, an Array field type must be converted using the following plug-ins and cannot be entered directly.
Note: This plug-in is configured in the data source settings, not when you define the application schema. You can set a plug-in only after a data source is configured.
Configuration item | Description | Example |
JsonKeyValueExtractor | Extracts the value of a specified key from a JSON-formatted source field. The extracted value is used as the content of the destination table field. Only the value of a single key can be extracted. | Extracts the value of the key title from {"title":"the content","body":"the content"}. If the value is in JsonArray format, it is converted to an Array field type in the system. "Ensure that the type of the extracted value matches the destination table field type; otherwise, the corresponding data is lost." The JsonArray format here refers to the JsonArray format defined by our system. For example, the literal_array field type: {"tags":["a","b","c"]}, or the int_array field type: {"tags":[1,2,3]}. |
MultiValueSpliter | Splits a source field value into multiple values using a delimiter. The resulting values become the content of the destination table field, which must be configured as an ARRAY type. (If the delimiter is a non-printable character, use a Unicode character to represent it, such as \u001D.) | The data source content is 1,2,3. Specify the delimiter as "," by entering an English comma directly. |
KeyValueExtractor | Extracts the value of a specified key from a key-value (KV) formatted source field. The extracted value is used as the content of the destination table field. Only the value of a single key can be extracted. The delimiter is optional. | The actual content is key1:value1,value2;key2:value3, with keys key1 and key2, a key delimiter of semicolon, a key-value delimiter of colon, and a multi-value delimiter of comma. If a multi-value delimiter is configured, the value is converted to an Array field type in the system. "Ensure that the type of the extracted value matches the destination table field type; otherwise, the corresponding data is lost." If two identical keys exist, only the value of the latter key is extracted. |
StringCatenateExtractor | Concatenates multiple specified fields into a single string in the specified order. This plug-in does not support the int field type; the literal field type is recommended. Separate the field list with commas (the fields must come from the destination fields). | Concatenates the content of field1 and field2 into a new field value using '_'. In addition, the system variable $table can retrieve the current table name ($table is displayed only when a table sharding wildcard is configured). |
HTMLTagRemover | Removes HTML tags from a source field and copies the resulting content to the current field. If the source field is configured as the current field, the content is replaced. | Original text: <div id="copyright">OpenSearch</div>. After the plug-in is applied, it is parsed as "OpenSearch". |
For best practices, see Configure the MultiValueSpliter plugin.