Fields in the DataSource data structure - OpenSearch - Alibaba Cloud Documentation Center

Specifies the data source configuration for an Open Search application, including the source system type, field mappings, data processing plug-ins, and connection parameters.

Example

{
  "tableName": "news",
  "type": "rds",
  "fields": [
    { "id": "id" },
    { "caption": "caption" },
    { "content": "content" },
    { "categoryId": "category_id" }
  ],
  "plugins": {
    "caption": {
      "name": "HTMLTagRemover",
      "parameters": {}
    },
    "content": {
      "name": "HTMLTagRemover",
      "parameters": {}
    }
  },
  "keyField": "id",
  "parameters": {
    "filter": "",
    "instanceId": "rm-abc",
    "dbName": "my_db",
    "dbTableName": "my_table",
    "dbUser": "my_user",
    "dbPassword": "my-password",
    "autoSync": true
  }
}

Structure

Field	Type	Description
`tableName`	String	The name of the table in the source system.
`type`	String	The type of the data source. Valid values: `rds`, `odps`, `opensearch`, and `polardb`. The value determines which fields are required in `parameters`.
`fields[]`	Object	Field mappings between the source table and the destination table. Each entry is a `{key: value}` pair where `key` is the source field name and `value` is the destination field name.
`plugins`	Object	Data processing plug-ins applied to specific fields before indexing. Each key is a destination field name; each value is a Plugin object.
`keyField`	String	The primary key of the data source table.
`parameters`	Object	Connection and sync parameters for the data source. The required fields depend on the value of `type`. See rds, odps, opensearch, and polardb.

Plugin

A plug-in processes the value of a source field before it is mapped to the destination field.

Example

{
  "name": "HTMLTagRemover",
  "parameters": {}
}

Structure

Field	Type	Description
`name`	String	The plug-in to apply. Valid values: `JsonKeyValueExtractor`, `MultiValueSpliter`, `KeyValueExtractor`, `StringCatenateExtractor`, and `HTMLTagRemover`.
`parameters`	Object	Plug-in configuration. The required fields depend on the value of `name`. See the plug-in sections below.

JsonKeyValueExtractor

Extracts the value of a specified key from a source field that contains JSON. The extracted value is written to the destination field. Only one key can be extracted per plug-in instance.

Example

{
  "key": "my_field"
}

Structure

Field	Type	Description
`key`	String	The key to extract from the JSON source field.

MultiValueSpliter

Splits a source field value into multiple values using a delimiter. The resulting values are written as array elements to the destination field.

The destination field must be of the ARRAY type. To use a non-printable character as the delimiter, specify it as a Unicode escape sequence (for example, \u001D).

Example

{
  "seperator": "|"
}

Structure

Field	Type	Description
`seperator`	String	The delimiter used to split the source field value.

KeyValueExtractor

Extracts the value of a specified key from a source field that contains key-value pairs. The extracted value is written to the destination field. Delimiters are optional.

Example

{
  "key": "my_field",
  "fieldSep": ";",
  "kvSep": ":"
}

Structure

Field	Type	Description
`key`	String	The key whose value to extract.
`fieldSep`	String	The delimiter that separates multiple key-value pairs.
`kvSep`	String	The delimiter that separates a key from its value.

StringCatenateExtractor

Concatenates the values of multiple destination fields into a single string using a specified connector, and writes the result to the target field.

This plug-in cannot concatenate fields of integer types. We recommend that you use fields of literal types. All fields listed in sources must be destination fields.

Example

{
  "connector": "-",
  "sources": "id,title"
}

Structure

Field	Type	Description
`connector`	String	The string inserted between concatenated field values.
`sources`	String	A comma-separated list of destination field names to concatenate.

HTMLTagRemover

Strips HTML tags from the value of a source field and writes the plain-text result to the destination field. No additional parameters are required.

Parameters by data source type

The parameters object contains connection and sync settings. The required fields differ based on the value of type.

rds

Connects to an ApsaraDB RDS instance.

Example

{
  "instanceId": "rds-instance-id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "dbUser": "my_user",
  "dbPassword": "my_passwd",
  "filter": "",
  "autoSync": true
}

Structure

Field	Type	Description
`instanceId`	String	The ID of the RDS instance.
`dbName`	String	The name of the database.
`dbTableName`	String	The name of the table in the database.
`dbUser`	String	The account used to connect to the database.
`dbPassword`	String	The password used to connect to the database.
`filter`	String	Filter conditions that limit which rows are synchronized. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).
`autoSync`	Boolean	Specifies whether to enable automatic data synchronization. Valid values: `true` and `false`.

odps

Connects to a MaxCompute (ODPS) table.

Example

{
  "projectName": "my_project",
  "tableName": "my_table",
  "partition": "",
  "done": "no",
  "donePrefix": "",
  "timestamp": ""
}

Structure

Field	Type	Description
`projectName`	String	The name of the MaxCompute project.
`tableName`	String	The name of the table.
`partition`	String	The partition to read from. For partition syntax, see Configure a MaxCompute data source.
`done`	String	Specifies whether to use a DONE file to signal that a data export is complete. Valid values: `no` and `yes`.
`donePrefix`	String	The prefix of the DONE file. Required when `done` is set to `yes`.
`timestamp`	String	The timestamp of the full data import.

opensearch

Uses another Open Search application as the data source.

Example

{
  "appId": "12345678",
  "tableName": "my_table",
  "appType": "standard",
  "filter": ""
}

Structure

Field	Type	Description
`appId`	String	The ID of the source Open Search application.
`tableName`	String	The name of the table in the source application.
`appType`	String	The type of the source application. Valid values: `standard`, `advance`, and `enhanced`. `standard` indicates a standard application. `advance` indicates an advanced application of a previous generation. `enhanced` indicates an advanced application of the current generation. New applications cannot use the `advance` type.
`filter`	String	Filter conditions that limit which records are synchronized. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).

polardb

Connects to a PolarDB cluster.

Example

{
  "clusterId": "my_cluster_id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "aliyunUserId": "123455789",
  "dbUser": "my_user",
  "dbPassword": "my_passwd",
  "autoSync": true,
  "filter": "id>100"
}

Structure

Field	Type	Description
`clusterId`	String	The ID of the PolarDB cluster.
`dbName`	String	The name of the database.
`dbTableName`	String	The name of the table.
`aliyunUserId`	String	The ID of the Alibaba Cloud account.
`dbUser`	String	The account used to connect to the database.
`dbPassword`	String	The password used to connect to the database.
`autoSync`	Boolean	Specifies whether to enable automatic data synchronization. Valid values: `true` and `false`.
`filter`	String	Filter conditions that limit which rows are synchronized. Supported operators: `<`, `>`, `=`, `!=`, `>=`, and `<=`. Separate multiple conditions with commas (,).