DataSource - OpenSearch - Alibaba Cloud Documentation Center

Description

The information about a data source.

Example

{
  "tableName": "news",
  "type": "rds",
  "fields": [
    {
      "id": "id"
    },
    {
      "caption": "caption"
    },
    {
      "content": "content"
    },
    {
      "categoryId": "category_id"
    }
  ],
  "plugins": {
    "caption": {
      "name": "HTMLTagRemover",
      "parameters": {}
    },
    "content": {
      "name": "HTMLTagRemover",
      "parameters": {}
    }
  },
  "keyField": "id",
  "parameters": {
    "filter": "",
    "instanceId": "rm-abc",
    "dbName": "my_db",
    "dbTableName": "my_table",
    "dbUser": "my_user",
    "dbPassword": "my-password",
    "autoSync": true
  }
}

Structure

Field	Type	Description
tableName	String	The name of the table.
type	String	The type of the data source. Valid values: rds, odps, opensearch, and polardb.
fields[]	Object	The information about field mappings. The data about each field mapping is in the format of {key:value}. key indicates the field from the source table. value indicates the field in the destination table of the application.
plugins	Object	The plug-ins that are used for data processing.
plugins.*	Object	The information about a plug-in. For more information, see the Plugin section of this topic. The asterisk (*) indicates the name of the plug-in.
keyField	String	The primary key.
parameters	Object	The parameters vary based on the type of the data source. For more information, see the following sections: - rds - odps - opensearch - polardb

Plugin

Example

{
  "name": "HTMLTagRemover",
  "parameters": {}
}

Structure

Field

Type

Description

name

String

The name of the plug-in. Valid values: JsonKeyValueExtractor, MultiValueSpliter, KeyValueExtractor, StringCatenateExtractor, and HTMLTagRemover.

parameters

Object

The parameters vary based on the plug-in. For more information, see the following sections:

- JsonKeyValueExtractor

- MultiValueSpliter

- KeyValueExtractor

- StringCatenateExtractor

- HTMLTagRemover

JsonKeyValueExtractor

This plug-in extracts the value of the specified key from source fields in the JSON format. The extracted key value is used as the value of the mapped destination field. Only the value of the specified key can be extracted. Example

{
  "key": "my_field"
}

Structure

Field	Type	Description
key	String	The name of the key.

MultiValueSpliter

This plug-in uses delimiters to divide the value of a source field into multiple values. The values are used as the array elements of the mapped destination field. The destination field must be of the ARRAY type. If you use a non-printable character as the delimiter, you must use Unicode characters such as \u001D to identify the non-printable character. Example

{
  "seperator": "|"
}

Structure

Field	Type	Description
seperator	String	The delimiter.

KeyValueExtractor

This plug-in extracts the value of the specified key from source fields that are key-value pairs. The extracted key value is used as the value of the mapped destination field. Only the value of the specified key can be extracted. Delimiters are optional. Example

{
  "key": "my_field",
  "fieldSep": ";",
  "kvSep": ":"
}

Structure

Field	Type	Description
key	String	The name of the key.
fieldSep	String	The delimiter that is used to separate keys.
kvSep	String	The delimiter that is used to separate keys and values.

StringCatenateExtractor

This plug-in concatenates the values of specified fields into a string in the specified sequence. This plug-in cannot concatenate fields of integer types. We recommend that you use fields of literal types. Separate multiple fields with commas (,). The fields must be destination fields. Example

{
  "connector": "-",
  "sources": "id,title"
}

Structure

Field	Type	Description
connector	String	The connector.
sources	String	The fields to be concatenated.

HTMLTagRemover

This plug-in removes HTML tags from the value of a source field. Then, the value of the mapped destination field is replaced by the value without HTML tags. Note: This plug-in does not require additional parameters.

rds

Example

{
  "instanceId": "rds-instance-id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "dbUser": "my",
  "dbPassword": "my_passwd",
  "filter":"",
  "autoSync": true
}

Structure

Field	Type	Description
instanceId	String	The ID of the instance.
dbName	String	The name of the database.
dbTableName	String	The name of the table in the database.
dbUser	String	The account that is used to connect to the database.
dbPassword	String	The password that is used to connect to the database.
filter	String	The filter conditions. The following operators are supported: <, >, =, !=, >=, and <=. Separate multiple conditions with commas (,).
autoSync	Boolean	Indicates whether automatic data synchronization is enabled. Valid values: true and false.

odps

Example

{
  "projectName": "my_project",
  "tableName": "my_table",
  "partition": "",
  "done": "no",
  "donePrefix": "",
  "timestamp":""
}

Structure

Field	Type	Description
projectName	String	The name of the MaxCompute project.
tableName	String	The name of the table.
partition	String	For more information about partitions, see Configure a MaxCompute data source.
done	String	Indicates whether the DONE file is used. Valid values: no and yes.
donePrefix	String	The prefix of the DONE file. This parameter is required if the done parameter is set to yes.
timestamp	String	The timestamp when full data was imported.

opensearch

Example

{
  "appId": "12345678",
  "tableName": "my_table",
  "appType": "standard",
  "filter": ""
}

Structure

Field	Type	Description
appId	String	The ID of the application.
tableName	String	The name of the table in the application.
appType	String	The type of the application. Valid values: standard, advance, and enhanced. The value of standard indicates a standard application. The value of advance indicates an advanced application that is of a previous application type, whereas the value of enhanced indicates an advanced application that is of a new application type. New applications cannot be of the advance type.
filter	String	The filter conditions. The following operators are supported: <, >, =, !=, >=, and <=. Separate multiple conditions with commas (,).

polardb

Example

{
  "clusterId": "my_cluster_id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "aliyunUserId":"123455789",
  "dbUser":"my_user",
  "dbPassword":"my_passwd",
  "autoSync":true,
  "filter": "id>100"
}

Structure

Field	Type	Description
clusterId	String	The ID of the cluster.
dbName	String	The name of the database.
dbTableName	String	The name of the table.
aliyunUserId	String	The ID of the Alibaba Cloud account.
dbUser	String	The account that is used to connect to the database.
dbPassword	String	The password that is used to connect to the database.
autoSync	Boolean	Indicates whether automatic data synchronization is enabled. Valid values: true and false.
filter	String	The filter conditions. The following operators are supported: <, >, =, !=, >=, and <=. Separate multiple conditions with commas (,).