All Products
Search
Document Center

DataSource

Last Updated: Sep 09, 2021

Description

The information about a data source.

Example

{
  "tableName": "news",
  "type": "rds",
  "fields": [
    {
      "id": "id"
    },
    {
      "caption": "caption"
    },
    {
      "content": "content"
    },
    {
      "categoryId": "category_id"
    }
  ],
  "plugins": {
    "caption": {
      "name": "HTMLTagRemover",
      "parameters": {}
    },
    "content": {
      "name": "HTMLTagRemover",
      "parameters": {}
    }
  },
  "keyField": "id",
  "parameters": {
    "filter": "",
    "instanceId": "rm-abc",
    "dbName": "my_db",
    "dbTableName": "my_table",
    "dbUser": "my_user",
    "dbPassword": "my-password",
    "autoSync": true
  }
}

Structure

Field

Type

Description

tableName

String

The name of the table.

type

String

The type of the data source. Valid values:- rds- odps- opensearch- polardb

fields[]

Object

The information about field mappings. The data about each field mapping is in the format of {key:value}. key indicates the field from the source table. value indicates the field in the destination table of the application.

plugins

Object

The plug-ins that are used for data processing.

plugins.*

Object

The information about a plug-in. For more information, see the "Plugin" section. The asterisk (*) indicates the name of the field.

keyField

String

The primary key.

parameters

Object

The parameters for different types of data sources are different. For more information, see the following sections:- rds- odps- opensearch- polardb

Plugin

Example

{
  "name": "HTMLTagRemover",
  "parameters": {}
}

Structure

Field

Type

Description

name

String

The name of the plug-in. Valid values:- JsonKeyValueExtractor- MultiValueSpliter- KeyValueExtractor- StringCatenateExtractor- HTMLTagRemover

parameters

Object

The parameters for different plug-ins are different. For more information, see the following sections:- JsonKeyValueExtractor- MultiValueSpliter- KeyValueExtractor- StringCatenateExtractor- HTMLTagRemover

JsonKeyValueExtractor

This plug-in extracts the specified key value from source fields in JSON format. The extracted key value is used as the name of the destination field. Only the value of the specified key can be extracted. Example

{
  "key": "my_field"
}

Structure

Field

Type

Description

key

String

The name of the key.

MultiValueSpliter

This plug-in uses delimiters to divide the value of a source field into multiple values. The values are used as the array elements of the corresponding destination field. The destination fields must be of an array type. If you use a non-printable character as the delimiter, you must use Unicode characters, for example, \u001D, to identify the non-printable character. Example

{
  "seperator": "|"
}

Structure

Field

Type

Description

seperator

String

The delimiter.

KeyValueExtractor

This plug-in extracts the specified key value from source fields which are key-value pairs. The extracted key value is used as the value of the destination field. Only the value of the specified key can be extracted. Delimiters are optional. Example

{
  "key": "my_field",
  "fieldSep": ";",
  "kvSep": ":"
}

Structure

Field

Type

Description

key

String

The name of the key.

fieldSep

String

The delimiter that is used to separate keys.

kvSep

String

The delimiter that is used to separate keys and values.

StringCatenateExtractor

This plug-in concatenates the values of specified fields into a string in a specified sequence. This plug-in cannot concatenate fields of integer types. We recommend that you use fields of literal types. Separate multiple fields with commas (,). Example

{
  "connector": "-",
  "sources": "id,title"
}

Structure

Field

Type

Description

connector

String

The connector.

sources

String

The fields to be concatenated.

HTMLTagRemover

This plug-in removes HTML tags from the value of a source field. Then, the value of the destination field is replaced by the value without HTML tags. Note: This plug-in does not require additional parameters.

rds

Example

{
  "instanceId": "rds-instance-id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "dbUser": "my",
  "dbPassword": "my_passwd",
  "filter":"",
  "autoSync": true
}

Structure

Field

Type

Description

instanceId

String

The ID of the instance.

dbName

String

The name of the database.

dbTableName

String

The name of the table in the database.

dbUser

String

The account that is used to connect to the database.

dbPassword

String

The password that is used to connect to the database.

filter

String

The filter conditions. The following operators are supported: <, >, =, !=,>=, and <=. Separate multiple conditions with commas (,).

autoSync

Boolean

Indicates whether automatic data synchronization is enabled. Valid values:- true- false

odps

Example

{
  "projectName": "my_project",
  "tableName": "my_table",
  "partition": "",
  "done": "no",
  "donePrefix": "",
  "timestamp":""
}

Structure

Field

Type

Description

projectName

String

The name of the MaxCompute project.

tableName

String

The name of the table.

partition

String

For more information about partitions, see Configure a MaxCompute data source.

done

String

Indicates whether the DONE file is used. Valid values:- no- yes

donePrefix

String

The prefix of the DONE file. This parameter is required when the value of the done parameter is yes.

timestamp

String

The timestamp when full data was imported.

opensearch

Example

{
  "appId": "12345678",
  "tableName": "my_table",
  "appType": "standard",
  "filter": ""
}

Structure

Field

Type

Description

appId

String

The ID of the application.

tableName

String

The name of the table in the application.

appType

String

The type of the application. Valid values:- standard: a standard application.- advance: an advanced application which is of an old application type. New applications cannot be of this type.- enhanced: an advanced application which is of a new application type.

filter

String

The filter conditions. The following operators are supported: <, >, =, !=,>=, and <=. Separate multiple conditions with commas (,).

polardb

Example

{
  "clusterId": "my_cluster_id",
  "dbName": "my_db",
  "dbTableName": "my_table",
  "aliyunUserId":"123455789",
  "dbUser":"my_user",
  "dbPassword":"my_passwd",
  "autoSync":true,
  "filter": "id>100"
}

Structure

Field

Type

Description

clusterId

String

The ID of the cluster.

dbName

String

The name of the database.

dbTableName

String

The name of the table.

aliyunUserId

String

The ID of the Alibaba Cloud account.

dbUser

String

The account that is used to connect to the database.

dbPassword

String

The password that is used to connect to the database.

autoSync

Boolean

Indicates whether automatic data synchronization is enabled. Valid values:- true- false

filter

String

The filter conditions. The following operators are supported: <, >, =, !=,>=, and <=. Separate multiple conditions with commas (,).