Write data to Sensors Data using Sensors Data Writer - DataWorks

DataWorks Data Integration supports writing data to Sensors Data using Sensors Data Writer. Sensors Data Writer supports three data types — events, users, and items — and optional field-level data type conversion before writing.

Limits

You can add a Sensors Data source only in the China (Shenzhen) region. Support for other regions is coming soon.
You cannot read data directly from Sensors Data. Use a plugin such as Hive or HDFS to read data, based on the storage type of your Sensors Data instance.

Add a Sensors Data source

Before configuring a synchronization task, add Sensors Data as a data source in DataWorks. Follow the steps in Data source management. Parameter descriptions are available in the DataWorks console when you add the data source.

Configure a synchronization task

Sensors Data Writer supports three data types, each mapping to a Sensors Data entity:

`type` value	Sensors Data entity
`track`	Event
`user`	User
`item`	Item (property)

The type value determines which sub-parameters are required in the writer step. Configure the writer using the code editor, then map source columns to destination fields using the column parameter.

For the task configuration procedure, see Configure a task in the code editor.

Script example and parameter reference

Writer script example

The following example writes item data (a course catalog) from a MySQL source to Sensors Data. Field mappings use 0-based column indexes from the source table.

{
    "type": "job",
    "steps": [
        {
            "stepType": "mysql",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "sahistory",
            "parameter": {
                "type": "item",
                "item": {
                    "itemType": "course",
                    "typeIsColumn": false,
                    "itemIdColumn": "course_id"
                },
                "column": [
                    {
                        "name": "course_id",
                        "index": 0
                    },
                    {
                        "name": "course_name",
                        "index": 1
                    },
                    {
                        "name": "course_schedule",
                        "index": 2,
                        "dataConverters": [
                            {
                                "type": "Long2Date"
                            }
                        ]
                    }
                ],
                "sdkDataAddress": "http://bigdata-project.datasink.sensorsdata.cn/sa?project=default&token=1111111111111111"
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "throttle": true,
            "concurrent": 2,
            "mbps": "12"
        }
    }
}

Top-level writer parameters

Parameter	Description	Required	Default
`type`	The type of data to write to Sensors Data. Valid values: `track`, `user`, `item`. The value determines which sub-parameters apply under the writer step.	Yes	None
`column`	Field mappings between source and destination tables. Each entry requires `name` (destination field name) and `index` (0-based source column index).	Yes	None
`sdkDataAddress`	The URL that Sensors Data uses to receive data. Format: `http://localhost:8106/sa?project=default`. To get this URL, go to Sensors Analytics > Basic Settings > Data Ingestion > Server Tracking > Copy HTTP Data Receiving Address.	Yes	None

Sub-parameters when `type` is `track`

Use these parameters to configure event data.

Parameter	Description	Required	Default
`distinctIdColumn`	The source field name to use as the event's Distinct ID in Sensors Data. Must also appear in the `column` list. The field value cannot be empty.	Optional	None
`eventName`	The event name in Sensors Data.	Optional	None
`isLoginId`	Specifies whether the Distinct ID field is a logon ID (unique user identifier). Valid values: `true`, `false`.	Optional	`true`

Example:

"track": {
    "distinctIdColumn": "track_id",
    "eventName": "testEventName",
    "isLoginId": true
}

Sub-parameters when `type` is `user`

Use these parameters to configure user data.

Parameter	Description	Required	Default
`distinctIdColumn`	The source field name to use as the user's Distinct ID in Sensors Data. Must also appear in the `column` list. The field value cannot be empty.	Optional	None
`isLoginId`	Specifies whether the Distinct ID field is a logon ID (unique user identifier). Valid values: `true`, `false`.	Optional	`true`

Example:

"user": {
    "distinctIdColumn": "user_id",
    "isLoginId": true
}

Sub-parameters when `type` is `item`

Use these parameters to configure item (property) data.

Parameter	Description	Required	Default
`itemIdColumn`	The source field name to use as the item's Distinct ID in Sensors Data. Must also appear in the `column` list. The field value cannot be empty.	Optional	None
`itemType`	The item type name in Sensors Data. If this field appears in `column`, its value cannot be empty and `typeIsColumn` must be `true`. Otherwise, use a constant as the item type.	Optional	None
`typeIsColumn`	Specifies whether `itemType` is read from the `column` list. Set to `false` to use a constant value. Valid values: `true`, `false`.	Optional	`true`

Example:

"item": {
    "itemIdColumn": "item_id",
    "itemType": "testItem",
    "typeIsColumn": false
}

Data converters

Sensors Data Writer supports data type conversion before writing to the destination. Add a dataConverters array to a field entry in column and specify the converter type.

Converter type	Converts from	Converts to	Parameters
`Date2Str`	Date	String	`pattern` (Optional): Output time format, e.g., `yyyy-MM-dd`.
`Date2Long`	Date	Long	None
`Number2Str`	Numeric	String	None
`Str2Long`	String	Long	None
`Str2Date`	String	Date	`pattern` (Optional): Output date format. Built-in formats: `yyyy-MM-dd`, `yyyy-MM-dd HH:mm:ss`, `yyyy-MM-dd HH:mm:ss.SSS`, `yyyy-MM`, `yyyyMM`, `yyyyMMddHHmmss`, `yyyyMMddHHmmssSSS`. `formats` (Optional): Array of additional input string formats for source data with mixed time formats.
`BigInt2Date`	BigInt	Date	None
`Str2Int`	String	Int	None
`Str2Double`	String	Double	None
`Str2BigDecimal`	String	BigDecimal	None
`IfNull2Default`	—	—	`default`: The constant to set when the field value is null. `dataConverters` (Optional): Nested converters to further transform the default value.
`NotNull2Null`	—	—	None. Converts any non-null field value to null.
`IfElse`	—	—	See IfElse parameters.
`IfNull2Column`	—	—	`targetColumnName`: The name of the column whose value is used when the current field is null. The referenced column must be configured before the column using this converter.

IfElse parameters

IfElse evaluates a JavaScript conditional expression and returns one of two values based on the result.

Parameter	Description
`if`	A JavaScript expression that must return a Boolean. Has access to `targetColumnName`, `value`, `param`, and `resolvedValues`.
`value`	The value returned when `if` evaluates to `true`. Parsed as JavaScript. Has access to the same variables as `if`.
`else`	The value returned when `if` evaluates to `false`. Parsed as JavaScript. Has access to the same variables as `if`.
`sharedPool`	A shared JavaScript scope. Variables and constants defined here are accessible in `if`, `value`, and `else`.

Example — validate an ID number format and set the field to null if the check fails:

"dataConverters": [
    {
        "type": "IfElse",
        "param": {
            "if": "return true;",
            "value": "if(resolvedValues.get(\"check_card_type\") == 'NAN' && value == null ){return true;} if(resolvedValues.get(\"check_card_type\") == 'ID_Card'){ if(java.util.regex.Pattern.compile(regEx).matcher(value).find() && java.lang.Integer.parseInt(value.substring(6, 10)) >=1900 && java.lang.Integer.parseInt(value.substring(6, 10)) <= nowYear){return true;}} return null;",
            "sharedPool": "var nowYear = java.time.LocalDate.now().getYear(); var regEx = \"(110|120|310|510|150|650|540|640|450|230|220|210|120|140|630|370|410|320|340|320|340|330|350|360|430|420|440|460|620|610|520|530)\\d{7}(0[1-9]|1[0-2])(0[1-9]|[1-2][0-9]|3[0-1])\\d{3}(\\d|X)\";"
        }
    }
]

IfNull2Default example

The following example handles a null timestamp field: it sets the value to "2021-07-01", converts the string to a Date, then converts the Date to a Long (milliseconds).

"dataConverters": [
    {
        "type": "IfNull2Default",
        "param": {
            "default": "2021-07-01",
            "dataConverters": [
                {
                    "type": "Str2Date",
                    "param": {
                        "pattern": "yyyy-MM-dd"
                    }
                },
                {
                    "type": "Date2Long"
                }
            ]
        }
    }
]

Limits

Add a Sensors Data source

Configure a synchronization task

Script example and parameter reference

Writer script example

Top-level writer parameters

Sub-parameters when type is track

Sub-parameters when type is user

Sub-parameters when type is item

Data converters

IfElse parameters

IfNull2Default example

Sub-parameters when `type` is `track`

Sub-parameters when `type` is `user`

Sub-parameters when `type` is `item`