DataFormat component - Security Center - Alibaba Cloud Documentation Center

The DataFormat component provides data processing actions for Security Orchestration, Automation and Response (SOAR) playbooks, including joining, grouping, concatenating, and transforming data.

Actions

Each action serves a distinct purpose. Use this table to pick the right action for your use case.

Action	What it does	Use when	SQL equivalent
`joindata`	Joins multiple data nodes and outputs specified fields	Combining two or more datasets where rows in both must match	INNER JOIN
`leftJoin`	Joins two data nodes, keeping all rows from the left node	Combining datasets where you need all left-side rows, even without a match on the right	LEFT JOIN
`groupbydata`	Groups rows by a key and computes aggregations	Summarizing data — for example, counting events by attack type	GROUP BY
`concatFieldtoBatch`	Concatenates field values in batches into strings	Merging array values into delimited strings with optional prefix/suffix	—
`convertToJson`	Converts a string to a JSONArray or JSONObject	Preparing string data for downstream actions that require JSON input	CAST / CONVERT
`uniondata`	Combines rows from multiple data nodes into one dataset	Merging results from parallel branches into a single list	UNION ALL
`formatdata`	Creates a new data row with specified field-value pairs	Generating structured output from static or upstream values	SELECT ... AS

Save the sample JSON data as a file before importing examples as test playbooks. To learn how to import a playbook, see Import a playbook.

Component configuration examples

The following sections cover the configuration details and sample inputs and outputs for each action.

joindata

Joins multiple data nodes on a shared key and outputs specified fields. This is an inner join — only rows that match in all input nodes appear in the output.

Parameters

Parameter	Description
`inputNodes`	The data nodes to join. Each node's data must be a JSONArray. There is no limit on nesting depth — a JSONArray can contain nested JSONArrays.
`joinFields`	The fields to join on. Format: `[nodeName].[fieldName]`. Only first-level fields are supported. JSONArray and String field types are both accepted. Nested fields (for example, `uuid-key` inside `uuid`) cannot be used as join keys.
`outputFields`	The fields to include in the output. Format: `[nodeName].[fieldName]`.

When referencing a node in inputNodes, prefix the value with JSONArray: to ensure the data is treated as a JSONArray. Without this prefix, data cannot be read from the upstream node.

Example

Input data:

leftJSON: [{"action":"11","uuid":[{"uuid-key":"1"}]},{"action":"21","uuid":[{"uuid-key":"2"}]}]
rightJSON: [{"uuid":[{"uuid-key":"1"}],"host":"196.1.*.*"},{"uuid":[{"uuid-key":"1"}],"host":"196.2.*.*"}]

Parameter values:

Parameter	Configuration
`inputNodes`	Row 1 — Left: `leftJSON`, Right: `${JSONArray:leftJSON.datalist.*}`
	Row 2 — Left: `rightJSON`, Right: `${JSONArray:rightJSON.datalist.*}`
`joinFields`	Left: `leftJSON.uuid`, Right: `rightJSON.uuid`
`outputFields`	Row 1 — Left: `myhost`, Right: `rightJSON.host`
	Row 2 — Left: `myaction`, Right: `leftJSON.action`
	Row 3 — Left: `uuid`, Right: `leftJSON.uuid`

Output:

[
    {
        "myaction": "11",
        "uuid": [{"uuid-key": "1"}],
        "myhost": "196.1.*.*"
    },
    {
        "myaction": "11",
        "uuid": [{"uuid-key": "1"}],
        "myhost": "196.2.*.*"
    }
]

The row with action: "21" does not appear in the output because no row in rightJSON has uuid-key: "2". Use leftJoin instead if you need to keep unmatched left-side rows.

leftJoin

Performs a left join using the left node as the primary table. All rows from the left node appear in the output. Rows from the right node that do not match are discarded; rows from the left node that do not match are included with empty values for right-node fields.

Parameters

Parameter	Description
`leftNode`	The primary (left) data node. Format must be a JSONArray of JSON objects. Nested JSONArrays are supported.
`rightNode`	The secondary (right) data node. Format must be a JSONArray of JSON objects. Nested JSONArrays are supported.
`joinFields`	The fields to join on. Format: `leftNode.[fieldName]` and `rightNode.[fieldName]`. Only first-level fields are supported — nested fields (for example, `uuid-key` inside `uuid`) cannot be used as join keys.
`leftOutputFilelds`	Fields to output from the left node, with optional key renaming. Left input: new name. Right input: original field name. Use `` in the left box to keep the original name; use `` in the right box to select all fields. Separate multiple fields with a comma.
`rightOutputFields`	Fields to output from the right node, with optional key renaming. Same convention as `leftOutputFilelds`.

Both leftNode and rightNode data must be converted to JSONArray type before use.

Example

Input data:

leftNode:

[
    {"action": "11", "uuid": [{"uuid-key": "1"}]},
    {"action": "21", "uuid": [{"uuid-key": "2"}]}
]

rightNode:

[
    {"uuid": [{"uuid-key": "1"}], "host": "196.1.*.*"},
    {"uuid": [{"uuid-key": "1"}], "host": "196.2.*.*"}
]

Parameter values:

Parameter	Configuration
`joinFields`	Left: `leftNode.uuid`, Right: `rightNode.uuid`
`leftOutputFilelds`	Left: `*`, Right: `action,uuid`
`rightOutputFields`	Left: `site`, Right: `host`

Output:

[
    {
        "action": "11",
        "uuid": [{"uuid-key": "1"}],
        "site": "196.1.*.*"
    },
    {
        "action": "11",
        "uuid": [{"uuid-key": "1"}],
        "site": "196.2.*.*"
    },
    {
        "action": "21",
        "uuid": [{"uuid-key": "2"}]
    }
]

The row with action: "21" is retained even though it has no matching row in rightNode. The site field is absent for that row.

groupbydata

Groups rows by a key and computes aggregated values for each group. Similar to a SQL GROUP BY statement — rows with the same key are grouped together, then an expression is applied to produce summary values.

Parameters

Parameter	Description
`inputNode`	The data to group. Must be a JSONArray of JSON objects.
`groupByKeys`	The field to group by. Rows with the same value for this field are placed in the same group.
`outputFields`	The computed fields to include for each group. Supports jq expressions applied to the grouped data.

Example

Scenario: Group security events by Attack Type, then compute the total number of attacks and the most recent attack time.

Input data (`inputNode`):

[
    {"Attack Type": "web shell", "count": "1", "startTime": "1646115503000", "endTime": "1646115503000", "securityEventIds": "330706"},
    {"Attack Type": "web shell", "count": "2", "startTime": "1646114346000", "endTime": "1646114346000", "securityEventIds": "330700"},
    {"Attack Type": "application whitelist", "count": "1", "startTime": "1646114098000", "endTime": "1646114098000", "securityEventIds": "330699"}
]

Parameter values:

Parameter	Configuration
`groupByKeys`	`Attack Type`
`outputFields`	`[{"fieldName":"Attack Count","fieldType":"jq","fieldValue":".datalist \| map_values(.count)\|add\|length"},{"fieldName":"Latest Attack Time","fieldType":"jq","fieldValue":"[.datalist[].endTime\|tonumber]\|max"}]`

The two jq expressions above:

.datalist | map_values(.count)|add|length — counts the total number of attack events in the group
[.datalist[].endTime|tonumber]|max — converts endTime values to numbers and returns the maximum (most recent time)

Output:

[
    {"Attack Type": "web shell", "Attack Count": 2, "Latest Attack Time": 1646115503000},
    {"Attack Type": "application whitelist", "Attack Count": 1, "Latest Attack Time": 1646114098000}
]

concatFieldtoBatch

Splits a JSONArray field into mini-batches and concatenates the values in each batch into a string. Supports custom separators, prefixes, and suffixes.

Parameters

Parameter	Description
`inputField`	The data to concatenate. Must be a JSONArray of JSON objects.
`batchSize`	Number of items per batch. For example, `2` groups two objects together. Use `-1` to combine all objects into one string.
`concatStr`	The separator inserted between concatenated values. Enclose in double quotation marks to preserve spaces — for example, `" ## "`.
`fieldAddPrefix`	A prefix added to the start of each concatenated string.
`fieldAddSuffix`	A suffix added to the end of each concatenated string.

Example

Scenario: Concatenate Attack Type values in groups of two, using " ## " as the separator, with prefix Prefix and suffix Suffix.

Input data (`inputField`):

[
    {"Attack Type": "web shell"},
    {"Attack Type": "web shell"},
    {"Attack Type": "web shell"},
    {"Attack Type": "application whitelist"},
    {"Attack Type": "application whitelist"}
]

Parameter values:

Parameter	Value
`batchSize`	`2`
`concatStr`	`" ## "`
`fieldAddPrefix`	`Prefix`
`fieldAddSuffix`	`Suffix`

Output:

[
    {"concatField": "PrefixWebShellSuffix ## PrefixWebShellSuffix"},
    {"concatField": "PrefixWebShellSuffix ## PrefixApplicationWhitelistSuffix"},
    {"concatField": "PrefixApplicationWhitelistSuffix"}
]

The five input items are split into batches of two (groups 1 and 2) plus one remaining item (group 3).

convertToJson

Converts a string to a JSON object. This action only performs type conversion — it does not modify the data.

If the input is a JSONArray string, all elements are placed directly into the datalist.
If the input is a JSONObject string, it is treated as a single row in the datalist.

This action is typically used to prepare upstream string data before passing it to actions such as joindata, leftJoin, or groupbydata, which require JSONArray input.

uniondata

Combines rows from multiple upstream data nodes into a single dataset. All rows from all specified nodes are included — equivalent to a SQL UNION ALL.

Parameters

Parameter	Description
`inputNodes`	The node data to combine.

The left input box labels (for example, leftJSON, rightJSON) are identifiers only and have no effect on the output. Prefix each right-box value with JSONArray: to ensure the data is read correctly from upstream nodes.

Example

Input data:

left: [{"action":"11","uuid":[{"uuid-key":"1"}]},{"action":"21","uuid":[{"uuid-key":"2"}]}]
right: [{"uuid":[{"uuid-key":"1"}],"host":"196.1.*.*"},{"uuid":[{"uuid-key":"1"}],"host":"196.2.*.*"}]

Parameter values:

Parameter	Configuration
`inputNodes`	Row 1 — Left: `leftJSON`, Right: `${JSONArray:left.datalist.*}`
	Row 2 — Left: `rightJSON`, Right: `${JSONArray:right.datalist.*}`

Output:

[
    {"action": "11", "uuid": [{"uuid-key": "1"}]},
    {"action": "21", "uuid": [{"uuid-key": "2"}]},
    {"uuid": [{"uuid-key": "1"}], "host": "196.1.*.*"},
    {"uuid": [{"uuid-key": "1"}], "host": "196.2.*.*"}
]

formatdata

Transforms data and aggregates data from multiple sources into a new table. Use this action to create structured output with explicitly defined field-value pairs from static values or upstream node references.

Parameters

Parameter	Description
`outputFields`	The field-value pairs to include in the output. Left input: field name. Right input: field value.

Example

Parameter values:

Left (field name)	Right (field value)
`name`	`test`
`age`	`18`

Output:

{
    "name": "test",
    "age": "18"
}