Output configuration of data transformation results - Simple Log Service

Configure output destinations for datasets produced by a data transformation task (new version), including dynamic routing to projects or Logstores and shard-level hash key assignment.

Dataset overview

A data transformation task (new version) references datasets defined in SPL rules and writes them to one or more output targets. Named datasets are referenced by name, while unnamed datasets use the identifier __UNNAMED__. For more information about dataset definitions in the Simple Log Service SPL syntax, see General Reference.

Example

In this example, the data transformation task produces two datasets: valid and __UNNAMED__.

-- Output unnamed dataset __UNNAMED__
*
| where mode is null;

-- Define named dataset src, do not output
.let src = *
| where mode is not null;

-- Define dataset valid, do not output
.let valid = $src
| where mode = 'a'
| parse-regexp content, '(\S+)\s+(\S+)\s+(\S+)' as x, y, z
| project x, y, z;

-- Output named dataset valid
$valid;

-- Output unnamed dataset __UNNAMED__
$src
| where mode = 'b'
| parse-csv content as u, v
| project u, v;

Dynamic output to a project or Logstore

You can use SPL rules to dynamically route output to a specific project or Logstore. The following reserved fields control dynamic routing:

Reserved field	Description
`__tag__:__sls_etl_output_project__`	Specifies the destination project for dynamic output.
`__tag__:__sls_etl_output_logstore__`	Specifies the destination Logstore for dynamic output. Warning Do not configure the destination store as the current source store (same-source configuration). Otherwise, logs may be written in a loop, which incurs additional storage and traffic costs. You are responsible for the resource consumption and costs incurred.

Examples

Example 1: Specify the destination project for the data transformation result as the value of the dst_project field.

| extend "__tag__:__sls_etl_output_project__"=dst_project
Example 2: Specify the destination Logstore for the data transformation result as a concatenated string.

| extend "__tag__:__sls_etl_output_logstore__"=concat('log-', cast(status as VARCHAR))

Specify a hash key for output

You can use SPL rules to specify a hash key that routes data transformation results to a specific shard in a Logstore. For more information about shard routing with hash keys, see Write Logs.

__tag__:__sls_etl_output_hash_key__: The hash key used to route the output to a specific shard in the destination Logstore.

Example

Use the MD5 hash of the field value as the hash key for writing to the destination Logstore.

| extend "__tag__:__sls_etl_output_hash_key__"=to_hex(md5(to_utf8(field)))