DSL vs SPL Syntax Differences for Log Data Transformation - Simple Log Service

This topic compares the rule syntax between the old and new versions of data transformation.

For more information about the comparison between the Structured Process Language (SPL) of Simple Log Service and SQL in data processing scenarios, see Scenario comparison between SPL and SQL.

Data Synchronization (No Processing Logic Required)

Version	Script description
Old version	The old version of data transformation DSL script is empty.
New version	The new version of data transformation SPL rule is empty.

Data Screening and Filtering: Exact Match for Text Type

Version

Script description

Old version

e_keep(v("level") == "ERROR") or

e_drop(v("level") != "ERROR") or

e_if(v("level") != "ERROR", e_drop()) or

e_keep(e_search("level==ERROR"))

New version

| where level='ERROR'

Data Screening and Filtering: Numeric Type Filtering

Version	Script description
Old version	`e_keep(ct_int(v("status"))>=400)`
New version	`\| where cast(status as bigint)>=400`

Data Screening and Filtering: Fuzzy Match

Version

Script description

Old version

e_keep(op_in(v("level"), "ERROR")) or

e_keep(e_search("level: ERROR") or

e_if(op_not_in(v("level"), "ERROR"), e_drop())

New version

| where level like '%ERROR%'

Add a Field, Such as Extracting or Constructing a Single Key Information Item

Version	Script description
Old version	Extract a single information item using a regular expression. `e_set("version", regex_select(v("data"), r'"version":\d+'))` Extract a single information item using JSON. For more information about the JSON query syntax in the old version of data transformation, see JMESPath syntax. `e_set("version", json_select(v("data"), "version"))`
New version	Extract a single information item using a regular expression. `\| extend version=regexp_extract(data, '"version":\d+')` Extract a single piece of information using JSON. For more information about JSON object path references in the new version of data transformation, see JsonPath on GitHub. `\| extend version=json_extract(data, '$.version')`

Parse and Format Time Information

Version	Script description
Old version	Extract the log time field `__time__`. `e_set( "__time__", dt_parsetimestamp( v("time"), fmt="%Y/%m/%d %H-%M-%S", ), )` Standardize the time format. `e_set( "time", dt_strftime( dt_parse( v("time"), fmt="%Y/%m/%d %H-%M-%S", ), fmt="%Y-%m-%d %H:%M:%S", ), )`
New version	Extract the log time field `__time__`. `\| extend time=date_parse(time, '%Y/%m/%d %H-%i-%S')` `\| extend __time__=cast(to_unixtime(time) as bigint)` Standardize the time format. `\| extend time=date_parse(time, '%Y/%m/%d %H-%i-%S')` `\| extend time=date_format(time, '%Y-%m-%d %H:%i:%S')`

Field Processing and Filtering

Version	Script description
Old version	Select fields precisely. `e_keep_fields("__tag__:node", "path", regex=False)` Select fields by pattern. `e_keep_fields("__tag__:.", regex=True)` Rename some fields in place. `e_rename("__tag__:node", node)` Exclude fields by pattern. `e_drop_fields("__tag__:.", regex=True)`
New version	Select fields precisely. `\| project node="__tag__:node", path` Select fields by pattern. `\| project -wildcard "__tag__:"` Rename some fields in place. `\| project-rename node="__tag__:node"` Exclude fields by pattern. `\| project-away -wildcard "__tag__:"`

Regex-based Extraction of Multiple Fields

Version	Script description
Old version	`e_regex("data", r"(\S+)\s+(\w+)", ["time", "level"])`
New version	`\| parse-regexp data, '(\S+)\s+(\w+)' as time, level`

Expand JSON Object Key-Value Information into Data Fields

Version

Script description

Old version

For more information about the JSON query syntax in the old version of data transformation, see JMESPath syntax.

e_json("data", depth=1, jmes="x.y.z")

New version

For more information about the JSON object path reference in the new version of data transformation, see JsonPath on GitHub.

| parse-json -path='$.x.y.z' data

Extract CSV Format Content as Data Fields

Version	Script description
Old version	`e_csv("data", ["time", "addr", "user"], sep="\0", quote='"')`
New version	For single-character separators, see CSV RFC 4180. `\| parse-csv -delim='\0' -quote='"' data as time, addr, user` Multi-character delimiters. `\| parse-csv -delim='^_^' data AS time, addr, user`

Logical Branch Processing: Parallel Branches

Version

Script description

Old version

e_if(
    e_has("a"), e_set("mode_a", "1"), 
    e_has("b"), e_set("mode_b", "1"),
)

This is equivalent to the following Python code structure:

if e_has("a"):
    e_set("mode_a", "1")
if e_has("b"):
    e_set("mode_b", "1")

New version

.let a = *
| where a is not null
| extend mode_a='1';

.let b = *
| where b is not null
| extend mode_b='1';

$a;
$b;

Logical Branch Processing: Mutually Exclusive Branches (if-else/switch)

Version

Script description

Old version

e_switch(
    e_has("a"), e_keep_fields("x", "y", "z"), 
    e_has("b"), e_keep_fields("u", "v"),
    default=e_keep_fields("w"),
)

This is equivalent to the following Python code structure:

if e_has("a"):
    e_keep_fields("x", "y", "z")
elif e_has("b"):
    e_keep_fields("u", "v")
else:
    e_keep_fields("w")

New version

.let src = *
| extend mode=case
    when a is not null then 1
    when b is not null then 2
    else 0
  end;

.let a = $src | where mode=1 | project x, y, z;
.let b = $src | where mode=2 | project u, v;
.let c = $src | where mode=0 | project w;

$a;
$b;
$c;

Dynamically Select the Destination Project/LogStore for Data Based on Rules

Version

Script description

Old version

e_output(project=v("dst_project"), logstore=v("dst_logstore"))

New version

| extend "__tag__:__sls_etl_output_project__"=dst_project

| extend "__tag__:__sls_etl_output_logstore__"=dst_logstore

Route Transformation Results to the Corresponding Destination Shard by Specifying a HashKey

Version	Script description
Old version	`e_output(hash_key_field="key_field")`
New version	`\| extend "__tag__:__sls_etl_output_hash_key__"=to_hex(md5(to_utf8(key_field))`

Package Fields and Serialize to JSON in a New Field

Version

Script description

Old version

Package all fields.

e_pack_fields("content", include="\w+")

Use a regular expression to extract values from the dict field, package the values, and then assign them to the name field.

e_regex("dict", r"(\w+):(\d+)", {r"k_\1": r"\2"}, pack_json="name")

New version

Package all fields.

| pack-fields -include='\w+' as content

Use a regular expression to extract values from the dict field, package the values, and then assign them to the name field.

| parse-kv -prefix='k_' -regexp dict, '(\w+):(\d+)' | pack-fields -include='k_.*' as name

Convert Log Format to Time Series Storage Format

Version	Script description
Old version	`e_to_metric(names="rt", labels="host")`
New version	`\| log-to-metric -names='["rt"]' -labels='["host"]'`