Simple Log Service:Export logs to multiple destinations with different fields

Scenario

A raw log contains fields f1, f2, f3, f4, and f5. Each destination requires a different subset of these fields:

• target1: Exclude f1 and f2. Retain f3, f4, and f5.

• target2: Exclude f3 and f4. Retain f1, f2, and f5.

Solution overview

Export the same log to multiple destinations with different field sets by applying the tag-split-route pattern:

1. Tag -- Add a tag field to the log that lists all destination names.

2. Split -- Split the log into separate copies, one per destination.

3. Route -- For each copy, drop the unwanted fields and send the log to the corresponding destination.

This pattern creates independent copies of the original log before applying any field-level modifications. Each destination receives exactly the fields it requires without affecting the other copies.

Configuration example

Raw log

__time__ : 1591754815
f1: GET
f2: https
f3: aliyun
f4: 200
f5: standard

Transformation rules

The following rules implement the tag-split-route pattern:

1. Call e_set to add a tag field with the value target1, target2.

2. Call e_split to split the log into two copies based on the tag field. One copy contains tag: target1, the other contains tag: target2.

3. For the tag: target1 copy, call e_compose inside e_if to drop fields f1 and f2, then send the result to target1 with e_output. Because e_output terminates processing for that copy, subsequent rules do not apply to it.

4. For the tag: target2 copy (which continues to the next rule), drop fields f3 and f4, then send the result to target2 with e_output.

e_set("tag", "target1, target2")
e_split("tag")
e_if(e_search("tag==target1"), e_compose(e_drop_fields("f1", "f2", regex=False), e_output("target1")))
e_drop_fields("f3", "f4", regex=False)
e_output("target2")

Result for target1

__time__ : 1591754815
f3: aliyun
f4: 200
f5: standard

Result for target2

__time__ : 1591754815
f1: GET
f2: https
f5: standard

Common mistake: using e_coutput instead of e_output

The following transformation rules produce the correct output for target1 but cause data loss for target2:

e_drop_fields("f1", "f2", regex=False)
e_coutput("target1")
e_drop_fields("f3", "f4", regex=False)
e_output("target2")

Root cause

e_output terminates processing for the current log entry after sending it to a destination. In contrast, e_coutput sends a copy to the destination and continues executing subsequent rules on the already modified log.

In this example:

1. e_drop_fields("f1", "f2") removes f1 and f2 from the log.

2. e_coutput("target1") sends the modified log (without f1 and f2) to target1 -- correct so far.

3. Processing continues with the same modified log (f1 and f2 already removed).

4. e_drop_fields("f3", "f4") removes f3 and f4 from that modified log.

5. e_output("target2") sends a log containing only f5 to target2 -- losing both f1 and f2.

Incorrect result for target2

__time__ : 1591754815
f5: standard

Expected fields f1 and f2 are missing because they were already dropped in step 1.

Fix

Use the tag-split-route pattern shown in the Configuration example section. This pattern creates independent copies of the original log before applying any field-level modifications, so each destination receives the correct field set.