This topic describes how to export a log to different storage destinations. The fields in the log vary based on the storage destination.
Background information
You want to export a log to different storage destinations and the fields in the log vary based on the storage destination. In this example, the fields in a raw log are f1, f2, f3, f4, and f5.
To export the log to target 1, discard f1 and f2, and retain f3, f4, and f5 .
To export the log to target 2, discard f3 and f4, and retain f1, f2, and f5.
Configuration example
Raw log
__time__ : 1591754815
f1: GET
f2: https
f3: aliyun
f4: 200
f5: standard
Transformation syntax
For more information about the data transformation functions that are used in the following example, see Function overview.
Use the e_set function to add a new field named
tag: target1, target2
to the log.Use the e_split function to split the log into two logs based on the tag field. One log contains the
tag: target1
field and the other log contains thetag: target2
field.Discard f1 and f2 from the log that contains the
tag: target1
field and export f3, f4, and f5 totarget1
by using thee_output
function. The subsequent transformation rules are not executed for the log that contains thetag: target1
field.Discard f3 and f4 from the log that contains the
tag: target2
field.
e_set("tag", "target1, target2")
e_split("tag")
e_if(e_search("tag==target1"), e_compose(e_drop_fields("f1", "f2", regex=False), e_output("target1")))
e_drop_fields("f3", "f4", regex=False)
e_output("target2")
Export fields to target1
__time__ : 1591754815
f3: aliyun
f4: 200
f5: standard
Export fields to target2
__time__ : 1591754815
f1: GET
f2: https
f5: standard
Incorrect settings
If you use the following transformation syntax, f3, f4, and f5 are exported to target1. Then, target1 is used as the input for exporting data to target 2. As a result, f1 and f2 are not exported to target2. However, target2 is expected to contain f1 and f2.
e_drop_fields("f1", "f2", regex=False)
e_coutput("target1")
e_drop_fields("f3", "f4", regex=False)
e_output("target2")