All Products
Search
Document Center

Simple Log Service:How do I export a log to different storage destinations?

Last Updated:Jul 04, 2024

This topic describes how to export a log to different storage destinations. The fields in the log vary based on the storage destination.

Background information

You want to export a log to different storage destinations and the fields in the log vary based on the storage destination. In this example, the fields in a raw log are f1, f2, f3, f4, and f5.

  • To export the log to target 1, discard f1 and f2, and retain f3, f4, and f5 .

  • To export the log to target 2, discard f3 and f4, and retain f1, f2, and f5.

Configuration example

Raw log

__time__ : 1591754815
f1: GET
f2: https
f3: aliyun
f4: 200
f5: standard

Transformation syntax

Note

For more information about the data transformation functions that are used in the following example, see Function overview.

  1. Use the e_set function to add a new field named tag: target1, target2 to the log.

  2. Use the e_split function to split the log into two logs based on the tag field. One log contains the tag: target1 field and the other log contains the tag: target2 field.

  3. Discard f1 and f2 from the log that contains the tag: target1 field and export f3, f4, and f5 to target1 by using the e_output function. The subsequent transformation rules are not executed for the log that contains the tag: target1 field.

  4. Discard f3 and f4 from the log that contains the tag: target2 field.

e_set("tag", "target1, target2")
e_split("tag")
e_if(e_search("tag==target1"), e_compose(e_drop_fields("f1", "f2", regex=False), e_output("target1")))
e_drop_fields("f3", "f4", regex=False)
e_output("target2")

Export fields to target1

__time__ : 1591754815
f3: aliyun
f4: 200
f5: standard

Export fields to target2

__time__ : 1591754815
f1: GET
f2: https
f5: standard

Incorrect settings

If you use the following transformation syntax, f3, f4, and f5 are exported to target1. Then, target1 is used as the input for exporting data to target 2. As a result, f1 and f2 are not exported to target2. However, target2 is expected to contain f1 and f2.

e_drop_fields("f1", "f2", regex=False)
e_coutput("target1")
e_drop_fields("f3", "f4", regex=False)
e_output("target2")