Log Service allows you to dispatch transformed data to multiple target Logstores across different accounts.

Transformed data can be dispatched in two methods: dispatch to multiple target Logstores across accounts and dynamic dispatch to target Logstores. The following table lists the advantages and disadvantages of the two methods.
Method Advantage Disadvantage
Configure multiple target Logstores Logstores of different accounts can be configured as target Logstores. It is complex to configure and use multiple target Logstores. Target Logstores are specified statically in the code. Currently, you can specify a maximum of 20 target Logstores.
Configure a few target Logstores and reset the project and logstore fields in the code. The project and logstore fields of the target Logstores can be dynamically obtained and set in the LOG domain specific language (DSL) rules. You can dispatch transformed data to more than 20 target Logstores. The used AccessKey comes from the configuration of each target Logstore and cannot be dynamically modified. You can specify the target Logstores of a maximum of 20 different accounts.
Note Currently, data transformation tasks can be dispatched only in the same region.

Scenario 1: dispatch to multiple target Logstores across accounts

  • Raw logs
    """
    All the following logs are stored in the same Logstore. The logical name of this Logstore defaults to target0.
    """
    "Log 1"
    http_host:  m1.abcd.com
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    "Log 2"
    http_host:  m2.abcd.com
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    "Log 3"
    http_host:  m3.abcd.com
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    "Log 4"
    http_host:  m4.abcd.com
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https
  • Dispatch objectives
    • Retain the log events whose http_status is 2XX in the source Logstore target0 and set their topic to success_event.
    • Dispatch the log events whose http_status is 3XX to the target Logstore target1 and set their topic to redirection_event.
    • Dispatch the log events whose http_status is 4XX to the target Logstore target2, and set their topic to unauthorized_event.
    • Dispatch the log events whose http_status is 5XX to the target Logstore target3 and set their topic to internal_server_error_event.
  • LOG DSL orchestration
    e_switch(e_match("status", r"2\d+"), e_set("__topic__", "success_event"),
             e_match("status", r"3\d+"), e_compose(e_set("__topic__", "redirection_event"), e_output("target1")),
             e_match("status", r"4\d+"), e_compose(e_set("__topic__", "unauthorized_event"), e_output("target2")),
             e_match("status", r"5\d+"), e_compose(e_set("__topic__", "internal_server_error_event`"), e_output("target3"))
        )
  • Transformation result
    """
    Source Logstore: target0
    """
    __topic__:  success_event
    http_host:  m1.abcd.com
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    """
    Target Logstore: target1
    """
    __topic__:  redirection_event
    http_host:  m2.abcd.com
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    """
    Target Logstore: target2
    """
    __topic__: unauthorized_event
    http_host:  m3.abcd.com
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    """
    Target Logstore: target3
    """
    __topic__: internal_server_error_event
    http_host:  m4.abcd.com
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https
    • You can set the logical names of different Logstores, such as target0 and target1, in the LOG DSL rule code. In addition, you can set multiple projects and Logstores of different accounts by using AccessKeys.Data dispatch
    • After you call the e_output function, the corresponding log event will be deleted from the source Logstore. If you want to retain the log event in the source Logstore after the e_output function is called, you can call the e_coutput function instead.

Scenario 2: dynamic dispatch to target Logstores

  • Raw logs
    "project1 logstore1"
    __tag__:type: dynamic_dispatch
    host:  a.b.c.com
    project: project1
    logstore: logstore1
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    "project1 logstore2"
    __tag__:type: dynamic_dispatch
    host:  m.n.q.com
    project: project1
    logstore: logstore2
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    "project2 logstore1"
    __tag__:type:  dynamic_dispatch
    host:   e.f.d.com
    project: project2
    logstore: logstore1
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    "project2 logstore2"
    __tag__:type: dynamic_dispatch
    host:   p.q.t.com
    project: project2
    logstore: logstore2
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https
  • Dispatch objectives
    • Dynamically dispatch log events based on different values of the project and logstore fields.
    • Add the tag __tag:__type to log events, with its value set to dynamic_dispatch.
  • LOG DSL orchestration
    e_output(project=v("project"), logstore=v("logstore"), tags={"type": "dynamic_dispatch"})
    • The default AccessKey used for dynamic dispatch to target Logstores is the AccessKey that corresponds to the first target Logstore configured in the configuration items of the LOG DSL rules.
    • The preceding transformation rules are not affected by the project and logstore fields of the first target Logstore, because the e_output function dynamically extracts the values of the project and logstore fields for log event dispatch.
  • Transformation result
    "project1 logstore1"
    __tag__:type: dynamic_dispatch
    host:  a.b.c.com
    project: project1
    logstore: logstore1
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    "project1 logstore2"
    __tag__:type: dynamic_dispatch
    host:  m.n.q.com
    project: project1
    logstore: logstore2
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    "project2 logstore1"
    __tag__:type:  dynamic_dispatch
    host:   e.f.d.com
    project: project2
    logstore: logstore1
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    "project2 logstore2"
    __tag__:type: dynamic_dispatch
    host:   p.q.t.com
    project: project2
    logstore: logstore2
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https

Scenario 3: dynamic dispatch to target Logstores across accounts

Note This scenario is a combination of scenario 1 and scenario 2.
  • Raw logs
    """
    project1 belongs to account 1. It has two Logstores: logstore1 and logstore2.
    project2 belongs to account 2. It has two Logstores: logstore1 and logstore2.
    """
    "Log 1"
    host:  a.b.c.com
    project: project1
    logstore: logstore1
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    "Log 2"
    host:  m.n.q.com
    project: project1
    logstore: logstore2
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    "Log 3"
    host:   e.f.d.com
    project: project2
    logstore: logstore1
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    "Log 4"
    host:   p.q.t.com
    project: project2
    logstore: logstore2
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https
  • Dispatch objectives
    • Dynamically dispatch log events based on different values of the project and logstore fields.
    • Dispatch data to the target Logstores of different accounts. project1 belongs to account 1, and project2 belongs to account 2.
  • LOG DSL orchestration
    e_switch(e_match(v("project"), "project1"), e_output(name="target0", project=v("project"), logstore=v("logstore")),
             e_match(v("project"), "project2"), e_output(name="target1", project=v("project"), logstore=v("logstore")))
    • In the task configuration items, configure the AccessKeys of accounts 1 and 2 for the target Logstores target0 and target1, respectively.
    • The preceding transformation rules are not affected by the project and logstore fields of the target Logstores target0 and target1, because the e_output function dynamically extracts the values of the project and logstore fields for log event dispatch.
  • Transformation result
    """
    Account 1
    project1 logstore1
    """
    host:  a.b.c.com
    project: project1
    logstore: logstore1
    http_status:  200
    request_method:  GET
    request_uri:  /pic/icon.jpg
    scheme:  https
    
    """
    Account 1
    project1 logstore2
    """
    host:  m.n.q.com
    project: project1
    logstore: logstore2
    http_status:  301
    request_method:  POST
    request_uri:  /data/data.php
    scheme:  http
    
    """
    Account 2
    project2 logstore1
    """
    host:   e.f.d.com
    project: project2
    logstore: logstore1
    http_status:  404
    request_method:  GET
    request_uri:  /category/abc/product_id
    scheme:  https
    
    """
    Account 2
    project2 logstore2
    """
    host:   p.q.t.com
    project: project2
    logstore: logstore2
    http_status:  504
    request_method:  GET
    request_uri:  /data/index.html
    scheme:  https