All Products
Search

This Product

    Simple Log Service:Distribute data to multiple destination Logstores

    Document Center

    Simple Log Service:Distribute data to multiple destination Logstores

    Last Updated:Mar 13, 2026

    This topic describes various scenarios for distributing data to multiple destination Logstores and the required steps. The scenarios include dynamic distribution, cross-account distribution, dynamic cross-account distribution, and dynamic distribution in complex scenarios.

    Background information

    The data transformation feature of Simple Log Service supports distributing transformation results to multiple destination Logstores. When you save the transformation results, you can set the AccessKey information for different accounts to distribute the results to the Logstores of those accounts. You can also use the e_output or e_coutput function to dynamically obtain the destination project and Logstore and distribute the results to different Logstores.

    Note

    • You can distribute transformation results to Logstores across a maximum of 20 accounts.

    • If you use the e_output function, logs sent to the specified LogStore are not processed by subsequent transformation rules. To apply subsequent transformation rules to logs after they are sent to the specified LogStore, use the e_coutput function. For more information, see e_outputLogStoreut. This topic uses the e_output function as an example.

    Scenario 1: Cross-account distribution

    For example, all access logs for a website are stored in one Logstore. You want to distribute and store logs with different statuses into Logstores under different accounts.

    To meet this requirement, you can use the data transformation feature of Simple Log Service.

    • Raw logs

      http_host:  example.com
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg
scheme:  https

http_host:  example.org
http_status:  301
request_method:  POST
request_uri:  /data/data.php
scheme:  http

http_host:  example.net
http_status:  404
request_method:  GET
request_uri:  /category/abc/product_id
scheme:  https

http_host:  aliyundoc.com
http_status:  504
request_method:  GET
request_uri:  /data/index.html
scheme:  https

    • Distribution requirements

      • Store logs where http_status is 2XX in Logstore0 of storage target target0. Set the log topic to success_event.

      • Distribute logs where http_status is 3XX to Logstore1 of storage target target1. Set the log topic to redirection_event.

      • Distribute logs where http_status is 4XX to Logstore2 of storage target target2. Set the log topic to unauthorized_event.

      • Distribute logs where http_status is 5XX to Logstore3 of storage target target3. Set the log topic to internal_server_error_event.

      target0 is under Account A. target1, target2, and target3 are under Account B.

    • Transformation rule

      e_switch(e_match("status", r"2\d+"), e_set("__topic__", "success_event"),
         e_match("status", r"3\d+"), e_compose(e_set("__topic__", "redirection_event"), e_output("target1")),
         e_match("status", r"4\d+"), e_compose(e_set("__topic__", "unauthorized_event"), e_output("target2")),
         e_match("status", r"5\d+"), e_compose(e_set("__topic__", "internal_server_error_event`"), e_output("target3"))
    )

    • Storage targets

      In the Create Data Transformation Job panel, you can configure the storage targets. For more information, see Create a data transformation job.数据分发

      Label

      Storage Target

      Destination Project and Logstore

      AccessKey

      1

      target0

      Project0, LogStore0

      AccessKey information for Account A

      2

      target1

      Project1, Logstore1

      AccessKey information for Account B

      3

      target2

      Project2, Logstore2

      AccessKey information for Account B

      4

      target3

      Project3, Logstore3

      AccessKey information for Account B

    • Transformation result

      ## Logs with an http_status of 2XX are distributed to logstore0 of Account A.

__topic__:  success_event
http_host:  example.com
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg
scheme:  https

## Logs with an http_status of 3XX are distributed to Logstore1 of Account B.

__topic__:  redirection_event
http_host:  example.org
http_status:  301
request_method:  POST
request_uri:  /data/data.php
scheme:  http

## Logs with an http_status of 4XX are distributed to Logstore2 of Account B.

__topic__: unauthorized_event
http_host:  example.net
http_status:  404
request_method:  GET
request_uri:  /category/abc/product_id
scheme:  https

## Logs with an http_status of 5XX are distributed to Logstore3 of Account B.

__topic__: internal_server_error_event
http_host:  aliyundoc.com
http_status:  504
request_method:  GET
request_uri:  /data/index.html
scheme:  https

    Scenario 2: Simple dynamic distribution

    For example, all access logs for a website are stored in one Logstore. You want to distribute and store the logs in different Logstores based on the project and LogStore fields.

    To meet this requirement, you can use the data transformation feature of Simple Log Service.

    • Raw logs 

      __tag__:type: dynamic_dispatch
host:  example.aliyundoc.com
project: Project1
logstore: Logstore1
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg
scheme:  https

__tag__:type: dynamic_dispatch
host:  demo.aliyundoc.com
project: Project1
logstore: Logstore2
http_status:  301
request_method:  POST
request_uri:  /data/data.php
scheme:  http

__tag__:type:  dynamic_dispatch
host:   learn.aliyundoc.com
project: Project2
logstore: Logstore1
http_status:  404
request_method:  GET
request_uri:  /category/abc/product_id
scheme:  https

__tag__:type: dynamic_dispatch
host:   guide.aliyundoc.com
project: Project2
logstore: Logstore2
http_status:  504
request_method:  GET
request_uri:  /data/index.html
scheme:  https

    • Distribution requirements

      • Dynamically distribute logs based on the values of the project and LogStore fields.

      • Add the __tag:__type field to the logs with a value of dynamic_dispatch.

    • Transformation rule

      e_output(project=v("project"), logstore=v("logstore"), tags={"type": "dynamic_dispatch"})

      The e_output function dynamically extracts the values of the project and LogStore fields to distribute the logs.

    • Storage target

      In the Create Data Transformation Job panel, you can set a default storage target. This target is used to store other logs that are not dropped during transformation.

      Note

      In this scenario, the destination project and Logstore for dynamic distribution are determined by the configuration in the e_output function. They are not related to the destination project and Logstore configured for the default storage target (labeled 1) in the Create Data Transformation Job panel.

      动态分发

    • Transformation result

      ## Distributed to Logstore1 in Project1.

__tag__:type: dynamic_dispatch
host:  example.aliyundoc.com
project: Project1
logstore: Logstore1
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg
scheme:  https

## Distributed to Logstore2 in Project1.

__tag__:type: dynamic_dispatch
host:  demo.aliyundoc.com
project: Project1
logstore: Logstore2
http_status:  301
request_method:  POST
request_uri:  /data/data.php
scheme:  http

## Distributed to Logstore1 in Project2.

__tag__:type:  dynamic_dispatch
host:   learn.aliyundoc.com
project: Project2
logstore: Logstore1
http_status:  404
request_method:  GET
request_uri:  /category/abc/product_id
scheme:  https

## Distributed to Logstore2 in Project2.

__tag__:type: dynamic_dispatch
host:   guide.aliyundoc.com
project: Project2
logstore: Logstore2
http_status:  504
request_method:  GET
request_uri:  /data/index.html
scheme:  https

    Scenario 3: Dynamic cross-account distribution

    For example, all access logs for a website are stored in one Logstore. You want to dynamically distribute logs to Logstores under different accounts based on the project and LogStore fields.

    To meet this requirement, you can use the data transformation feature of Simple Log Service.

    • Raw logs 

      host:  example.aliyundoc.com
project: Project1
logstore: Logstore1
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg
scheme:  https

host:  demo.aliyundoc.com
project: Project1
logstore: Logstore2
http_status:  301
request_method:  POST
request_uri:  /data/data.php
scheme:  http

host:   learn.aliyundoc.com
project: Project2
logstore: Logstore1
http_status:  404
request_method:  GET
request_uri:  /category/abc/product_id
scheme:  https

host:   guide.aliyundoc.com
project: Project2
logstore: Logstore2
http_status:  504
request_method:  GET
request_uri:  /data/index.html
scheme:  https

    • Distribution requirements

      Dynamically distribute logs based on the values of the project and LogStore fields. The destination projects belong to different accounts. Project1, which contains two Logstores (LogStore1 and LogStore2), belongs to Account A. Project2, which also contains two Logstores (LogStore1 and LogStore2), belongs to Account B.

    • Transformation rule

      e_switch(e_match(v("project"), "Project1"), e_output(name="target1", project=v("project"), logstore=v("logstore")),
         e_match(v("project"), "Project2"), e_output(name="target2", project=v("project"), logstore=v("logstore")))

    • Storage targets

      In the Create Data Transformation Job panel, you can configure the storage targets. For more information, see Create a data transformation job.

      Note

      In this scenario, the destination project and Logstore are determined by the configuration in the e_output function. They are not related to the destination project and Logstore configured for the default storage target (labeled 1) in the Create Data Transformation Job panel.

      跨账号动态分发

      Label

      Storage Target

      Destination Project and Logstore

      AccessKey

      1

      target0

      Project0, Logstore0

      Not applicable

      2

      target1

      Select any. The destination is determined by the e_output function.

      AccessKey information for Account A

      3

      target2

      Select any. The destination is determined by the e_output function.

      AccessKey information for Account B

    • Transformation result

      ## Distributed to Logstore1 in Project1 of Account A.

host:  example.aliyundoc.com
project: Project1
logstore: Logstore1
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg
scheme:  https

## Distributed to Logstore2 in Project1 of Account A.
host:  demo.aliyundoc.com
project: Project1
logstore: Logstore2
http_status:  301
request_method:  POST
request_uri:  /data/data.php
scheme:  http

## Distributed to Logstore1 in Project2 of Account B.

host:   learn.aliyundoc.com
project: Project2
logstore: Logstore1
http_status:  404
request_method:  GET
request_uri:  /category/abc/product_id
scheme:  https

## Distributed to Logstore2 in Project2 of Account B.

host:   guide.aliyundoc.com
project: Project2
logstore: Logstore2
http_status:  504
request_method:  GET
request_uri:  /data/index.html
scheme:  https

    Scenario 4: Dynamic distribution in a complex scenario

    For example, a company advertises a game and stores all API request information for the game in one Logstore. The company wants to parse the user-agent request header, distribute and store requests from different devices (iOS, Android, and Windows), and analyze the method to obtain the ad conversion rate.

    To meet these requirements, you can use the data transformation and query and analysis features of Simple Log Service.

    • Raw log

      __source__:127.0.0.0
__tag__:__receive_time__: 1589541467
ip:10.0.0.0
request_method: GET
user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:50.0) Gecko/20100101 Firefox/50.0

    • Distribution requirements

      • Distribute and store request information from Windows users in Logstore1 of storage target target1.

      • Distribute and store request information from iOS users in Logstore2 of storage target target2.

      • Distribute and store request information from Android users in Logstore3 of storage target target3.

    • Transformation rule

      In this example, you can use the ua_parse_os function to parse the user_agent field. You can use the dct_get function to retrieve the operating system information from the request header. Then, you can use the e_set function to add an os field that contains the operating system information. Finally, you can use the e_output function and conditional functions for dynamic distribution.

      e_set("os", dct_get(ua_parse_os(v("user_agent")),"family"))
e_if(e_search("os==Windows"),e_output(name="target1"))
e_if(e_search("os=iOS"),e_output(name="target2"))
e_if(e_search("os==Android"),e_output(name="target3"))

    • Storage targets

      In the Create Data Transformation Job panel, you can configure the storage targets. For more information, see Create a data transformation job.

      默认存储目标

      Label

      Storage Target

      Destination Project and Logstore

      1

      target0

      Project0 and LogStore0

      2

      target1

      Project1, Logstore1

      3

      target2

      Project2, Logstore2

      4

      target3

      Project3, Logstore3

    • Query and analysis

      Perform query and analysis operations in the destination Logstores to obtain the ad conversion rate. The following query results show that the ad conversion rate for Android users is higher. For more information, see Quick start for query and analysis.

      • On the query and analysis page of Logstore2, you can enter the following query statement to view the ratio of GET and POST requests from iOS users.

        * | SELECT Request_method, COUNT(*) as number GROUP BY Request_method

        查询分析

      • On the query and analysis page of Logstore3, you can enter the following query statement to view the ratio of GET and POST requests from Android users.

        * | SELECT Request_method, COUNT(*) as number GROUP BY Request_method

        查询分析