Replicate and distribute data

Use case

A data analytics company replicates each log entry in a source Logstore and distributes it to two Logstores based on content. The transformation uses e_set to tag each log, e_split to split logs by tag, and e_output to route logs to different Logstores. The following figure shows this workflow. Before you begin:

• Capacity for destination Projects (target-a and target-b) is planned, including shard count. Performance guide.

• Destination resources are created: the target-a Project with Logstore-a, and the target-b Project with Logstore-b. Manage Projects, Manage a Logstore.

Procedure

1. Log on to the Simple Log Service console.

2. In the Projects section, click the one you want.

   

3. On the Log Storage > Logstores tab, click the logstore you want.

   

4. On the query and analysis page, click Data Transformation to enter data transformation mode.

5. In the Data Transformation editor, enter the following transformation script.

   e_set("tags", "target-a,target-b")
   e_split("tags")
   e_if(op_eq(v("tags"), "target-a"), e_output("target-a"))
   e_if(op_eq(v("tags"), "target-b"), e_output("target-b"))
   e_drop()

   • e_set adds the tags target-a and target-b to each raw log. e_set.

   • e_split splits logs into multiple entries based on tags. e_split.

   • e_output distributes logs to the destinations target-a and target-b. e_output.

   • e_drop() discards unmatched logs to prevent unintended distribution. e_drop.

6. Click Preview Data.

   The original log is split into tagged entries. Logs tagged target-a go to the target-a destination, and logs tagged target-b go to the target-b destination.

7. Click Save as Transformation Job.

8. In the Create Data Transformation Job panel, configure the following parameters.

   1. Configure the basic information.

      Parameter	Description
      Job name	Enter test.
      Authorization method	Permission to read from the source Logstore. Select Default Role.

   2. Configure the storage destination for target-a.

      Parameter	Description
      Destination name	Enter target-a.
      Destination region	Select China (Hangzhou).
      Destination project	Enter target-a.
      Target store	Enter LogStore-a.
      Authorization method	Permission to write to the target-a destination. Select Default Role.

   3. Configure the storage destination for target-b.

      Parameter	Description
      Destination name	Enter target-b.
      Destination region	Select China (Hangzhou).
      Destination project	Enter target-b.
      Target store	Enter LogStore-b.
      Authorization method	Permission to write to the target-b destination. Select Default Role.

   4. Configure the processing time range.

      Parameter	Description
      Time range	Select All to process data continuously from the start time.

9. Click OK.