Log Service allows you to replicate data from a source Logstore and distribute the data to multiple destination Logstores. Before you can replicate and distribute the data, you must create a data transformation rule for the source Logstore. This topic describes how to replicate data from a source Logstore and distribute the data to multiple destination Logstores in a typical use scenario.

Use scenario

A data analytics company wants to replicate all log entries from a source Logstore and distribute the log entries to two destination Logstores. The replication and distribution features of Log Service allow the company to use the e_set function to specify tags, use the e_split function to categorize the log entries based on the tags that you specify, and then use the e_output function to distribute each category of log entries to the destination Logstore that matches the tag for the category. The following figure shows the basic logic of the replication and distribution features. Replicate and distribute dataBefore you begin, make sure that the following operations are complete:
  • Evaluate the performance of the two destination projects, target-a and target-b. For example, evaluate the number of shards in each of these Logstores. For more information, see Performance guide.
  • Create two projects, target-a and target-b. Then, create two Logstores, logstore-a and logstore-b. For more information, see Manage a project and Manage a Logstore.

Procedure

  1. Log on to the Log Service console.
  2. In the Projects section, click the name of the project that you want to view.
  3. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to view.
  4. In the upper-right corner of the page that appears, click Data Transformation.
  5. In the editor that appears, enter the following statements:
    e_set("tags", "target-a,target-b")
    e_split("tags")
    e_if(op_eq(v("tags"), "target-a"), e_output("target-a"))
    e_if(op_eq(v("tags"), "target-b"), e_output("target-b"))
    e_drop()
    • Use the e_set function to specify target-a and target-b as tags for raw log entries. For more information, see e_set.
    • Use the e_split function to categorize raw log entries based on the tags that you specify. For more information, see e_split.
    • Use the e_output function to distribute the log entries that are categorized by the e_split function to the target-a and target-b storage destinations. For more information, see e_output.
    • Use the e_drop() function to specify conditions. If a log entry does not meet the specified conditions, the log entry is dropped and is not distributed. For more information, see e_drop.
  6. Click Preview Data.
    On the Transformation Results tab, the specified tags are added to raw log entries. The raw log entries with the target-a tag are distributed to the target-a storage destination, and the raw log entries with the target-b tag are distributed to the target-b storage destination.
  7. Click Save as Transformation Rule.
  8. In the Create Data Transformation Rule panel, configure the parameters.
    1. Configure the basic parameters.
      Parameter Description
      Rule Name The name of the data transformation rule. For this example, enter test.
      Authorization Method The role that is attached to Log Service. The role specifies the permissions that Log Service has to read data from the source Logstore. For this example, select Default Role.
    2. Configure the parameters for the target-a storage destination.
      Parameter Description
      Target Name The name of the storage destination. For this example, enter target-a.
      Target Region The region where the destination project resides. For this example, select China (Hangzhou).
      Target Project The name of the destination project. For this example, enter target-a.
      Target Logstore The name of the destination Logstore. For this example, enter logstore-a.
      Authorization Method The role that is attached to Log Service. The role specifies the permissions that Log Service has to read and write data to the target-a storage destination.

      For this example, select Default Role.

    3. Configure the parameters for the target-b storage destination.
      Parameter Description
      Target Name The name of the storage destination. For this example, enter target-b.
      Target Region The region where the destination project resides. For this example, select China (Hangzhou).
      Target Project The name of the destination project. For this example, enter target-b.
      Target Logstore The name of the destination Logstore. For this example, enter logstore-b.
      Authorization Method The role that is attached to Log Service. The role specifies the permissions that Log Service has to read and write data to the target-b storage destination.

      For this example, select Default Role.

    4. Configure the parameters in the Processing Range section.
      Parameter Description
      Time Range The time range over which the log entries that are processed by Log Service are generated. If you select All, Log Services transforms all log entries in the source Logstore.
  9. Click OK.

Result

  • Open the target-a project. On the Logstores tab of the Log Storage page, select the logstore-a Logstore. Then, you can view the log entries that are distributed to the logstore-a Logstore.
  • Open the target-b project. On the Logstores tab of the Log Storage page, select the logstore-b Logstore. Then, you can view the log entries that are distributed to the logstore-b Logstore.