Quick start

Create a data transformation task

1. Log on to the Log Service console.
2. In the Projects section, click the project from which you want to transform data.
3. Enable the data transformation mode.
   You can enable the data transformation mode by using one of the following methods:

   • Choose Log Management > Logstores. Click the source Logstore. On the search and analytics page that appears, turn on the Data Transformation switch. The data transformation page appears.
   • Choose Log Management > Logstores. On the Logstores tab, find the source Logstore, and click the > icon of the Logstore. Choose Data Transformation > Data Transformation. Click the plus sign (+) next to Data Transformation. The data transformation page appears.
4. Select a time range for received logs on the page.
   Make sure that log data exists on the Data Transformation page.
5. Enter the following rule in the text box:

   e_kv("http_host")
   e_if(e_search("status==200"),e_compose(e_set("__topic__","login_success_event"),e_output(name="200-target", project="project1", logstore="200_log")))
   e_if(e_search("status==400"),e_compose(e_set("__topic__","login_fail_event"),e_output(name="400-target", project="project1", logstore="400_log")))

   • The e_kv function extracts key-value pairs from the http_host field. In this example, key-value pairs are extracted from the field value m.abcd.com/s? q=asd&a=1&b=2 in the following format:

     http_host: m.abcd.com/s? q=asd&a=1&b=2
     q: asd
     a: 1
     b: 2

   • The e_if(condition,operation) function performs an operation when the corresponding condition is met.
     • The e_search("status==200") function checks whether the value of the status field is 200.
     • The e_compose(operation1,operation2) function performs the specified two operations in sequence.

     The e_set function is called first to set the value of the __topic__ field to login_success_event. Then, the e_output function is called to export log entries to the destination Logstores and specify topics, sources, and tags of the exported log entries. If you preview the log data, it is not exported to the destination Logstores, but to a temporary Logstore. The information about the destination Logstores is also displayed.

   For more information about data transformation functions, see Data transformation syntax.

6. Preview the log data.
   You can preview the log data in quick preview mode or advanced mode. For more information, see Configure preview modes.
7. View the transformation results.
   • All log entries whose status field value is 200 are exported to the 200_log Logstore. The values of the __topic__ field in these log entries are set to login_success_event.
   • All log entries whose status field value is 400 are exported to the 400_log Logstore. The values of the __topic__ field in these log entries are set to login_fail_event.

   
8. Save the transformation rule.
   1. Click Save as Transformation Rule.
   2. In the Create Data Transformation Rule dialog box, set the required parameters, as shown in the following figure.
      

      • Set Target Name to the value of the name parameter in the e_output function. Set Target Project to the value of the project parameter in the e_output function. Set Target Logstore to the value of the Logstore parameter in the e_output function. For information about other parameters, see Create a data transformation rule.
      • The e_output function writes transformed data to the destination Logstores.
      • You can include multiple operations in the e_compose function. For more information, see e_compose.
9. View the data transformation task.
   1. In the left-side navigation pane, click .
   2. In the left-side navigation pane of the page that appears, click the data transformation task that you want to manage.
   3. On the Data Transformation Overview page, you can perform the following operations on the task as needed:
      • View, modify, stop, restart, or delete the data transformation task.
      • View the dashboard of data transformation tasks. For more information, see Data transformation dashboard.

Use functions

• Basic usage of functions
  • Use the e_set function.
    The e_set function updates log fields. For more information, see e_set.

    • The following example shows how to add a new field named province to a log entry:

      e_set("province","Shanghai")

      When you preview the log entry, the province field is displayed.

      __source__:  log_service
      __tag__:__receive_time__:  1559799897
      __topic__:  
      body_bytes_sent:  740
      client_ip:  1.2.3.4
      host:  m.abcd.com
      http_host:  m.abcd.com/s? q=asd&a=1&b=2
      status:  200
      province: Shanghai

    • The following example shows how to copy the value of the status field in a log entry and assign the value to a new field. In this example, the v function is used to copy the value of the status field and assign the value to a new field named result.

      e_set("result",v("status"))

      When you preview the log entry, the result field is displayed and its value is the same as that of the status field.

      __source__:  log_service
      __tag__:__receive_time__:  1559799897
      __topic__:  
      body_bytes_sent:  740
      client_ip:  1.2.3.4
      host:  m.abcd.com
      http_host:  m.abcd.com/s? q=asd&a=1&b=2
      status:  200
      result: 200

  • Use the e_rename function.
    The e_rename function renames a field. For more information, see e_rename.

    • The following example shows how to rename the client_ip field to the ip field:

      e_rename("client_ip","ip")

      When you preview the log entry, the ip field is displayed.

      __source__:  log_service
      __tag__:__receive_time__:  1559799897
      __topic__:  
      body_bytes_sent:  740
      ip:  1.2.3.4
      host:  m.abcd.com
      http_host:  m.abcd.com/s? q=asd&a=1&b=2
      status:  200

    • The following example shows how to rename the client_ip field to the ip field and rename the body_bytes_sent field to the bytes_content field:

      e_rename("client_ip","ip","body_bytes_sent","bytes_content")

      When you preview the log entry, the ip field and the bytes_content field are displayed.

      __source__:  log_service
      __tag__:__receive_time__:  1559799897
      __topic__:  
      bytes_content:  740
      ip:  1.2.3.4
      host:  m.abcd.com
      http_host:  m.abcd.com/s? q=asd&a=1&b=2
      status:  200

  • Use the e_keep_fields and e_drop_fields functions.

    The e_keep_fields function retains the fields that meet a specified condition. For more information, see e_keep_fields.

    The e_drop_fields function deletes the fields that meet a specified condition. For more information, see e_drop_fields.

    • The following example shows how to use the e_keep_fields function to retain the status field:
      Note The __source__ field or the __topic__ field cannot be deleted.

      e_keep_fields("status")

      When you preview the log entry, the status field is retained.

      __source__:  
      __topic__:  
      status:  200

    • The following example shows how to use the e_drop_fields function to delete the http_host field:

      e_drop_fields("http_host")

      When you preview the log entry, the http_host field is deleted.

      __source__:  log_service
      __tag__:__receive_time__:  1559799897
      __topic__:  
      bytes_content:  740
      ip:  1.2.3.4
      host:  m.abcd.com
      status:  200

• Intermediate usage of functions
  • Use the e_if and e_search functions.

    The condition and operation parameters of the e_if function must be specified in pairs. The e_if function performs an operation on log fields based on the corresponding condition. If a log field meets the condition, the operation is performed. Otherwise, the operation is not performed and the function continues to check the next condition. If an operation is performed to delete a log entry, no subsequent operations are performed on the log entry.

    The e_search function uses a query string to check whether a field value in a log entry meets a specified condition. True or False is returned.

    If a field value meets the condition, the corresponding operation is performed. The following functions check whether the value of the status field is 200. If so, the value of the topic field is set to login_success_event.

    e_if(e_search("status==200"), e_set("__topic__", "login_success_event"))

    When you preview the log entry, the value of the __topic__ field is set to login_success_event.

    __source__:  log_service
    __tag__:__receive_time__:  1559799897
    __topic__:  login_success_event
    body_bytes_sent:  740
    client_ip:  1.2.3.4
    host:  m.abcd.com
    http_host:  m.abcd.com/s? q=asd&a=1&b=2
    status:  200

  • Use the e_kv function.

    The e_kv function extracts key-value pairs from multiple source fields by using parameters such as quote. For more information, see e_kv.

    The following example shows how to use the e_kv function to extract key-value pairs from the http_host field:

    e_kv("http_host")

    When you preview the log entry, the key-value pairs are extracted from the http_host field.

    __source__:  log_service
    __tag__:__receive_time__:  1559799897
    __topic__:  
    body_bytes_sent:  740
    client_ip:  1.2.3.4
    host:  m.abcd.com
    http_host:  m.abcd.com/s? q=asd&a=1&b=2
    status:  200
    q: asd
    a: 1
    b: 2

    In addition, the e_regex, e_kv_delimit, and e_json functions also extract log fields. For more information, see Value extraction functions.

• Advanced usage of functions

  Integrate the e_if, e_search, e_drop, and e_keep functions.

  The following example shows how to retain the log entries whose status field value is 200 and drop the log entries whose status field value is 400:

  • Raw log entries:

    """
    Log entry 1
    """
    __source__:  log_service
    __tag__:__receive_time__:  1559799897
    __topic__:  
    body_bytes_sent:  740
    client_ip:  1.2.3.4
    host:  m.abcd.com
    http_host:  m.abcd.com/s? q=asd&a=1&b=2
    status:  200
    
    
    """
    Log entry 2
    """
    __source__:  log_service
    __tag__:__receive_time__:  1559799897
    __topic__:  
    body_bytes_sent:  740
    client_ip:  1.2.3.4
    host:  m.abcd.com
    http_host:  m.abcd.com/s? q=asd&a=1&b=2
    status:  400

  • Transformation rule:

    e_if(e_search("status==400"),e_drop())
    e_if(e_search("status==200"),e_keep())

  • Transformation result:

    """
    Log entry 1
    """
    __source__:  log_service
    __tag__:__receive_time__:  1559799897
    __topic__:  
    body_bytes_sent:  740
    client_ip:  1.2.3.4
    host:  m.abcd.com
    http_host:  m.abcd.com/s? q=asd&a=1&b=2
    status:  200
    
    """
    Log entry 2 is dropped.
    """