In Quick preview mode, you can check whether the syntax of a data transformation script is valid and whether data is efficiently transformed as expected. You can perform the preceding operations in Quick preview mode free of charge. This topic describes how to configure the Quick preview mode.

Prerequisites

Data is collected. For more information, see Log collection methods.

Procedure

  1. Log on to the Log Service console.
  2. Go to the data transformation page.
    1. In the Projects section, click the name of the project that you want to view.
    2. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to view.
    3. On the query and analysis page, click Data Transformation.
  3. In the upper-right corner of the page, select a time range for the required log data.
    Make sure that the Raw Logs tab displays log data.
  4. In the edit box, enter a data transformation statement.
    For more information, see Data processing syntax.
    Note You can add comments to the data transformation statement to debug the statement line by line.
  5. Preview data.
    1. In the upper-right corner of the page, click Quick.
    2. Click the Data Testing tab.
    3. On the Data Testing tab, enter test data.
      Test data includes basic data and dimension table data. Data
      • On the Data tab, you can enter or specify basic data.

        You can click the Raw Logs tab, find a log entry, and then click Import Test Data. The log entry is added as test data. You can also enter a test log entry.

        Note
        • The size of the test data for a single preview cannot exceed 1 MB.
        • Each test log entry is separated by a blank line.
        • The Markdown syntax is used to indicate cross-line field values. Triple backticks (```) are used to indicate a whole field.
        • The test data that you enter or specify on the Data tab must be in the key-value pair or JSON format. Colons (:) are used to connect the field names and field values of data in the key-value pair format.
        • Example 1: The test data includes two log entries. The first log entry is in the key-value pair format and contains a cross-line field named traceback. The second log entry is in the JSON format.
          time_local: 25/May/2020:01:56:22
          user agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18
          "request:method": GET
          ```    
          traceback: Traceback (most recent call last):
            File "traceback_print_exc.py", line 20, in <module>
              produce_exception()
            File "/home/user/code/test.py", line 16, in produce_exception
              produce_exception(recursion_level-1)
            File "/home/user/code/test.py", line 18, in produce_exception
              raise RuntimeError()
          
          RuntimeError
          ```
          
          {
            "time_local": "25/May/2020:01:56:22",
            "user agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18",
            "request:method": "GET",
            "remote user": "john"
          }
        • Example 2: The test data includes three log entries in the JSON format.
          [
            {
              "time_local": "25/May/2020:01:56:22",
              "user agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18",
              "request:method": "GET",
              "remote user": "john"
            },
            {
              "time_local": "25/May/2020:01:56:22",
              "user agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18",
              "request:method": "GET",
              "remote user": "john"
            },
            {
              "time_local": "25/May/2020:01:56:22",
              "user agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18",
              "request:method": "GET",
              "remote user": "john"
            }
          ]
      • On the Dimension Table tab, you can enter or specify dimension table data.

        You can use dimension tables to preview the resources that are accessed by resource functions. You can enter sample dimension table data for preview and debugging.

        Note If you enter test data on the Dimension Table tab and use the res_rds_mysql or res_log_logstore_pull function to transform the data, the data must be in the CSV format. If you use the res_oss_file or res_local function, the data must be in the CSV or JSON format.

        Example: The test data includes two log entries. The first log entry is in the CSV format, and the second log entry is in the JSON format.

        ip,country,province
        127.0.0.1,China,Shanghai
        192.168.0.0,China,Zhejiang
        
        [
          {
            "ip": "127.0.0.1",
            "country": "China",
            "province": "Shanghai"
          },
          {
            "ip": "192.168.0.0",
            "country": "China",
            "province": "Zhejiang"
          }
        ]
    4. Click Preview Data.
      Note A maximum of 100 log entries can be returned for each data transformation test.

      After you configure the preview settings, you can preview the data transformation results on the Transformation Results tab.

      • If the data fails to be transformed because the syntax of the transformation statement or the permissions are invalid, troubleshoot the failure as prompted.
      • If the data is transformed as expected, you can save the transformation statement as a rule. For more information, see Create a data transformation task.

Quick preview example

  • Transformation statement

    Enter the following transformation statement in the editor:

    # e_set("insert_field", "test_value")
    e_table_map(
        res_rds_mysql(
            address="rm-uf6wjk5****.mysql.rds.aliyuncs.com",
            username="test_username",
            password="****",
            database="test_db",
            table="test_table",
        ),
        "ip",
        ["country", "province"],
    )
  • Test data

    On the Data Testing > Data tab, enter the following content:

    {
        "id": "1001", 
        "ip": "127.0.0.1"
    }
  • Dimension table data

    On the Data Testing > Dimension Table tab, enter the following content:

    ip,country,province
    127.0.0.1,China,Shanghai
    192.168.0.0,China,Zhejiang
  • Preview transformation resultsConfigure preview mode