All Products
Search
Document Center

Simple Log Service:Advanced preview

Last Updated:Aug 25, 2023

In Advanced preview mode, Simple Log Service accesses the specified Logstore and reads data from the Logstore to test the data and simulate the data transformation process. This topic describes how to configure the Advanced preview mode.

Procedure

  1. Log on to the Log Service console.

  2. Go to the data transformation page.

    1. In the Projects section, click the project that you want to manage.

    2. On the Log Storage > Logstores tab, click the Logstore that you want to manage.

    3. On the query and analysis page, click Data Transformation.

  3. In the upper-right corner of the page, select a time range for the required log data.

    Make sure that the Raw Logs tab displays log data.

  4. In the edit box, enter a data transformation statement.
    For more information, see Data processing syntax.
    Note You can add comments to the data transformation statement to debug the statement line by line.
  5. Preview data.

    1. In the upper-right corner of the page, click Advanced.

    2. Click Preview Data.

    3. In the Add Preview Settings panel, set the parameters and click OK. The following table describes the parameters.

      When you preview data for the first time, you must set the parameters. After you set the parameters, you can click Modify Preview Settings to modify the parameters.

      Parameter

      Description

      Authorization Method

      You can authorize a data transformation task to read data from the current source Logstore by using one of the following methods:

      • Default Role: authorizes the data transformation job to assume the system role AliyunLogETLRole to read data from the source Logstore.

        You must click You must authorize the system role AliyunLogETLRole. Then, you must configure other parameters as prompted to complete the authorization. For more information, see Access data by using a default role.

        Note
        • If the authorization is complete within your Alibaba Cloud account, you can skip this operation.

        • If you use an Alibaba Cloud account that has assumed the role, you can skip this operation.

      • Custom Role: authorizes the data transformation job to assume a custom role to read data from the source Logstore.

        You must grant the custom role the permissions to read from the source Logstore. Then, you must enter the Alibaba Cloud Resource Name (ARN) of the custom role in the Role ARN field. For more information about authorization, see Access data by using a custom role.

      • AccessKey Pair: authorizes the data transformation job to use the AccessKey pair of an Alibaba Cloud account or a RAM user to read data from the source Logstore.

        • Alibaba Cloud account: The AccessKey pair of an Alibaba Cloud account has permissions to read from the source Logstore. You can directly enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account in the AccessKey ID and AccessKey Secret fields. For more information about how to obtain an AccessKey pair, see AccessKey pair.

        • RAM user: You must grant the RAM user the permissions to read from the source Logstore. Then, you can enter the AccessKey ID and AccessKey secret of the RAM user in the AccessKey ID and AccessKey Secret fields. For more information about authorization, see Access data by using AccessKey pairs.

      Advanced Parameter Settings

      Simple Log Service allows you to set the passwords that are required in the transformation statement in the key-value pair format. For example, you can set a password that is used to connect to a database in the key-value pair format. You can reference passwords in the transformation statement by using the ${key} variable.

      You can click the plus sign (+) to add more key-value pairs. For example, you can add config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj, which indicates the ID of the virtual private cloud (VPC) to which an ApsaraDB RDS instance belongs. Advanced Parameter Settings

      After you configure the preview settings, you can preview the data transformation results on the Transformation Results tab.

      • If the data fails to be transformed because the syntax of the transformation statement or the permissions are invalid, troubleshoot the failure as prompted.

      • If the data is transformed as expected, you can save the transformation statement as a rule. For more information, see Create a data transformation job.

Advanced preview example

  • Transformation statement

    # e_set("insert_field", "test_value")
    e_table_map(
        res_rds_mysql(
            address="rm-uf6wjk5****.mysql.rds.aliyuncs.com",
            username="test_username",
            password="****",
            database="test_db",
            table="test_table",
        ),
        "ip",
        ["country", "province"],
    )
  • Raw log entry

    {
      "__source__": "192.0.2.0",
      "__time__": 1624523917,
      "__topic__": "topic",
      "id": "1001",
      "ip": "127.0.0.1"
    }
  • Advanced preview settingsAdvanced preview

  • Transformation resultTransformation result