In Quick preview mode, you can check whether the syntax of a data transformation script
is valid and whether data is efficiently transformed as expected. You can perform
the preceding operations in Quick preview mode free of charge. This topic describes
how to configure the Quick preview mode.
Procedure
- Log on to the Log Service console.
- Go to the data transformation page.
- In the Projects section, click the name of the project that you want to view.
- Choose . On the Logstores tab, click the Logstore that you want to view.
- On the query and analysis page, click Data Transformation.
- In the upper-right corner of the page, select a time range for the required log data.
Make sure that the Raw Logs tab displays log data.
- In the edit box, enter a data transformation statement.
For more information, see
Data processing syntax.
Note You can add comments to the data transformation statement to debug the statement line
by line.
- Preview data.
- In the upper-right corner of the page, click Quick.
- Click the Data Testing tab.
- On the Data Testing tab, enter test data.
Test data includes basic data and dimension table data.
- On the Data tab, you can enter or specify basic data.
You can click the Raw Logs tab, find a log entry, and then click Import Test Data. The log entry is added as test data. You can also enter a test log entry.
Note
- The size of the test data for a single preview cannot exceed 1 MB.
- Each test log entry is separated by a blank line.
- The Markdown syntax is used to indicate cross-line field values. Triple backticks
(
```
) are used to indicate a whole field.
- The test data that you enter or specify on the Data tab must be in the key-value pair or JSON format. Colons (:) are used to connect
the field names and field values of data in the key-value pair format.
- Example 1: The test data includes two log entries. The first log entry is in the key-value
pair format and contains a cross-line field named traceback. The second log entry
is in the JSON format.
time_local: 25/May/2020:01:56:22
user agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18
"request:method": GET
```
traceback: Traceback (most recent call last):
File "traceback_print_exc.py", line 20, in <module>
produce_exception()
File "/home/user/code/test.py", line 16, in produce_exception
produce_exception(recursion_level-1)
File "/home/user/code/test.py", line 18, in produce_exception
raise RuntimeError()
RuntimeError
```
{
"time_local": "25/May/2020:01:56:22",
"user agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18",
"request:method": "GET",
"remote user": "john"
}
- Example 2: The test data includes three log entries in the JSON format.
[
{
"time_local": "25/May/2020:01:56:22",
"user agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18",
"request:method": "GET",
"remote user": "john"
},
{
"time_local": "25/May/2020:01:56:22",
"user agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18",
"request:method": "GET",
"remote user": "john"
},
{
"time_local": "25/May/2020:01:56:22",
"user agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18",
"request:method": "GET",
"remote user": "john"
}
]
- On the Dimension Table tab, you can enter or specify dimension table data.
You can use dimension tables to preview the resources that are accessed by resource
functions. You can enter sample dimension table data for preview and debugging.
Note If you enter test data on the Dimension Table tab and use the res_rds_mysql or res_log_logstore_pull function to transform the
data, the data must be in the CSV format. If you use the res_oss_file or res_local
function, the data must be in the CSV or JSON format.
Example: The test data includes two log entries. The first log entry is in the CSV
format, and the second log entry is in the JSON format.
ip,country,province
127.0.0.1,China,Shanghai
192.168.0.0,China,Zhejiang
[
{
"ip": "127.0.0.1",
"country": "China",
"province": "Shanghai"
},
{
"ip": "192.168.0.0",
"country": "China",
"province": "Zhejiang"
}
]
- Click Preview Data.
Note A maximum of 100 log entries can be returned for each data transformation test.
After you configure the preview settings, you can preview the data transformation
results on the Transformation Results tab.
- If the data fails to be transformed because the syntax of the transformation statement
or the permissions are invalid, troubleshoot the failure as prompted.
- If the data is transformed as expected, you can save the transformation statement
as a rule. For more information, see Create a data transformation task.
Quick preview example
- Transformation statement
Enter the following transformation statement in the editor:
# e_set("insert_field", "test_value")
e_table_map(
res_rds_mysql(
address="rm-uf6wjk5****.mysql.rds.aliyuncs.com",
username="test_username",
password="****",
database="test_db",
table="test_table",
),
"ip",
["country", "province"],
)
- Test data
On the tab, enter the following content:
{
"id": "1001",
"ip": "127.0.0.1"
}
- Dimension table data
On the tab, enter the following content:
ip,country,province
127.0.0.1,China,Shanghai
192.168.0.0,China,Zhejiang
- Preview transformation results