pull data from other logstores for data joining and data filtering - Simple Log Service

This topic describes how to use resource functions to retrieve data from another Logstore for data enrichment.

Background information

A hotel stores guest personal information in a Logstore named user_logstore and check-in information in a Logstore named check-in_logstore. The hotel wants to retrieve some fields from check-in_logstore and join them with the data in user_logstore. For this purpose, Simple Log Service provides the res_log_logstore_pull function to retrieve data from check-in_logstore and the e_table_map function to perform data enrichment.

Join logs from different Logstores

Raw data

Logstore for personal information (user_logstore)

topic:xxx
city:xxx
cid:12345
name:maki

topic:xxx
city:xxx
cid:12346
name:vicky

topic:xxx
city:xxx
cid:12347
name:mary

Logstore for check-in information (check-in_logstore)

time:1567038284
status:check in
cid:12345
name:maki
room_number:1111

time:1567038284
status:check in
cid:12346
name:vicky
room_number:2222

time:1567038500
status:check in
cid:12347
name:mary
room_number:3333

time:1567038500
status:leave
cid:12345
name:maki
room_number:1111

Transformation rule
Note
The res_log_logstore_pull function lets you set a time range. You can specify a time interval or only a start time.
- In the transformation rule, setting from_time=1567038284,to_time=1567038500 retrieves only the Logstore data within this time range.
- In the transformation rule, setting from_time="begin" continuously retrieves Logstore data.
For more information about the parameters of the res_log_logstore_pull function, see res_log_logstore_pull.
- e_table_map function
  This function matches logs based on the common cid field in both Logstores. A match is successful only if the cid field values are identical. When a match is successful, the room_number field and its value are returned from check-in_logstore and joined with the data in user_logstore to generate new data.
```
e_table_map(res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, 
        fields=["cid","room_number"],
        from_time="begin",
        ), "cid","room_number")
```

Transformation result

e_table_map function

topic:xxx
city:xxx
cid:12345
name:maki
room_nuber:1111

topic:xxx
city:xxx
cid:12346
name:vicky
room_number:2222

topic:xxx
city:xxx
cid:12347
name:mary
room_number:3333

Filter data using blacklists and whitelists

Set a whitelist

Transformation rule
Use the fetch_include_data parameter to set a whitelist. For example, fetch_include_data="room_number:1111" retrieves only the data where the value of the room_number field is 1111.
```
res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, ["cid","name","room_number","status"],from_time=1567038284,to_time=1567038500,fetch_include_data="room_number:1111")
```

Retrieved data

status: check in
cid:12345
name:maki
room_number:1111

status:leave
cid:12345
name:maki
room_number:1111

Set a blacklist

Transformation rule
Use the fetch_exclude_data parameter to set a blacklist. For example, fetch_exclude_data="room_number:1111" discards the data where the value of the room_number field is 1111.
```
res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, ["cid","name","room_number","status"],from_time=1567038284,to_time=1567038500,fetch_exclude_data="room_number:1111")
```

Retrieved data

status:check in
cid:12346
name:vicky
room_number:2222


status:check in
cid:12347
name:mary
room_number:3333

Set a blacklist and a whitelist

Transformation rule
When you set both a blacklist and a whitelist, the blacklist is applied first, and then the whitelist is applied. For example, fetch_exclude_data="time:1567038285",fetch_include_data="status:check in" first discards data where the time value is 1567038285. Then, it retrieves data where the status value is 'check in'.
```
res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, ["cid","name","room_number","status"],from_time=1567038284,to_time=1567038500,fetch_exclude_data="time:1567038285",fetch_include_data="status:check in")
```

Retrieved data

status:check in
cid:12345
name:maki
room_number:1111


status:check in
cid:12346
name:vicky
room_number:2222


status:check in
cid:12347
name:mary
room_number:3333

Enable primary key maintenance to retrieve data from the destination Logstore

If you want to delete retrieved data before it is transformed, you can enable the primary key maintenance feature. For example, you may want to retrieve the check-in information of guests who have checked in but not checked out from the check-in_logstore. If the retrieved data contains status:leave, it indicates that the guest has checked out. You can set the primary_keys parameter of the res_log_logstore_pull function to specify a primary key and prevent this data from being transformed.

Note

The primary_keys parameter supports only a single string, and the string must also be specified in the fields parameter.
When you enable primary key maintenance, the Logstore from which data is pulled can have only one shard.
When you enable primary key maintenance, the delete_data parameter cannot be set to None.

Transformation rule

res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, ["cid","name","room_number","status","time"],from_time=1567038284,to_time=None,primary_keys="cid",delete_data="status:leave")

Retrieved data
For the guest with the name maki, the final status is status:leave. This indicates that the guest has checked out. Therefore, the data related to this guest is not transformed.
```
time:1567038284
status:check in
cid:12346
name:vicky
room_number:2222

time:1567038500
status:check in
cid:12347
name:mary
room_number:3333
```