Simple Log Service:Pull a CSV file from OSS to enrich data

Background

OSS is well-suited for storing reference data that changes infrequently — for example, account-to-nickname mappings, IP-to-region tables, or error code descriptions. When your log data in Log Service is incomplete, you can enrich it by pulling reference data from an OSS CSV object.

The data transformation feature uses three functions together:

• res_oss_file — retrieves the raw content of an OSS object

• tab_parse_csv — parses that content into a structured table

• e_table_map — matches a field in the log against the table and appends the corresponding fields to the log

Prerequisites

Before you begin, ensure that you have:

• Create an AccessKey to access your OSS bucket. For more information, see Create an AccessKey.

  Create two AccessKeys: one with read-only permissions to retrieve objects from OSS, and one with write-only permissions to upload objects to OSS. For more information about authorization, see RAM policy overview.

• Upload the CSV object that contains your reference data to an OSS bucket. For more information, see Upload objects.

  Use an AccessKey with write-only permissions for the upload.

Enrich log data with an OSS CSV file

The following example shows how to match an account field in a Logstore against an OSS CSV object and append the corresponding nickname field to each log entry.

• Raw log in a Logstore

  account :  Sf24asc4ladDS

• Data in a CSV object in an OSS bucket

  Id	Account	Nickname
  1	Sf24asc4ladDS	Doflamingo
  2	Sf24asc4ladSA	Kaido
  3	Sf24asc4ladCD	Roger

• Transformation rule

  The rule matches the account field in the Logstore against the account column in the CSV object. A match succeeds only when the account values are identical. When a match is found, the nickname field and its value are appended to the log.

  e_table_map(tab_parse_csv(res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                          ak_id=res_local("AK_ID"),
                                          ak_key=res_local("AK_KEY"), 
                                          bucket='test',
                                          file='account.csv',change_detect_interval=30)),
              "account","nickname")

  In e_table_map, the first string argument ("account") specifies the field used to match rows in the CSV table, and the second ("nickname") specifies the field retrieved from the matching row and added to the log.

  The following table describes the key parameters of res_oss_file. For all parameters, see Resource functions.

  Parameter	Description
  endpoint	The endpoint for accessing OSS. For more information, see res_oss_file.
  ak_id	The AccessKey ID with read-only permissions for OSS. Set this to res_local("AK_ID") to retrieve the value from Advanced Parameter Settings instead of embedding credentials in the rule. To configure Advanced Parameter Settings, see Create a data transformation job. In the Advanced options section, add two key-value pairs: AK_ID set to your AccessKey ID and AK_KEY set to your AccessKey secret.
  ak_key	The AccessKey secret with read-only permissions for OSS. Set the value to res_local("AK_KEY") to retrieve it from Advanced Parameter Settings.
  bucket	The OSS bucket where the CSV object is stored.
  file	The path of the target OSS object, for example test/data.txt. The path cannot start with a forward slash (/).
  change_detect_interval	How often Log Service checks whether the OSS object has changed, in seconds. The default value is 0, which disables periodic checks — the object is fetched once when the data transformation job starts. Set a non-zero value to enable periodic checks; Log Service re-fetches the object whenever a change is detected.

• Result

  account :  Sf24asc4ladDS
  nickname: Doflamingo