This topic describes how to use resource functions to pull data from Object Storage Service (OSS) objects to enrich log data in Log Service.

Procedure

  1. Create AccessKey pairs that are used to access OSS.

    We recommend that you create a read-only AccessKey pair and a write-only AccessKey pair. The read-only AccessKey pair is used to pull objects from OSS. The write-only AccessKey pair is used to upload objects to OSS.

    For more information, see RAM policy.

    After creating AccessKey pairs, you can upload an object to an OSS bucket. For more information, see Upload objects.

    For a list of OSS endpoints, see Regions and endpoints.

  2. Use the AccessKey pairs to set a transformation rule.
    Use the write-only AccessKey pair to upload an object named test.text to a bucket named test. The object content is test text.Enrichment ruleThe transformation rule shown in the preceding figure is as follows:
    e_set("test_oss",res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                     ak_id=res_local("AK_ID"),
                                                     ak_key=res_local("AK_KEY"),
                                                     bucket='test', file='test.text',
                                                     format='text',change_detect_interval=0))
    Rule description:
    • The endpoint parameter indicates that OSS is deployed in the China (Hangzhou) region.
    • The ak_id and ak_key parameters indicate that the AccessKey pair is read-only and is pulled from the advanced parameter configurations.
    • The bucket parameter indicates the bucket in OSS. For more information about buckets, see Basic concepts.
    • The file parameter indicates the test.text object.
    • The format parameter indicates that the test.text object pulled from OSS is in the TXT format. The binary byte stream format is also supported.
    The log content pulled is as follows:
    test_oss: test text
  3. Pull all data from the specified OSS object.
    If the data in the specified OSS object is automatically refreshed at regular intervals, and you want to automatically pull the data at regular intervals, set the transformation rule as follows:
    e_set("test_oss",res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                     ak_id=res_local("AK_ID"),
                                                     ak_key=res_local("AK_KEY"),
                                                     bucket='test', file='test.text',
                                                     format='text',change_detect_interval=300))
    At an interval of five minutes, this rule checks whether the test.text object in OSS is updated. If so, all data in the object is pulled again. TXT data is returned.

    change_detect_interval: checks whether an update interval is specified for the test.text object. The default value is 0, which means not to refresh the object.

Enrich an IP address

The following example enriches an IP address with its geolocation data such as country, province, and city.
  • Raw log
    ip: 1.2.3.4
  • DSL orchestration
    In DSL orchestration, if the res_oss_file function is used to pull the IP address pool, the format parameter must be set to binary. Data is automatically pulled from OSS at an interval of 200 seconds. By default, the exported fields are city_name, region_name, and country_name. The following orchestration rule renames these fields in tuple format. The free version of IP geolocation database is used. For more information, visit IPIP.NET.
    e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                     ak_id='your ak_id',
                                                     ak_key='your ak_key',
                                                     bucket='your bucket', file='ipipfree.ipdb',
                                                                   format='binary',change_detect_interval=200),keep_fields=(("city_name","city"),("country_name","country"),("region_name","province"))))
  • Enrichment result
    ip: 1.2.3.4
    city: Hangzhou
    province: Zhejiang
    country: China

Use the e_table_map function to enrich log data

The following example searches CSV data obtained from OSS for the columns corresponding to the specified fields. Data in these fields is returned. For more information, see Enrich complex data by using search-based mapping functions.
  • Raw log
    account :  Sf24asc4ladDS
  • OSS object data
    id account nickname
    1 Sf24asc4ladDS Doflamingo
    2 Sf24asc4ladSA Kaidou
    3 Sf24asc4ladCD Roger
  • DSL orchestration
    In the following DSL orchestration rule, the res_oss_file function is used to pull CSV data from OSS. The tab_parse_csv function is used to create a table. The e_table_map function is used search for and return data.
    e_table_map(tab_parse_csv(res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',ak_id=ak_id='your ak_id',ak_key='your ak_key', bucket='your bucket',file='data.csv',change_detect_interval=30)),"account","nickname")
  • Enrichment result
    account :  Sf24asc4ladDS
    nickname: Doflamingo