All Products
Search
Document Center

Simple Log Service:Obtain the IPIP library from OSS and enrich IP address data

Last Updated:Dec 13, 2023

You can use the data transformation feature of Simple Log Service to obtain the IPIP library from Object Storage Service (OSS) and use the library to identify the city, state, and country to which an IP address belongs.

Prerequisites

  • AccessKey pairs are created to access an OSS bucket. For more information, see Create an AccessKey pair.

    We recommend that you create an AccessKey pair that has read-only permissions on the OSS bucket and an AccessKey pair that has write-only permissions on OSS. This way, you can use the first AccessKey pair to read data from the OSS bucket and use the second AccessKey pair to write data to the OSS bucket. For more information, see Overview.

  • The IPIP library is downloaded from the IPIP.NET website and uploaded to OSS. For more information, see Upload objects.

    We recommend that you use an AccessKey pair that has write-only permissions on OSS to upload the library.

Background information

IPIP.NET provides a global IP address library that allows you to identify geographic locations of IP addresses around the world. You can download the IP address library from the IPIP.NET website, and upload the library to OSS. If you want to identify the city, state, and country to which an IP address belongs during data transformation, you can use the res_oss_file function to obtain the IPIP library. Then you can use the Structured data functions function to parse IP addresses and use the e_set function to add new fields to log entries.

Examples

  • Raw log entry:

    ip: 192.0.2.1
  • Transformation rule:

    e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                     ak_id=res_local("AK_ID"),
                                                     ak_key=res_local("AK_KEY"),
                                                     bucket='your bucket', file='ipipfree.ipdb',
                                                     format='binary',change_detect_interval=200)))

    The following table describes the fields in the res_oss_file function.

    Field

    Description

    endpoint

    The endpoint of OSS. For more information, see Regions and endpoints.

    ak_id

    The AccessKey ID that has read permissions on OSS.

    For security concerns, we recommend that you set the value to res_local("AK_ID") in the Advanced Parameter Settings field. For information about how to set the Advanced Parameter Settings field, see Create a data transformation job.AccessKey

    ak_key

    The AccessKey secret that has read permissions on OSS.

    For security concerns, we recommend that you set the value to res_local("AK_KEY") in the Advanced Parameter Settings field.

    bucket

    The OSS bucket used to store the IP address library.

    file

    The name of the IP address library that you have uploaded.

    format

    The format of the data obtained from the IP address library. Set the value to binary.

  • Result:

    ip: 192.0.2.1
    city: Hangzhou
    province: Zhejiang
    country: China