IP address parsing functions - Simple Log Service - Alibaba Cloud Documentation Center

Functions


Function	Description
geo_parse	Identifies the city, province, and country based on an IP address.
ip_cidrmatch	Checks whether an IP address belongs to a Classless Inter-Domain Routing (CIDR) block.
ip_version	Checks whether the version of an IP address is IPv4 or IPv6.
ip_type	Identifies the type of an IP address and checks whether the type of the IP address is private or public.
ip_makenet	Converts an IP address to a CIDR block.
ip_to_format	Converts the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.
ip_overlaps	Checks whether two CIDR blocks overlap.
ip2long	Converts an IP address to a value of the long type.
long2ip	Converts a value of the long type to an IP address.

geo_parse

The geo_parse function is used to identify the city, province, and country based on an IP address.

Syntax

geo_parse(ip, ip_db="SLS-GeoIP", keep_fields=None, provider="ipip", ip_sep=None)

Parameters


Parameter	Type	Required	Description
ip	String	Yes	The IP address that you want to parse to obtain the city, province, and country to which the IP address belongs. If you want to enter multiple IP addresses, you can specify the delimiter by using the `ip_sep` parameter.
ip_db	String	Yes	The IP address database that is used to parse an IP address into the city, province, and country to which the IP address belongs. Valid values: SLS-GeoIP: the built-in IP address database of Log Service. This is the default value. To ensure accuracy, the built-in IP address database of Log Service is updated once a day. You can use the database without the need for additional configuration. Custom IP address database: Set the value to `res_oss_file(endpoint, ak_id, ak_key, bucket, file, format='binary', change_detect_interval=0,fetch_interval=2,refresh_retry_max=60,encoding='utf8',error='ignore')`. For more information about the parameters in the res_oss_file function, see res_oss_file.
keep_fields	Tuple	No	The keys that are included in the response. If you use the built-in IP address database to parse an IP address, the following keys are included in the response by default: city: the name of the city province: the name of the province country: the name of the country city_en: the administrative region code or name of the city province_en: the administrative region code or name of the province country_en: the code or name of the country or region isp: the name of the Internet service provider (ISP) lat: the latitude of the location to which the IP address belongs lon: the longitude of the location to which the IP address belongs If you use a custom IP address database to parse an IP address, the following keys are included in the response by default: city: the name of the city province: the name of the province country: the name of the country For example, `keep_fields=("city","country")` indicates that the `city` and `country` keys are returned. The `keep_fields` parameter can also be used to rename the keys. For example, `(("city","cty"),("country","state"))` indicates that the city and country keys are renamed `cty` and `state` in the returned result.
provider	String	No	This parameter is valid only when the `ip_db` parameter is set to a `custom IP address database`. Valid values: ipip: The binary IP address database that is provided by IPIP in the IPDB format is used to parse an IP address. To download the database, visit ipip. This is the default value. ip2location: The global binary IP address database that is provided by IP2Location is used to parse an IP address. To download the database, visit ip2location. Only a binary IP address database is supported.
ip_sep	String	No	The IP address delimiter. The delimiter is used to delimit a string of IP addresses into multiple IP addresses. The response is in the JSON format. Default value: None. This value specifies that a string of IP addresses is not delimited.

Response

A dictionary is returned in the following format:

{
  "city": "...",
  "province":"...",
  "country": "..."
}

Examples

Example 1: Use the built-in IP address database of Log Service to query data.

Raw log:
```
ip : 203.0.113.1
```
Transformation rule:
```
e_set("geo", geo_parse(v("ip")))
```

Result:

ip : 203.0.113.1
geo: {"city":"Hangzhou","province":"Zhejiang province","country":"China","isp":"China Mobile","lat":30.16,"lon":120.12}

Example 2: Use the built-in IP address database of Log Service to query data. The function parses a log field that contains multiple IP addresses and returns the city, province, and country to which each IP address belongs.

Raw log:

ip : 203.0.113.4, 192.0.2.2, 198.51.100.2

Transformation rule:

e_set("geo", geo_parse(v("ip"), ip_sep=","))

Result:

ip : 203.0.113.4, 192.0.2.2, 198.51.100.2
geo : {"203.0.113.4": {"country_en": "CN", "province_en": "330000", "city_en": "330200", "country": "China", "province": "Zhejiang province", "city": "Ningbo", "isp": "China Telecom", "lat": 29.8782, "lon": 121.549}, "192.0.2.2": {"country_en": "CN", "province_en": "320000", "city_en": "321300", "country": "China", "province": "Jiangsu province", "city": "Suqian", "isp": "China Telecom", "lat": 33.9492, "lon": 118.296}, "198.51.100.2": {"country_en": "CN", "province_en": "330000", "city_en": "330500", "country": "China", "province": "Zhejiang province", "city": "Huzhou", "isp": "China Telecom", "lat": 30.8703, "lon": 120.093}}

Example 3: Use a custom IP address database to query data.

Raw log:
```
ip : 203.0.113.1
```

Transformation rule:

e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                 ak_id='your ak_id',
                                                 ak_key='your ak_key',
                                                 bucket='your bucket', file='ipipfree.ipdb',
                                                               format='binary',change_detect_interval=20)))

Result:

ip : 203.0.113.1
geo : {"city": "Hangzhou", "province":"Zhejiang province","country": "China"}

Example 4: Use a custom IP address database to query data. The function returns the specified keys and renames the keys.

Raw log:
```
ip : 203.0.113.1
```

Transformation rule:

e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                 ak_id='your ak_id',
                                                 ak_key='your ak_key',
                                                 bucket='your bucket', file='ipipfree.ipdb',
                                                               format='binary',change_detect_interval=20),keep_fields=(("city","cty"),("country","state"),("province","pro"))))

Result:

ip : 203.0.113.1
geo : { "state": "China","pro": "Zhejiang province","cty": "Hangzhou"}

Example 5: Use a custom IP address database to query data. The function returns the specified keys.

Raw log:
```
ip : 203.0.113.1
```

Transformation rule:

e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                 ak_id='your ak_id',
                                                 ak_key='your ak_key',
                                                 bucket='your bucket', file='ipipfree.ipdb',
                                                               format='binary',change_detect_interval=20),keep_fields=("country","province")))

Result:

ip : 203.0.113.1
geo : { "country": "China","province": "Zhejiang province"}

Example 6: Use a custom IP address database to query data and use the global binary IP address database that is provided by IP2Location to parse the data. The function returns the specified keys.

Raw log:
```
ip : 203.0.113.2
```

Transformation rule:

e_set("geo", geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',ak_id="your ak_id", ak_key="your ak_secret", bucket='log-etl-staging', file='your ip2location bin file', format='binary', change_detect_interval=20),provider="ip2location"))

Result:

ip : 203.0.113.2
geo : {"city":"Dearborn","province":"Michigan","country":"United States"}

If you set the value of the provider parameter to ip2location, the open source SDK for Python that is provided by IP2Location is used for data transformation. The SDK for Python that is provided by IP2Location can be used to parse the following fields. If a field fails to be parsed, you must check whether the field is included in the IP address database provided by IP2Location.

country_short
country_long / The country field is specified for data transformation.
region / The province field is specified for data transformation.
city
isp
latitude
longitude
domain
zipcode
timezone
netspeed
idd_code
area_code
weather_code
weather_name
mcc
mnc
mobile_brand
elevation
usage_type

For more information, visit IP2Location Python SDK.

Example 7: Use a custom IP address database to query data. The function parses a log field that contains multiple IP addresses and returns the city, province, and country to which each IP address belongs.

Raw log:

ip : 203.0.113.3, 192.0.2.1, 198.51.100.1

Transformation rule:

e_set("geo", geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                               ak_id="ak_id",
                                                               ak_key="ak_secret",
                                                               bucket='log-etl-staging',
                                                               file='calendar.csv/IP2LOCATION-LITE-DB3.BIN',
                                                               format='binary', change_detect_interval=20),
                                provider="ip2location", ip_sep=","))

Result:

ip : 203.0.113.3, 192.0.2.1, 198.51.100.1
geo : {"203.0.113.3": {"city": "Dearborn", "province": "Michigan", "country": "United States"}, "192.0.2.1": {"city": "Hangzhou", "province": "Zhejiang", "country": "China"}, "198.51.100.1": {"city": "Hangzhou", "province": "Zhejiang", "country": "China"}}

ip_cidrmatch

The ip_cidrmatch function is used to return a Boolean value based on whether an IP address matches a specified CIDR subnet. This function checks whether an IP address belongs to a CIDR block. If the IP address belongs to the CIDR block, the function returns true. Otherwise, the function returns false. The IP address can be an IPv4 address or an IPv6 address.

Syntax

ip_cidrmatch(cidr_subnet, ip, default="")

Parameters


Parameter	Type	Required	Description
cidr_subnet	String	Yes	The CIDR block. Example: 192.168.1.0/24.
ip	String	Yes	The IP address.
default	String	No	If the IP address does not belong to the CIDR block, the value of this parameter is returned. You can leave this parameter empty.

Response
If the specified IP address belongs to the specified CIDR block, the function returns true. Otherwise, the function returns false.

Examples

Example 1: The specified IPv4 address belongs to the specified CIDR block. The function returns true.

Raw log:

cidr_subnet: 192.168.1.0/24
ip: 192.168.1.100

Transformation rule:

e_set("is_belong",ip_cidrmatch(v("cidr_subnet"),v("ip")))

Result:

cidr_subnet: 192.168.1.0/24
ip: 192.168.1.100
is_belong: true

Example 2: The specified IPv4 address does not belong to the specified CIDR block. The function returns false.

Raw log:

cidr_subnet: 192.168.1.0/24
ip: 10.10.1.100

Transformation rule:

e_set("is_belong",ip_cidrmatch(v("cidr_subnet"),v("ip")))

Result:

cidr_subnet: 192.168.1.0/24
ip: 10.10.1.100
is_belong: false

Example 3: The function cannot determine whether the specified IP address belongs to the specified CIDR block and returns unknown.
- Raw log:
```
cidr_subnet: 192.168.1.0/24
ip: a
```
- Transformation rule:
```
e_set("is_belong",ip_cidrmatch(v("cidr_subnet"),v("ip"),default="unknown"))
```
- Result:
```
cidr_subnet: 192.168.1.0/24
ip: a
is_belong: unknown
```

ip_version

The ip_version function is used to check whether the version of an IP address is IPv4 or IPv6. If the version of the IP address is IPv4, the function returns IPv4. If the version of the IP address is IPv6, the function returns IPv6.

Syntax
```
ip_version(ip, default="")
```

Parameters


Parameter	Type	Required	Description
ip	String	Yes	The IP address.
default	String	No	If the version of the specified IP address fails to be identified, the value of this parameter is returned. You can leave this parameter empty.

Response
IPv6 or IPv4 is returned.
Examples
- Example 1: The specified IP address is an IPv4 address. The function returns IPv4.
  - Raw log:
```
ip: 192.168.1.100
```
  - Transformation rule:
```
e_set("version",ip_version(v("ip")))
```
  - Result:
```
ip: 192.168.1.100
version: IPv4
```
- Example 2: The specified IP address is an IPv6 address. The function returns IPv6.
  - Raw log:
```
ip: ::1
```
  - Transformation rule:
```
e_set("version",ip_version(v("ip")))
```
  - Result:
```
ip: ::1
version: IPv6
```

ip_type

The ip_type function is used to identify the type of an IP address and check whether the type of the IP address is private or public. Valid values: private, reserved, loopback, public, and allocated ripe ncc.

Syntax
```
ip_type(ip, default="")
```

Parameters


Parameter	Type	Required	Description
ip	String	Yes	The IP address.
default	String	No	If the type of the specified IP address fails to be identified, the value of this parameter is returned. You can leave this parameter empty.

Response
A value that indicates the type of an IP address is returned. Valid values: private, reserved, loopback, public, and allocated ripe ncc.
Examples
- Example 1: Identify the type of the specified IP address. The function returns loopback.
  - Raw log:
```
ip: 127.0.0.1
```
  - Transformation rule:
```
e_set("type",ip_type(v("ip")))
```
  - Result:
```
ip: 127.0.0.1
type: loopback
```
- Example 2: Identify the type of the specified IP address. The function returns private.
  - Raw log:
```
ip: 47.100.XX.XX
```
  - Transformation rule:
```
e_set("type",ip_type(v("ip")))
```
  - Result:
```
ip: 47.100.XX.XX
type: private
```
- Example 3: Identify the type of the specified IP address. The function returns public.
  - Raw log:
```
ip: 47.100.XX.XX
```
  - Transformation rule:
```
e_set("type",ip_type(v("ip")))
```
  - Result:
```
ip: 47.100.XX.XX
type: public
```
- Example 4: Identify the type of the specified IPv6 address. The function returns loopback.
  - Raw log:
```
ip: ::1
```
  - Transformation rule:
```
e_set("type",ip_type(v("ip")))
```
  - Result:
```
ip: ::1
type: loopback
```
- Example 5: Identify the type of the specified IPv6 address. The function returns allocated ripe ncc.
  - Raw log:
```
ip: 2001:0658:022a:cafe:0200::1
```
  - Transformation rule:
```
e_set("type",ip_type(v("ip")))
```
  - Result:
```
ip: 2001:0658:022a:cafe:0200::1
type: allocated ripe ncc
```

ip_makenet

The ip_makenet function is used to convert an IP address to a CIDR block.

Syntax

ip_makenet(ip, subnet_mask=None, default="")

Parameters


Parameter	Type	Required	Description
ip	String	Yes	The IP address.
subnet_mask	String	Yes	The subnet mask. Example: 255.255.255.0. Note If you set the ip parameter to an IP address range, you can leave the subnet_mask parameter empty.
default	String	No	If the specified IP address fails to be converted to a CIDR block, the value of this parameter is returned. You can leave this parameter empty.

Response
A CIDR block is returned.

Examples

Example 1: Convert an IP address to a CIDR block.

Raw log:
```
ip: 192.168.1.0
```

Transformation rule:

e_set("makenet",ip_makenet(v("ip"),"255.255.255.0"))

Result:
```
ip: 192.168.1.0
makenet: 192.168.1.0/24
```

Example 2: Convert an IP address range to a CIDR block.

Raw log:
```
ip: 192.168.1.0-192.168.1.255
```
Transformation rule:
```
e_set("makenet",ip_makenet(v("ip")))
```

Result:

ip: 192.168.1.0-192.168.1.255
makenet: 192.168.1.0/24

Example 3: Convert an IP address range to a CIDR block.

Raw log:
```
ip: 192.168.1.0/255.255.255.0
```
Transformation rule:
```
e_set("makenet",ip_makenet(v("ip")))
```

Result:

ip: 192.168.1.0/255.255.255.0
makenet: 192.168.1.0/24

ip_to_format

The ip_to_format function is used to convert the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.

Syntax

ip_to_format(cidr_subnet, want_prefix_len=0, default="")

Parameters


Parameter	Type	Required	Description
cidr_subnet	String	Yes	The CIDR block. Example: 192.168.1.0/24.
want_prefix_len	Int	No	The format of the output CIDR block. Default value: 0. Valid values: 0: returns the original CIDR block. 1: returns an IP address and the prefix length of the IP address. 2: returns an IP address and the netmask of the IP address. 3: returns an IP address range.
default	String	No	If the format of the specified CIDR block fails to be converted to the specified format, the value of this parameter is returned. You can leave this parameter empty.

Response
A CIDR block of the specified format is returned.
Examples
- Example 1: The format of a CIDR block is not converted.
  - Raw log:
```
ip: 192.168.1.0/24
```
  - Transformation rule:
```
e_set("strNormal",ip_to_format(v("ip"),0))
```
  - Result:
```
ip: 192.168.1.0/24
strNormal: 192.168.1.0/24
```
- Example 2: Convert the format of a CIDR block to the format that specifies the prefix length of the CIDR block.
  - Raw log:
```
 ip: 192.168.1.0/24
```
  - Transformation rule:
```
e_set("strNormal",ip_to_format(v("ip"),1))
```
  - Result:
```
ip: 192.168.1.0/24
strNormal: 192.168.1.0/24
```
- Example 3: Convert the format of a CIDR block to the format that specifies the netmask of the CIDR block.
  - Raw log:
```
ip: 192.168.1.0/24
```
  - Transformation rule:
```
e_set("strNormal",ip_to_format(v("ip"),2))
```
  - Result:
```
ip: 192.168.1.0/24
strNormal: 192.168.1.0/255.255.255.0
```
- Example 4: Convert a CIDR block to an IP address range.
  - Raw log:
```
ip: 192.168.1.0/24
```
  - Transformation rule:
```
e_set("strNormal",ip_to_format(v("ip"),3))
```
  - Result:
```
ip: 192.168.1.0/24
strNormal: 192.168.1.0-192.168.1.255
```

ip_overlaps

The ip_overlaps function is used to check whether two CIDR blocks overlap.

Syntax

ip_overlaps(cidr_subnet, cidr_subnet2, default="")

Parameters


Parameter	Type	Required	Description
cidr_subnet	String	Yes	The first CIDR block.
cidr_subnet2	String	Yes	The second CIDR block.
default	String	No	If the function cannot determine whether the CIDR blocks overlap, the value of this parameter is returned. You can leave this parameter empty.

Response
- If the specified CIDR blocks do not overlap, the function returns 0.
- If the specified CIDR blocks overlap at the end of the blocks, the function returns 1.
- If the specified CIDR blocks overlap at the start of the blocks, the function returns -1.

Examples

Example 1: The specified two CIDR blocks do not overlap.

Raw log:

cidr1: 192.168.0.0/23
cidr2: 192.168.2.0/24

Transformation rule:

e_set("overlaps",ip_overlaps(v("cidr1"),v("cidr2")))

Result:

cidr1: 192.168.0.0/23
cidr2: 192.168.2.0/24
overlaps: 0

Example 2: The specified two CIDR blocks overlap at the start of the blocks.

Raw log:

cidr1: 192.168.1.0/24
cidr2: 192.168.0.0/23

Transformation rule:

e_set("overlaps",ip_overlaps(v("cidr1"),v("cidr2")))

Result:

cidr1: 192.168.1.0/24
cidr2: 192.168.0.0/23
overlaps: -1

Example 3: The specified two CIDR blocks overlap at the end of the blocks.

Raw log:

cidr1: 192.168.0.0/23
cidr2: 192.168.1.0/24

Transformation rule:

e_set("overlaps",ip_overlaps(v("cidr1"),v("cidr2")))

Result:

cidr1: 192.168.0.0/23
cidr2: 192.168.1.0/24
overlaps: 1

ip2long

The ip2long function is used to convert an IP address to a value of the long type.

Syntax
```
ip2long(value,default=0)
```

Parameters


Parameter	Type	Required	Description
value	String	Yes	The value that you want to convert.
default	String	No	The value that is converted from an invalid IP address. You can use a custom value. Example: 0.

Response
The value that is converted from a valid IP address is returned. The value is of the long type.
Examples
- Example 1: Convert a valid IP address. This is the default scenario.
  - Raw log:
```
ip: 192.168.0.100
```
  - Transformation rule:
```
e_set("long_ip",ip2long(v("ip")))
```
  - Result:
```
ip: 192.168.0.100
long_ip: 167772160
```
- Example 2: Convert an invalid IP address.
  - Raw log:
```
ip: 47.100.XX.XX
```
  - Transformation rule:
```
e_set("long_ip",ip2long(v("ip"), "ignore"))
```
  - Result:
```
ip:47.100.XX.XX
long_ip:ignore
```

long2ip

The long2ip function is used to convert a value of the long type to an IP address.

Syntax
```
long2ip(value,default="")
```

Parameters


Parameter	Type	Required	Description
value	String	Yes	The value that you want to convert.
default	String	No	The empty string that is converted from an invalid value of the long type. You can use a custom string.

Response
The IP address that is converted from a valid value of the long type is returned.
Examples
- Example 1: Convert a valid value of the long type. This is the default scenario.
  - Raw log:
```
long: 167772160
```
  - Transformation rule:
```
e_set("ip",long2ip(v("long")))
```
  - Result:
```
long: 167772160
ip: 192.168.0.100
```
- Example 2: Convert an invalid value of the long type.
  - Raw log:
```
long: 4294967296
```
  - Transformation rule:
```
e_set("ip",long2ip(v("long")))
```
  - Result:
```
long: 4294967296
ip: 
```
- Example 3: Convert an invalid value of the long type and set the default parameter to a custom string.
  - Raw log:
```
long: 4294967296
```
  - Transformation rule:
```
e_set("ip",long2ip(v("long"),default="xxx"))
```
  - Result:
```
long: 4294967296
ip: xxx
```