IP address parsing functions in SLS data transformation - Simple Log Service

Functions

Function	Description
geo_parse	Parses an IP address to get its country, province, and city.
ip_cidrmatch	Checks whether an IP address belongs to a Classless Inter-Domain Routing (CIDR) block.
ip_version	Checks whether an IP address is an IPv4 or IPv6 address.
ip_type	Checks whether an IP address is a private or public address.
ip_makenet	Converts an IP address to a CIDR block.
ip_to_format	Formats a CIDR block to use a prefix length or a subnet mask.
ip_overlaps	Checks whether two CIDR blocks overlap.
ip2long	Converts an IP address from a string to a long integer.
long2ip	Converts a long integer to an IP address string.

geo_parse

Parses an IP address to retrieve its country, province, and city.

Syntax

geo_parse(ip, ip_db="SLS-GeoIP", keep_fields=None, provider="ipip", ip_sep=None)

Parameters

Parameter	Parameter type	Required	Description
ip	String	Yes	The IP address to parse. To parse multiple IP addresses in a single string, use the ip_sep parameter to specify a separator.
ip_db	String	Yes	The IP address library used for parsing. Valid values: SLS-GeoIP: The built-in SLS IP database. Default. Updated daily. No additional configuration required. Custom IP database: Use `res_oss_file(endpoint, ak_id, ak_key, bucket, file, format='text', change_detect_interval=0,fetch_interval=2,refresh_retry_max=60,encoding='utf8',error='ignore')`. For parameter details, see res_oss_file.
keep_fields	Tuple	No	The keys to include in the returned result. If you use the built-in SLS IP database, the following information is returned by default: city: The city name. province: The province name. country: The country name. city_en: The administrative region code or English name of the city. province_en: The administrative region code or English name of the province. country_en: The code or English name of the country or region. isp: The name of the Internet Service Provider (ISP). lat: The latitude of the IP address location. lon: The longitude of the IP address location. If you use a custom IP database, the following information is returned by default: city: The city name. province: The province name. country: The country name. For example, `keep_fields=("city","country")` returns only the `city` and `country` fields. You can also use `keep_fields` to rename keys. For example, `(("city","cty"),("country","state"))` renames the output fields to `cty` and `state`.
provider	String	No	This parameter is valid only when ip_db is set to a custom IP database. Valid values: ipip: The default value. Uses the binary IP address database in the IPDB format provided by IPIP. To download the database, visit ipip. ip2location: Uses the global binary IP address database from IP2Location. To download the database, visit ip2location. Only binary packages are supported.
ip_sep	String	No	The separator for multiple IP addresses in a single string. Results are returned in JSON format. Default: None (no splitting).

Response

Returns a dictionary:

{
  "city": "...",
  "province":"...",
  "country": "..."
}

Examples

Example 1: Query data using the built-in SLS IP database.

Raw log
```
ip : 203.0.113.1
```
Transformation rule
```
e_set("geo", geo_parse(v("ip")))
```

Result

ip : 203.0.113.1
geo: {"city":"Hangzhou","province":"Zhejiang","country":"China","isp":"China Mobile","lat":30.16,"lon":120.12}

Example 2: Parse multiple IP addresses from a single field using the built-in SLS IP database.

Raw log

ip : 203.0.113.4, 192.0.2.2, 198.51.100.2

Transformation rule

e_set("geo", geo_parse(v("ip"), ip_sep=","))

Result

ip : 203.0.113.4, 192.0.2.2, 198.51.100.2
geo : {"203.0.113.4": {"country_en": "CN", "province_en": "330000", "city_en": "330200", "country": "China", "province": "Zhejiang", "city": "Ningbo", "isp": "China Telecom", "lat": 29.8782, "lon": 121.549}, "192.0.2.2": {"country_en": "CN", "province_en": "320000", "city_en": "321300", "country": "China", "province": "Jiangsu", "city": "Suqian", "isp": "China Telecom", "lat": 33.9492, "lon": 118.296}, "198.51.100.2": {"country_en": "CN", "province_en": "330000", "city_en": "330500", "country": "China", "province": "Zhejiang", "city": "Huzhou", "isp": "China Telecom", "lat": 30.8703, "lon": 120.093}}

Example 3: Query data using a custom IP database.

Raw log
```
ip : 203.0.113.1
```

Transformation rule

e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                 ak_id='your ak_id',
                                                 ak_key='your ak_key',
                                                 bucket='your bucket', file='ipipfree.ipdb',
                                                               format='binary',change_detect_interval=20)))

Result

ip : 203.0.113.1
geo : {"city": "Hangzhou", "province":"Zhejiang","country": "China"}

Example 4: Query with a custom IP database and rename output fields.

Raw log
```
ip : 203.0.113.1
```

Transformation rule

e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                 ak_id='your ak_id',
                                                 ak_key='your ak_key',
                                                 bucket='your bucket', file='ipipfree.ipdb',
                                                               format='binary',change_detect_interval=20),keep_fields=(("city","cty"),("country","state"),("province","pro"))))

Result

ip : 203.0.113.1
geo : { "state": "China","pro": "Zhejiang","cty": "Hangzhou"}

Example 5: Query with a custom IP database and select output fields.

Raw log
```
ip : 203.0.113.1
```

Transformation rule

e_set("geo",geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                 ak_id='your ak_id',
                                                 ak_key='your ak_key',
                                                 bucket='your bucket', file='ipipfree.ipdb',
                                                               format='binary',change_detect_interval=20),keep_fields=("country","province")))

Result

ip : 203.0.113.1
geo : { "country": "China","province": "Zhejiang"}

Example 6: Query with a custom IP database using the IP2Location provider.

Raw log
```
ip : 203.0.113.2
```

Transformation rule

e_set("geo", geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',ak_id="your ak_id", ak_key="your ak_secret", bucket='log-etl-staging', file='your ip2location bin file', format='binary', change_detect_interval=20),provider="ip2location"))

Result

ip : 203.0.113.2
geo : {"city":"Dearborn","province":"Michigan","country":"United States"}

Data transformation uses the open source IP2Location SDK for Python, which supports the fields listed below. If a field cannot be parsed, verify that your IP2Location database includes it.

country_short
country_long / Data transformation uses the country field instead.
region / Data transformation uses the province field instead.
city
isp
latitude
longitude
domain
zipcode
timezone
netspeed
idd_code
area_code
weather_code
weather_name
mcc
mnc
mobile_brand
elevation
usage_type

IP2Location Python SDK.

Example 7: Parse multiple IP addresses using a custom IP database.

Raw log

ip : 203.0.113.3, 192.0.2.1, 198.51.100.1

Transformation rule

e_set("geo", geo_parse(v("ip"), ip_db=res_oss_file(endpoint='http://oss-cn-hangzhou.aliyuncs.com',
                                                               ak_id="ak_id",
                                                               ak_key="ak_secret",
                                                               bucket='log-etl-staging',
                                                               file='calendar.csv/IP2LOCATION-LITE-DB3.BIN',
                                                               format='binary', change_detect_interval=20),
                                provider="ip2location", ip_sep=","))

Result

ip : 203.0.113.3, 192.0.2.1, 198.51.100.1
geo : {"203.0.113.3": {"city": "Dearborn", "province": "Michigan", "country": "United States"}, "192.0.2.1": {"city": "Hangzhou", "province": "Zhejiang", "country": "China"}, "198.51.100.1": {"city": "Hangzhou", "province": "Zhejiang", "country": "China"}}

ip_cidrmatch

Checks whether an IP address belongs to a CIDR block. Returns True if matched, False otherwise. Supports IPv4 and IPv6.

Syntax

ip_cidrmatch(cidr_subnet, ip,default="")

Parameters

Parameter Name	Parameter type	Required	Description
cidr_subnet	String	Yes	The CIDR block. Example: 192.168.1.0/24.
ip	String	Yes	The IP address to check.
default	String	No	The fallback value if the IP address cannot be matched. Can be empty.

Response

Returns True if the IP address matches the CIDR block. Otherwise, it returns False.

Examples

Example 1: An IPv4 address matches the CIDR block and the function returns True.

Raw log

cidr_subnet: 192.168.1.0/24
ip: 192.168.1.100

Transformation rule

e_set("is_belong",ip_cidrmatch(v("cidr_subnet"),v("ip")))

Result

cidr_subnet: 192.168.1.0/24
ip: 192.168.1.100
is_belong: true

Example 2: An IPv4 address does not match the CIDR block and the function returns False.

Raw log

cidr_subnet: 192.168.1.0/24
ip: 10.10.1.100

Transformation rule

e_set("is_belong",ip_cidrmatch(v("cidr_subnet"),v("ip")))

Result

cidr_subnet: 192.168.1.0/24
ip: 10.10.1.100
is_belong: false

Example 3: The IP address cannot be matched with the CIDR block and the function returns unknown.

Raw log
```
cidr_subnet: 192.168.1.0/24
ip: a
```

Transformation rule

e_set("is_belong",ip_cidrmatch(v("cidr_subnet"),v("ip"),default="unknown"))

Result

cidr_subnet: 192.168.1.0/24
ip: a
is_belong: unknown

ip_version

Returns the version of an IP address: IPv4 or IPv6.

Syntax
```
ip_version(ip,default="")
```

Parameters

Parameter	Parameter type	Required	Description
ip	String	Yes	The IP address.
default	String	No	The fallback value if the version cannot be determined. Can be empty.

Response

Returns IPv4 or IPv6.
Examples
- Example 1: Check the version of an IPv4 address.
  - Raw log
```
ip: 192.168.1.100
```
  - Transformation rule
```
e_set("version",ip_version(v("ip")))
```
  - Result
```
ip: 192.168.1.100
version: IPv4
```
- Example 2: Check the version of an IPv6 address.
  - Raw log
```
ip: ::1
```
  - Transformation rule
```
e_set("version",ip_version(v("ip")))
```
  - Result
```
ip: ::1
version: IPv6
```

ip_type

Returns the type of an IP address: private, reserved, loopback, public, or allocated ripe ncc.

Syntax
```
ip_type(ip,default="")
```

Parameters

Parameter	Type	Required	Description
ip	String	Yes	The IP address.
default	String	No	The fallback value if the type cannot be determined. Can be empty.

Response

Returns the IP address type. Valid values are private, reserved, loopback, public, and allocated ripe ncc.
Examples
- Example 1: Check a loopback address.
  - Raw log
```
ip: 127.0.0.1
```
  - Transformation rule
```
e_set("type",ip_type(v("ip")))
```
  - Result
```
ip: 127.0.0.1
type: loopback
```
- Example 2: Checks whether an IP address is private.
  - Raw log
```
ip: 47.100.XX.XX
```
  - Transformation rule
```
e_set("type",ip_type(v("ip")))
```
  - Result
```
ip: 47.100.XX.XX
type: private
```
- Example 3: Check a public address.
  - Raw log
```
ip: 47.100.XX.XX
```
  - Transformation rule
```
e_set("type",ip_type(v("ip")))
```
  - Result
```
ip: 47.100.XX.XX
type: public
```
- Example 4: Check the type of an IPv6 address that is a loopback address.
  - Raw log
```
ip: ::1
```
  - Transformation rule
```
e_set("type",ip_type(v("ip")))
```
  - Result
```
ip: ::1
type: loopback
```
- Example 5: Check the type of an IPv6 address that is an allocated ripe ncc address.
  - Raw log
```
ip: 2001:0658:022a:cafe:0200::1
```
  - Transformation rule
```
e_set("type",ip_type(v("ip")))
```
  - Result
```
ip: 2001:0658:022a:cafe:0200::1
type: allocated ripe ncc
```

ip_makenet

Converts an IP address to a CIDR block.

Syntax

ip_makenet(ip, subnet_mask=None,default="")

Parameters

Parameter name	Parameter type	Required	Description
ip	String	Yes	The IP address.
subnet_mask	String	Yes	The subnet mask. Example: 255.255.255.0. Note If you specify an IP address range for the ip parameter, you can leave the subnet_mask parameter empty.
default	String	No	The fallback value if conversion fails. Can be empty.

Response

Returns a CIDR block.

Examples

Example 1: Convert an IP address to a CIDR block.

Raw log
```
ip: 192.168.1.0
```

Transformation rule

e_set("makenet",ip_makenet(v("ip"),"255.255.255.0"))

Result
```
ip: 192.168.1.0
makenet: 192.168.1.0/24
```

Example 2: Convert an IP address range to a CIDR block.

Raw log
```
ip: 192.168.1.0-192.168.1.255
```
Transformation rule
```
e_set("makenet",ip_makenet(v("ip")))
```

Result

ip: 192.168.1.0-192.168.1.255
makenet: 192.168.1.0/24

Example 3: Convert an IP address range to a CIDR block.

Raw log
```
ip: 192.168.1.0/255.255.255.0
```
Transformation rule
```
e_set("makenet",ip_makenet(v("ip")))
```

Result

ip: 192.168.1.0/255.255.255.0
makenet: 192.168.1.0/24

ip_to_format

Converts a CIDR block to a specified format, such as a prefix length, a subnet mask, or an IP address range.

Syntax

ip_to_format(cidr_subnet, want_prefix_len=0,default="")

Parameters

Parameter name	Type	Required	Description
cidr_subnet	String	Yes	The CIDR block. Example: 192.168.1.0/24
want_prefix_len	Int	No	The output format. Default: 0. 0: Unformatted. 1: Prefix format. 2: Netmask format. 3: IP address range.
default	String	No	The fallback value if formatting fails. Can be empty.

Response

Returns an IP address in the specified format.

Examples

Example 1: Return the input CIDR block without changing the format.

Raw log
```
ip: 192.168.1.0/24
```

Transformation rule

e_set("strNormal",ip_to_format(v("ip"),0))

Result

ip: 192.168.1.0/24
strNormal: 192.168.1.0/24

Example 2: Format a CIDR block to use a prefix.

Raw log
```
 ip: 192.168.1.0/24
```

Transformation rule

e_set("strNormal",ip_to_format(v("ip"),1))

Result

ip: 192.168.1.0/24
strNormal: 192.168.1.0/24

Example 3: Format a CIDR block to use a subnet mask.

Raw log
```
ip: 192.168.1.0/24
```

Transformation rule

e_set("strNormal",ip_to_format(v("ip"),2))

Result

ip: 192.168.1.0/24
strNormal: 192.168.1.0/255.255.255.0

Example 4: Format a CIDR block as an IP address range.

Raw log
```
ip: 192.168.1.0/24
```

Transformation rule

e_set("strNormal",ip_to_format(v("ip"),3))

Result

ip: 192.168.1.0/24
strNormal: 192.168.1.0-192.168.1.255

ip_overlaps

Determines whether two CIDR blocks overlap.

Syntax

ip_overlaps(cidr_subnet, cidr_subnet2,default="")

Parameters

Parameter name	Parameter type	Required	Description
cidr_subnet	String	Yes	The first CIDR block.
cidr_subnet2	String	Yes	The second CIDR block.
default	String	No	The fallback value if overlap cannot be determined. Can be empty.

Response
- The function returns 0 if the two CIDR blocks do not overlap.
- The function returns 1 if the two CIDR blocks overlap at the end.
- The function returns -1 if the two CIDR blocks overlap at the start.

Examples

Example 1: The two CIDR blocks do not overlap.

Raw log

cidr1: 192.168.0.0/23
cidr2: 192.168.2.0/24

Transformation rule

e_set("overlaps",ip_overlaps(v("cidr1"),v("cidr2")))

Result

cidr1: 192.168.0.0/23
cidr2: 192.168.2.0/24
overlaps: 0

Example 2: The two CIDR blocks overlap at the start.

Raw log

cidr1: 192.168.1.0/24
cidr2: 192.168.0.0/23

Transformation rule

e_set("overlaps",ip_overlaps(v("cidr1"),v("cidr2")))

Result

cidr1: 192.168.1.0/24
cidr2: 192.168.0.0/23
overlaps: -1

Example 3: The two CIDR blocks overlap at the end.

Raw log

cidr1: 192.168.0.0/23
cidr2: 192.168.1.0/24

Transformation rule

e_set("overlaps",ip_overlaps(v("cidr1"),v("cidr2")))

Result

cidr1: 192.168.0.0/23
cidr2: 192.168.1.0/24
overlaps: 1

ip2long

Converts an IP address from a string to a long integer.

Syntax
```
ip2long(value,default=0)
```

Parameters

Parameter Name	Parameter type	Required	Description
value	String	Yes	The value to convert.
default	String	No	The fallback value for an invalid IP address. Example: 0.

Response

Returns the long integer representation of a valid IP address.

Examples

Example 1: Default case

Raw log
```
ip: 192.168.0.100
```
Transformation rule
```
e_set("long_ip",ip2long(v("ip")))
```
Result
```
ip: 192.168.0.100
long_ip: 167772160
```

Example 2: Invalid IP address

Raw log
```
ip: 47.100.XX.XX
```

Transformation rule

e_set("long_ip",ip2long(v("ip"), "ignore"))

Result
```
ip:47.100.XX.XX
long_ip:ignore
```

long2ip

Converts a long integer to a string-format IP address.

Syntax
```
long2ip(value,default="")
```

Parameters

Parameter Name	Parameter type	Required	Description
value	String	Yes	The value to convert.
default	String	No	The fallback value for an invalid long integer. Default: empty string.

Response

Returns the string-format IP address converted from a valid long integer.
Examples
- Example 1: Default case
  - Raw log
```
long: 167772160
```
  - Transformation rule
```
e_set("ip",long2ip(v("long")))
```
  - Result
```
long: 167772160
ip: 192.168.0.100
```
- Example 2: Failed conversion from long integer to IP address
  - Raw log
```
long: 4294967296
```
  - Transformation rule
```
e_set("ip",long2ip(v("long")))
```
  - Result
```
long: 4294967296
ip: 
```
- Example 3: Custom error handling for a failed conversion
  - Raw log
```
long: 4294967296
```
  - Transformation rule
```
e_set("ip",long2ip(v("long"),default="xxx"))
```
  - Result
```
long: 4294967296
ip: xxx
```