Simple Log Service:General reference

Handle field name conflicts

During log upload or SPL execution, case-sensitive processing can cause field name conflicts. For example, a raw log might contain both Method and method fields. SPL resolves these conflicts differently depending on the scenario.

To avoid these situations, you can standardize the field names in your raw logs.

Conflicts in output results

For a complete list of reserved fields, see Reserved fields. SPL reads data from the LogGroup structure as input. For more information about the LogGroup definition, see Data encoding. If raw data written to Simple Log Service is not encoded in the standard LogGroup format, some reserved fields may be in LogContent instead of their standard location. SPL handles these reserved fields as follows:

• For the __source__, __topic__, __time__, and __time_ns_part__ fields, SPL reads their values from the standard LogGroup structure and ignores any fields with the same names in LogContent.

• For tag fields with the __tag__: prefix, SPL first attempts to read their values from the standard LogGroup structure. If a value is not found, SPL reads it from LogContent. For example, for the __tag__:ip field, SPL first attempts to read the field with the key ip from the LogTag list. If the field does not exist, SPL then reads the log field with the key __tag__:ip from the custom log fields in LogContent.

The __line__ field for full-text search

To filter raw logs in the console or when using the GetLogstoreLogs API operation, you can use the __line__ field.

Example

• Search for the keyword error in logs.

  * | where __line__ like '%error%'

• If a log contains a field named `__line__`, enclose the field name in backticks to reference it.

  * | where `__line__` ='20'

Inconsistent data types between old and new values

The original value of the input field is retained.

Example

• Example 1: A renamed field from a project instruction has a conflicting name.

  • SPL statement

    * 
    | extend status=cast(status as BIGINT) -- Convert the type of the status field to BIGINT.
    | project code=status -- The new value's type (BIGINT) differs from the old value's type (VARCHAR), so the old value is kept.

  • Input data

    status: '200'
    code: 'Success'

  • Output result

    code: 'Success'

• Example 2: An extracted field from a parse-json instruction has a conflicting name.

  • SPL statement

    * 
    | extend status=cast(status as BIGINT) -- Convert the type of the status field to BIGINT.
    | parse-json content -- The old type of status is BIGINT and the new type is VARCHAR. The old value is retained.

  • Input data

    status: '200'
    content: '{"status": "Success", "body": "this is test"}'

  • Output result

    content: '{"status": "Success", "body": "this is test"}'
    status: 200
    body: 'this is test'

Consistent data types between old and new values

If the input value is null, the new value is used. Otherwise, the behavior is determined by the mode parameter in the instruction, as defined in the following table.

Example

• Example 1: A renamed field from a project instruction has a conflicting name and the same type. The default mode is overwrite.

  • SPL statement

  * | project code=status -- The old and new types of code are both VARCHAR. The new value is used based on the overwrite mode.

  • Input data

    status: '200'
    code: 'Success'

  • Output result

    code: '200'

• Example 2: An extracted field from a parse-json instruction has a conflicting name and the same type. The default mode is overwrite.

  • SPL statement

    * | parse-json content -- The old and new types of status are both VARCHAR. The new value is used based on the overwrite mode.

  • Input data

    status: '200'
    content: '{"status": "Success", "body": "this is test"}'

  • Output result

    content: '{"status": "Success", "body": "this is test"}'
    status: 'Success'
    body: 'this is test'

• Example 3: An extracted field from a parse-json instruction has a conflicting name and the same type. The mode is set to preserve.

  • SPL statement

    * | parse-json -mode='preserve' content -- The old and new types of status are both VARCHAR. The old value is retained based on the preserve mode.

  • Input data

    status: '200'
    content: '{"status": "Success", "body": "this is test"}'

  • Output result

    content: '{"status": "Success", "body": "this is test"}'
    status: '200'
    body: 'this is test'

Data type conversion

Initial type

When SPL processes data, the initial data type of all input fields is VARCHAR, except for log time fields. If subsequent processing logic involves different data types, you must perform data type conversion.

Example

To filter access logs with a status code of 5xx, you must convert the status field to the BIGINT type before the comparison.

* -- The initial type of the status field is VARCHAR.
| where cast(status as BIGINT) >= 500 -- Convert the type of the status field to BIGINT, and then perform the comparison.

Type retention

During SPL data processing, after you use the extend instruction to convert the data type of a field, subsequent processing logic uses the converted data type.

Example

* -- A Logstore is used as input data. Except for time fields, all fields are initially of the VARCHAR type.
| where __source__='127.0.0.1' -- Filter on the __source__ field.
| extend status=cast(status as BIGINT) -- Convert the type of the status field to BIGINT.
| project status, content
| where status>=500 -- The type of the status field remains BIGINT, so it can be directly compared with the number 500.

Generating null values

During SPL data processing, null values are generated in the following two scenarios:

1. If a field used in an SPL expression does not exist in the input data, its value is treated as null during calculations.

2. If an exception occurs during the calculation of an SPL expression, the result is null. For example, a cast type conversion fails or an array index is out of bounds.

Example

1. If a field does not exist, its value is treated as null in calculations.

   • SPL statement

     * | extend withoutStatus=(status is null)

   • Input data

     # Entry 1
     status: '200'
     code: 'Success'
     
     # Entry 2
     code: 'Success'

   • Output result

     # Entry 1
     status: '200'
     code: 'Success'
     withoutStatus: false
     
     # Entry 2
     code: 'Success'
     withoutStatus: true

2. If an exception occurs during calculation, the result is null.

   • SPL statement

     *
     | extend code=cast(code as BIGINT) -- Failed to convert the code field to BIGINT.
     | extend values=json_parse(values)
     | extend values=cast(values as ARRAY(BIGINT))
     | extend last=arr[10] -- Array index out of bounds.

   • Input data

     status: '200'
     code: 'Success'
     values: '[1,2,3]'

   • Output result

     status: '200'
     code: null
     values: [1, 2, 3]
     last: null

Eliminating null values

To eliminate null values during calculations, you can use the COALESCE expression. This expression returns the first non-null value from a list of expressions. You can also set a default value to use if all expressions evaluate to null.

Example

Read the last element of an array. If the array is empty, the default value is 0.

• SPL statement

  *
  | extend values=json_parse(values)
  | extend values=cast(values as ARRAY(BIGINT))
  | extend last=COALESCE(values[3], values[2], values[1], 0)

• Input data

  # Entry 1
  values: '[1, 2, 3]'
  
  # Entry 2
  values: '[]'

• Output result

  # Entry 1
  values: [1, 2, 3]
  last: 3
  
  # Entry 2
  values: []
  last: 0

Single quotation marks

Single quotation marks are used to enclose a string literal. If the string literal contains a single quotation mark, you must use an additional single quotation mark to escape it.

Double quotation marks

Double quotation marks are used to enclose a field name. If the field name contains a double quotation mark, you must use an additional double quotation mark to escape it.