All Products
Search
Document Center

Simple Log Service:Multi-set operations

Last Updated:Aug 25, 2023

Simple Log Service supports associated monitoring for multiple data sets.

Set operations

In the alert monitoring system of Simple Log Service, a query result is considered as a set. Associated monitoring of multiple sets is supported.

Important
  • Simple Log Service supports associated monitoring for up to three sets.

  • By default, only the first 1,000 rows of data in a query result are selected for set operations. If you specify three query statements and do not set the Set Operations parameter to No Merge, only the first 100 rows of data in each query result are selected.

  • If you specify three query statements, the returned three query results are considered as three sets. In this case, the first two sets are used for a set operation, and a result is returned. Then, this result and the third set are used for a set operation. Examples:

    • Set A LEFT JOIN Set B LEFT JOIN Set C: The LEFT JOIN operation is first performed on Set A and Set B, and a result is returned. Then, the LEFT JOIN operation is performed on the result and Set C.

    • Set A JOIN Set B INNER JOIN Set C: The JOIN operation is first performed on Set A and Set B, and a result is returned. Then, the INNER JOIN operation is performed on the result and Set C.

    • Set A LEFT EXCLUDE JOIN Set B No Merge Set C: The LEFT EXCLUDE JOIN operation is performed on Set A and Set B, and Set C is ignored.

Multi-set operations

Set operation

Illustration

Description

No Merge

No Merge

The specified two sets are not associated.

Set A is a query result. The fields in Set B are only used as alert template variables that can be quoted in alert notifications.

CROSS JOIN

None

Arbitrary data from Set A combines with arbitrary data from Set B. This set operation is used to filter data for further evaluation.

JOIN

JOIN

Data in Set B is added to Set A and aligned by field.

INNER JOIN

INNER JOIN

Only the data that exists in Set B is retained in Set A. Set B is the whitelist of Set A.

LEFT JOIN

LEFT JOIN

Partial data from Set B is complemented to Set A. Set B is the dimension table of Set A.

RIGHT JOIN

RIGHT JOIN

Partial data from Set A is complemented to Set B. Set A is the dimension table of Set B.

FULL JOIN

FULL JOIN

Set A and Set B complement each other.

LEFT EXCLUDE JOIN

LEFT EXCLUDE JOIN

Data that exists in Set B is deleted in Set A. Set B is the blacklist of Set A.

RIGHT EXCLUDE JOIN

RIGHT EXCLUDE JOIN

Data that exists in Set A is deleted in Set B. Set A is the blacklist of Set B.

Examples

  • No Merge

    • Requirement

      NGINX access logs are monitored. If the number of 5xx error responses exceeds 500 every 15 minutes, an alert is triggered and an alert notification is sent. The information of the hosts that meet the trigger condition is included in the alert notification.

    • ConfigurationsNo Merge

    • Results

      • Result of Query Statistics 0

        The number of 5xx error responses within 15 minutes is calculated.

        cnt

        1234

      • Result of Query Statistics 1

        The top 5 hosts with the most 5xx error responses within 15 minutes and the related number of errors are calculated.

        host

        pv

        host1

        60

        host2

        55

        host3

        47

        host4

        45

        hosg5

        30

      • Result of the set operation

        If you set the Set Operations parameter to No Merge, the result of the set operation is the result of Query Statistics 0.

  • JOIN

    • Example 1

      • Requirement

        Two Logstores are used to store NGINX access logs. One Logstore resides in the China (Beijing) region, and the other Logstore resides in the China (Shanghai) region. The number of hosts that have more than 30 5xx error responses is calculated every 15 minutes. If the number of hosts that meet the specified condition in the two Logstores exceeds 10, an alert is triggered.

      • ConfigurationsJOIN

      • Results

        • Result of Query Statistics 0

          The hosts whose number of 5xx error responses within 15 minutes exceeds 30 and the related number of errors are calculated.

          host

          pv

          host1

          60

          host2

          55

          host3

          47

          host4

          45

          hosg5

          31

        • Result of Query Statistics 1

          The hosts whose number of 5xx error responses within 15 minutes exceeds 30 and the related number of errors are calculated.

          host

          pv

          hosta

          70

          hostb

          45

          hostc

          44

          hostd

          42

        • Result of the set operation

          If you set the Set Operations parameter to JOIN, the result of the set operation is displayed in the following table.

          host

          pv

          host1

          60

          host2

          55

          host3

          47

          host4

          45

          hosg5

          31

          hosta

          70

          hostb

          45

          hostc

          44

          hostd

          42

    • Other examples

      • If the fields in two query results are not completely matched, the non-matching fields are left blank after a set operation is performed.

        • Result of Query Statistics 0

          a

          b

          a1

          b1

          a2

          b2

        • Result of Query Statistics 1

          b

          c

          b1

          c1

          b2

          c2

        • Result of the set operation

          a

          b

          c

          a1

          b1

          None

          a2

          b2

          None

          None

          b1

          c1

          None

          b2

          c2

      • If you specify three query statements, the results of three Query Statistics are used for set operations. In this case, the results of Query Statistics 0 and Query Statistics 1 are used for a set operation, and a result is returned. Then, the JOIN operation is performed on the preceding result and the result of Query Statistics 2.

        • Result of Query Statistics 0

          a

          b

          a1

          b1

          a2

          b2

        • Result of Query Statistics 1

          a

          b

          a1

          b11

          a2

          b22

          a3

          b33

        • Operation result of the Query Statistics 0 result and Query Statistics 1 result

          If you set the Set Operations parameter to INNER JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.

          a

          $0.b

          $1.b

          a1

          b1

          b11

          a2

          b2

          b22

        • Result of Query Statistics 2

          a

          b

          a3

          b333

          a4

          b444

        • Result of the set operation

          If you set the Set Operations parameter to JOIN, the result of the set operation is displayed in the following table.

          Note

          The b field in the result of Query Statistics 2 is aligned with the $0.b field.

          a

          $0.b

          $1.b

          a1

          b1

          b11

          a2

          b2

          b22

          a3

          b333

          None

          a4

          b444

          None

  • INNER JOIN

    • Example 1

      • Requirement

        The number of 5xx error responses in each specified bucket is monitored. If the number of 5xx error responses within 15 minutes exceeds 1,000, an alert is triggered. To meet this requirement, you must add resource data to maintain the bucket whitelist.

      • ConfigurationsINNER JOIN

      • Results

        • Result of Query Statistics 0

          The buckets whose number of 5xx error responses exceeds 1,000 within 15 minutes are calculated.

          bucket

          pv

          bucket_01

          1600

          bucket_02

          1550

          bucket_03

          1470

          bucket_04

          1450

        • Result of Query Statistics 1

          The resource data of the specified buckets.

          bucket

          desc

          bucket_03

          for dev team

          bucket_04

          for test team

          bucket_05

          for service team

          bucket_06

          for support team

        • Result of the set operation

          If you set the Set Operations parameter to INNER JOIN,$0.bucket == $1.bucket, the result of the set operation is displayed in the following table.

          bucket

          pv

          desc

          bucket_03

          47

          for dev team

          bucket_04

          45

          for test team

    • Example 2

      • Requirement

        Two Logstores are used to store NGINX access logs. One Logstore resides in the China (Beijing) region, and the other Logstore resides in the China (Shanghai) region. The clients that have more than 30 5xx error responses are calculated every 15 minutes. If both Logstores have 5xx error responses, and the number of errors in the China (Beijing) region is greater than that in the China (Shanghai) region, an alert is triggered.

      • ConfigurationsINNER JOIN

      • Results

        • Result of Query Statistics 0

          The clients whose number of 500 error responses within 15 minutes exceed 30 and the related number of errors are calculated. These errors occur in the China (Beijing) region.

          client_ip

          pv

          192.0.2.4

          60

          192.0.2.5

          55

          192.0.2.6

          47

          192.0.2.7

          45

          192.0.2.8

          31

        • Result of Query Statistics 1

          The clients whose number of 500 error responses within 15 minutes exceed 30 and the related number of errors are calculated. These errors occur in the China (Shanghai) region.

          client_ip

          pv

          192.0.2.5

          70

          192.0.2.6

          45

          192.0.2.7

          44

          192.0.2.8

          42

          192.0.2.9

          42

        • Result of the set operation

          If you set the Set Operations parameter to INNER JOIN,$0.client_ip == $1.client_ip,$0.pv > $1.pv, the result of the set operation is displayed in the following table.

          client_ip

          pv

          192.0.2.6

          47

          192.0.2.7

          45

    • Other examples

      For example, a field in the result of Query Statistics 0 and a field in the result of Query Statistics 1 are not associated but share the same name. In this case, the two fields in the result of the specified set operation are automatically prefixed by $0 and $1.

      • Result of Query Statistics 0

        a

        b

        c

        d

        a1

        b1

        c1

        d1

        a2

        b2

        c2

        d2

        a3

        b3

        c3

        d3

      • Result of Query Statistics 1

        a

        b

        c

        a1

        b11

        c11

        a2

        b22

        c22

      • Result of the set operation

        If you set the Set Operations parameter to INNER JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.

        a

        $0.b

        $0.c

        d

        $1.b

        $1.c

        a1

        b1

        c1

        d1

        b11

        c11

        a2

        b2

        c2

        d2

        b22

        c22

  • LEFT EXCLUDE JOIN

    • Requirement

      The number of 5xx error responses in each bucket that is not specified is monitored. If the number of 5xx errors within 15 minutes exceeds 1,000, an alert is triggered. To meet this requirement, you must add resource data to maintain the bucket blacklist.

    • ConfigurationsLEFT EXCLUDE JOIN

    • Results

      • Result of Query Statistics 0

        The buckets whose number of 500 error responses exceeds 1,000 within 15 minutes are calculated.

        bucket

        pv

        bucket_01

        60

        bucket_02

        55

        bucket_03

        47

        bucket_04

        45

      • Result of Query Statistics 1

        The resource data of the specified buckets.

        bucket

        desc

        bucket_03

        for dev team

        bucket_04

        for test team

      • Result of the set operation

        If you set the Set Operations parameter to LEFT EXCLUDE JOIN,$0.bucket == $1.bucket, the result of the set operation is displayed in the following table.

        bucket

        pv

        bucket_01

        60

        bucket_02

        55

  • RIGHT EXCLUDE JOIN

    • Requirement

      The number of 5xx error responses in each bucket that is not specified is monitored. If the number of 5xx error responses within 15 minutes exceeds 1,000, an alert is triggered. To meet this requirement, you must add resource data to maintain the bucket blacklist.

    • ConfigurationsRIGHT EXCLUDE JOIN

    • Results

      • Result of Query Statistics 0

        The resource data of the specified buckets.

        bucket

        desc

        bucket_03

        for dev team

        bucket_04

        for test team

      • Result of Query Statistics 1

        The buckets whose number of 500 error responses exceeds 1,000 within 15 minutes are calculated.

        bucket

        pv

        bucket_01

        60

        bucket_02

        55

        bucket_03

        47

        bucket_04

        45

      • Result of the set operation

        If you set the Set Operations parameter to RIGHT EXCLUDE JOIN,$0.bucket == $1.bucket, the result of the set operation is displayed in the following table.

        bucket

        pv

        bucket_01

        60

        bucket_02

        55

  • CROSS JOIN

    • Example 1

      • Requirements

        Object Storage Service (OSS) access logs and Server Load Balancer (SLB) access logs are monitored at the same time. The number of 4xx error responses in OSS and the number of 5xx error responses in SLB are calculated every 15 minutes. If the total number of errors reaches 1,000, an alert is triggered.

      • ConfigurationsCROSS JOIN

      • Results

        • Result of Query Statistics 0

          The number of 4xx error responses within 15 minutes in OSS is calculated.

          pv

          890

        • Result of Query Statistics 1

          The number of 5xx error responses within 15 minutes in SLB is calculated.

          pv

          567

        • Result of the set operation

          If you set the Set Operations parameter to CROSS JOIN, the result of the set operation is displayed in the following table.

          $0.pv

          $1.pv

          890

          567

    • Other examples

      • Result of Query Statistics 0

        a

        b

        a1

        b1

        a2

        b2

        a5

        b5

      • Result of Query Statistics 1

        a

        c

        a1

        c1

        a3

        c3

      • Result of the set operation

        If you set the Set Operations parameter to CROSS JOIN, the result of the set operation is displayed in the following table.

        $0.a

        b

        $1.a

        c

        a1

        b1

        a1

        c1

        a1

        b1

        a3

        c3

        a2

        b2

        a1

        c1

        a2

        b2

        a3

        c3

        a5

        b5

        a1

        c1

        a5

        b5

        a3

        c3

  • LEFT JOIN

    • Result of Query Statistics 0

      a

      b

      a1

      b1

      a2

      b2

      a3

      b3

    • Result of Query Statistics 1

      a

      b

      c

      a1

      b11

      c1

      a2

      b22

      c2

    • Result of the set operation

      If you set the Set Operations parameter to LEFT JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.

      a

      $0.b

      $1.b

      c

      a1

      b1

      b11

      c1

      a2

      b2

      b22

      c2

      a3

      b3

      None

      None

  • RIGHT JOIN

    • Result of Query Statistics 0

      a

      b

      c

      a1

      b11

      c1

      a2

      b22

      c2

    • Result of Query Statistics 1

      a

      b

      a1

      b1

      a2

      b2

      a3

      b3

    • Result of the set operation

      If you set the Set Operations parameter to RIGHT JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.

      a

      $0.b

      c

      $1.b

      a1

      b11

      c1

      b1

      a2

      b22

      c2

      b2

      a3

      None

      None

      b3

  • FULL JOIN

    • Result of Query Statistics 0

      a

      b

      c

      a1

      b1

      c1

      a2

      b2

      c2

      a5

      b5

      c3

    • Result of Query Statistics 1

      a

      b

      d

      a1

      b11

      d1

      a2

      b22

      d2

      a3

      b33

      d3

    • Result of the set operation

      If you set the Set Operations parameter to FULL JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.

      a

      $0.b

      c

      $1.b

      d

      a1

      b1

      c1

      b11

      d1

      a2

      b2

      c2

      b22

      d2

      a5

      b5

      c3

      None

      None

      a3

      None

      None

      b33

      d3