Multi-set operations - Log Service - Alibaba Cloud Documentation Center

Simple Log Service supports associated monitoring for multiple data sets.

Set operations

In the alert monitoring system of Simple Log Service, a query result is considered as a set. Associated monitoring of multiple sets is supported.

Important

Simple Log Service supports associated monitoring for up to three sets.
By default, only the first 1,000 rows of data in a query result are selected for set operations. If you specify three query statements and do not set the Set Operations parameter to No Merge, only the first 100 rows of data in each query result are selected.
If you specify three query statements, the returned three query results are considered as three sets. In this case, the first two sets are used for a set operation, and a result is returned. Then, this result and the third set are used for a set operation. Examples:
- Set A LEFT JOIN Set B LEFT JOIN Set C: The LEFT JOIN operation is first performed on Set A and Set B, and a result is returned. Then, the LEFT JOIN operation is performed on the result and Set C.
- Set A JOIN Set B INNER JOIN Set C: The JOIN operation is first performed on Set A and Set B, and a result is returned. Then, the INNER JOIN operation is performed on the result and Set C.
- Set A LEFT EXCLUDE JOIN Set B No Merge Set C: The LEFT EXCLUDE JOIN operation is performed on Set A and Set B, and Set C is ignored.

Set operation	Illustration	Description
No Merge		The specified two sets are not associated. Set A is a query result. The fields in Set B are only used as alert template variables that can be quoted in alert notifications.
CROSS JOIN	None	Arbitrary data from Set A combines with arbitrary data from Set B. This set operation is used to filter data for further evaluation.
JOIN		Data in Set B is added to Set A and aligned by field.
INNER JOIN		Only the data that exists in Set B is retained in Set A. Set B is the whitelist of Set A.
LEFT JOIN		Partial data from Set B is complemented to Set A. Set B is the dimension table of Set A.
RIGHT JOIN		Partial data from Set A is complemented to Set B. Set A is the dimension table of Set B.
FULL JOIN		Set A and Set B complement each other.
LEFT EXCLUDE JOIN		Data that exists in Set B is deleted in Set A. Set B is the blacklist of Set A.
RIGHT EXCLUDE JOIN		Data that exists in Set A is deleted in Set B. Set A is the blacklist of Set B.

Examples

No Merge
- Requirement
  NGINX access logs are monitored. If the number of 5xx error responses exceeds 500 every 15 minutes, an alert is triggered and an alert notification is sent. The information of the hosts that meet the trigger condition is included in the alert notification.
- Configurations
- Results
  - Result of Query Statistics 0
    The number of 5xx error responses within 15 minutes is calculated.
    cnt
    1234
  - Result of Query Statistics 1
    The top 5 hosts with the most 5xx error responses within 15 minutes and the related number of errors are calculated.
    host
    pv
    host1
    60
    host2
    55
    host3
    47
    host4
    45
    hosg5
    30
  - Result of the set operation
    If you set the Set Operations parameter to No Merge, the result of the set operation is the result of Query Statistics 0.
JOIN
- Example 1
  - Requirement
    Two Logstores are used to store NGINX access logs. One Logstore resides in the China (Beijing) region, and the other Logstore resides in the China (Shanghai) region. The number of hosts that have more than 30 5xx error responses is calculated every 15 minutes. If the number of hosts that meet the specified condition in the two Logstores exceeds 10, an alert is triggered.
  - Configurations
  - Results
    - Result of Query Statistics 0
      The hosts whose number of 5xx error responses within 15 minutes exceeds 30 and the related number of errors are calculated.
      host
      pv
      host1
      60
      host2
      55
      host3
      47
      host4
      45
      hosg5
      31
    - Result of Query Statistics 1
      The hosts whose number of 5xx error responses within 15 minutes exceeds 30 and the related number of errors are calculated.
      host
      pv
      hosta
      70
      hostb
      45
      hostc
      44
      hostd
      42
    - Result of the set operation
      If you set the Set Operations parameter to JOIN, the result of the set operation is displayed in the following table.
      host
      pv
      host1
      60
      host2
      55
      host3
      47
      host4
      45
      hosg5
      31
      hosta
      70
      hostb
      45
      hostc
      44
      hostd
      42
- Other examples
  - If the fields in two query results are not completely matched, the non-matching fields are left blank after a set operation is performed.
    - Result of Query Statistics 0
      a
      b
      a1
      b1
      a2
      b2
    - Result of Query Statistics 1
      b
      c
      b1
      c1
      b2
      c2
    - Result of the set operation
      a
      b
      c
      a1
      b1
      None
      a2
      b2
      None
      None
      b1
      c1
      None
      b2
      c2
  - If you specify three query statements, the results of three Query Statistics are used for set operations. In this case, the results of Query Statistics 0 and Query Statistics 1 are used for a set operation, and a result is returned. Then, the JOIN operation is performed on the preceding result and the result of Query Statistics 2.
    - Result of Query Statistics 0
      a
      b
      a1
      b1
      a2
      b2
    - Result of Query Statistics 1
      a
      b
      a1
      b11
      a2
      b22
      a3
      b33
    - Operation result of the Query Statistics 0 result and Query Statistics 1 result
      If you set the Set Operations parameter to INNER JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.
      a
      $0.b
      $1.b
      a1
      b1
      b11
      a2
      b2
      b22
    - Result of Query Statistics 2
      a
      b
      a3
      b333
      a4
      b444
    - Result of the set operation
      If you set the Set Operations parameter to JOIN, the result of the set operation is displayed in the following table.
      Note
      The b field in the result of Query Statistics 2 is aligned with the $0.b field.
      a
      $0.b
      $1.b
      a1
      b1
      b11
      a2
      b2
      b22
      a3
      b333
      None
      a4
      b444
      None
INNER JOIN
- Example 1
  - Requirement
    The number of 5xx error responses in each specified bucket is monitored. If the number of 5xx error responses within 15 minutes exceeds 1,000, an alert is triggered. To meet this requirement, you must add resource data to maintain the bucket whitelist.
  - Configurations
  - Results
    - Result of Query Statistics 0
      The buckets whose number of 5xx error responses exceeds 1,000 within 15 minutes are calculated.
      bucket
      pv
      bucket_01
      1600
      bucket_02
      1550
      bucket_03
      1470
      bucket_04
      1450
    - Result of Query Statistics 1
      The resource data of the specified buckets.
      bucket
      desc
      bucket_03
      for dev team
      bucket_04
      for test team
      bucket_05
      for service team
      bucket_06
      for support team
    - Result of the set operation
      If you set the Set Operations parameter to INNER JOIN,$0.bucket == $1.bucket, the result of the set operation is displayed in the following table.
      bucket
      pv
      desc
      bucket_03
      47
      for dev team
      bucket_04
      45
      for test team
- Example 2
  - Requirement
    Two Logstores are used to store NGINX access logs. One Logstore resides in the China (Beijing) region, and the other Logstore resides in the China (Shanghai) region. The clients that have more than 30 5xx error responses are calculated every 15 minutes. If both Logstores have 5xx error responses, and the number of errors in the China (Beijing) region is greater than that in the China (Shanghai) region, an alert is triggered.
  - Configurations
  - Results
    - Result of Query Statistics 0
      The clients whose number of 500 error responses within 15 minutes exceed 30 and the related number of errors are calculated. These errors occur in the China (Beijing) region.
      client_ip
      pv
      192.0.2.4
      60
      192.0.2.5
      55
      192.0.2.6
      47
      192.0.2.7
      45
      192.0.2.8
      31
    - Result of Query Statistics 1
      The clients whose number of 500 error responses within 15 minutes exceed 30 and the related number of errors are calculated. These errors occur in the China (Shanghai) region.
      client_ip
      pv
      192.0.2.5
      70
      192.0.2.6
      45
      192.0.2.7
      44
      192.0.2.8
      42
      192.0.2.9
      42
    - Result of the set operation
      If you set the Set Operations parameter to INNER JOIN,$0.client_ip == $1.client_ip,$0.pv > $1.pv, the result of the set operation is displayed in the following table.
      client_ip
      pv
      192.0.2.6
      47
      192.0.2.7
      45
- Other examples
  For example, a field in the result of Query Statistics 0 and a field in the result of Query Statistics 1 are not associated but share the same name. In this case, the two fields in the result of the specified set operation are automatically prefixed by $0 and $1.
  - Result of Query Statistics 0
    a
    b
    c
    d
    a1
    b1
    c1
    d1
    a2
    b2
    c2
    d2
    a3
    b3
    c3
    d3
  - Result of Query Statistics 1
    a
    b
    c
    a1
    b11
    c11
    a2
    b22
    c22
  - Result of the set operation
    If you set the Set Operations parameter to INNER JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.
    a
    $0.b
    $0.c
    d
    $1.b
    $1.c
    a1
    b1
    c1
    d1
    b11
    c11
    a2
    b2
    c2
    d2
    b22
    c22
LEFT EXCLUDE JOIN
- Requirement
  The number of 5xx error responses in each bucket that is not specified is monitored. If the number of 5xx errors within 15 minutes exceeds 1,000, an alert is triggered. To meet this requirement, you must add resource data to maintain the bucket blacklist.
- Configurations
- Results
  - Result of Query Statistics 0
    The buckets whose number of 500 error responses exceeds 1,000 within 15 minutes are calculated.
    bucket
    pv
    bucket_01
    60
    bucket_02
    55
    bucket_03
    47
    bucket_04
    45
  - Result of Query Statistics 1
    The resource data of the specified buckets.
    bucket
    desc
    bucket_03
    for dev team
    bucket_04
    for test team
  - Result of the set operation
    If you set the Set Operations parameter to LEFT EXCLUDE JOIN,$0.bucket == $1.bucket, the result of the set operation is displayed in the following table.
    bucket
    pv
    bucket_01
    60
    bucket_02
    55
RIGHT EXCLUDE JOIN
- Requirement
  The number of 5xx error responses in each bucket that is not specified is monitored. If the number of 5xx error responses within 15 minutes exceeds 1,000, an alert is triggered. To meet this requirement, you must add resource data to maintain the bucket blacklist.
- Configurations
- Results
  - Result of Query Statistics 0
    The resource data of the specified buckets.
    bucket
    desc
    bucket_03
    for dev team
    bucket_04
    for test team
  - Result of Query Statistics 1
    The buckets whose number of 500 error responses exceeds 1,000 within 15 minutes are calculated.
    bucket
    pv
    bucket_01
    60
    bucket_02
    55
    bucket_03
    47
    bucket_04
    45
  - Result of the set operation
    If you set the Set Operations parameter to RIGHT EXCLUDE JOIN,$0.bucket == $1.bucket, the result of the set operation is displayed in the following table.
    bucket
    pv
    bucket_01
    60
    bucket_02
    55
CROSS JOIN
- Example 1
  - Requirements
    Object Storage Service (OSS) access logs and Server Load Balancer (SLB) access logs are monitored at the same time. The number of 4xx error responses in OSS and the number of 5xx error responses in SLB are calculated every 15 minutes. If the total number of errors reaches 1,000, an alert is triggered.
  - Configurations
  - Results
    - Result of Query Statistics 0
      The number of 4xx error responses within 15 minutes in OSS is calculated.
      pv
      890
    - Result of Query Statistics 1
      The number of 5xx error responses within 15 minutes in SLB is calculated.
      pv
      567
    - Result of the set operation
      If you set the Set Operations parameter to CROSS JOIN, the result of the set operation is displayed in the following table.
      $0.pv
      $1.pv
      890
      567
- Other examples
  - Result of Query Statistics 0
    a
    b
    a1
    b1
    a2
    b2
    a5
    b5
  - Result of Query Statistics 1
    a
    c
    a1
    c1
    a3
    c3
  - Result of the set operation
    If you set the Set Operations parameter to CROSS JOIN, the result of the set operation is displayed in the following table.
    $0.a
    b
    $1.a
    c
    a1
    b1
    a1
    c1
    a1
    b1
    a3
    c3
    a2
    b2
    a1
    c1
    a2
    b2
    a3
    c3
    a5
    b5
    a1
    c1
    a5
    b5
    a3
    c3
LEFT JOIN
- Result of Query Statistics 0
  a
  b
  a1
  b1
  a2
  b2
  a3
  b3
- Result of Query Statistics 1
  a
  b
  c
  a1
  b11
  c1
  a2
  b22
  c2
- Result of the set operation
  If you set the Set Operations parameter to LEFT JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.
  a
  $0.b
  $1.b
  c
  a1
  b1
  b11
  c1
  a2
  b2
  b22
  c2
  a3
  b3
  None
  None
RIGHT JOIN
- Result of Query Statistics 0
  a
  b
  c
  a1
  b11
  c1
  a2
  b22
  c2
- Result of Query Statistics 1
  a
  b
  a1
  b1
  a2
  b2
  a3
  b3
- Result of the set operation
  If you set the Set Operations parameter to RIGHT JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.
  a
  $0.b
  c
  $1.b
  a1
  b11
  c1
  b1
  a2
  b22
  c2
  b2
  a3
  None
  None
  b3
FULL JOIN
- Result of Query Statistics 0
  a
  b
  c
  a1
  b1
  c1
  a2
  b2
  c2
  a5
  b5
  c3
- Result of Query Statistics 1
  a
  b
  d
  a1
  b11
  d1
  a2
  b22
  d2
  a3
  b33
  d3
- Result of the set operation
  If you set the Set Operations parameter to FULL JOIN,$0.a == $1.a, the result of the set operation is displayed in the following table.
  a
  $0.b
  c
  $1.b
  d
  a1
  b1
  c1
  b11
  d1
  a2
  b2
  c2
  b22
  d2
  a5
  b5
  c3
  None
  None
  a3
  None
  None
  b33
  d3

host	pv
host1	60
host2	55
host3	47
host4	45
hosg5	31
hosta	70
hostb	45
hostc	44
hostd	42

bucket	pv
bucket_01	1600
bucket_02	1550
bucket_03	1470
bucket_04	1450

bucket	desc
bucket_03	for dev team
bucket_04	for test team
bucket_05	for service team
bucket_06	for support team

client_ip	pv
192.0.2.4	60
192.0.2.5	55
192.0.2.6	47
192.0.2.7	45
192.0.2.8	31

a	$0.b	$0.c	d	$1.b	$1.c
a1	b1	c1	d1	b11	c11
a2	b2	c2	d2	b22	c22

$0.a	b	$1.a	c
a1	b1	a1	c1
a1	b1	a3	c3
a2	b2	a1	c1
a2	b2	a3	c3
a5	b5	a1	c1
a5	b5	a3	c3

a	b
a3	b333
a4	b444

$0.pv	$1.pv
890	567