All Products
Search
Document Center

Simple Log Service:Common data structures

Last Updated:Oct 26, 2023

ConsumerGroup

The following table describes the data structure of a consumer group.

Parameter

Type

Example

Description

name

String

test-consumer-group

The name of the consumer group.

timeout

Integer

300

The timeout period. If Log Service does not receive heartbeats from a consumer within the timeout period, Log Service deletes the consumer. Unit: seconds.

order

Boolean

false

Specifies whether to consume data in sequence. Valid values:

  • true: consumes data in shards in sequence. If a shard is split, the data in the original shard is consumed first. Then, the data in the new shards is consumed at the same time. If shards are merged, the data in the original shards is consumed first. Then, the data in the new shard is consumed.

  • false: does not consume data in shards in sequence. The data in all shards is consumed at the same time. If a new shard is generated after a shard is split or shards are merged, the data in the new shard is immediately consumed.

EncryptConf

The following table describes the data structure of an encryption configuration.

Parameter

Type

Example

Description

enable

Boolean

true

Specifies whether to enable data encryption. Valid values:

  • true: enables data encryption.

  • false: does not enable data encryption.

encrypt_type

String

default

The encryption algorithm. Valid values: default and sm4. This parameter must be specified when enable is set to true.

user_cmk_info

EncryptUserCmkConf

{ "cmk_key_id" : "f5136b95-2420-ab31-xxxxxxxxx" "arn" : "acs:ram::13234:role/logsource" "region_id" : "cn-hangzhou" }

Optional. If you specify this parameter, the bring-your-own-key (BYOK) key is used. If you do not specify this parameter, the service key of Log Service is used.

EncryptUserCmkConf

The following table describes the data structure of a user-defined encryption configuration.

Parameter

Type

Example

Description

cmk_key_id

String

f5136b95-2420-ab31-xxxxxxxxx

The ID of the customer master key (CMK) that uses the BYOK.

arn

String

acs:ram::13234:role/logsource

The Alibaba Cloud Resource Name (ARN) of the RAM role.

region_id

String

cn-hangzhou

The ID of the region where the CMK resides.

Histogram

The following table describes the data structure of log distribution.

Parameter

Type

Example

Description

from

Integer

1409529600

The start time of the subinterval. The value is a timestamp that follows the UNIX time format. It is the number of seconds that have elapsed since 00:00:00 UTC, Thursday, January 1, 1970.

to

Integer

1409569200

The end time of the subinterval. The value is a timestamp that follows the UNIX time format. It is the number of seconds that have elapsed since 00:00:00 UTC, Thursday, January 1, 1970.

count

Long

2

The number of logs in the subinterval.

progress

String

Complete

Indicates whether the query result in the subinterval is complete. Valid values:

  • Complete: The query is successful, and the complete result is returned.

  • Incomplete: The query is successful, but the query result is incomplete. To obtain the complete result, you must call the operation again.

LogtailConfig

The following table describes the data structure of a Logtail configuration.

Parameter

Type

Example

Description

configName

String

test-config

The name of the Logtail configuration. The name must meet the following requirements:

  • The name must be unique in a project.

  • The name can contain letters, digits, underscores (_), and hyphens (-).

  • The name must be 2 to 64 characters in length.

  • The name must start and end with a letter or a digit.

logSample

String

2022-06-14 11:13:29.796 | DEBUG | __main__:<module>:1 - hello world

A sample of the logs that Logtail needs to collect.

inputType

String

file

The type of the data source from which Logtail collects logs.

inputDetail

Map

The configurations of the data source.

outputType

String

LogService

The type of the destination to which the collected logs are stored. Set the value to LogService.

outputDetail

Object

The configurations of the destination.

endpoint

String

cn-hangzhou-intranet.log.aliyuncs.com

The endpoint of the project.

logstoreName

String

test-logstore

The name of the Logstore to which the collected logs are stored.

region

String

cn-hangzhou

The region.

createTime

Long

1655176807

The time at which the Logtail configuration was created. The value is a UNIX timestamp.

lastModifyTime

Long

1655176807

The time at which the Logtail configuration was last modified. The value is a UNIX timestamp.

SavedSearch

The following table describes the data structure of a saved search.

Parameter

Type

Example

Description

savedsearchName

String

savedsearch-name

The name of the saved search.

searchQuery

String

status: 401 | SELECT remote_addr,COUNT(*) as pv GROUP by remote_addr ORDER by pv desc limit 5

The search statement or the query statement.

logstore

String

aliyun-test-logstore

The name of the Logstore.

topic

String

topic

The topic of the logs. Default value: empty string ("").

displayName

String

Method statistics

The display name of the saved search.

chart

The following table describes the data structure of a chart.

Parameter

Type

Example

Description

title

String

my-chart

The title of the chart.

type

String

linepro

The type of the chart.

search

Map

(*)| SELECT date_format(__time__ - __time__ % 60, '%H:%i:%s') as time, count(1) as count GROUP BY time ORDER BY time

The query configuration.

display

Map

The display configuration of the chart.

action

Map

action

dashboard

The following table describes the data structure of a dashboard.

Parameter

Type

Example

Description

dashboardName

String

dashboard-1609294922657-434834

The internal display name of the dashboard, which is not displayed on the dashboard page.

displayName

String

test-alert

The display name of the dashboard.

description

String

This is a dashboard.

The description of the dashboard.

attribute

Map

The attribute values of the dashboard.

charts

Array of chart

The charts on the dashboard.

etlJob

The following table describes the data structure of a data transformation job.

Parameter

Type

Example

Description

etlJobName

String

etl-name

The name of the data transformation job.

sourceConfig

Object

The configurations of the data source.

logstoreName

String

my-logstore

The name of the Logstore.

triggerConfig

Object

The trigger configuration of the data transformation job.

roleArn

String

acs:ram::13791******44:role/aliyunlogetlrole

The ARN of the role that is used for authorization. Example: acs:ram::13791******44:role/aliyunlogetlrole.

triggerInterval

Integer

5

The interval at which the data transformation job is triggered.

maxRetryTime

Integer

10

The maximum number of retries.

startingPosition

String

at-unixtime

The position at which the data transformation job starts to run.

startingUnixtime

Long

1655176807

The time at which the data transformation job starts to run. This parameter takes effect when startingPosition is set to at-unixtime.

functionConfig

Object

The configuration of the functions that are used.

functionProvider

String

FunctionCompute

The function provider.

endpoint

String

cn-hangzhou.log.aliyuncs.com

The Log Service endpoint.

accountId

String

13791******44

The ID of the Alibaba Cloud account.

regionName

String

cn-hangzhou

The region.

serviceName

String

my-service

The name of the service.

functionName

String

hello-world

The name of the function.

roleArn

String

acs:ram::13791******44:role/aliyunlogetlrole

The ARN of the role that is used for authorization. Example: acs:ram::13791******44:role/aliyunlogetlrole.

functionParameter

Map

The parameters.

enable

Boolean

true

Specifies whether to enable the data transformation job.

logConfig

Object

The logging configuration.

endpoint

String

cn-hangzhou.log.aliyuncs.com

The Log Service endpoint.

projectName

String

ali-test-project

The name of the project.

logstoreName

String

ali-test-logstore

The name of the Logstore.

etlMeta

The following table describes the data structure of etlMeta.

Parameter

Type

Example

Description

etlMetaName

String

MetaName

MetaName

etlMetaKey

String

MetaKey

MetaKey

etlMetaTag

String

MetaTag

MetaTag

enable

Boolean

true

Enable

etlMetaValue

String

MetaValue

MetaValue

externalStore

The following table describes the data structure of an external store.

Parameter

Type

Example

Description

externalStoreName

String

rds_store

The name of the external store. The name must be unique in a project and must be different from Logstore names.

storeType

String

rds-vpc

The storage type. Set the value to rds-vpc, which indicates an ApsaraDB RDS for MySQL database in a virtual private cloud (VPC).

parameter

Map

{ "vpc-id": "vpc-bp1aevy8sofi8mh1q****", "instance-id": "i-bp1b6c719dfa08exf****", "host": "192.168.XX.XX", "port": "3306", "username": "root", "password": "sfdsfldsfksfls****", "db": "meta", "table": "join_meta", "region": "cn-qingdao" }

The parameters that are configured for the external store.

index

The following table describes the data structure of an index configuration.

Parameter

Type

Example

Description

ttl

Integer

30

The log retention period. Unit: days.

max_text_len

Integer

2048

The maximum length of a field value that can be retained. Unit: bytes. Default value: 2048. The default value is equal to 2 KB. You can change the value of the max_text_len parameter. Valid values: 64 to 16384. Unit: bytes.

log_reduce_white_list

Array of String

name

The fields in the whitelist.

log_reduce_black_list

Array of String

time

The fields in the blacklist.

line

Object

The configuration of full-text indexing. You must configure at least one of the following parameters: line and keys. For more information, see Example.

chn

Boolean

false

Specifies whether to include Chinese characters. Valid values:

  • true

  • false (default)

caseSensitive

Boolean

true

Specifies whether to enable case sensitivity. Valid values:

  • true

  • false (default)

token

Array of String

,

The delimiter. Examples: ",", ".", "\r", and "\n".

include_keys

Array of String

includeField

The included fields.

exclude_keys

Array of String

excludeField

The excluded fields.

keys

Map

The configuration of field indexes. A field index is a key-value pair in which the key specifies the name of the field and the value specifies the index configuration of the field. You must configure at least one of the following parameters: line and keys. For more information, see Example.

log_reduce

Boolean

true

Specifies whether to enable the LogReduce feature. After you enable the LogReduce feature, either the whitelist or blacklist takes effect. Valid values:

  • true

  • false (default)

lastModifyTime

Long

1622186280

The time at which the index configuration was last modified. The value is a timestamp that follows the UNIX time format. It is the number of seconds that have elapsed since 00:00:00 UTC, Thursday, January 1, 1970.

logging

The following table describes the data structure of the service log feature for a project.

Parameter

Type

Example

Description

loggingProject

String

my-project

The name of the project to which service logs are stored.

loggingDetails

Array

The configuration of service logs.

type

String

consumergroup_log

The type of service logs. Valid values:

  • consumergroup_log: the consumption delay logs of consumer groups.

  • logtail_alarm: the alert logs of Logtail.

  • operation_log: the operation logs. You are charged for operation logs.

  • logtail_profile: the collection logs of Logtail.

  • metering: the metering logs.

  • logtail_status: the status logs of Logtail.

  • scheduledsqlalert: the job operational logs of Scheduled SQL jobs

  • etl_alert: the operational logs of data transformation jobs.

logstore

String

my-logstore

The name of the Logstore.

logstore

The following table describes the data structure of a Logstore.

Parameter

Type

Example

Description

logstoreName

String

my-logstore

The name of the Logstore.

ttl

Integer

1

The data retention period. Unit: days. Valid values: 1 to 3650. If you set the value to 3650, data is permanently stored.

shardCount

Integer

2

The number of shards.

enable_tracking

Boolean

false

Specifies whether to enable the web tracking feature. Default value: false. Valid values:

  • true: enables the web tracking feature.

  • false: disables the web tracking feature.

autoSplit

Boolean

true

Specifies whether to enable automatic sharding. Valid values:

  • true: enables automatic sharding.

  • false: disables automatic sharding.

maxSplitShard

Integer

6

The maximum number of readwrite shards that are obtained after shards are automatically split . Valid values: 1 to 64.

appendMeta

Boolean

false

Specifies whether to record public IP addresses. Default value: false. Valid values:

  • true: records public IP addresses.

  • false: does not record public IP addresses.

telemetryType

String

Metrics

The type of the log data that you want to query. Valid values:

  • Metrics: metric data

  • None: non-metric data

createTime

Integer

1453949705

The time at which the Logstore was created. The value is a timestamp that follows the UNIX time format. It is the number of seconds that have elapsed since 00:00:00 UTC, Thursday, January 1, 1970.

lastModifyTime

Integer

1524155379

The time at which the Logstore was last modified. The value is a timestamp that follows the UNIX time format. It is the number of seconds that have elapsed since 00:00:00 UTC, Thursday, January 1, 1970.

encrypt_conf

EncryptConf

The configuration of data encryption.

hot_ttl

Integer

60

The period of time during which you want to store logs in the hot storage tier of the Logstore. Minimum value: 30. Unit: days.

mode

String

standard

The type of the Logstore. Log Service provides the following types of Logstores: Standard Logstores and Query Logstores. Valid values:

  • standard: Standard Logstore. This type of Logstore supports the log analysis feature and is suitable for scenarios such as real-time monitoring and interactive analysis. You can also use this type of Logstore to build a comprehensive observability system.

  • query: Query Logstore. This type of Logstore supports high-performance queries. The index traffic fee of a Query Logstore is approximately half that of a Standard Logstore. Query Logstores do not support SQL analysis. Query Logstores are suitable for scenarios in which the volume of data is large, the log retention period is long, or log analysis is not required. Log retention periods of weeks or months are considered long.

productType

String

aliyun

The type of the service to which the logs belong.

machine

The following table describes the data structure of a machine.

Parameter

Type

Example

Description

ip

String

192.168.xx.x

The IP address of the machine.

machine-uniqueid

String

3B70F4F1-80F7-46C4-A6C1-100***CEE647

The unique identifier of the machine.

lastHeartbeatTime

Long

1447182247

The last heartbeat time. The value is a timestamp that follows the UNIX time format. It is the number of seconds that have elapsed since 00:00:00 UTC, Thursday, January 1, 1970.

userdefined-id

String

None

The custom identifier of the machine. This parameter takes effect when the machine belongs to a custom identifier-based machine group.

machineGroup

The following table describes the data structure of a machine group.

Parameter

Type

Example

Description

groupName

String

machineGroup-1

The name of the machine group.

groupType

String

""

The type of the machine group. Set the value to an empty string.

machineIdentifyType

String

ip

The type of the machine group identifier. Valid values:

  • ip: The machine group uses IP addresses as identifiers.

  • userdefined: The machine group uses custom identifiers.

groupAttribute

Object

The attribute of the machine group.

groupTopic

String

testtopic

The topic of the machine group.

externalName

String

testgroup

The identifier of the external management system on which the machine group depends.

machineList

Array of String

47.100.XX.XX

The IP address or custom identifier of the machine group.

project

The following table describes the data structure of a Log Service project.

Parameter

Type

Example

Description

createTime

String

2021-07-07 14:08:09

The time at which the project was created.

lastModifyTime

String

2022-04-18 13:30:19

The time at which the project was last modified.

description

String

Description of my-project

The description of the project.

owner

String

""

The Alibaba Cloud account to which the project belongs.

projectName

String

ali-test-project

The name of the project. The name is included in the value of the Host header. The name must be unique in a region. You cannot change the name after you create the project.

region

String

cn-hangzhou

The ID of the region where the project resides.

status

String

Normal

The status of the project. Valid values:

  • Normal: The status of the project is normal.

  • Disable: The status of the project is disabled.

resourceGroupId

String

rg-acf******sq

The ID of the resource group to which the project belongs.

shard

The following table describes the data structure of a shard.

Parameter

Type

Example

Description

shardID

Integer

0

shard id

status

String

readwrite

The status of the shard. After you create a shard, the shard enters the readwrite state. If you split a shard or merge shards, the status of the original shard changes to readonly. The newly generated shards are in the readwrite state. The status of a shard does not affect the read capacity of the shard. Data can be written to the shards that are in the readwrite state, but cannot be written to the shards that are in the readonly state. Valid values:

  • readwrite

  • readonly

inclusiveBeginKey

String

00000000000000000000000000000000

The start of the MD5 hash range. The value is included in the MD5 hash range of the shard.

exclusiveEndKey

String

8000000000000000000000000000000

The end of the MD5 hash range.

createTime

Integer

1524222931

The time at which the shard was created. The value is a timestamp that follows the UNIX time format. It is the number of seconds that have elapsed since 00:00:00 UTC, Thursday, January 1, 1970.