Simple Log Service:Common data structures

name

String

test-consumer-group

The name of the consumer group.

timeout

Integer

300

The timeout period. If Simple Log Service does not receive heartbeats from a consumer within the timeout period, Simple Log Service deletes the consumer. Unit: seconds.

order

Boolean

false

Specifies whether to consume data in sequence. Valid values:

• true: consumes data in shards in sequence. If a shard is split, the data in the original shard is consumed first. Then, the data in the new shards is consumed at the same time. If shards are merged, the data in the original shards is consumed first. Then, the data in the new shard is consumed.
• false: does not consume data in shards in sequence. The data in all shards is consumed at the same time. If a new shard is generated after a shard is split or shards are merged, the data in the new shard is immediately consumed.

enable

Boolean

true

Specifies whether to enable data encryption. Valid values:

• true
• false

encrypt_type

String

default

The encryption algorithm. Valid values: default and sm4. If enable is set to true, you must configure this parameter.

user_cmk_info

EncryptUserCmkConf

{ "cmk_key_id" : "f5136b95-2420-ab31-xxxxxxxxx" "arn" : "acs:ram::13234:role/logsource" "region_id" : "cn-hangzhou" }

Optional. If you configure this parameter, the bring-your-own-key (BYOK) key is used. If you do not configure this parameter, the service key of Simple Log Service is used.

cmk_key_id

String

f5136b95-2420-ab31-xxxxxxxxx

The ID of the customer master key (CMK) that uses the BYOK.

arn

String

acs:ram::13234:role/logsource

The Alibaba Cloud Resource Name (ARN) of the Resource Access Management (RAM) role.

region_id

String

cn-hangzhou

The ID of the region where the CMK resides.

from

Integer

1409529600

The start time of the subinterval. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

to

Integer

1409569200

The end time of the subinterval. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

count

Long

2

The number of logs that are generated during the subinterval.

progress

String

Complete

Specifies whether the query result in the subinterval is complete. Valid values:

• Complete: The query is successful, and the complete result is returned.
• Incomplete: The query is successful, but the query result is incomplete. To obtain the complete result, you must repeat the request.

Key

String

key-test

The name of the custom key.

Value

String

value-test

The value of the custom key.

configName

String

test-config

The name of the Logtail configuration. The name must be unique in the project to which the Logtail configuration belongs. After the Logtail configuration is created, you cannot change the name of the Logtail configuration. The name must meet the following requirements:

• The name can contain only lowercase letters, digits, hyphens (-), and underscores (_).
• The name must start and end with a lowercase letter or a digit.
• The name must be 2 to 128 characters in length.

logSample

String

2022-06-14 11:13:29.796 | DEBUG | __main__:<module>:1 - hello world

The sample log.

inputType

String

file

The collection method of logs. Valid values:

• plugin: Logs such as MySQL binary logs are collected by using Logtail plug-ins.
• file: Text logs are collected by using existing modes, including the full regex mode and delimiter mode.

inputDetail

Map

"logType": "common_reg_log", "logPath": "/var/log/httpd/", "filePattern": "access*.log", "localStorage": true, "timeFormat": "%Y/%m/%d %H:%M:%S", "logBeginRegex": ".*", "regex": "(\w+)(\s+)", "key" :["key1", "key2"], "filterKey":["key1"], "filterRegex":["regex1"], "fileEncoding":"utf8", "topicFormat": "none"

The configuration of the data source. For more information, see inputDetail.

outputType

String

LogService

The destination to which the collected logs are uploaded. Set the value to LogService. Collected logs can be uploaded to only Simple Log Service.

createTime

Long

1655176807

The time at which the Logtail configuration was created. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

lastModifyTime

Long

1655176807

The time at which the Logtail configuration was last modified. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

configName

String

test-config

The name of the configuration.

logSample

String

127.0.0.1 - - [10/Jun/2022:12:36:49 +0800] "GET /index.html HTTP/1.1" 200

The sample log.

global

Map

The global settings.

inputs

Array of Object

{ "Type": "input_file", "FilePaths": ["/var/log/*.log"] }

The data source plug-ins.

processors

Array of Object

{ "Type": "processor_parse_json_native", "SourceKey": "content" }

The processing plug-ins.

aggregators

Array of Object

The aggregation plug-ins.

flushers

Array of Object

{ "Type": "flusher_sls", "Project": "test_project", "Logstore": "test", "Region": "cn-hangzhou", "Endpoint": "cn-hangzhou-intranet.log.aliyuncs.com", "TelemetryType": "log" }

The data output plug-ins.

createTime

Long

1655176807

The creation time. The value is a UNIX timestamp.

lastModifyTime

Long

1655176807

The last modification time. The value is a UNIX timestamp.

annotationdataId

String

dc74b0f569126bb310e1ba6454c351ac

The unique identifier of the data.

dataHash

String

59db060bd89468245d76416a68a510ac

The hash value of the data.

datasetId

String

cb8cc4eb51a85e823471cdb368fae9be

The ID of the associated dataset.

value

String

xxx/xxx/xxx/

The data source.

valueType

String

oss

The type of the data source.

config

Map

The configuration of the data.

annotations

Map

The annotations of the data.

predictions

Map

The model preview results.

createTime

Long

1695094335

The creation time.

lastModifyTime

Long

1695094335

The last modification time.

datasetId

String

d9bd488f6dd42d294495fb780858e83d

The unique identifier of the dataset.

name

String

sls_builtin_dataset_metric.shapeclassification.anomalydetection

The name of the dataset.

description

String

Dataset A

The description of the dataset.

dataType

String

Metric

The type of the data.

labelId

String

a191ae4ca615b0ccb93c211fc8a998af

The ID of the tag system that corresponds to the dataset.

createTime

Long

1695090077

The creation time.

lastModifyTime

Long

1695090077

The last modification time.

createBy

String

sls-console

The creator.

settingType

String

Metric.ShapeClassification.AnomalyDetection

The configuration type of the tag table.

labelId

String

abbd488f6dd42d294495fb780858e83d

The unique identifier of the tag table.

name

String

Tag table

The name of the tag table.

description

String

Default tag table

The description of the tag table.

type

String

xxx

The type of the tag table.

createTime

Long

1695090077

The creation time.

lastModifyTime

Long

1695090077

The last modification time.

parameter

Map

parameter

input

Array of Object

input

savedsearchName

String

savedsearch-name

The name of the saved search.

searchQuery

String

status: 401 | SELECT remote_addr,COUNT(*) as pv GROUP by remote_addr ORDER by pv desc limit 5

The search statement or the query statement.

logstore

String

aliyun-test-logstore

The name of the Logstore.

topic

String

topic

The topic of the logs. Default value: empty string ("").

displayName

String

Method statistics

The display name of the saved search.

title

String

my-chart

The title of the chart.

type

String

linepro

The type of the chart.

search

Map

(*)| SELECT date_format(__time__ - __time__ % 60, '%H:%i:%s') as time, count(1) as count GROUP BY time ORDER BY time

The query configuration.

display

Map

The display configuration of the chart.

action

Map

action

dashboardName

String

dashboard-1609294922657-434834

The internal display name of the dashboard, which is not displayed on the dashboard page.

displayName

String

test-alert

The display name of the dashboard.

description

String

This is a dashboard.

The description of the dashboard.

attribute

Map

The attribute values of the dashboard.

etlJobName

String

etl-name

The name of the data transformation job.

functionParameter

Map

The parameters.

enable

Boolean

true

Specifies whether to enable the data transformation job.

etlMetaName

String

MetaName

MetaName

etlMetaKey

String

MetaKey

MetaKey

etlMetaTag

String

MetaTag

MetaTag

enable

Boolean

true

Enable

etlMetaValue

String

MetaValue

MetaValue

externalStoreName

String

rds_store

The name of the external store. The name must be unique in a project and must be different from Logstore names.

storeType

String

rds-vpc

The storage type. Set the value to rds-vpc, which indicates a database on an ApsaraDB RDS for MySQL instance in a virtual private cloud (VPC).

parameter

Map

{ "vpc-id": "vpc-bp1aevy8sofi8mh1q****", "instance-id": "i-bp1b6c719dfa08exf****", "host": "192.168.XX.XX", "port": "3306", "username": "root", "password": "sfdsfldsfksfls****", "db": "meta", "table": "join_meta", "region": "cn-qingdao" }

The parameters that are configured for the external store.

ttl

Integer

30

The log retention period. Unit: days.

max_text_len

Integer

2048

The maximum length of a field value that can be retained. Default value: 2048. Unit: bytes. The default value is equal to 2 KB. You can change the value of the max_text_len parameter. Valid values: 64 to 16384. Unit: bytes.

log_reduce_white_list

Array of String

name

The fields in the whitelist.

log_reduce_black_list

Array of String

time

The fields in the blacklist.

keys

Map

The configuration of field indexes. A field index is a key-value pair in which the key specifies the name of the field and the value specifies the index configuration of the field. You must configure this parameter, the line parameter, or both parameters. For more information, see Example.

log_reduce

Boolean

true

Specifies whether to enable the LogReduce feature. After you enable the LogReduce feature, either the whitelist or blacklist takes effect. Valid values:

• true
• false (default)

lastModifyTime

Long

1622186280

The time at which the index configuration was last modified. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

loggingProject

String

my-project

The name of the project to which service logs are stored.

logstoreName

String

my-logstore

The name of the Logstore.

ttl

Integer

1

The log retention period. Unit: days. Valid values: 1 to 3650. If you set this parameter to 3650, data is permanently stored.

shardCount

Integer

2

The number of shards.

enable_tracking

Boolean

false

Specifies whether to enable the web tracking feature. Default value: false. Valid values:

• true
• false

autoSplit

Boolean

true

Specifies whether to enable automatic sharding. Valid values:

• true
• false

maxSplitShard

Integer

6

The maximum number of shards into which existing shards can be automatically split. Valid values: 1 to 64.

appendMeta

Boolean

false

Specifies whether to record public IP addresses. Default value: false. Valid values:

• true
• false

telemetryType

String

Metrics

The type of the data that you want to query. Valid values:

• Metrics: metric data.
• None: non-metric data.

createTime

Integer

1453949705

The time at which the Logstore was created. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

lastModifyTime

Integer

1524155379

The time at which the Logstore was last modified. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

encrypt_conf

EncryptConf

The configuration of data encryption.

hot_ttl

Integer

60

The retention period of data in the hot storage tier of the Logstore. Minimum value: 30. Unit: days.

infrequentAccessTTL

Integer

30

The retention period of data in the Infrequent Access (IA) storage tier of the Logstore.

mode

String

standard

The type of the Logstore. Simple Log Service provides two types of Logstores: Standard Logstores and Query Logstores. Valid values:

• standard: Standard Logstore. This type of Logstore supports the log analysis feature and is suitable for scenarios such as real-time monitoring and interactive analysis. You can also use this type of Logstore to build a comprehensive observability system.
• query: Query Logstore. This type of Logstore supports high-performance queries. The index traffic fee of a Query Logstore is approximately half that of a Standard Logstore. Query Logstores do not support SQL analysis. Query Logstores are suitable for scenarios in which the amount of data is large, the log retention period is long, or log analysis is not required. If logs are stored for weeks or months, the log retention period is considered long.

productType

String

aliyun

The type of the service to which the logs belong.

ip

String

192.168.xx.x

The IP address of the machine.

machine-uniqueid

String

3B70F4F1-80F7-46C4-A6C1-100***CEE647

The unique identifier of the machine.

lastHeartbeatTime

Long

1447182247

The last heartbeat time. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

userdefined-id

String

None

The custom identifier of the machine. This parameter takes effect when the machine belongs to a custom identifier-based machine group.

groupName

String

machineGroup-1

The name of the machine group.

groupType

String

""

The type of the machine group. Set the value to an empty string.

machineIdentifyType

String

ip

The type of the machine group identifier. Valid values:

• ip: The machine group uses IP addresses as identifiers.
• userdefined: The machine group uses custom identifiers.

machineList

Array of String

47.100.XX.XX

The IP address or custom identifier of the machine group.

createTime

String

2021-07-07 14:08:09

The time at which the project was created.

lastModifyTime

String

2022-04-18 13:30:19

The time at which the project was last modified.

description

String

Description of my-project

The description of the project.

owner

String

""

The ID of the Alibaba Cloud account that is used to create the project.

projectName

String

ali-test-project

The name of the project. The name is included in the value of the Host header. The name must be unique in a region. You cannot change the name after you create the project.

region

String

cn-hangzhou

The region to which the project belongs.

status

String

Normal

The status of the project. Valid values:

• Normal
• Disable

resourceGroupId

String

rg-acf******sq

The ID of the resource group to which the project belongs.

shardID

Integer

0

The shard ID.

status

String

readwrite

The status of the shard. After you create a shard, the shard enters the read/write state. If you split a shard or merge shards, the shard status changes to readonly. The newly generated shards are in the read/write state. The status of a shard does not affect the performance of read operations that are performed on the shard. Data can be written to the shards that are in the read/write state, but data cannot be written to the shards that are in the readonly state. Valid values:

• readwrite
• readonly

inclusiveBeginKey

String

00000000000000000000000000000000

The start of the MD5 hash range. The value is included in the MD5 hash range of the shard.

exclusiveEndKey

String

8000000000000000000000000000000

The end of the MD5 hash range.

createTime

Integer

1524222931

The time at which the shard was created. The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.