Simple Log Service:Service log types

Type

Overview

Logstore name

Details

Description

Detailed logs

Records operations performed on all resources in your project, including create, modify, update, delete, read, and write operations.

internal-operation_log

operation logs

Detailed operation logs for all API requests and operations, including requests from clients such as the console, consumer groups, and SDKs.

important logs

Includes Logstore-level consumer group delay logs, and Logtail error, heartbeat, and statistics logs.

internal-diagnostic_log

consumer group delay logs

Consumer group delay logs are generated every 2 minutes. To query consumer group delay logs, specify __topic__: consumergroup_log in the query statement.

Logtail alert logs

Records Logtail errors.

A log is generated every 30 seconds. If the same type of error occurs multiple times within this period, the system records only the total number of occurrences and one random error message. To query Logtail alert logs, specify __topic__: logtail_alarm in the query statement.

Logtail collection logs

Records Logtail collection statistics.

A log is generated every 10 minutes. To query Logtail collection logs, specify __topic__: logtail_profile in the query statement.

Logtail status logs

Records periodic status logs from Logtail.

A log is generated every minute. To query Logtail status logs, specify __topic__: logtail_status in the query statement.

operational logs

Records the operational metrics and operational logs for data transformation, Scheduled SQL, data import, and data shipping jobs in a project.

internal-diagnostic_log

Scheduled SQL operational logs

A log is generated for each Scheduled SQL instance after it completes.

To query Scheduled SQL operational logs, specify __topic__: scheduled_sql_alert in the query statement.

Operational logs for data transformation, data import, and data shipping jobs

Logs are generated every minute, but only if the job's data source contains data.

Category

Request method

Read operation

Read operation logs are generated when you call the following API operations:

• GetHistograms

• GetLogs

• PullLogs

• GetCursor

• GetCursorTime

Write operation

Write operation logs are generated when you call the following API operations:

• PostLogStoreLogs

• PutWebTracking

• WebTrackingImg

• BatchPostLogStoreLogs

• WebTracking

• PutData

Resource operation

All other API operations generate resource operation logs.

For example, CreateProject and DeleteProject.

Parameter

Description

Example

Logstore

The name of the Logstore.

Logstore-1

config_name

The name of the Logtail configuration. The name is globally unique and is formatted as ##ConfigVersion##ProjectName$ConfigName.

##1.0##project-1$LogStore-1

error_line

The raw log entry that caused the error.

M_INFO_COL,all_status_monitor,T22380,0,2018-04-17 10:48:25.0,AY66K,AM5,2018-04-17 10:48:25.0,2018-04-17 10:48:30.561,i-23xebl5ni.1569395.715455,901,00789b

file_dev

The device ID of the log file.

1​23

file_inode

The inode of the log file.

124

file_name

The full path of the log file or the value logstore_statistics.

/abc/file_1

file_size

The size of the log file, in bytes.

12345

history_data_failures

The cumulative number of data processing failures.

0

last_read_time

The UNIX timestamp of the latest read operation within the time window.

1525346677

Project

The name of the Project.

project-1

logtail_version

The version of Logtail.

0.14.2

os

The operating system.

Windows

parse_failures

The number of lines that failed to be parsed within the time window.

12

read_avg_delay

The average difference between the file size and the read offset for each read operation within the time window.

65

read_count

The number of read operations within the time window.

10

read_offset

The current read offset in the file, in bytes.

12345

regex_match_failures

The number of regular expression matching failures.

1

send_failures

The number of failed send operations within the time window.

12

source_ip

The IP address of the server where Logtail is installed.

47.100..

succeed_lines

The number of successfully processed log lines.

123

time_format_failures

The number of time format matching failures.

122

total_bytes

The total number of bytes read.

12345

Parameter

Description

Example

send_block_flag

Indicates whether the send queue is blocked at the end of the time window.

false

send_discard_error

The number of packets discarded within the time window due to data errors or insufficient permissions.

0

send_network_error

The number of packets that failed to be sent within the time window due to network errors.

12

send_queue_size

The number of unsent packets in the send queue at the end of the time window.

3

send_quota_error

The number of packets that failed to be sent within the time window because the quota was exceeded.

0

send_success_count

The number of packets successfully sent within the time window.

12345

sender_valid_flag

Indicates whether the send flag for the Logstore is valid at the end of the time window. Valid values:

• true: The flag is valid.

• false: The flag is invalid, which can be caused by network or quota errors.

true

max_send_success_time

The UNIX timestamp of the latest successful send operation within the time window.

1525342763

max_unsend_time

The UNIX timestamp of the most recent packet that failed to be sent from the send queue within the time window. The value is 0 if the queue is empty.

1525342764

min_unsend_time

The UNIX timestamp of the earliest packet that failed to be sent from the send queue within the time window. The value is 0 if the queue is empty.

1525342764

Parameter

Description

Example

config_count

The number of Logtail configurations.

​1

config_get_last_time

The last time the Logtail configuration was fetched.

2021-07-20 16:19:22

config_update_count

The number of Logtail configuration updates since Logtail started.

​1

config_update_item_count

The total number of Logtail configuration items updated since Logtail started.

​1

config_update_last_time

The timestamp of the last Logtail configuration update.

2021-07-20 16:18:42

env_config

Indicates whether environment variables are used to create the Logtail configuration.

false

event_tps

The number of events per second (TPS).

1​

last_read_event_time

The timestamp of the last event read.

2021-07-20 16:18:42

last_send_time

The timestamp when data was last sent.

2021-07-20 16:18:42

multi_config

Indicates whether the same file is being collected by multiple Logtail configurations.

false

net_err_stat

The number of network errors that occurred in the last 1, 5, and 15 minutes.

0,0,0

open_fd

The number of currently open file descriptors.

​1

plugin_enabled

Indicates whether plugin functionality is enabled.

• true: Plugin functionality is enabled.

• false: Plugin functionality is disabled.

false

poll_modify_size

The number of files being monitored for modification events.

​1

polling_dir_cache

The number of directories in the polling cache.

​1

polling_file_cache

The number of files in the polling cache.

​1

process_bytes_ps

The log volume processed per second, in bytes.

​1000

process_lines_ps

The number of log lines processed per second.

​1000

process_queue_full

The number of full processing queues.

​1

process_queue_total

The number of processing queues.

​10

process_tps

The number of processing transactions per second (TPS).

​0

reader_count

The number of files currently being read.

​1

region

The region where the host machine is located.

​cn-hangzhou

register_handler

The number of directories registered for monitoring.

​1

send_bytes_ps

The volume of raw logs sent per second, in bytes.

​11111

send_lines_ps

The number of log lines sent per second.

1000​

send_net_bytes_ps

The volume of network traffic sent per second, in bytes.

1000​

send_queue_full

The number of full send queues.

​1

send_queue_total

The number of send queues.

​12

send_request_concurrency

The maximum number of concurrent sending requests.

10

send_tps

The number of sending transactions per second (TPS).

​0.075

sender_invalid

The number of send queues in an error state.

​0

start_time

The startup time of the Logtail process.

2021-07-20 16:19:22

used_sending_concurrency

The current number of active sending requests.

0

Parameter

Description

Example

action

The operation the job performs. Valid values:

• ingest: Reads data from the data source.

• deliver: Writes data to the destination.

ingest

connector

The type of the data source or destination.

• sls: Simple Log Service

• oss: Object Storage Service (OSS)

• maxcompute: MaxCompute

• cms: CloudMonitor

• cloudtrail: AWS CloudTrail

• kafka: Apache Kafka

• elasticsearch: Elasticsearch

• mysql: MySQL (open source, from Alibaba Cloud, or third-party)

• mssql: SQL Server (open source, from Alibaba Cloud, or third-party)

oss

instance

The name of the source or destination instance.

my-oss-bucket

task_id

The task ID.

1669277

task_name

The task name.

1652765276578_task_0

task_type

The type of the task. The value is always CONNECTOR.

CONNECTOR

Parameter

Description

Example

state

The operational status of the job.

• 0: Normal. The job is running as expected.

• 1: Error. An error that can slow the job has occurred.

• 2: Failure. The job has failed and is blocked.

0

error

The job's error message.

This field is present only when an error occurs.

fail to pull logs, unauthorized

lags

The progress of the job.

The lags field is present in the logs only for data read jobs, that is, when the value of the action field is ingest. The value of this field is related to the data source type (the connector field).

• When the connector field is sls, the lags field indicates the Shard consumption delay in seconds.

• When the connector field is kafka, the value of the lags field indicates the Topic-level consumption Lag.

• When the connector field is cms, the lags field specifies the read latency of Cloud Monitor in seconds.

0.813

desc

For data read jobs, that is, when the action field is set to ingest, the desc field is present in the log. The value of this field is related to the data source type (the connector field).

• When the connector field is sls, the value of the desc field is in seconds (s).

• When the connector field is Kafka, the value of the desc field is the lag description.

• When the connector field is cms, the value of the desc field is in seconds.

s

events

The number of data entries processed in the reporting interval.

13245

failed

The number of data entries that failed to process in the reporting interval.

3

native_bytes

The raw size of the data processed in the reporting interval, in bytes.

7539125

events_bytes

The size of the parsed data processed in the reporting interval, in bytes.

13295475

pub_net_bytes

The Internet traffic generated by the data processed in the reporting interval, in bytes.

45678

req_count

The number of requests sent to the data source or destination in the reporting interval.

89

rep_time

The average request latency for the data source or destination in the reporting interval.

38

rate_limit_hits

The number of rate limit hits on the data source or destination in the reporting interval.

1

extras

Additional information. This field is included only if a value is present.

The value of this field depends on the data source type (the connector field).

{"events":"898","objectName":"shenzhen/2022/05/18/15/19_1652858350000000000_315eb857efeb9f00.snappy","objectSize":"212792"}