Log Service provides the service log feature. This feature can record different types of logs. This topic describes the log types and the fields for each log type.

Log types

The following table describes the log types that you can select when you enable the service log feature.
Log type Description Logstore Log source Remarks
Detailed Logs Detailed logs record the operations to create, modify, update, and delete resources in each project. Detailed logs also record data read and write operations. internal-operation_log Operation logs The logs about all API requests, including requests sent by using the Log Service console, consumer groups, and SDKs.
Important Logs Important logs record the consumption delay of consumer groups, and the errors, heartbeats, and statistics of Logtail for each Logstore. internal-diagnostic_log Consumption delay logs of consumer groups The consumption delay logs of consumer groups. These logs are recorded at 2-minute intervals. If you want to query the consumption delay logs of a consumer group, you must include __topic__: consumergroup_log in the query statement.
Logtail alert logs The alert logs of Logtail, which record errors on Logtail.

Alert logs are recorded at 30-second intervals. If the same error occurs multiple times within a 30-second interval, only one alert log is recorded. The alert log contains the total number of times that the error occurs and one of the error messages. If you want to query Logtail alert logs, you must include __topic__: logtail_alarm in the query statement.

Logtail collection logs The collection logs of Logtail, which record statistics about Logtail configurations.

These logs are recorded at 10-minute intervals. If you want to query Logtail collection logs, you must include __topic__: logtail_profile in the query statement.

Logtail status logs

The status logs of Logtail, which is reported at regular intervals.

These logs are recorded at 1-minute intervals. If you want to query Logtail status logs, you must include __topic__: logtail_status in the query statement.

Operation logs

Operation logs are categorized into read, write, and resource operation logs based on the Method field.
Category API operation
Read operation log Read operation logs are generated when you call the following API operations:
  • GetHistograms
  • GetLogs
  • PullLogs
  • GetCursor
  • GetCursorTime
Write operation log Write operation logs are generated when you call the following API operations:
  • PutLogs
  • PutWebtracking
Resource operation log Resource operation logs are generated when you call the following API operations:

Other operations, such as CreateProject and DeleteProject

The following table describes the common fields in operation logs.
Field Description Example
APIVersion The version number of the API. 0.6.0
AccessKeyId The AccessKey ID that is used to access Log Service. LTAI4FkSqNGBsVTqVZYx****
CallerType The type of the API caller. Subuser
InvokerUid The account ID of the API caller. 175921811532****
Latency The latency of the request. Unit: microseconds. 123279
LogStore The name of the Logstore. logstore-1
Method The API operation for which the log is recorded. GetLogStoreLogs
NetOutFlow The volume of read traffic. Unit: bytes. 120
NetworkOut The volume of read traffic that is routed over the Internet. Unit: bytes. 10
Project The name of the project. project-1
RequestId The ID of the request. 8AEADC8B0AF2FA2592C9****
SourceIP The IP address of the client that sends the request. 1.2.3.4
Status The HTTP status code in the response to the request. 200
UserAgent The agent that is used by the client to call the API operation. sls-java-sdk-v-0.6.1
The following table describes the fields specific to read operation logs.
Field Description Example
BeginTime The start time of the request. The value is a UNIX timestamp. 1523868463
DataStatus The response to the request. Valid values include Complete, OK, and Unknown. OK
EndTime The end time of the request. The value is a UNIX timestamp. 1523869363
Offset The read offset that you specify when you call the GetLogs operation. 20
Query The original query statement. UserAgent: [consumer-group-java]*
RequestLines The number of lines requested by the user. 100
ResponseLines The number of returned lines. 100
Reverse Indicates whether logs are returned in descending order of timestamps.
  • 1: Logs are returned in descending order of timestamps.
  • 0: Logs are returned in ascending order of timestamps. This is the default value.
0
TermUnit The number of keywords that are included in the search statement after tokenization. 0
Topic The topic of the data that is read. topic-1
The following table describes the fields specific to write operation logs.
Field Description Example
InFlow The size of raw data that you want to write. Unit: bytes. 200
InputLines The number of lines that you want to write. 10
NetInflow The size of compressed data that you want to write. Unit: bytes. 100
Shard The ID of the shard to which data is written. 1
Topic The topic of the data that is written. topic-1

Consumption delay logs of consumer groups

The following table describes the fields in consumption delay logs.
Field Description Example
consumer_group The name of the consumer group. consumer-group-1
fallbehind The interval between the current consumption checkpoint and the point in time at which the last write operation log is recorded. Unit: seconds. 12345
logstore The name of the Logstore. logstore-1
project The name of the project. project-1
shard The ID of the shard whose data is consumed. 1

Logtail alert logs

The following table describes the fields in Logtail alert logs.
Field Description Example
alarm_count The number of times that alerts are generated in the specified time window. 10
alarm_message The sample of the raw log that triggers the alert. M_INFO_COL,all_status_monitor,T22380,0,2018-04-17 10:48:25.0,AY66K,AM5,2018-04-17 10:48:25.0,2018-04-17 10:48:30.561,i-23xebl5ni.1569395.715455,901,00789b
alarm_type The type of the alert. REGISTER_INOTIFY_FAIL_ALARM
logstore The name of the Logstore. logstore-1
os The operating system, such as Linux or Windows. Linux
project The name of the project. project-1
source_ip The IP address of the server on which Logtail runs. 1.2.3.4
version The version of Logtail. 0.14.2

Logtail collection logs

Logtail collection logs are categorized into the following types based on the file_name field:
  • Statistics about a Logtail configuration for a log file.
  • Statistics about a Logtail configuration for a Logstore. In the configuration, file_name is set to logstore_statistics.
The following table describes the fields in Logtail collection logs.
Field Description Example
logstore The name of the Logstore. logstore-1
config_name The name of the Logtail configuration, which is globally unique. The name must be in the following format: ##Logtail configuration version##Project name$Logstore name. ##1.0##project-1$logstore-1
error_line The raw log that causes an error. M_INFO_COL,all_status_monitor,T22380,0,2018-04-17 10:48:25.0,AY66K,AM5,2018-04-17 10:48:25.0,2018-04-17 10:48:30.561,i-23xebl5ni.1569395.715455,901,00789b
file_dev The device ID of the log file.
Note If the file_name field is set to logstore_statistics, this field is invalid.
123
file_inode The inode of the log file.
Note If the file_name field is set to logstore_statistics, this field is invalid.
124
file_name The full path of the log file, or logstore_statistics. /abc/file_1
file_size The size of the log file. Unit: bytes. 12345
history_data_failures The number of data processing failures. 0
last_read_time The last read time in the specified time window. The value is a UNIX timestamp. 1525346677
project The name of the project. project-1
logtail_version The version of Logtail. 0.14.2
os The operating system. Windows
parse_failures The number of lines that fail to be parsed in the specified time window. 12
read_avg_delay The average difference between the offset and the remaining file size for each read in the specified time window. 65
read_count The number of reads in the specified time window. 10
read_offset The last read offset of the log file. Unit: bytes. 12345
regex_match_failures The number of regular expression mismatches. 1
send_failures The number of times that logs fail to be sent in the specified time window. 12
source_ip The IP address of the server on which Logtail runs. 1.2.3.4
succeed_lines The number of log lines that are successfully processed. 123
time_format_failures The number of log time mismatches. 122
total_bytes The total size of data that is read. Unit: bytes. 12345
The following table describes the fields specific to Logstore statistics collected when the file_name field is set to logstore_statistics.
Field Description Example
send_block_flag Indicates whether the send queue is blocked when the specified time window ends. false
send_discard_error The number of packets that are discarded due to data errors or insufficient permissions in the specified time window. 0
send_network_error The number of packets that fail to be sent due to network errors in the specified time window. 12
send_queue_size The number of unsent packets in the current send queue when the specified time window ends. 3
send_quota_error The number of packets that fail to be sent due to quota exhaustion in the specified time window. 0
send_success_count The number of packets that are successfully sent in the specified time window. 12345
sender_valid_flag Indicates whether the send flag of the current Logstore is valid when the specified time window ends. Valid values:
  • true: The flag is valid.
  • false: The flag is disabled due to network or quota errors.
true
max_send_success_time The last time when packets were successfully sent in the specified time window. The value is a UNIX timestamp. 1525342763
max_unsend_time The last time when packets in the send queue failed to be sent in the specified time window. The value is a UNIX timestamp. If the send queue is empty, the value is 0. 1525342764
min_unsend_time The first time when packets in the send queue failed to be sent in the specified time window. The value is a UNIX timestamp. If the send queue is empty, the value is 0. 1525342764

Logtail status logs

The following table describes the fields in Logtail status logs.
Field Description Example
cpu The CPU load of the Logtail process. 0.001333156
hostname The hostname. abc2.****
instance_id The instance ID. This ID is randomly assigned. 05AFE618-0701-11E8-A95B-00163E025256_10.11.12.13_151745****
ip The IP address. 1.0.1.0
load The average system load. 0.01 0.04 0.05 2/376 5277
memory The memory space that is occupied by the Logtail process. Unit: MB. 12
detail_metric The metrics in the JSON format. For more information, see detail_metric. detail_metric
os The operating system. Linux
os_cpu The CPU utilization of the system. 0.004120005
os_detail The details about the operating system. 2.6.32-220.23.8.tcp1.34.el6.x86_64
status The status of Logtail.
  • ok
  • busy
  • many_log_files
  • process_block
  • send_block
  • send_error
For more information, see View Logtail status.
busy
user The username. root
user_defined_id The user-defined ID. aliyun-log-id
uuid The universally unique identifier (UUID) of the server. 64F28D10-D100-492C-8FDC-0C62907F****
version The version of Logtail. 0.14.2
project The project to which the Logtail configuration belongs. my-project
The following table describes the fields that are included in the detail_metric field.
Field Description Example
config_count The number of Logtail configurations. 1
config_get_last_time The last time when the Logtail configuration was obtained. 2021-07-20 16:19:22
config_update_count The number of Logtail configuration updates after Logtail was started. 1
config_update_item_count The total number of configuration items updated after Logtail was started. 1
config_update_last_time The time of the last configuration update after Logtail was started. 2021-07-20 16:18:42
env_config Indicates whether environment variables are used to create the Logtail configuration. false
event_tps The transactions per second (TPS). 1
last_read_event_time The last time when data was read. 2021-07-20 16:18:42
last_send_time The last time when data was sent. 2021-07-20 16:18:42
multi_config Indicates whether multiple Logtail configurations are enabled to collect logs from the same file. false
net_err_stat The numbers of network sending errors in the previous 1, 5, and 15 minutes. 0,0,0
open_fd The number of log files that are open. 1
plugin_enabled Indicates whether Logtail plug-ins are enabled. false
poll_modify_size The number of monitored log files that are modified. 1
polling_dir_cache The number of scanned directories. 1
polling_file_cache The number of scanned files. 1
process_bytes_ps The size of log data that is processed per second. Unit: bytes. 1000
process_lines_ps The number of log lines that are processed per second. 1000
process_queue_full The number of processing queues that reach the maximum processing capacity. 1
process_queue_total The total number of processing queues. 10
process_tps The number of data processing transactions per second. 0
reader_count The number of log files that are being processed. 1
region The region to which Logtail belongs. cn-hangzhou,cn-shanghai
register_handler The number of directories that need to be monitored. 1
send_bytes_ps The size of raw log data that is sent per second. Unit: bytes. 11111
send_lines_ps The number of log lines that are sent per second. 1000
send_net_bytes_ps The amount of network data that is sent per second. Unit: bytes. 1000
send_queue_full The number of send queues that reach the maximum sending capacity. 1
send_queue_total The total number of send queues. 12
send_request_concurrency The maximum number of packets that can be concurrently sent from send queues. 10
send_tps The number of data sending transactions per second. 0.075
sender_invalid The number of abnormal send queues. 0
start_time The start time. 2021-07-20 16:19:22
used_sending_concurrency The number of packets that are concurrently sent. 0