The service log feature of Log Service can record multiple types of logs. This topic describes these log types and their log fields.

Log types

When you enable the service log feature for a project, you can select one of the following log types:
  • Detailed logs. Detailed logs include operations logs generated when you create, modify, delete, or update resources in the project. Detailed logs also include log data about read and write operations. These logs are stored in the internal-operation_log Logstore of a specified project.
  • Important logs. Important logs include log data about consumption delay of consumer groups, Logtail errors, heartbeats, and collection statistics in each Logstore. These logs are stored in the internal-diagnostic_log Logstore of a specified project.
Log type Logstore Log source Description
Detailed log internal-operation_log Operations logs Logs about all API requests, including requests sent by using the console and SDK.
Important log internal-diagnostic_log Consumption delay logs of consumer groups The consumption delay logs of consumer groups. These logs are recorded every 2 minutes. To query consumption delay logs of a consumer group, you must include __topic__: consumergroup_log in the query statement.
Logtail alert logs The error logs of Logtail.

Error logs are recorded every 30 seconds. If the same type of error log entries occur within 30 seconds, only the total number of error log entries and one of the error log entries are recorded in the Logstore. To query Logtail alert logs, you must include __topic__: logtail_alarm in the query statement.

Logtail collection logs The logs about the log data collected by Logtail.

These logs are recorded every 10 minutes. To query Logtail collection logs, you must include __topic__: logtail_profile in the query statement.

Logtail status logs

The Logtail status logs that are recorded at regular intervals.

These logs are recorded every 1 minute. To query Logtail status logs, you must include __topic__: logtail_status in the query statement.

Operations logs

Operations logs can be divided into three types based on the method field: read operations log, write operations log, and resource operations log.
Type API
Read operations log Read operations logs are generated when you call the following API operations:
  • GetLogStoreHistogram
  • GetLogStoreLogs
  • PullData
  • GetCursor
  • GetCursorTime
Write operations log Write operations logs are generated when you call the following API operations:
  • PostLogStoreLogs
  • WebTracking
Resource operations log Resource operations logs are generated when you call the following API operations:

CreateProject, DeleteProject, and other API operations

The following table describes the common fields of operations logs.
Table 1. Common field
Field Description Example
APIVersion The version of the API. 0.6.0
InvokerUid The account ID of the user who performs the operation. 1759218115323050
NetworkOut The traffic that is read over the public network. Unit: bytes. 10
Latency The latency of the request. Unit: microseconds. 123279
LogStore The name of the Logstore. logstore-1
Method The API operation that generates the log. GetLogStoreLogs
Project The name of the project. project-1
NetOutFlow The read data traffic. Unit: bytes. 120
RequestId The ID of the request. 8AEADC8B0AF2FA2592C9509E
SourceIP The IP address of the client that sends the request. 1.2.3.4
Status The HTTP response status code. 200
UserAgent The agent that is used by the client to call API operations. sls-java-sdk-v-0.6.1
The following table describes the unique fields in read operations logs.
Table 2. Unique fields in read operations logs
Field Description Example
BeginTime The start time of the request. The value is a Unix timestamp. 1523868463
DataStatus The status that indicates whether all data is read. Valid values: Complete, OK, Unknown, and other values. OK
EndTime The end time of the request. The value is a Unix timestamp. 1523869363
Offset The offset of log entries returned when you call the GetLogStoreLogs or GetLogStoreHistogram API operation. 20
Query The original query statement. UserAgent: [consumer-group-java]*
RequestLines The number of lines that the user requests to return. 100
ResponseLines The number of returned lines. 100
Reverse Indicates whether logs are returned in the descending order of timestamps.
  • 1: Logs are returned in the descending order of timestamps.
  • 0: The default value. Logs are returned in the ascending order of timestamps.
0
TermUnit The number of keywords in the query statement. 0
Topic The topic of the read data. topic-1
The following table describes the unique fields in write operations logs.
Table 3. Unique fields in write operations logs
Field Description Example
InFlow The raw data size. Unit: bytes. 200
InputLines The number of lines that the user requests to write. 10
NetInflow The size of the compressed data. 100
Shard The ID of the shard to which data is written. 1
Topic The topic of the written data. topic-1

Consumption delay logs of consumer groups

The following table describes the fields in consumption delay logs.
Table 4. Fields in consumption delay logs
Field Description Example
consumer_group The name of the consumer group. consumer-group-1
fallbehind The difference between the time when the current consumption data was generated and the time when the latest log was written. Unit: seconds. 12345
logstore The name of the Logstore. logstore-1
project The name of the project. project-1
shard The ID of the shard from which data is consumed. 1

Logtail alert logs

The following table describes the fields in Logtail alert logs.
Table 5. Fields in Logtail alert logs
Field Description Example
alarm_count The number of alerts during the specified time window. 10
alarm_message The sample raw log data that triggers the alert. M_INFO_COL,all_status_monitor,T22380,0,2018-04-17 10:48:25.0,AY66K,AM5,2018-04-17 10:48:25.0,2018-04-17 10:48:30.561,i-23xebl5ni.1569395.715455,901,00789b
alarm_type The type of the alert. REGISTER_INOTIFY_FAIL_ALARM
logstore The name of the Logstore. logstore-1
source_ip The IP address of the server on which Logtail runs. 1.2.3.4
os The operating system of the server on which Logtail runs, such as Linux and Windows. Linux
project The name of the project. project-1
version The version of Logtail. 0.14.2

Logtail collection logs

Based on the file_name field, Logtail collection logs are divided into the following two categories:
  • Collection statistics of a single log file.
  • Logstore statistics whose file_name is logstore_statistics.
The following table describes the fields in Logtail collection logs.
Table 6. Fields in Logtail collection logs
Field Description Example
logstore The name of the Logstore. logstore-1
config_name The globally unique name of the configuration file. The name is in the format of ##configuration version##projectName$configuration name. ##1.0##project-1$logstore-1
error_line The raw log that records the error. M_INFO_COL,all_status_monitor,T22380,0,2018-04-17 10:48:25.0,AY66K,AM5,2018-04-17 10:48:25.0,2018-04-17 10:48:30.561,i-23xebl5ni.1569395.715455,901,00789b
file_dev The device ID of the log file.
Note This field is invalid if the file_name parameter value is logstore_statistics.
123
file_inode The inode of the log file.
Note This field is invalid if the file_name parameter value is logstore_statistics.
124
file_name The full path of the log file, or logstore_statistics. /abc/file_1
file_size The size of the current log file. Unit: bytes. 12345
history_data_failures The number of data processing failures. 0
last_read_time The last read time is a Unix timestamp. 1525346677
project The name of the project. project-1
logtail_version The version of Logtail. 0.14.2
os The operating system of the server on which Logtail runs. Windows
parse_failures The number of lines that fail to be parsed during the specified time window. 12
read_avg_delay The average difference between the current offset and the file size at each read during the specified time window. 65
read_count The number of times that logs are read during the specified time window. 10
read_offset The read offset of the current file. Unit: bytes. 12345
regex_match_failures The number of regular expression mismatches. 1
send_failures The number of log data sending failures during the specified time window. 12
source_ip The IP address of the server on which Logtail runs. 1.2.3.4
succeed_lines The number of processed log lines. 123
time_format_failures The number of log time mismatches. 122
total_bytes The total size of read data. Unit: bytes. 12345
The following table describes the fields that are valid only when file_name is logstore_statistics.
Table 7. Unique fields in Logstore statistics
Field Description Example
send_block_flag Indicates whether the data sending queue is blocked when the specified time window expires. false
send_discard_error The number of packets dropped due to data errors or insufficient permissions during the specified time window. 0
send_network_error The number of packets that fail to be sent due to network errors during the specified time window. 12
send_queue_size The number of unsent data packets in the current sending queue when the specified time window expires. 3
send_quota_error The number of data packets that fail to be sent during the specified time window because the quota is exhausted. 0
send_success_count The number of data packets sent during the time window. 12345
sender_valid_flag Indicates whether the sending flag of the current Logstore is valid when the specified time window expires. Valid values:
  • true: The flag is valid.
  • false: The flag is disabled due to a network error or quota error.
true
max_send_success_time The latest time when log data packets are last sent in the specified time window. The value is a Unix timestamp. 1525342763
max_unsend_time The latest time when log data packets in the sending queue fail to be sent in the specified time window. The value is a Unix timestamp. The value is 0 if the queue is empty. 1525342764
min_unsend_time The time when log data packets in the sending queue fail to be sent for the first time in the specified time window. The value is a Unix timestamp. The value is 0 if the queue is empty. 1525342764

Logtail status logs

The following table describes the fields in Logtail status logs.
Field Description Example
cpu The load of the CPU. ​0.001333156
hostname The hostname of the server. abc2.et12
instance_id The unique ID of Logtail. This ID is randomly assigned. 05AFE618-0701-11E8-A95B-00163E025256_10.11.12.13_1517456122
ip The IP address of the server that Logtail receives. ​1.0.1.0
load The average system load. ​0.01 0.04 0.05 2/376 5277
memory The memory size occupied by the Logtail process. Unit: MB. 12
detail_metric The metric values in the JSON format. For more information, see detail_metric. detail_metric
os The operating system of the server on which Logtail runs. Linux
os_cpu The CPU utilization of the system. ​0.004120005
os_detail The details of the operating system. ​2.6.32-220.23.8.tcp1.34.el6.x86_64
status The status of Logtail. Valid values:
  • ok
  • busy
  • many_log_files
  • process_block
  • send_block
  • send_error
For more information, see Monitor the running status of Logtail.
busy
user The username used to log on to the server. root
user_defined_id The user-defined ID of the server. ​aliyun-log-id
uuid The UUID of the server. ​64F28D10-D100-492C-8FDC-0C62907F1234
version The version of Logtail. ​0.14.2
project The project to which the Logtail configuration files belong. my-project
The following table describes the values of the detail_metric field.
Table 8. Values of the detail_metric field
Field Description Example
config_count The number of Logtail configuration files. ​1
config_get_last_time The last time when the configuration file was obtained. ​1525686673
config_update_count The number of configuration updates after Logtail was started. ​1
config_update_item_count The total number of updated configuration items after Logtail was started. ​1
config_update_last_time The time of the last configuration update after Logtail was started. ​1525686663
event_tps The transactions per second (TPS). 1​
last_read_event_time The time when events were last read. ​1525686663
last_send_time The time when data was last sent. ​1525686663
open_fd The number of open log files. ​1
poll_modify_size The number of modified log files that are monitored. ​1
polling_dir_cache The number of scanned folders. ​1
polling_file_cache The number of scanned files. ​1
process_byte_ps The size of log data processed per second. Unit: bytes. ​1000
process_lines_ps The number of log entries processed per second. ​1000
process_queue_full The number of queues that reach the maximum data processing capacity. ​1
process_queue_total The number of queues that are processing data. ​10
process_tps The number of data processing transactions per second. ​0
reader_count The number of log files that are being processed. ​1
region The region to which Logtail belongs. ​cn-hangzhou, cn-shanghai
register_handler The number of folders to monitor. ​1
send_byte_ps The size of raw log data sent per second. Unit: bytes. ​11111
send_line_ps The number of log entries sent per second. 1000​
send_net_bytes_ps The size of data packets sent per second. Unit: bytes. 1000​
send_queue_full The number of queues that reach the maximum data sending capacity. ​1
send_queue_total The number of queues that send log data. ​12
send_tps The number of data sending transactions per second. ​0.075
sender_invalid The number of abnormal queues that send data. ​0