Errors may occur during log collection by Logtail, such as regular expression parsing failures, incorrect file paths, and traffic exceeding the shard service capability. Currently, the query function is provided for debugging log collection errors.
- Enter the error diagnosis page
Enter the Logstore List page by clicking a project name. Click Diagnose at the right of the Logstore. The Log Collection Error dialog box appears.
- View log collection errors.
You can view the list of Logtail log collection errors of a specified Logstore in the Log Collection Error dialog box.
- Query log collection errors of a specified machine
To query all log collection errors occurred to a specific machine, enter the IP address of the machine in the search box on the query page. Logtail reports errors every 5 minutes.
After fixing these errors and resuming business, check if the errors persist based on the timeframe. Historical error reports are still displayed before they expire. Ignore the historical error reports and only check whether new errors occurred after these historical errors are fixed.
What to do next
|Error type||Description||Handling method|
|LOGFILE_PERMINSSION_ALARM||Logtail has no permission to read the specified file.||Check the Logtail startup account on the server. We recommend that Logtail be started by a root user.|
|SPLIT_LOG_FAIL_ALARM||The regular expression cannot match with the beginning of a line and cannot separate logs into lines.||Check whether the regular expression is correct or not.
|MULTI_CONFIG_MATCH_ALARM||A file can only be collected by one Logtail configuration.||Check whether a file is collected by multiple Logtail configurations. If yes, delete the redundant configurations.|
|REGEX_MATCH_ALARM||In the regular expression parsing mode, the log content does not match with the regular expression.||Copy the log sample in the error content to retry the matching and generate new regular expression.|
|PARSE_LOG_FAIL_ALARM||Logtail fails to parse logs because the log format does not conform to the definition in the parsing modes such as JSON and delimiter.||Click the error to view relevant details.|
|CATEGORY_CONFIG_ALARM||The Logtail collection configuration is invalid.||A common error is that the regular expression fails to extract the file path as a topic. For other errors, open a ticket.|
|LOGTAIL_CRASH_ALARM||Logtail has crashed because it exceeds the upper limit of machine resource usage.||For more information, see Configure startup parameters to modify the upper limits of CPU usage and memory usage. Open a ticket if you have any questions.|
|REGISTER_INOTIFY_FAIL_ALARM||Logtail fails to register log listening in Linux possibly because Logtail does not have permissions to access the folder or the folder has been deleted.||Check if Logtail has permissions to access the folder or the folder is deleted.|
|DISCARD_DATA_ALARM||This error is caused by the insufficient CPU resources configured for Logtail or the flow control of network sending.||For more information, see Configure startup parameters to modify the upper limit of CPU usage or the limit on concurrent sending by using network. Open a ticket if you have any questions.|
|SEND_DATA_FAIL_ALARM||(1) The main account does not create any AccessKey. (2) The Logtail client cannot connect to Log Service, or the network link quality is bad. (3) The writing quota of Log Service is insufficient.||1. Use the main account to log on to the AccessKey console to create an AccessKey. 2. Check the local configuration file /usr/local/ilogtail/ilogtail_config.json. Run the curl <endpoint>(3) Add shards to the Logstore to support writing larger data volumes.|
|PARSE_TIME_FAIL_ALARM||Logtail fails to parse the time field according to the time parsing expression.||Configure the time parsing expression correctly according to the log time.|
|REGISTER_INOTIFY_FAIL_ALARM||Logtail fails to register inotify watcher for the log directory.||Check whether the directory exists and the directory permission settings.|
|SEND_QUOTA_EXCEED_ALARM||The traffic of writing logs exceeds the limit.||Split the shard for expansion in the console.|
|READ_LOG_DELAY_ALARM||Log collection lags behind log generation. Generally, this error is caused by the insufficient CPU resources configured for Logtail or the network bandwidth throttling.||For more information, see LogtailConfigure startup parametersto modify the upper limit of CPU usage or the limit on concurrent sending by using network. Open a ticket if you have any questions.|
|DROP_LOG_ALARM||Log collection lags behind log generation, and the number of unprocessed log rotations is more than 20. Generally, this error is caused by the insufficient CPU resources configured for Logtail or the flow control of network sending.||For more information, see LogtailConfigure startup parametersto modify the upper limit of CPU usage or the limit on concurrent sending by using network. Open a ticket if you have any questions.|
|LOGDIR_PERMINSSION_ALARM||Logtail has no permission to read the log monitoring directory.||Check whether the log monitoring directory exists. If yes, check the directory permission settings.|
|ENCODING_CONVERT_ALARM||Logtail fails to convert the encoding.||Check whether the log encoding format configuration is consistent with the log encoding format.|
|OUTDATED_LOG_ALAR||Logs expire with a time lag of more than 12 hours. Possible causes: Log parsing lags behind by more than 12 hours, the user-defined time field is incorrectly configured, or the time output of the logging program is abnormal.||Check whether READ_LOG_DELAY_ALARM exists. If yes, handle the error according to the instructions of READ_LOG_DELAY_ALARM. If not, check the time field. If the time field is correctly configured, check whether the time output of the logging program is normal. Open a ticket if you have any questions.|
|STAT_LIMIT_ALARM||The number of files in the log collection configuration directory exceeds the limit.||Check whether or not the log collection configuration directory contains many files and subdirectories, and properly configure the monitored root directory and the maximum monitoring depth of the directory.|
|DROP_DATA_ALARM||Timeout for flushing logs to the local machine when the process exits. Logs without being flushed are discarded.||Generally, this error is caused by the severe collection obstruction. For more information, see Configure startup parameters to modify the upper limit of CPU usage or the limit on concurrent sending by using network. Open a ticket if you have any questions.|
|INPUT_COLLECT_ALARM||An exception occurred when collecting the input sources.||Handle the error as instructed by the error message.|
|HTTP_LOAD_ADDRESS_ALARM||The entered HTTP address is invalid.||Check the address validity.|
|HTTP_COLLECT_ALARM||An exception occurred when collecting HTTP.||Troubleshoot the error as instructed by the error message. Generally, this error is caused by timeout.|
|FILTER_INIT_ALARM||An exception occurred when initiating the filter.||Generally, this error is caused by the invalid regular expression of the filter. Handle the error as instructed.|
|INPUT_CANAL_ALARM||An exception occurred when running MySQL binlog.||Troubleshoot the error as instructed by the error message. The canal service may be restarted when the configuration is updated. Therefore, you can ignore the service restart error.|
|CANAL_INVALID_ALARM||An exception in the internal status of MySQL binlog.||The table schema modification during the running time leads to the inconsistent meta and then causes this error. Check if the table schema is modified when the error is reported. For other cases, open a ticket.|
|MYSQL_INIT_ALARM||An exception occurred when initiating MySQL.||Handle the error as instructed by the error message.|
|MYSQL_CHECKPOING_ALARM||An exception in the MySQL checkpoint format.||Check whether or not to modify the checkpoint configurations. For other cases, open a ticket.|
|MYSQL_TIMEOUT_ALARM||Timeout for querying MySQL.||Check whether the MySQL server and the network are abnormal.|
|MYSQL_PARSE_ALARM||Logtail fails to parse the MySQL query results.||Check whether the checkpoint format configured by MySQL matches with the format of the specified field.|
|To view all the complete log lines that are discarded because of parsing failure, log on to the machine to view the /usr/local/ilogtail/ilogtail.LOG file.|