This topic describes the release notes for Simple Log Service Logtail and provides links to the relevant references.
The input_command plug-in is added to collect command execution results.
The processor_log_to_sls_metric plug-in is added to parse logs into metrics.
The processor_json plug-in can be used to parse JSON arrays.
Custom root file system (rootfs) paths and automatically discovered paths are supported when data is collected from a containerd container.
The reporting logic of the container metadata preview feature is optimized.
Container stdout cannot be collected when the path to the stdout file is a symbolic link. This issue is fixed.
A container cannot exit due to file descriptor (FD) locking when only a container deletion event occurs. This issue is fixed.
The nil panic error is reported by the service_go_profile plug-in. This issue is fixed.
Collection errors occur when disk-related metrics are collected. This issue is fixed by adding a timeout configuration.
The metadata of profile data is inaccurate in container scenarios. This issue is fixed.
OpenKruise metrics can be collected by Full-stack Monitoring.
Container metadata is added to Logtail profile data.
FDs are closed due to container exit. This issue is fixed.
Cache errors may occur when you configure environment variables in a Kubernetes cluster. This issue is fixed.
The startup parameter data_endpoint_policy is added to configure an endpoint switchover policy for Logtail.
The goProfile pull mode is supported by the profiling feature.
The processor_string_replace plug-in is added to replace strings.
HTTP_PROXY can be used to configure a network proxy for Logtail.
The performance of the processor_split_key_value plug-in is optimized, and multi-character quotes are added.
The statistical data of the disk usage metric that is collected by using the metric_system_v2 plug-in is inaccurate. This issue is fixed.
The Pyroscope protocol is supported by the HTTP input service.
Container information can be reported from clients to enhance the observability of container collection configurations.
More Logstore-specific configurations such as cold storage configurations are supported when data is automatically collected by running the env command.
The prometheus plug-in supports Staleness data.
If the system time fails to be used or the log time fails to be parsed, time-zone adjustment options may be ignored. This issue is fixed.
Logs may be repeatedly collected due to inode reuse. This issue is fixed.
The Grok plug-in may not work when the plug-in parses Chinese characters. This issue is fixed.
The memory that is occupied by containers in Logtail V1.2.1 is excessively high. This issue is fixed.
OpenTelemetry logs can be collected over HTTP.
The processor_desensitize plug-in is added. For more information, see Mask sensitive data.
In the container environment, HTTPS is supported when you use environment variables to create Simple Log Service resources.
In the container environment, Logstore specifications are configurable when you use custom resource definitions (CRDs) to create Logstores.
If Logtail plug-ins are used to process data, the offsets of content into files can be obtained.
By default, continuous context is retained when container stdout and stderr are collected.
Memory usage is optimized to collect Prometheus data.
In the Docker environment, FD leaks and event omissions may occur. This issue is fixed.
When a Logtail configuration is updated, file handle leaks may occur. This issue is fixed.
Special hostnames cannot be correctly resolved into IP addresses. This issue is fixed.
If multiple file paths are specified and several paths are nested, data is repeatedly collected from the same files. This issue is fixed.
JMX performance metrics can be collected.
The aggregator_context plug-in is added to aggregate contextual logs. For more information, see aggregators configuration.
The contextual query and LiveTail features are available for logs that are processed by using Logtail plug-ins. The __topic__ field can be extracted from the logs that are processed by using Logtail plug-ins.
The Grok plug-in is added to extract log fields. For more information, see Form configuration.
Multiple Grok expressions can be specified at a time to match multiple formats of logs.
The stdout and stderr of containers that exit in seconds after the containers are started can be collected.
The timestamps that are accurate to the microsecond in Apsara logs cannot be parsed. This issue is fixed.
A new Logtail plug-in is added to process CSV data. For more information, see Form configuration.
eBPF can be used to analyze network traffic at Layer 4 and Layer 7. The HTTP, MySQL, PostgreSQL, Redis, and Domain Name System (DNS) protocols are supported.
The netping plug-in supports httping and DNS resolution, which are time-consuming.
The SkyWalking Logging API is added.
The quick release of file handles is supported for files in a containerd container that is stopped.
The Kubernetes labels of containerd containers cannot be used for matching. This issue is fixed.
When Logtail collects text logs, the extended setting "enable_precise_timestamp": true and the processor_strptime plug-in can be used to parse high-precision time values.
The detection mechanism of the rootfs in a Kubernetes cluster is optimized.
The recognition mechanism of container runtimes in a Kubernetes cluster is optimized.
Exceptions that are related to the netping plug-in occur in a Windows system. This issue is fixed.
Environment variables can be used to create a Logtail configuration.
The netping plug-in is added. You can use the plug-in to collect data about ping commands when you ping a destination by IP address.
The gotime plug-in can be used to convert the extracted log time to the timestamp format.
When Logtail collects syslog logs, the_client_ip_ field is added to indicate the IP address of the syslog client that transfers logs.
The memory that is used to collect container stdout and stderr is optimized.
The Prometheus plug-in can be used to collect Prometheus data from the same machine by using multiple Logtail configurations.
Windows nodes in Container Service for Kubernetes clusters support authentication by using add-on tokens.
The process collection plug-in encounters errors in the threadNum and fdNum metrics in Linux. This issue is fixed.
The plug-ins of SkyWalking encounter the ConfigurationDiscoveryService not implement error. This issue is fixed.
When Logtail collects container stdout and stderr, labels cannot be matched by using regular expressions. This issue occurs in Logtail V1.0.27 and Logtail V1.0.28. This issue is fixed.
Simple Network Management Protocol (SNMP) data can be collected.
In the plug-ins of SkyWalking V3, instances can be filtered by using instance attributes.
The Incomplete Entry Upload parameter can be configured when the delimiter mode is used.
The ID of a span in the plug-ins of SkyWalking V2 is incorrect. This issue is fixed.
The processor_regex plug-in is added.
Multi-region control can be configured.
The IO Counter metric can be collected from hosts.
The service_http_server plug-in does not release UNIX links. This issue is fixed.
When Logtail runs multiple Logtail configurations that use the metric_meta_kubernetes plug-in, conflicts occur. This issue is fixed.
Process metrics can be collected.
File handles and TCP data can be collected from hosts.
The metadata of Kubernetes clusters can be collected.
The metadata of hosts can be collected.
The gRPC output plug-in is added.
When Logtail collects container logs, semantic recognition can be performed in Kubernetes clusters.
The trace data of SkyWalking V2 can be collected.
The input_canal plug-in is supported on Windows systems that use the i386 architecture.
The accuracy of metrics that are collected from hosts is improved. Containers are deployed on the hosts.
When you import historical data, a breakdown may occur. This issue is fixed.
The logic that processes inaccurate metadata is improved. The metadata is returned by the readdir function of file systems.
When Logtail is just started, the data that Logtail sends does not contain a custom identifier. This issue is fixed.
When network interruptions occur in global acceleration mode, Logtail may stop submitting status data that is generated by the system to Simple Log Service. This issue is fixed.
As a minor version of Logtail V1.0, Logtail V1.0.21 is the first version that is released in all regions. Logtail V1.0.21 includes all features of Logtail V0.16.64 and provides the following new features:
The exactly_once_concurrency parameter is added to record the fine-grained checkpoints by file on local disks. For more information, see Logtail configurations.
The enable_log_time_auto_adjust parameter is added to allow the system to automatically set the log time to the local time of the server. For more information, see Configure the startup parameters of Logtail.
The enable_log_position_meta parameter is introduced to add the metadata of a source log file to a collected log. For more information, see Logtail configurations.
The specified_year parameter is added to supplement the log time by using the current year or a specified year. For more information, see Logtail configurations.
When Logtail collects container stdout and stderr, Simple Log Service fails to parse the collected data because the P (partial) tag of the data cannot be identified. This issue is fixed.
When the service_skywalking_agent_v3 plug-in is installed on multiple application servers, the ID of a span and the ID of a parent span in the Links parameter are incorrect. This issue is fixed.
The timeout period for requesting a container engine is increased from 3 seconds to 30 seconds. The DOCKER_CLIENT_REQUEST_TIMEOUT environment variable is added to specify the timeout period for requesting a container engine.
The ID of a parent span in the service_skywalking plug-in is incorrect. This issue is fixed.
When a container engine has an exception, the code that is used to create a Logtail configuration based on environment variables stops running. This issue is fixed.
If you are using Logtail V0.16.58 or Logtail V0.16.60, we recommend that you update Logtail to V0.16.62.
Data fails to be sent due to out-of-order data. This issue is fixed.
Container data can be collected in the containerd environment.
The net_err_stat metric in service logs is adjusted to record only sending errors that are caused by network issues.
The net_err_stat metric is added to service logs. The number of sending errors that occur in the last 1, 5, or 15 minutes can be recorded.
If a large number of Logtail configurations are created to collect container stdout and stderr and logs in container files, we recommend that you update Logtail to V0.16.52 or later. This helps you reduce the CPU utilization.
The CPU utilization is reduced during container data collection.
The service_telegraf plug-in can be installed at runtime. This feature is available only for Elastic Compute Service (ECS) users.
The service_telegraf plug-in is optimized. Multiple configurations are supported on a single machine.
If you update Logtail to V0.16.46 or later in the China (Hangzhou), China (Shanghai), or China (Beijing) region, Logtail does not switch endpoints even when network jitter occurs.
The network types that are available for Logtail are strictly limited.
The service_telegraf plug-in is added to collect metric data.
When you use a blacklist to filter data, multi-level matching is supported. Example: /path/**/log.
The policy that is used to obtain local IP addresses is optimized. If a policy is invalid, the first IP address in the IP address list is obtained.
The metric_system_v2 plug-in is added to collect the data about host status.
The max_docker_config_update_times parameter that corresponds to the ALIYUN_LOGTAIL_MAX_DOCKER_CONFIG_UPDATE_TIMES environment variable is added. This parameter is suitable for scenarios in which you need to frequently create short-lived jobs in a Kubernetes environment.
The CPU utilization is reduced for scenarios in which a large number of Logtail configurations are created to collect container data.
The processor_split_log_string plug-in generates blank lines. This issue is fixed.
A custom name is supported for the time field in full regex mode.
The KeepSourceIfParseError parameter is added in the processor_json, processor_regex, and processor_split_char plug-ins. If raw data fails to be parsed, the raw data can be retained. For more information, see Use Logtail plug-ins to process data.
The processor_encrypt plug-in is added for encryption.
An HTTP probe is added for Kubernetes health check.
Core dumps are caused by libcurl in some environments. This issue is fixed.
When you install Logtail in CentOS 8, the GNU Libidn library cannot be found. This issue is fixed.
The IgnoreFirstConnector parameter is added in the processor_json plug-in. For more information, see Expand JSON fields.
If you run Logtail V0.16.30 for a long period of time, files may fail to be opened. We recommend that you update Logtail to the latest version.
When Logtail collects Docker stdout and stderr and logs in container files from Kubernetes clusters, logs can be filtered based on a Kubernetes cluster whitelist.
The concurrency competition among Logstores in the same region due to poor network connections is diluted.
When a file is opened, checkpoint loss may occur due to logic errors. This issue is fixed.
A parameter is added to specify a tail size for the file from which logs are collected for the first time.
The logic of container metadata collection is optimized to reduce the impact of faulty containers.
The docker_stdout process causes memory leaks in complex environments. This issue is fixed.
In JSON mode, Logtail does not provide full support for timestamps with millisecond precision. This issue is fixed.
containerd logs can be collected.
The checkpoints of rotated files cannot be found. This issue is fixed.
If the /usr/local/ilogtail/user_log_config.json file does not exist, the local Logtail configuration /etc/ilogtail/user_config.d is not loaded. This issue is fixed.
The working_ip and working_hostname parameters can be configured by using environment variables.
The force_quit_read_timeout parameter is added to specify the timeout period before Logtail is forced to quit when events are blocked and fail to be read.
Tags such as path and topic can be passed to plug-ins.
The aggregator_shardhash plug-in is added, and the shardhash parameter can be configured in the plug-in.
The processor_gotime, processor_rename, processor_add_fields, processor_json, and processor_packjson plug-ins are added for data processing. For more information, see Use Logtail plug-ins to process data.
LogtailInsight is updated. The progress viewing feature is added. This feature can be used only after you configure the mark_offset_global_flag or customized_fields.mark_offset parameter.
If the Journal plug-in runs for a long period of time and causes high memory usage, the memory is released at the earliest opportunity.
The interval at which an on-premises machine obtains its first Logtail configuration is reduced.
If multiple Logtail configurations are used, duplicate data may be collected. This issue is fixed.
Timestamps with millisecond or microsecond precision do not support JSON int64. This issue is fixed.
Timestamps with millisecond or microsecond precision (%s) are supported.
Multiple local Logtail configurations can be loaded. File path: /etc/ilogtail/config.d/.
Multiple local user configurations can be loaded. File path: /etc/ilogtail/user_config.d/.
The oas_connect_timeout and oas_request_timeout parameters are added to provide support for the scenarios of slow networks.
The limits on the inputs parameter in hybrid configurations (file + plug-in) are removed.
Custom static topics can be configured.
Blacklist-based filtering is supported.
The EnableEventMeta parameter is added to the service_canal plug-in. You can configure the parameter to collect the header information of MySQL binary logs.
The stop mechanism of plug-ins is optimized.
GBK logs may cause memory leaks. This issue is fixed.
The memory usage for container logs and plug-in-based collection is reduced.
The collection of multi-line stdout from containers is optimized.
The resources for Kubernetes audit logs can be automatically created.
The startup parameters can be configured by using environment variables. The startup parameters include parameters related to CPUs, memory, and sending concurrency.
The upload of custom tags can be configured by using environment variables.
Logtail configurations can be automatically created in Sidecar mode. For more information, see Use CRDs to collect container text logs in Sidecar mode.
AliUid files can be automatically saved to your computer.
When Logtail collects logs from container files, a breakdown may occur. This issue is fixed.
IncludeLabelparameter of a Logtail configuration that is created by using environment variables does not take effect in Kubernetes. This issue is fixed.
When Logtail collects MySQL binary logs, the global transaction identifier (GTID) feature is automatically enabled.
When you import historical data, specified wildcard characters are supported for file names.
Indexes can be automatically created for Kubernetes logs.
If a log fails to be split into multiple lines, the error can be troubleshot by checking the
discardUnMatchparameter, and the log can be reported.
If an unknown sending error occurs, Logtail retries to collect data. This prevents data loss in some cases, such as when a packet is modified during the sending process.
When you import historical data, the wildcard mode is supported.
default_tail_limit_kbstartup parameter is added to specify the size of data that is collected from a file the first time Logtail reads the file. Default value: 1024. Unit: KB.
batch_send_secondsparameter is added to specify the interval at which data packets are sent during log collection.
batch_send_bytesparameter is added to specify the size of a data packet that is sent during log collection.
When Logtail collects container stdout, the logs that are split by Docker Engine can be automatically merged.
Log collection can be configured by using environment variables.
The metadata of MySQL binary logs can be collected, such as the
When Logtail is installed in a VPC, the parameters in the installation script can be automatically selected.
When Logtail is installed, the global acceleration mode can be used. For more information, see Configure log collection acceleration for Logtail.
When Logtail collects MySQL binary logs, the information of the _filename_ and _offset_ fields can be collected.
When Logtail collects container stdout in multi-line mode, exceptions may occur. This issue is fixed.
The method that is used to collect container stdout is upgraded.
The leak of socket fd may occur. This issue is fixed.
The limits on the update frequency of Logtail configurations are added for container files.
The Lumberjack plug-in is added to collect data from Logstash and Beats. For more information, see Collect data from Beats and Logstash.
The inotify blacklist feature is added.
The parameters in earlier versions of installation packages are inconsistent. This issue is fixed.
When you install Logtail in some systems, the versions of the operating systems cannot be obtained. This issue is fixed.
Host monitoring data can be collected.
Redis monitoring data can be collected.
DDL statements in MySQL binary logs can be collected.
When Logtail collects container stdout and logs in container files, logs can be filtered by using Docker environment variables.
Data cannot be collected from a MySQL table that does not have a primary key. This issue is fixed.
Event loss occurs due to unstable subscription channels of container engines in container collection mode. This issue is fixed.
The multi-line mode is supported to collect container stdout. For more information, see Examples of Logtail configurations for multi-line logs.
The deployment scenarios of Docker and Kubernetes are supported.
Container stdout and logs in container files can be collected. For more information, see Use the Simple Log Service console to collect container stdout and stderr in DaemonSet mode and Use the Simple Log Service console to collect container text logs in DaemonSet mode.
The processor_geoip plug-in is added. For more information, see Convert IP addresses.
MySQL binary logs, MySQL query results, and HTTP data can be collected. For more information, see Use Logtail plug-ins to collect data.
The following configuration items can be combined to parse data: regular expressions, anchors, delimiters, and filters.