Log Service:Release notes

1.7.1

• New features

  • The input_command plug-in is added to collect command execution results.

  • The processor_log_to_sls_metric plug-in is added to parse logs into metrics.

• Optimization

  • The processor_json plug-in can be used to parse JSON arrays.

  • Custom root file system (rootfs) paths and automatically discovered paths are supported when data is collected from a containerd container.

  • The reporting logic of the container metadata preview feature is optimized.

• Fixed issues

  • Container stdout cannot be collected when the path to the stdout file is a symbolic link. This issue is fixed.

  • A container cannot exit due to file descriptor (FD) locking when only a container deletion event occurs. This issue is fixed.

  • The nil panic error is reported by the service_go_profile plug-in. This issue is fixed.

  • Collection errors occur when disk-related metrics are collected. This issue is fixed by adding a timeout configuration.

  • The metadata of profile data is inaccurate in container scenarios. This issue is fixed.

1.6.0

• New features

  OpenKruise metrics can be collected by Full-stack Monitoring.

• Optimization

  Container metadata is added to Logtail profile data.

• Fixed issues

  • FDs are closed due to container exit. This issue is fixed.

  • Cache errors may occur when you configure environment variables in a Kubernetes cluster. This issue is fixed.

1.5.1

• New features

  • The startup parameter data_endpoint_policy is added to configure an endpoint switchover policy for Logtail.

  • The goProfile pull mode is supported by the profiling feature.

  • The processor_string_replace plug-in is added to replace strings.

  • HTTP_PROXY can be used to configure a network proxy for Logtail.

• Optimization

  The performance of the processor_split_key_value plug-in is optimized, and multi-character quotes are added.

• Fixed issues

  The statistical data of the disk usage metric that is collected by using the metric_system_v2 plug-in is inaccurate. This issue is fixed.

1.4.0

• New features

  • The Pyroscope protocol is supported by the HTTP input service.

  • Container information can be reported from clients to enhance the observability of container collection configurations.

• Optimization

  More Logstore-specific configurations such as cold storage configurations are supported when data is automatically collected by running the env command.

1.3.2

• Optimization

  The prometheus plug-in supports Staleness data.

• Fixed issues

  • If the system time fails to be used or the log time fails to be parsed, time-zone adjustment options may be ignored. This issue is fixed.

  • Logs may be repeatedly collected due to inode reuse. This issue is fixed.

  • The Grok plug-in may not work when the plug-in parses Chinese characters. This issue is fixed.

  • The memory that is occupied by containers in Logtail V1.2.1 is excessively high. This issue is fixed.

1.3.1

• New features

  • OpenTelemetry logs can be collected over HTTP.

  • The processor_desensitize plug-in is added. For more information, see Mask sensitive data.

• Optimization

  • In the container environment, HTTPS is supported when you use environment variables to create Simple Log Service resources.

  • In the container environment, Logstore specifications are configurable when you use custom resource definitions (CRDs) to create Logstores.

  • If Logtail plug-ins are used to process data, the offsets of content into files can be obtained.

  • By default, continuous context is retained when container stdout and stderr are collected.

  • Memory usage is optimized to collect Prometheus data.

• Fixed issues

  • In the Docker environment, FD leaks and event omissions may occur. This issue is fixed.

  • When a Logtail configuration is updated, file handle leaks may occur. This issue is fixed.

  • Special hostnames cannot be correctly resolved into IP addresses. This issue is fixed.

  • If multiple file paths are specified and several paths are nested, data is repeatedly collected from the same files. This issue is fixed.

1.2.1

• New features

  • The query results from SQL Server databases and PostgreSQL databases can be collected. For more information, see Collect SQL Server query results and Collect PostgreSQL query results.

  • JMX performance metrics can be collected.

  • The aggregator_context plug-in is added to aggregate contextual logs. For more information, see aggregators configuration.

    The contextual query and LiveTail features are available for logs that are processed by using Logtail plug-ins. The __topic__ field can be extracted from the logs that are processed by using Logtail plug-ins.

  • The Grok plug-in is added to extract log fields. For more information, see Form configuration.

    Multiple Grok expressions can be specified at a time to match multiple formats of logs.

• Optimization

  The stdout and stderr of containers that exit in seconds after the containers are started can be collected.

• Fixed issues

  The timestamps that are accurate to the microsecond in Apsara logs cannot be parsed. This issue is fixed.

1.1.1

• New features

  • A new Logtail plug-in is added to process CSV data. For more information, see Form configuration.

  • eBPF can be used to analyze network traffic at Layer 4 and Layer 7. The HTTP, MySQL, PostgreSQL, Redis, and Domain Name System (DNS) protocols are supported.

1.1.0

• New features

  The netping plug-in supports httping and DNS resolution, which are time-consuming.

1.0.34

• New features

  The SkyWalking Logging API is added.

• Optimization

  The quick release of file handles is supported for files in a containerd container that is stopped.

• Fixed issues

  The Kubernetes labels of containerd containers cannot be used for matching. This issue is fixed.

1.0.32

• New features

  When Logtail collects text logs, the extended setting "enable_precise_timestamp": true and the processor_strptime plug-in can be used to parse high-precision time values.

• Optimization

  • The detection mechanism of the rootfs in a Kubernetes cluster is optimized.

  • The recognition mechanism of container runtimes in a Kubernetes cluster is optimized.

• Fixed issues

  Exceptions that are related to the netping plug-in occur in a Windows system. This issue is fixed.

1.0.31

• New features

  • Environment variables can be used to create a Logtail configuration.

  • The netping plug-in is added. You can use the plug-in to collect data about ping commands when you ping a destination by IP address.

  • The gotime plug-in can be used to convert the extracted log time to the timestamp format.

  • When Logtail collects syslog logs, the_client_ip_ field is added to indicate the IP address of the syslog client that transfers logs.

• Optimization

  The memory that is used to collect container stdout and stderr is optimized.

1.0.30

• New features

  • The Prometheus plug-in can be used to collect Prometheus data from the same machine by using multiple Logtail configurations.

  • Windows nodes in Container Service for Kubernetes clusters support authentication by using add-on tokens.

• Fixed issues

  • The process collection plug-in encounters errors in the threadNum and fdNum metrics in Linux. This issue is fixed.

  • The plug-ins of SkyWalking encounter the ConfigurationDiscoveryService not implement error. This issue is fixed.

1.0.29

• Fixed issues

  When Logtail collects container stdout and stderr, labels cannot be matched by using regular expressions. This issue occurs in Logtail V1.0.27 and Logtail V1.0.28. This issue is fixed.

1.0.28

• New features

  • Simple Network Management Protocol (SNMP) data can be collected.

  • In the plug-ins of SkyWalking V3, instances can be filtered by using instance attributes.

  • The Incomplete Entry Upload parameter can be configured when the delimiter mode is used.

• Fixed issues

  The ID of a span in the plug-ins of SkyWalking V2 is incorrect. This issue is fixed.

1.0.27

• New features

  • The processor_regex plug-in is added.

  • Multi-region control can be configured.

• Optimization

  The IO Counter metric can be collected from hosts.

• Fixed issues

  • The service_http_server plug-in does not release UNIX links. This issue is fixed.

  • When Logtail runs multiple Logtail configurations that use the metric_meta_kubernetes plug-in, conflicts occur. This issue is fixed.

1.0.26

• New features

  • Process metrics can be collected.

  • File handles and TCP data can be collected from hosts.

  • The metadata of Kubernetes clusters can be collected.

  • The metadata of hosts can be collected.

  • The gRPC output plug-in is added.

  • When Logtail collects container logs, semantic recognition can be performed in Kubernetes clusters.

  • The trace data of SkyWalking V2 can be collected.

  • The input_canal plug-in is supported on Windows systems that use the i386 architecture.

• Optimization

  The accuracy of metrics that are collected from hosts is improved. Containers are deployed on the hosts.

1.0.25

• Fixed issues

  When you import historical data, a breakdown may occur. This issue is fixed.

• Optimization

  The logic that processes inaccurate metadata is improved. The metadata is returned by the readdir function of file systems.

1.0.24

• Fixed issues

  When Logtail is just started, the data that Logtail sends does not contain a custom identifier. This issue is fixed.

1.0.22

• Fixed issues

  When network interruptions occur in global acceleration mode, Logtail may stop submitting status data that is generated by the system to Simple Log Service. This issue is fixed.

1.0.21

As a minor version of Logtail V1.0, Logtail V1.0.21 is the first version that is released in all regions. Logtail V1.0.21 includes all features of Logtail V0.16.64 and provides the following new features:

• New features

  • The exactly_once_concurrency parameter is added to record the fine-grained checkpoints by file on local disks. For more information, see Logtail configurations.

  • The enable_log_time_auto_adjust parameter is added to allow the system to automatically set the log time to the local time of the server. For more information, see Configure the startup parameters of Logtail.

  • The enable_log_position_meta parameter is introduced to add the metadata of a source log file to a collected log. For more information, see Logtail configurations.

  • The specified_year parameter is added to supplement the log time by using the current year or a specified year. For more information, see Logtail configurations.

0.16.68

• Fixed issues

  • When Logtail collects container stdout and stderr, Simple Log Service fails to parse the collected data because the P (partial) tag of the data cannot be identified. This issue is fixed.

  • When the service_skywalking_agent_v3 plug-in is installed on multiple application servers, the ID of a span and the ID of a parent span in the Links parameter are incorrect. This issue is fixed.

0.16.64

• Optimization

  The timeout period for requesting a container engine is increased from 3 seconds to 30 seconds. The DOCKER_CLIENT_REQUEST_TIMEOUT environment variable is added to specify the timeout period for requesting a container engine.

• Fixed issues

  • The ID of a parent span in the service_skywalking plug-in is incorrect. This issue is fixed.

  • When a container engine has an exception, the code that is used to create a Logtail configuration based on environment variables stops running. This issue is fixed.

0.16.62

• Fixed issues

  Data fails to be sent due to out-of-order data. This issue is fixed.

0.16.60

• New features

  Container data can be collected in the containerd environment.

0.16.56

• Optimization

  The net_err_stat metric in service logs is adjusted to record only sending errors that are caused by network issues.

0.16.54

• New features

  The net_err_stat metric is added to service logs. The number of sending errors that occur in the last 1, 5, or 15 minutes can be recorded.

0.16.52

If a large number of Logtail configurations are created to collect container stdout and stderr and logs in container files, we recommend that you update Logtail to V0.16.52 or later. This helps you reduce the CPU utilization.

• Optimization

  The CPU utilization is reduced during container data collection.

0.16.50

• New features

  The service_telegraf plug-in can be installed at runtime. This feature is available only for Elastic Compute Service (ECS) users.

0.16.48

• Optimization

  The service_telegraf plug-in is optimized. Multiple configurations are supported on a single machine.

0.16.46

• Optimization

  The network types that are available for Logtail are strictly limited.

0.16.44

• New features

  The service_telegraf plug-in is added to collect metric data.

0.16.42

• New features

  When you use a blacklist to filter data, multi-level matching is supported. Example: /path/**/log.

• Optimization

  The policy that is used to obtain local IP addresses is optimized. If a policy is invalid, the first IP address in the IP address list is obtained.

0.16.40

• New features

  • The metric_system_v2 plug-in is added to collect the data about host status.

  • The max_docker_config_update_times parameter that corresponds to the ALIYUN_LOGTAIL_MAX_DOCKER_CONFIG_UPDATE_TIMES environment variable is added. This parameter is suitable for scenarios in which you need to frequently create short-lived jobs in a Kubernetes environment.

• Optimization

  The CPU utilization is reduced for scenarios in which a large number of Logtail configurations are created to collect container data.

• Fixed issues

  The processor_split_log_string plug-in generates blank lines. This issue is fixed.

0.16.38

• New features

  • A custom name is supported for the time field in full regex mode.

  • The KeepSourceIfParseError parameter is added in the processor_json, processor_regex, and processor_split_char plug-ins. If raw data fails to be parsed, the raw data can be retained. For more information, see Use Logtail plug-ins to process data.

0.16.36

• New features

  The processor_encrypt plug-in is added for encryption.

0.16.34

• New features

  An HTTP probe is added for Kubernetes health check.

• Fixed issues

  • Core dumps are caused by libcurl in some environments. This issue is fixed.

  • When you install Logtail in CentOS 8, the GNU Libidn library cannot be found. This issue is fixed.

0.16.32

• New features

  The IgnoreFirstConnector parameter is added in the processor_json plug-in. For more information, see Expand JSON fields.

0.16.30

• New features

  When Logtail collects Docker stdout and stderr and logs in container files from Kubernetes clusters, logs can be filtered based on a Kubernetes cluster whitelist.

• Optimization

  The concurrency competition among Logstores in the same region due to poor network connections is diluted.

• Fixed issues

  When a file is opened, checkpoint loss may occur due to logic errors. This issue is fixed.

0.16.28

• New features

  A parameter is added to specify a tail size for the file from which logs are collected for the first time.

• Optimization

  The logic of container metadata collection is optimized to reduce the impact of faulty containers.

• Fixed issues

  • The docker_stdout process causes memory leaks in complex environments. This issue is fixed.

  • In JSON mode, Logtail does not provide full support for timestamps with millisecond precision. This issue is fixed.

0.16.26

• New features

  containerd logs can be collected.

• Fixed issues

  • The checkpoints of rotated files cannot be found. This issue is fixed.

  • If the /usr/local/ilogtail/user_log_config.json file does not exist, the local Logtail configuration /etc/ilogtail/user_config.d is not loaded. This issue is fixed.

0.16.24

• New features

  • The working_ip and working_hostname parameters can be configured by using environment variables.

  • The force_quit_read_timeout parameter is added to specify the timeout period before Logtail is forced to quit when events are blocked and fail to be read.

  • Tags such as path and topic can be passed to plug-ins.

  • The aggregator_shardhash plug-in is added, and the shardhash parameter can be configured in the plug-in.

  • The processor_gotime, processor_rename, processor_add_fields, processor_json, and processor_packjson plug-ins are added for data processing. For more information, see Use Logtail plug-ins to process data.

  • LogtailInsight is updated. The progress viewing feature is added. This feature can be used only after you configure the mark_offset_global_flag or customized_fields.mark_offset parameter.

• Optimization

  • If the Journal plug-in runs for a long period of time and causes high memory usage, the memory is released at the earliest opportunity.

  • The interval at which an on-premises machine obtains its first Logtail configuration is reduced.

• Fixed issues

  • If multiple Logtail configurations are used, duplicate data may be collected. This issue is fixed.

  • Timestamps with millisecond or microsecond precision do not support JSON int64. This issue is fixed.

0.16.23

• New features

  • Timestamps with millisecond or microsecond precision (%s) are supported.

  • Multiple local Logtail configurations can be loaded. File path: /etc/ilogtail/config.d/.

  • Multiple local user configurations can be loaded. File path: /etc/ilogtail/user_config.d/.

  • The processor_split_key_value and processor_strptime plug-ins are added for data processing. For more information, see Form configuration and Extract log time.

  • The oas_connect_timeout and oas_request_timeout parameters are added to provide support for the scenarios of slow networks.

• Optimization

  The limits on the inputs parameter in hybrid configurations (file + plug-in) are removed.

0.16.21

• New features

  • Custom static topics can be configured.

  • Blacklist-based filtering is supported.

  • The EnableEventMeta parameter is added to the service_canal plug-in. You can configure the parameter to collect the header information of MySQL binary logs.

• Optimization

  The stop mechanism of plug-ins is optimized.

• Fixed issues

  GBK logs may cause memory leaks. This issue is fixed.

0.16.18

• New features

  • Docker events can be collected. For more information, see Collect Docker events.

  • systemd-journald logs can be collected. For more information, see Collect systemd journal logs.

  • The processor_pick_key and processor_drop_last_key plug-ins are added for data processing.

• Optimization

  • The memory usage for container logs and plug-in-based collection is reduced.

  • The collection of multi-line stdout from containers is optimized.

0.16.16

• New features

  • The resources for Kubernetes audit logs can be automatically created.

  • The startup parameters can be configured by using environment variables. The startup parameters include parameters related to CPUs, memory, and sending concurrency.

  • The upload of custom tags can be configured by using environment variables.

  • Logtail configurations can be automatically created in Sidecar mode. For more information, see Use CRDs to collect container text logs in Sidecar mode.

• Optimization

  AliUid files can be automatically saved to your computer.

• Fixed issues

  • When Logtail collects logs from container files, a breakdown may occur. This issue is fixed.

  • The IncludeLabel parameter of a Logtail configuration that is created by using environment variables does not take effect in Kubernetes. This issue is fixed.

0.16.15

• New features

  • When Logtail collects MySQL binary logs, the global transaction identifier (GTID) feature is automatically enabled.

  • When you import historical data, specified wildcard characters are supported for file names.

  • Indexes can be automatically created for Kubernetes logs.

• Optimization

  • If a log fails to be split into multiple lines, the error can be troubleshot by checking the discardUnMatch parameter, and the log can be reported.

  • If an unknown sending error occurs, Logtail retries to collect data. This prevents data loss in some cases, such as when a packet is modified during the sending process.

0.16.14

• New features

  • When you import historical data, the wildcard mode is supported.

  • The default_tail_limit_kb startup parameter is added to specify the size of data that is collected from a file the first time Logtail reads the file. Default value: 1024. Unit: KB.

  • The batch_send_seconds parameter is added to specify the interval at which data packets are sent during log collection.

  • The batch_send_bytes parameter is added to specify the size of a data packet that is sent during log collection.

• Optimization

  When Logtail collects container stdout, the logs that are split by Docker Engine can be automatically merged.

0.16.13

0.16.11

• Optimization

  When Logtail collects MySQL binary logs, the information of the _filename_ and _offset_ fields can be collected.

• Fixed issues

  When Logtail collects container stdout in multi-line mode, exceptions may occur. This issue is fixed.

0.16.10

• Optimization

  The method that is used to collect container stdout is upgraded.

0.16.9

• Fixed issues

  • The leak of socket fd may occur. This issue is fixed.

  • The limits on the update frequency of Logtail configurations are added for container files.

0.16.8

• New features

  • The Lumberjack plug-in is added to collect data from Logstash and Beats. For more information, see Collect data from Beats and Logstash.

  • The inotify blacklist feature is added.

• Fixed issues

  • The parameters in earlier versions of installation packages are inconsistent. This issue is fixed.

  • When you install Logtail in some systems, the versions of the operating systems cannot be obtained. This issue is fixed.

0.16.6

• New features

  • Host monitoring data can be collected.

  • Redis monitoring data can be collected.

  • DDL statements in MySQL binary logs can be collected.

  • When Logtail collects container stdout and logs in container files, logs can be filtered by using Docker environment variables.

• Fixed issues

  • Data cannot be collected from a MySQL table that does not have a primary key. This issue is fixed.

  • Event loss occurs due to unstable subscription channels of container engines in container collection mode. This issue is fixed.

0.16.5

• New features

  The multi-line mode is supported to collect container stdout. For more information, see Examples of Logtail configurations for multi-line logs.

0.16.4

• New features

  • The deployment scenarios of Docker and Kubernetes are supported.

  • Container stdout and logs in container files can be collected. For more information, see Use the Simple Log Service console to collect container stdout and stderr in DaemonSet mode and Use the Simple Log Service console to collect container text logs in DaemonSet mode.

0.16.2

• New features

  The processor_geoip plug-in is added. For more information, see Convert IP addresses.

0.16.0

• New features

  • MySQL binary logs, MySQL query results, and HTTP data can be collected. For more information, see Use Logtail plug-ins to collect data.

  • The following configuration items can be combined to parse data: regular expressions, anchors, delimiters, and filters.