Alibaba Cloud Logstash allows you to specify a keyword and a time range in the Elasticsearch console to query specific logs of your Logstash cluster. You can use the logs to identify cluster issues and perform cluster O&M in an efficient manner. This topic describes how to query logs and describes common types of logs.

Procedure

  1. Log on to the Elasticsearch console.
  2. Navigate to the desired cluster.
    1. In the top navigation bar, select the region where the cluster resides.
    2. In the left-side navigation pane, click Logstash Clusters. On the Logstash Clusters page, find the cluster and click its ID.
  3. In the left-side navigation pane of the page that appears, click Logs. Then, you can view the logs of the cluster.
    The Logs page contains the following tabs: Cluster Log, Slow Log, GC Log, Debug Log, and Asynchronous Write Log.
  4. On a tab of the Logs page, enter a query string, select the start time and end time, and then click Search.
    You can query logs that are generated within the last seven days. By default, the logs are displayed by time in descending order. The Lucene query syntax is supported. For more information, see Query string syntax.
    In this example, the logs that meet the following conditions are queried on the Cluster Log tab: The value of the level field is info, the value of the host field is 172.16.xx.xx, and the value of the content field contains the running keyword. In this case, the query string is host:172.16.xx.xx AND level:info AND content:running. Log query example
    Notice
    • AND in the query string must be uppercase.
    • If you do not specify an end time, the current system time is used as the end time. If you do not specify a start time, the start time is one hour earlier than the end time.
    After you click Search, the logs that match your query string are displayed.

Common types of logs

Operational logs

The Cluster Log tab displays the operational logs of the cluster. Each operational log contains the following information: Time, Node IP, and Content. Logstash log query results
  • Time: the time when the log is generated.
  • Node IP: the IP address of the node that generates the log.
  • Content: consists of the level, host, time, and content fields.
    Field Description
    level The level of the log. Log levels include trace, debug, info, warn, and error.
    Note Garbage collection (GC) logs do not contain the level field.
    host The IP address of the node that generates the log.
    time The time when the log is generated.
    content The content of the log.

GC logs

By default, GC logs are enabled. Each GC log contains the following information: Time, Node IP, and Content. For more information, see Operational logs. Logstash GC logs

Debug logs

If a Logstash pipeline is incorrectly configured, the output data of the pipeline may not meet your requirements. In this case, you must repeatedly check the format of the data in the destination and modify the pipeline configuration in the console. This increases time and labor costs. To address this issue, you can use the pipeline configuration debugging feature provided by Logstash. This feature allows you to view the output data of your Logstash pipeline in debug logs in the console after you create and deploy the pipeline. This reduces your debugging costs. For more information, see Use the pipeline configuration debugging feature.

Debug logs are disabled by default. You can perform the following operations to enable debug logs:
  1. Install the logstash-output-file_extend plug-in for the cluster. For more information, see Install and remove a plug-in.
  2. Configure the file_extend parameter in the output configuration of the pipeline. For more information, see Use configuration files to manage pipelines.
After you enable debug logs, you can obtain the output data of the pipeline on the Debug Logs tab. Logstash debug logs

References

ListLogstashLog