Use Logtail to collect text files

Last Updated: Jul 06, 2017

The Logtail client can help Log Service users collect logs from ECS instances through the console.

After creating a Logstore, the system will prompt you to create Logtail configuration. In the pop-up box, click Create Logtail Config. In addition, you can also create Logtail configuration on the Logstore List page.

Prerequisites

You must install Logtail before using it to collect logs. Logtail supports Windows and Linux operating systems. For installation methods, refer to Install Logtail on Windows and Install Logtail on Linux.

Constraints

  • A single file can only be collected using one configuration.
  • Fot details of operating system versions, refer to Using Logtail to write logs.

Procedure

  1. Log on to the Log Service console. Select the desired project, and click Manage.

    1

  2. Click Create in the upper-right corner.

    2

  3. Select text as the data source type and click Next.

    3

  4. Specify the Configuration Name.

    The configuration name can be 3 to 63 characters in length, and can contain lowercase letters, digits, hyphens (-), and underscores (_). It must begin and end with lowercase letters or digits.

    Note: Once the configuration name is set, it cannot be modified.

  5. Specify the log directory structure.

    All files under the specified folder (including all levels of the directories) conforming to the file name will be monitored. The log file name must be a complete file name or a name that contains wildcards. Linux file paths must start with “/“; for example, /apsara/nuwa/…/app.Log. Windows file paths must start with a drive; for example, C:\Program Files\Intel\...\*.Log.

    Note: A single file can only be collected by one configuration.

    4

  6. Set the collection mode.

    Logtail supports simple mode, delimiter mode, JSON mode, regular expression mode, and other log collection methods. For details, refer to Collection modes. In this example, simple mode and regular expression mode are used to introduce the collection mode settings.

    • Simple mode

      In simple mode, namely single line mode, one line of data is treated as a log by default. Two logs are separated by a line break. In single line mode, the system does not extract fields (that is, the regular expression is (.*) by default), and uses the system time of the current server as the log generation time. If, you need to use more detailed settings, you can change the configuration to regular expression mode and configure all the settings. For details about how to modify the Logtail configuration, refer to Logtail configuration.

      In simple mode, you only need to specify the file directory and file name, Logtail will collect a log line by line. Logtail will not extract fields from the log content. In addition, the log time is set to the time the log was captured.

      simple

    • Full mode

      If you need to configure more personalized extraction settings for log contents (for example, cross-row logs or field extraction), select Full mode. For details on the specific meanings and setting methods for these parameters, refer to Logtail log collection parameters .

      1. Enter the Log Sample.

        The purpose of providing a log sample is to facilitate the Log Service console in automatically extracting the regular expression matching mode in logs. Be sure to use a log from the actual environment.

      2. Set Singleline.

        Singleline mode is the default option. This means that the log is partitioned line by line. If you need to collect cross-line logs (such as Java program logs), you must disable Singleline and set Regular Expression.

      3. Set Regular Expression.

        This option provides two functions: automatic generation and manual input. After entering the log sample, click Auto Generate to automatically generate a regular expression. If it fails, you can switch to manual mode and input a regular expression for verification.

      4. Set Extract Field.

        If you need to analyze and process fields one by one in the log content, use the Extract Field function to convert the specified field into a key-value pair before sending it to the server. Therefore, you need to specify a method for parsing the log content (specifically, a regular expression).

        The Log Service console allows you to specify a regular expression for parsing in two ways. The first option is to automatically generate a regular expression through simple interactions. You can use the “Drag Select” method on the log sample to indicate the fields to be extracted, and then click Generate RegEx, the Log Service console will automatically generate a regular expression.

        Additionally, you can manually input regular expressions. You can click Manually Input Regular Expression to switch to manual input mode. After the expression is inputted, click Validate on the right to verify whether the expression can parse and extract the sample log.

        Regardless of whether the regular expression for log parsing is automatically generated or manually inputted, you need to name each extracted field (that is, set the key for the field).

        3

      5. Set Use System Time.

        Use System Time is set by default. If it is disabled, you must specify a certain field (value) to be used as the time field during field extraction and name this field time. After selecting a time field, you can click Auto Generate in Time Format to generate a method to parse this field. For more information on log time formats, refer to Logtail date format.

  7. After configuring the settings, click Next.

  8. Select the desired machine group and click Apply to Machine Group to apply the configuration.

    If you have not created a machine group, you must create one first. For information about creating machine groups, refer to Create a machine group.

Note:

  • It may take up to 3 minutes for the Logtail configuration to come into effect after being pushed.
  • If you need to collect IIS access logs, you must first refer to the IIS log collection best practices to configure IIS.
  • After creating Logtail configuration, view the Logtail configuration list where you can modify or delete Logtail configurations. For details, refer to Logtail configuration.

Additional operations

After the configuration is complete, Log Service can collect logs. You can view the collected logs. For information about log queries, refer to Query logs.

An example of a log collected on the server in simple mode is shown below. All log contents are displayed under the key named content.

easy

An example of log content collected on the server in regular expression mode is shown below. The log contents are collected on the server according to the set key-value.

full

More information

Logtail configuration

Configuration Item Description
Log path Indicates the root directory of collected log files. The patha can be a complete file name or a name that contains wildcards.
Log file name Indicates the name of a collected log file. The name is case-sensitive and may contain wildcards, for example, *.log. The file name wildcards in Linux include \*, ?, and […]. In Windows, MS-DOS and Windows wildcards are supported, for example, *.doc and readme.???.
Local storage Indicates whether to enable the local cache to temporarily store logs that cannot be sent due to short-term network interruptions.
First-line log header Indicates the starting line of a multiline log by means of a regular expression. Lines cannot be used to separate individual logs in multiline log collection (for example, application logs with stack information). You need to specify a starting line to delimit multiline logs. Because the starting line (for example, timestamp) of each log may be different, you need to specify a starting line match rule. A regular expression is used as a match rule here.
Log parsing expression Indicates how to extract a piece of log information and convert it into a log format supported by Log Service. You need to specify a regular expression to extract the required log fields and then name each field. For details, refer to Sample.
Log time format Defines how to parse the time format of the timestamp string in log data. For details, refer to Logtail log time format.

Other collection mode

In addition to using Logtail to collect logs, Log Service also provides APIs and SDKs help you write logs.

Use API to write logs

Log Service provides RESTful APIs to help you write logs. You can use the PostLogStoreLogs interface to write data. For a complete API reference, refer to API Reference.

Use SDKs to write logs

In addition to APIs, Log Service also provides SDKs in multiple languages (Java, .NET, PHP, and Python) that allow you to easily write logs. For a complete SDK reference, refer to SDK Reference.

Thank you! We've received your feedback.