The Logtail client can help Log Service users collect logs from ECS instances through the console.
After creating a Logstore, the system will prompt you to create Logtail configuration. In the dialog box, click Create Logtail Config. In addition, you can also create Logtail configuration on the Logstore List page.
You must install Logtail before using it to collect logs. Logtail supports Windows and Linux operating systems. For installation methods, refer to Install Logtail on Windows and Install Logtail on Linux.
- A single file can only be collected using one configuration. If you need to collect files using more than one configuration, soft link is recommended. For example, files under
/home/log/nginx/logneed to be collected using 2 configurations. One configure original path, and the other configures soft link
ln -s /home/log/nginx/log /home/log/nginx/link_log, which was created for the folder.
- Fot details of operating system versions, refer to Using Logtail to write logs.
Log on to the Log Service console. Select the Logstore, and click Manage.
Click Create in the upper-right corner.
Select text as the data source type and click Next.
Specify the Configuration Name.
The configuration name can be 3 to 63 characters in length, and can contain lowercase letters, digits, hyphens (-), and underscores (_). It must begin and end with lowercase letters or digits.
Note: Once the configuration name is set, it cannot be modified.
Specify the log directory and file name.
The directory structure must be a full path or a path that contains wildcards.
?can be used as wildcards in the directory.
The log file name must be a complete file name or a name that contains wildcards. For the rules of file names, refer to Wildcard matching.
The search mode of log files is the multi-level directory matching mode, which means in the specified folder (including all levels of the subdirectories of this folder), all of the files that conform to the file name search mode will be monitored. Here are two examples:
/apsara/nuwa/ … /*.Logmeans the files whose suffix is
.Logand exist in the
/apsara/nuwadirectory (including its recursive subdirectories).
/var/logs/app_* … /*.Log*means the files whose file name contains
.Logand exist in all of the directories that conform to the
app_*search mode (including their recursive subdirectories) under the
Note: A single file can only be collected by one configuration.
Set the collection mode.
Logtail supports simple mode, delimiter mode, JSON mode, regular expression mode, and other log collection methods. For details, refer to Collection modes. In this example, simple mode and regular expression mode are used to introduce the collection mode settings.
In simple mode, namely single line mode, one line of data is treated as a log by default. Two logs are separated by a line break. In single line mode, the system does not extract fields (that is, the regular expression is (.*) by default), and uses the system time of the current server as the log generation time. If, you need to use more detailed settings, you can change the configuration to regular expression mode and configure all the settings. For details about how to modify the Logtail configuration, refer to Logtail configuration.
In simple mode, you only need to specify the file directory and file name, Logtail will collect a log line by line. Logtail will not extract fields from the log content. In addition, the log time is set to the time the log was captured.
If you need to configure more personalized extraction settings for log contents (for example, cross-row logs or field extraction), select Full mode. For details on the specific meanings and setting methods for these parameters, refer to Logtail log collection parameters .
Enter the Log Sample.
The purpose of providing a log sample is to facilitate the Log Service console in automatically extracting the regular expression matching mode in logs. Be sure to use a log from the actual environment.
Singleline mode is the default option. This means that the log is partitioned line by line. If you need to collect cross-line logs (such as Java program logs), you must disable Singleline and set Regular Expression.
Set Regular Expression.
This option provides two functions: automatic generation and manual input. After entering the log sample, click Auto Generate to automatically generate a regular expression. If it fails, you can switch to manual mode and input a regular expression for verification.
Set Extract Field.
If you need to analyze and process fields one by one in the log content, use the Extract Field function to convert the specified field into a key-value pair before sending it to the server. Therefore, you need to specify a method for parsing the log content (specifically, a regular expression).
The Log Service console allows you to specify a regular expression for parsing in two ways. The first option is to automatically generate a regular expression through simple interactions. You can use the “Drag Select” method on the log sample to indicate the fields to be extracted, and then click Generate RegEx, the Log Service console will automatically generate a regular expression.
Additionally, you can manually input regular expressions. You can click Manually Input Regular Expression to switch to manual input mode. After the expression is inputted, click Validate on the right to verify whether the expression can parse and extract the sample log.
Regardless of whether the regular expression for log parsing is automatically generated or manually inputted, you need to name each extracted field (that is, set the key for the field).
Set Use System Time.
Use System Time is set by default. If it is disabled, you must specify a certain field (value) to be used as the time field during field extraction and name this field time. After selecting a time field, you can click Auto Generate in Time Format to generate a method to parse this field. For more information on log time formats, refer to Logtail date format.
Set Advanced Options according to the circumstances.
Set Local Cache, Topic Generation Mode, Log File Encoding, Maximum Monitor Directory Depth, Timeout and Filter Configuration depending on the requirement. Otherwise, leave that as the default.
Topic Generation Mode defaults to Null, do not generate topic. That means Topic will be defined as empty string, and there is no need to enter the topic when you query logs. You can choose Machine Group Topic Attributes to differentiate log data of different server, or choose File Path Regular to differentiate log data of users and instance.
After configuring the settings, click Next.
Select the machine group and click Apply to Machine Group to apply the configuration.
If you have not created a machine group, you must create one first. For information about creating machine groups, refer to Create a machine group.
- It may take up to 3 minutes for the Logtail configuration to come into effect after being pushed.
- If you need to collect IIS access logs, you must first refer to the IIS log collection best practices to configure IIS.
- After creating Logtail configuration, view the Logtail configuration list where you can modify or delete Logtail configurations. For details, refer to Logtail configuration.
After the configuration is complete, Log Service can collect logs. You can view the collected logs. For information about log queries, refer to Query logs.
An example of a log collected on the server in simple mode is shown below. All log contents are displayed under the key named content.
An example of log content collected on the server in regular expression mode is shown below. The log contents are collected on the server according to the set key-value.
|Log path||Indicates the root directory of collected log files. The patha can be a complete file name or a name that contains wildcards.|
|Log file name||Indicates the name of a collected log file. The name is case-sensitive and may contain wildcards, for example, |
|Local storage||Indicates whether to enable the local cache to temporarily store logs that cannot be sent due to short-term network interruptions.|
|First-line log header||Indicates the starting line of a multiline log by means of a regular expression. Lines cannot be used to separate individual logs in multiline log collection (for example, application logs with stack information). You need to specify a starting line to delimit multiline logs. Because the starting line (for example, timestamp) of each log may be different, you need to specify a starting line match rule. A regular expression is used as a match rule here.|
|Log parsing expression||Indicates how to extract a piece of log information and convert it into a log format supported by Log Service. You need to specify a regular expression to extract the required log fields and then name each field.|
|Log time format||Defines how to parse the time format of the timestamp string in log data. For details, refer to Logtail log time format.|
In addition to using Logtail to collect logs, Log Service also provides APIs and SDKs help you write logs.
In addition to APIs, Log Service also provides SDKs in multiple languages (Java, .NET, PHP, and Python) that allow you to easily write logs. For a complete SDK reference, refer to SDK Reference.