Log Service allows you to collect logs in delimiter mode. After logs are collected, you can transform and ship logs, and perform multidimensional log analysis. This topic describes how to configure Logtail in delimiter mode in the Log Service console to collect logs.
Prerequisites
- A project and a Logstore are created. For more information, see Create a project and Create a Logstore.
- Ports 80 and 443 of the server from which you want to collect logs are enabled.
Procedure
Appendix: delimiters and sample log entries
DSV formatted logs use line feeds as boundaries. Each line indicates a log entry. The fields of each log entry are delimited by using single-character delimiters or multi-character delimiters. If a field contains delimiters, you can enclose the field with a pair of quotes.
- Single-character delimiter
The following examples list sample log entries with single-character delimiters:
05/May/2016:13:30:28,10.10. *. *,"POST /PutData? Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java 05/May/2016:13:31:23,10.10. *. *,"POST /PutData? Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",401,23472,aliyun-sdk-java
For the log entries with single-character delimiters, you must specify each delimiter. You can also use a pair of quotes in the log entry.- Delimiter: Available single-character delimiters include the tab (\t), vertical bar
(|), space, comma (,), and semicolon (;). You can also specify a non-printable character
as the delimiter. A double quotation mark (") cannot be used as the delimiter.
However, a double quotation mark (") can be used as a quote. You can place the double quotation mark at the border of a field, or in the field. If a double quotation mark (") is included in a log field but is not used as a quote, it must be escaped as double quotation marks (""). When Log Service parses log fields, the double quotation marks ("") are automatically converted into a double quotation mark ("). For example, assume that you specify a comma (,) as the delimiter and double quotation mark (") as the quote in a log field. You must enclose the field that contains commas by using a pair of quotes. In addition, you must escape the double quotation mark (") in the field to double quotation marks (""). The log format after processing is 1999,Chevy,"Venture ""Extended Edition, Very Large""","",5000.00. The log entry can be parsed into five fields: 1999, Chevy, Venture "Extended Edition, Very Large", an empty field, and 5000.00.
- Quote: If a log field contains delimiters, you must specify a pair of quotes to enclose
the field. Log Service parses the content that is enclosed in a pair of quotes into
a new complete field.
Available quotes include the tab (\t), vertical bar (|), space, comma (,), semicolon (;), and non-printable characters.
For example, assume that you specify a comma (,) as the delimiter and double quotation mark (") as the quote in log fields. The log entry is 1997,Ford,E350,"ac, abs, moon",3000.00. The log entry can be parsed into five fields: 1997, Ford, E350, ac, abs, moon, and 3000.00.
- Delimiter: Available single-character delimiters include the tab (\t), vertical bar
(|), space, comma (,), and semicolon (;). You can also specify a non-printable character
as the delimiter. A double quotation mark (") cannot be used as the delimiter.
- Multi-character delimiter
The following example lists sample log entries with multi-character delimiters:
05/May/2016:13:30:28&&10.200. **. **&&POST /PutData? Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1&&200&&18204&&aliyun-sdk-java 05/May/2016:13:31:23&&10.200. **. **&&POST /PutData? Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1&&401&&23472&&aliyun-sdk-java
A multi-character delimiter can contain two or three characters, such as ||, &&&, and ^_^). Log Service parses log field based on delimiters. You do not need to use quotes to enclose the log field to be parsed.
Note You must ensure that the delimiters in a field cannot be parsed into a new field. Otherwise, Log Service does not parse the field as expected.For example, assume that you specify && as the delimiter. The log entry 1997&&Ford&&E350&&ac&abs&moon&&3000.00 is parsed into the following five fields: 1997, Ford, E350, ac&abs&moon, and 3000.00.