edit-icon download-icon

Basic questions for log collection

Last Updated: Apr 23, 2018

Question list

  1. What to do if logs failed to be collected?
  2. What to do if the collected logs are gibberish?
  3. What types of logs does Log Service collect?
  4. In what ways does Log Service collect logs and which way is applicable to my case?
  5. How does Log Service collect logs from ECS instances?
  6. Does Log Service collect historical logs?
  7. How is the log collection capability of Log Service and does it have any limits?
  8. What to pay attention to when using Logtail to collect logs on NAS?

1. What to do if logs failed to be collected?

  1. Check if the matching rule is passed (generally because the log sample and the logs actually collected are inconsistent).
  2. Check if the log file is updated in real time. Log Service does not collect old logs.
  3. The log time must contain date information such as the year.
  4. A latency exists (about 1–2 minutes are needed for Log Service to read data). Wait patiently.
  5. Check if the Logtail heartbeat status of a machine in a machine group is FAIL.
  6. Log Service does not support collecting data that is not UTF-8 encoded.
  7. Check the log time. Generally, logs are discarded for timeout because of the time zone problem.

If the issue persists, contact our after-sales technical support.

2. What to do if the collected logs are gibberish?

Log Service supports collecting UTF-8 encoded logs. Gibberish may appear if the collected logs are not UTF-8 encoded.

To use SDKs to collect logs, transcode the character sets when writing codes. To use Logtail to collect logs, check the encoding of the log file monitored by Logtail.

If the issue persists, contact our after-sales technical support.

3. What types of logs does Log Service collect?

Log Service supports collecting text logs and syslog data with timestamp, and the log time must be within the last seven days and no more than 15 minutes later than the current time.

4. In what ways does Log Service collect logs and which way is applicable to my case?

Log Service allows you to write logs directly by using APIs and provides you with Logtail, which can be installed on Windows machines and Linux machines, to collect logs that are updated in real time from disk files.

  1. If logs generated by applications are not flushed into the disk, you can directly use APIs to write these logs to Log Service.
  2. If logs are written to the disk in real time, you can use Logtail to collect these logs.

5. How does Log Service collect logs from ECS instances?

You can use Logtail to collect logs that are flushed into the disk from Elastic Compute Service (ECS) instances as follows:

  1. Create a Logstore in the Log Service console.
  2. Complete the Logtail configuration.
  3. Create a machine group.
  4. Install the Logtail client by using the installation script.
  5. Apply the Logtail configuration to the created machine group.

6. Does Log Service collect historical logs?

You can only write logs generated within the last seven days by using APIs. However, you cannot use Logtail to collect historical logs.

7. How is the log collection capability of Log Service and does it have any limits?

You can adjust the number of shards in a Logstore as needed. Logtail collects logs from an ECS instance at a maximum rate of 1 MB/s.

8. What to pay attention to when using Logtail to collect logs on NAS?

For example, to collect Nginx access logs, generally the Nginx configurations of Web servers are the same and traditionally logs are written to files with the same name on different machines (in this case, Logtail collects logs normally). After you use Network Attached Storage (NAS), Logtail will have missing logs or encounter a collection error if the Nginx logs on multiple machines are written to the same file on NAS (that is, to write logs concurrently to the same file). To avoid this problem, make sure that the logs on different Web servers are written to different files on NAS.

Thank you! We've received your feedback.