This document introduces a new log collection tool for Docker: log-pilot.  Log-pilot is a log collection image we provide for you.  You can deploy a log-pilot  instance on each machine to collect all the Docker application logs. Docker of Linux version is supported, while Docker of Windows or Mac version is not supported.

Log-pilot has the following features:

  • A separate log process collects the logs of all the containers on the machine.  No need to start a log process for each container.
  • Log-pilot supports file logs and stdout logs. Docker log driver or Logspout can only process stdout, while log-pilot supports collecting the stdout logs and the file logs.
  • Declarative configuration.  When your container has logs to collect, log-pilot will automatically collect logs of the new container if the path of the log file to be collected is declared by using the label. No other configurations need to be changed.
  • Log-pilot supports multiple log storage methods  and can deliver the logs to the correct location for powerful Alibaba Cloud Log Service, popular ElasticSearch combination, or even Graylog.
  • Open-source.  Log-pilot is fully open-sourced. You can download the codes from log-pilot GitHub project.  If the current features cannot meet your requirements, welcome to raise an issue.

Quick start

See a simple scenario as follows: start a log-pilot and then start a Tomcat container, letting log-pilot collect Tomcat  logs.  For simplicity, here Alibaba Cloud Log Service or ELK is not involved.  To run locally, you only need a machine that runs Docker.

First, start log-pilot.

Note
When log-pilot is started in this way, all the collected logs will be directly output to the console because no log storage is configured for backend use. Therefore, this method is mainly for debugging.

Open the terminal and enter the following commands:

docker run --rm -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /:/host \
    --privileged \
    registry.cn-hangzhou.aliyuncs.com/acs-sample/log-pilot:0.1
You will see the startup logs of log-pilot.


Do not close the terminal. Open a new terminal to start Tomcat. The Tomcat image is among the few Docker images that use stdout and file logs at the same time, and is suitable for the demonstration here.

docker run -it --rm -p 10080:8080 \
-v /usr/local/tomcat/logs \
--label aliyun.logs.catalina=stdout \
--label aliyun.logs.access=/usr/local/tomcat/logs/localhost_access_log. *.txt \
tomcat

Note:

  • aliyun.logs.catalina=stdout tells log-pilot that this container wants to collect stdout logs.
  • aliyun.logs.access=/usr/local/tomcat/logs/localhost_access_log. *.txt indicates to collect all log files whose names comply with the localhost_access_log. *.txt format under the /usr/local/tomcat/logs/ directory in the container.  The label usage will be introduced in details later.
Note
If you deploy Tomcat locally, instead of in the Alibaba Cloud Container Service, specify -v /usr/local/tomcat/logs. Otherwise,  log-pilot cannot read log files.  Container Service has implemented the optimization and you do not need to specify -v on your own.
Log-pilot will monitor the events in the Docker container. When it finds any container with aliyun.logs.xxx,  it will automatically parse the container configuration and start to collect the corresponding logs. After you start Tomcat, you will find many contents are output immediately by the log-pilot terminal, including the stdout logs output at the Tomcat startup, and some debugging information output by log-pilot itself.


You can access the deployed Tomcat in the browser, and find that similar records are displayed on the log-pilot terminal every time you refresh the browser.  Wherein, the contents after message are the logs collected from /usr/local/tomcat/logs/localhost_access_log.XXX.txt .

Use ElasticSearch + Kibana

Deploy ElastichSearch + Kibana. See Use ELK in Container Service to deploy ELK in Alibaba Cloud Container Service, or deploy them directly on your machine by following the ElasticSearch/Kibana documents.  This document assumes that you have deployed the two components.

If you are still running the log-pilot, close it first, and then start it again by using the following commands:

Note
Before running the following commands, replace the two variables ELASTICSEARCH_HOST and ELASTICSEARCH_PORT with the actual values you are using. ELASTICSEARCH_PORT is generally 9200.
docker run --rm -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /:/host \
    --privileged \
    -e FLUENTD_OUTPUT=elasticsearch \
    -e ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST} \
    -e ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT}
    registry.cn-hangzhou.aliyuncs.com/acs-sample/log-pilot:0.1

Compared with the previous log-pilot startup method, here three environment variables are added:

  • FLUENTD_OUTPUT=elasticsearch: Send the logs to ElasticSearch.
  • ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST}: The domain name of ElasticSearch.
  • ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT}: The port number of ElasticSearch.

Continue to run the Tomcat started previously, and access it again to make Tomcat generate some logs. All these newly generated logs will be sent to ElasticSearch.

Open Kibana, and no new logs are visible yet. Create an index first. Log-pilot will write logs to the specific index of ElasticSearch. The rules are as follows:

If label aliyun.logs.tags is used in the application, and  tags contains target, use target  as the index of ElasticSearch. Otherwise, use XXX in the label aliyun.logs.XXX as the index.

In the previous example about Tomcat, the label aliyun.logs.tags is not used, so access and catalina are used by default as the index. First create the index access.


After the index is created, you can view the logs.


Use log-pilot in Alibaba Cloud Container Service

Container Service makes some special optimization for log-pilot, which adapts to running log-pilot best.

To run log-pilot in Container Service, create an application by using the following orchestration file.  For how to create an application, see Create an application.

pilot:
  image: registry.cn-hangzhou.aliyuncs.com/acs-sample/log-pilot:0.1
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - /:/host
  privileged: true
  environment:
    FLUENTD_OUTPUT: elasticsearch #Replace based on your requirements  
    ELASTICSEARCH_HOST: ${elasticsearch} #Replace based on your requirements
    ELASTICSEARCH_PORT: 9200
  labels:
    aliyun.global: true

Then, you can use the aliyun.logs.xxx label on the application that you want to collect logs.

Label description

When Tomcat is started, the following two labels are declared to tell log-pilot the location of the container logs.

--label aliyun.logs.catalina=stdout 
--label aliyun.logs.access=/usr/local/tomcat/logs/localhost_access_log. *.txt

You can also add more labels on the application container.

  • aliyun.logs.$name = $path
    • The variable name is the log name and can only contain 0–9, a–z, A–Z, and hyphens (-).
    • The variable path is the path of the logs to be collected. The path must specify the file, and cannot only be a directory. Wildcards are supported as part of the file name, for example, /var/log/he.log and /var/log/*.log are both correct. However, /var/log is not valid because the path cannot be only a directory. stdout is a special value, indicating standard output.
  • aliyun.logs.$name.format: The log format. Currently, the following formats are supported.
    • none: Unformatted plain text.
    • json: JSON format. One complete JSON string in each line.
    • csv: CSV format.
  • aliyun.logs.$name.tags: The additional field added when the logs are reported. The format is k1=v1,k2=v2. The key-value pairs are separated by commas, for example, aliyun.logs.access.tags="name=hello,stage=test". Then, the logs reported to the storage will contain the name field and the stage field.

    If ElasticSearch  is used for log storage, the target tag will have a special meaning, indicating the corresponding index in ElasticSearch.

Log-pilot extension

For most users, the existing features of log-pilot can meet their requirements. If log-pilot cannot meet your requirements, you can: