The LogHub feature of Log Service allows you to use Logtail to collect logs.

Log management scenario

A server or container stores a large amount of log data generated by applications in different directories.

  • Developers publish or unpublish applications.
  • The server can scale out during peak hours and scale in during off-peak hours.
  • The log data is queried, monitored, and warehoused based on changing requirements.
Log management scenario

Challenges

  • Fast application publishing and increasing log types

    Each application generates access, operations, logic, and error logs. When new applications are associated with each other or with existing applications, the volume of logs explodes.

    The following table lists the different types of logs collected for a takeout website.

    Type Application Log name
    Web NGINX wechat-nginx, which stores WeChat server NGINX logs
    Web error NGINX alipay-nginx, which stores Alipay server NGINX logs
    NGINX server-access, which stores server-side access logs
    NGINX error alipay-nginx, which stores NGINX error logs
    NGINX error ...
    Web app Tomcat alipay-app, which stores Alipay server application logic
    Tomcat ...
    App Mobile app deliver-app, which stores status logs of the courier app
    App error Mobile app deliver-error, which stores delivery error logs
    Web HTML5 web-click, which stores HTML5 page view (PV) logs
    Server Server Internal logic logs on the server side
    Syslog Server Server system logs
  • Log consumption for different purposes

    For example, access logs can be downloaded for metering and billing. Operations logs can be queried by a database administrator (DBA). These logs require business intelligence (BI) analysis and end-to-end monitoring.

  • Business and environment changes
    With the rapid development of the Internet, you need to adapt to continuous business and environment changes:
    • Application server scale-out
    • Servers as machines
    • New application deployment
    • New log consumers

Ideal management architecture

An ideal management architecture requires:

  • A well-defined, low-cost framework
  • A stable, reliable, and unattended mechanism, for example, a mechanism that allows you to scale in or out servers as needed
  • Standardized application deployment without complicated configurations
  • High performance that meets log processing requirements

Log Service solution

Log Service LogHub uses Logtail to collect logs. This process involves the following components:

  • Project: a management container.
  • Logstore: the source of a type of logs.
  • Machine group: the directory and format of logs.
  • Configuration: the path to the log source.

The relationships between these components are as follows:

  • A project includes multiple Logstores, machine groups, and configurations. Different projects can be used to meet different business requirements.
  • Each application can have multiple types of logs. Each type of log has a Logstore and a fixed directory (with the same configuration).
    app --> logstore1, logstore2, logstore3
    app --> config1, config2, config3 
  • A single application can be deployed on multiple machine groups. Multiple applications can be deployed on a single machine group.
    app --> machineGroup1, mahcineGroup2
    machineGroup1 --> app1, app2,app3
  • The collection directories defined in the Logtail configurations are applied to the specified machine groups, and logs are collected into any Logstore.
    config1 * machineGroup1 --> Logstore1
    config1 * machineGroup2 --> Logstore1
    config2 * machineGroup1 --> Logstore2

Benefits

  • High efficiency: The Web Console or SDK is provided for batch management.
  • Large scale: Millions of machines and applications can be managed.
  • Real-time response: Collection configurations take effect within minutes.
  • High elasticity
    • The machine identification feature supports auto scaling of servers.
    • LogHub supports auto scaling.
  • High stability and reliability: No human intervention is required.
  • Abundant query capabilities in log processing, such as real-time computing, offline analysis, and indexing:
    • LogHub: real-time collection and consumption. LogHub uses more than 30 methods to collect massive data for real-time downstream consumption.
    • LogShipper: stable and reliable log shipping. LogShipper ships log data from LogHub to Object Storage Service (OSS), MaxCompute, or Table Store for storage and big data analysis.
    • LogSearch: real-time data indexing and querying. LogSearch allows you to query logs in a centralized manner no matter where active server logs are located.