The LogHub feature of Log Service allows you to use Logtail to collect logs.
Log management scenario
A server or container stores a large amount of log data generated by applications in different directories.
- Developers publish or unpublish applications.
- The server can scale out during peak hours and scale in during off-peak hours.
- The log data is queried, monitored, and warehoused based on changing requirements.
- Fast application publishing and increasing log types
Each application generates access, operations, logic, and error logs. When new applications are associated with each other or with existing applications, the volume of logs explodes.
The following table lists the different types of logs collected for a takeout website.
Type Application Log name Web NGINX wechat-nginx, which stores WeChat server NGINX logs Web error NGINX alipay-nginx, which stores Alipay server NGINX logs NGINX server-access, which stores server-side access logs NGINX error alipay-nginx, which stores NGINX error logs NGINX error
Web app Tomcat alipay-app, which stores Alipay server application logic Tomcat
App Mobile app deliver-app, which stores status logs of the courier app App error Mobile app deliver-error, which stores delivery error logs Web HTML5 web-click, which stores HTML5 page view (PV) logs Server Server Internal logic logs on the server side Syslog Server Server system logs
- Log consumption for different purposes
For example, access logs can be downloaded for metering and billing. Operations logs can be queried by a database administrator (DBA). These logs require business intelligence (BI) analysis and end-to-end monitoring.
- Business and environment changes
With the rapid development of the Internet, you need to adapt to continuous business and environment changes:
- Application server scale-out
- Servers as machines
- New application deployment
- New log consumers
Ideal management architecture
An ideal management architecture requires:
- A well-defined, low-cost framework
- A stable, reliable, and unattended mechanism, for example, a mechanism that allows you to scale in or out servers as needed
- Standardized application deployment without complicated configurations
- High performance that meets log processing requirements
Log Service solution
Log Service LogHub uses Logtail to collect logs. This process involves the following components:
- Project: a management container.
- Logstore: the source of a type of logs.
- Machine group: the directory and format of logs.
- Configuration: the path to the log source.
The relationships between these components are as follows:
- A project includes multiple Logstores, machine groups, and configurations. Different projects can be used to meet different business requirements.
- Each application can have multiple types of logs. Each type of log has a Logstore
and a fixed directory (with the same configuration).
app --> logstore1, logstore2, logstore3 app --> config1, config2, config3
- A single application can be deployed on multiple machine groups. Multiple applications
can be deployed on a single machine group.
app --> machineGroup1， mahcineGroup2 machineGroup1 --> app1, app2，app3
- The collection directories defined in the Logtail configurations are applied to the
specified machine groups, and logs are collected into any Logstore.
config1 * machineGroup1 --> Logstore1 config1 * machineGroup2 --> Logstore1 config2 * machineGroup1 --> Logstore2
- High efficiency: The Web Console or SDK is provided for batch management.
- Large scale: Millions of machines and applications can be managed.
- Real-time response: Collection configurations take effect within minutes.
- High elasticity
- The machine identification feature supports auto scaling of servers.
- LogHub supports auto scaling.
- High stability and reliability: No human intervention is required.
- Abundant query capabilities in log processing, such as real-time computing, offline
analysis, and indexing:
- LogHub: real-time collection and consumption. LogHub uses more than 30 methods to collect massive data for real-time downstream consumption.
- LogShipper: stable and reliable log shipping. LogShipper ships log data from LogHub to Object Storage Service (OSS), MaxCompute, or Table Store for storage and big data analysis.
- LogSearch: real-time data indexing and querying. LogSearch allows you to query logs in a centralized manner no matter where active server logs are located.