The accelerating demand for software and service delivery is now increasingly driving startups and large Internet companies to apply the DevOps model. This is because the model enables effective cooperation between developers and operations teams, ensuring fast and efficient response to customer requirements and continuous delivery.
Comparison of log query solutions
Logs are increasingly important under the DevOps model. Troubleshooting, security audit, operation, and other businesses all require the strong support of logs. Therefore, to give the full play of the DevOps model, an appropriate log solution is essential.
- Latency: a time interval between the time when logs are written and the time when they can be queried.
- Query capability: the amount of data that can be scanned per unit time.
- Query type: query by keyword, query by multiple conditions, fuzzy match, numeric value comparison, and contextual query.
- Elasticity: rapid response to a hundred times of traffic surges.
- Cost: calculated based on the number of gigabytes of the log data.
- Reliability: the capability to prevent against log data loss
- Self-built ELK: uses the Elasticsearch, Logstash, and Kibana stack.
- Offline Hadoop and Hive suite: stores data in Hadoop and uses Hive or Presto to query logs (analytics excluded).
- Log Service: uses the log search feature.
The following table compares the preceding three solutions by using application logs and NGINX access logs as an example (10 GB a day).
|Feature||Self-built ELK||Offline Hadoop and Hive suite||Log Service|
|Latency for logs to become queryable||1 to 60 seconds (depending on the refresh_interval parameter)||Minutes to hours||Real-time|
|Query latency||Less than 1 second||Minutes||Less than 1 second|
|Query for massive volumes of data||Dozens of seconds to minutes||Minutes||Seconds for billions of log entries|
|Query by keyword||Supported||Supported||Supported|
|Perform a context query||Not supported||Not supported||Supported|
|Numeric value comparison||Supported||Supported||Supported|
|String query||Supported||Supported||Not supported|
|Elasticity||Servers required||Servers required||Capable of expanding capacities by 10 times in seconds|
|Write cost||USD 5/GB for data writes and no fee incurred for data queries||USD 0.3/GB for data queries and no fee incurred for data writes||USD 0.5/GB for data writes and no fee incurred for data queries|
|Storage cost||Less than USD 3.36/GB per day||Less than USD 0.035/GB per day||Less than USD 0.016/GB per day|
|Reliability||Depending on the number of copies||Depending on the number of copies||SLA greater than 99.9%, data availability greater than 99.99999999%|