The accelerating demand for software and service delivery is now increasingly driving startups and large Internet companies to apply the DevOps model. This is because the model enables effective cooperation between developers and operations teams, ensuring fast and efficient response to customer requirements and continuous delivery.

Comparison of log query solutions

Logs are increasingly important under the DevOps model. Troubleshooting, security audit, operation, and other businesses all require the strong support of logs. Therefore, to give the full play of the DevOps model, an appropriate log solution is essential.

The following lists the considerations for the comparison of the log search feature provided by Log Service, the Elasticsearch, Logstash, and Kibana (ELK) stack, and the Hadoop and Hive suite.
  • Latency: a time interval between the time when logs are written and the time when they can be queried.
  • Query capability: the amount of data that can be scanned per unit time.
  • Query type: query by keyword, query by multiple conditions, fuzzy match, numeric value comparison, and contextual query.
  • Elasticity: rapid response to a hundred times of traffic surges.
  • Cost: calculated based on the number of gigabytes of the log data.
  • Reliability: the capability to prevent against log data loss
Common solutions and comparisons
  • Self-built ELK: uses the Elasticsearch, Logstash, and Kibana stack.
  • Offline Hadoop and Hive suite: stores data in Hadoop and uses Hive or Presto to query logs (analytics excluded).
  • Log Service: uses the log search feature.

The following table compares the preceding three solutions by using application logs and NGINX access logs as an example (10 GB a day).

Feature Self-built ELK Offline Hadoop and Hive suite Log Service
Latency for logs to become queryable 1 to 60 seconds (depending on the refresh_interval parameter) Minutes to hours Real-time
Query latency Less than 1 second Minutes Less than 1 second
Query for massive volumes of data Dozens of seconds to minutes Minutes Seconds for billions of log entries
Query by keyword Supported Supported Supported
Fuzzy match Supported Supported Supported
Perform a context query Not supported Not supported Supported
Numeric value comparison Supported Supported Supported
String query Supported Supported Not supported
Elasticity Servers required Servers required Capable of expanding capacities by 10 times in seconds
Write cost USD 5/GB for data writes and no fee incurred for data queries USD 0.3/GB for data queries and no fee incurred for data writes USD 0.5/GB for data writes and no fee incurred for data queries
Storage cost Less than USD 3.36/GB per day Less than USD 0.035/GB per day Less than USD 0.016/GB per day
Reliability Depending on the number of copies Depending on the number of copies SLA greater than 99.9%, data availability greater than 99.99999999%