Compare Log Service against ELK (search class) and Hadoop/Hive in DevOps scenario

To handle the accelerating demand for software and service delivery, startup teams and big IT companies have switched or are switching to the DevOps mode. With the effective collaboration between developers and Operation & Maintenance (O&M) personnel, they implement the collaboration across departments, respond to customer requirements quickly, and conduct continuous delivery.

In the DeveOps mode, logs play an important support role in aspects such as problem investigation, security audit, and operation support. An appropriate log solution is important to DevOps.

Compare LogSearch against ELK and Hadoop/Hive solutions in the following aspects:
  • When the user can perform query after the log is generated
  • Query capability: The data volume scanned in unit time.
  • Query function: The keyword query, condition combination query, fuzzy query, numerical comparison, and context query.
  • Rapid response to rise of hundred times of traffic
  • Cost: The cost per GB.
  • Reliability: The log data is secure and will not be lost.
Common solutions and comparison
  • Self-built ELK: Use Elastic, Logstash, and Kibana for comparison.
  • Offline Hadoop + Hive: The data is stored in Hadoop, and Hive or Presto is used for query (not analysis).
  • Use Log Service (LogSearch).

Compare these solutions by using application logs and Nginx access logs as an example (10 GB per day).

Function ELK system Hadoop + Hive Log Service
Latency that can be queried 1–60 seconds (controlled by refresh_interval) Several minutes to several hours Real time
Query latency Less than 1 second In minutes Less than 1 second
Super large query Tens of seconds to several minutes In minutes In seconds (query one billion logs)
Keyword query Supported Supported Supported
Fuzzy search Supported Supported Supported
Context query Not supported Not supported Supported
Context query Supported Supported Supported
Consecutive string query Supported Supported Not supported
Elasticity Prepare machines in advance Prepare machines in advance 10 times of expansion in seconds
Write cost USD 5/GB for write. No charge for query No charge for write. USD 0.3/GB for one query USD 0.5/GB for write. No charge for query
Storage cost  Less than or equal to USD 3.36/GB * day  Less than or equal to USD 0.035/GB * day  Less than or equal to USD 0.016/GB * day
Reliability Set the number of copies Set the number of copies SLA > 99.9%. Data > 99.99999999%