Compare Log Service against ELK (search class) and Hadoop/Hive in DevOps scenario
To handle the accelerating demand for software and service delivery, startup teams and big IT companies have switched or are switching to the DevOps mode. With the effective collaboration between developers and Operation & Maintenance (O&M) personnel, they implement the collaboration across departments, respond to customer requirements quickly, and conduct continuous delivery.
In the DeveOps mode, logs play an important support role in aspects such as problem investigation, security audit, and operation support. An appropriate log solution is important to DevOps.
- When the user can perform query after the log is generated
- Query capability: The data volume scanned in unit time.
- Query function: The keyword query, condition combination query, fuzzy query, numerical comparison, and context query.
- Rapid response to rise of hundred times of traffic
- Cost: The cost per GB.
- Reliability: The log data is secure and will not be lost.
- Self-built ELK: Use Elastic, Logstash, and Kibana for comparison.
- Offline Hadoop + Hive: The data is stored in Hadoop, and Hive or Presto is used for query (not analysis).
- Use Log Service (LogSearch).
Compare these solutions by using application logs and Nginx access logs as an example (10 GB per day).
|Function||ELK system||Hadoop + Hive||Log Service|
|Latency that can be queried||1–60 seconds (controlled by refresh_interval)||Several minutes to several hours||Real time|
|Query latency||Less than 1 second||In minutes||Less than 1 second|
|Super large query||Tens of seconds to several minutes||In minutes||In seconds (query one billion logs)|
|Context query||Not supported||Not supported||Supported|
|Consecutive string query||Supported||Supported||Not supported|
|Elasticity||Prepare machines in advance||Prepare machines in advance||10 times of expansion in seconds|
|Write cost||USD 5/GB for write. No charge for query||No charge for write. USD 0.3/GB for one query||USD 0.5/GB for write. No charge for query|
|Storage cost||Less than or equal to USD 3.36/GB * day||Less than or equal to USD 0.035/GB * day||Less than or equal to USD 0.016/GB * day|
|Reliability||Set the number of copies||Set the number of copies||SLA > 99.9%. Data > 99.99999999%|