Simple Log Service provides Elasticsearch-compatible API. After you collect logs to Simple Log Service, you can use Elasticsearch to query the logs.
This topic is proprietary information of Alibaba Cloud, and describes the capabilities that are provided by Alibaba Cloud to interact with third-party services. Therefore, the names of third-party companies and services may be referenced in this topic.
Background information
For example, User A uses the Elasticsearch, Logstash, and Kibana (ELK) system to collect and visualize logs. However, as the log size increases, the ELK system is challenged by the performance and costs of the existing log system. To address the challenges, User A considers Simple Log Service as a substitute for the existing log system. Simple Log Service provides Elasticsearch-compatible API. After you collect logs to Simple Log Service, you can use Elasticsearch to query data.
Benefit
The Elasticsearch-compatible query solution based on Simple Log Service provides the following benefits:
Simple Log Service provides full management capabilities and a highly available storage and query engine. No investment is required for O&M.
Real-time log query is supported because data transfer over Logstash is no longer required.
Less machine resources are occupied because Kafka and Elasticsearch are no longer used. You need to only pay for the resources that are used.
Simple Log Service is compatible with the Elasticsearch ecosystem and programs, such as Kibana and API.
Log system architecture before switchover
The following figure shows an example of the ELK architecture.
Use Filebeat to collect data.
Use three Kafka machines to cache data.
Use two Logstash machines to synchronize data from Kafka to Elasticsearch.
Use five Elasticsearch machines to store and query data.
Use Kibana to query data in a visualized manner, and use Grafana to generate reports and configure alerts.
Switchover objective
Perform a smooth switchover. Prevent impacts on the original Elasticsearch link during the switchover.
Retain the Elasticsearch usage habits whenever possible.
Switchover solution
Step 1: Implement log dual-write
During the switchover, the following links are used to collect logs. The links do not affect each other.
Link 1: Use Filebeat to collect logs to Elasticsearch.
Link 2: Use Logtail to collect logs to Simple Log Service.
After you collect logs to Simple Log Service, you can still use Grafana and Kinbana to query, analyze, and visualize the logs. For more information, see Use Kibana to access the Elasticsearch-compatible API of Simple Log Service and Use Grafana to access the Elasticsearch-compatible API of Simple Log Service.
Step 2: Delete the Filebeat collection link
After you confirm that the Simple Log Service link in the dual-write links meets your requirements, delete the Filebeat collection link to complete the switchover.