Alibaba Cloud Logstash is a server-side data processing pipeline. It allows you to dynamically collect data from a variety of sources, transform the data, and then save the data in your desired destination. Alibaba Cloud Logstash can process and transform all types of events by using input, output, and filter plug-ins. Alibaba Cloud Logstash is compatible with all the capabilities of open-source Logstash.
Alibaba Cloud Logstash offers a plug-in center that is ideal for various scenarios. It includes plug-ins such as logstash-input-sls, logstash-input-oss, and logstash-output-oss. It also supports all open-source built-in plug-ins. This plug-in center is connected to the data ecosystem on the cloud and provides powerful data processing and migration capabilities.
Elasticsearch, Logstash, and Kibana form an ecosystem. Elasticsearch is a distributed, real-time search and analytics engine. Kibana provides a visual interface for Elasticsearch. Logstash allows you to collect, transform, process, and generate data. You can use Alibaba Cloud Elasticsearch in real-time log processing, full-text search, and data analytics.
Data transmission mechanism
- Data collection and input: Logstash allows you to collect data from a variety of sources such as logs, metrics, applications, and storage. It can then transmit data in streaming mode.
- Real-time parsing and data transformation: Logstash uses filters to parse all types of events and identifies defined fields to build schemas. Then, it transforms the schemas into common data types and transmits data to a repository.
- Data storage and export: Logstash allows you to send data to your desired destination.
- Enables fast deployment, easy management, and simplified O&M.
- Integrates all open-source input, output, and filter plug-ins.
- Supports the input and output plug-ins of other Alibaba Cloud services such as Log Service and Object Storage Service (OSS).
- Provides an open, flexible plug-in center.
- Connects to Elasticsearch clusters for centralized management of pipelines.