Log Service allows for sending logs to a security information and event management (SIEM) system. This ensures that all logs related to regulations and audits on Alibaba Cloud can be imported to your security operations center (SOC).
- SIEM: security information and event management (SIEM) systems, such as Splunk and IBM QRadar.
- Splunk HEC: Splunk HTTP Event Collector (HEC) can be used to receive and send logs over HTTP or HTTPS.
- Hardware specifications:
- Operating system: Linux, such as Ubuntu x64.
- CPU: 2.0+ GHz x 8 cores.
- Memory: 32 GB (recommended) or 16 GB.
- Network interface controller (NIC): 1 Gbit/s.
- Available disk space: at least 2 GB. We recommend that you have an available disk space of 10 GB or greater.
- Network specifications:
The bandwidth between your network environment and Alibaba Cloud must be greater than the speed at which data is generated on Alibaba Cloud. Otherwise, logs cannot be consumed in real time. Assume that the peak speed for data generation is about twice that of the average speed and 1 TB of raw logs are generated every day. If data is compressed at a ratio of 5:1 before transmission, we recommend that you use a bandwidth of around 4 MB/s (32 Mbit/s).
- Python: You can use Python to consume logs. For more information about using Java, see Use a consumer group to consume logs.
- We recommend that you use a standard CPython interpreter.
- You can run the python3 -m pip install aliyun-log-python-sdk -U command to install the Log Service SDK for Python.
- For more information about how to use the Log Service SDK for Python, see User Guide.
The consumer library is an advanced log consumption mode in Log Service. The consumer library provides consumer groups to facilitate consumer management. In comparison to reading data by using the SDK, you can focus on the business logic rather than worrying about the implementation details of Log Service. In addition, the consumer library allows you to ignore failover and load balancing between consumers.
- Each shard can only be allocated to one consumer.
- One consumer can have multiple shards at the same time.
The consumer library can also store checkpoints, which allows you to consume data starting from a breakpoint after a program crash is fixed. This ensures that the data is consumed only once.
Spark Streaming, Storm, and Flink Connector are all implemented based on the consumer library.