Simple Log Service imposes the following limits on importing data from Kafka.
Collection limits
| Limit | Description |
| Compression format | Kafka Producer must use gzip, zstd, lz4, or snappy compression. Data compressed in other formats is discarded. Discarded entries are counted as Deliver Failed on the Data Processing Insight dashboard. For more information, see View the data import configuration. |
| Maximum number of topics | A single data import configuration supports up to 10,000 topics. |
| Maximum log size | The maximum size of a single log is 3 MB. Logs that exceed this limit are discarded. Discarded logs are counted as Deliver Failed on the Data Processing Insight dashboard. For more information, see View the data import configuration. |
| Starting position | The starting position can only be set to Earliest or Latest. Importing data from a specific point in time is not supported. |
Configuration limits
| Limit | Description |
| Number of data import configurations | A single Project supports up to 100 data import configurations of all types. If you require a higher limit, submit a ticket. |
| Bandwidth limit | The default maximum network bandwidth is 128 MB/s when a data import task reads data from an Alibaba Cloud Kafka cluster over a VPC. If you require more bandwidth, submit a ticket. |
Performance limits
| Limit | Description |
| Number of concurrent subtasks | Simple Log Service automatically creates concurrent subtasks based on the number of topics. Each subtask processes decompressed data at up to 50 MB/s.
If you require a higher limit, submit a ticket. |
| Number of topic partitions | More partitions in a Kafka topic improve throughput by allowing subtasks to scale out. For high-volume topics, we recommend increasing the number of partitions to at least 16. |
| Number of Logstore Shards | Write performance depends on the number of Shards in the destination Logstore. Each Shard supports a write throughput of 5 MB/s. For high-volume ingestion, increase the number of Shards. For more information, see Manage shards. |
| Data compression | For large data volumes, compress data before writing it to Kafka to significantly reduce network traffic. Network transmission is typically slower than data decompression, especially when importing data over the internet. |
| Network | If your Alibaba Cloud Kafka cluster is in a VPC, reading data over the VPC network saves internet traffic and provides faster speeds, with bandwidth reaching over 100 MB/s. Importing data over the internet does not guarantee network performance or bandwidth, which can lead to latency. |
Other limits
| Limit | Description |
| Metadata synchronization latency | The import task synchronizes metadata from the Kafka cluster every 10 minutes. Newly added topics and partitions may take approximately 10 minutes to be recognized. Note If the starting position is set to Latest, the initial data written to a new topic (up to 10 minutes of data) might be skipped. |
| Topic offset validity period | The maximum validity period for a topic offset is 7 days. If no data is read from a topic for 7 consecutive days, its previous offset is discarded. When new data arrives, the import task sets the new offset based on the starting position in the data import configuration. |