This topic describes the limits on importing data from Elasticsearch/OpenSearch to Simple Log Service.
Collection limits
Limitations | Description |
Size of a single data record | The maximum size of a single data record is 3 MB. If a record exceeds this limit, it is discarded. The number of failed writes on the Data Processing Traffic dashboard indicates the number of discarded records. For more information, see View a data import configuration. |
Data latency | When you use the incremental auto import mode, the most recent data written to Elasticsearch/OpenSearch is not immediately imported to Simple Log Service. The import is affected by the Maximum Latency (Seconds) configuration item. For example, if you set Maximum Latency (Seconds) to 300, the most recent data in Elasticsearch/OpenSearch is imported to Simple Log Service after 300 seconds. |
Configuration limits
Limits | Description |
Number of data import configurations | The total number of data import configurations of all types that you can create in a single project is 100. If you require a higher quota, submit a ticket. |
Bandwidth limit | When a data import task reads data from an Alibaba Cloud Elasticsearch/OpenSearch cluster over a VPC network, the default maximum network bandwidth is 128 MB/s. If you require more bandwidth, submit a ticket. |
Performance limits
Limitations | Description |
Number of concurrent tasks | A data import task uses the Scroll mode to pull data from Elasticsearch/OpenSearch. The Elasticsearch/OpenSearch server has a limit on the maximum number of supported Scroll requests. The default limit is 500. You can change this limit by configuring |
Capabilities of the Elasticsearch/OpenSearch server | The larger the Elasticsearch/OpenSearch cluster and the better the machine configurations, the higher the overall throughput capacity. |
Query complexity | If a data import task involves complex Elasticsearch/OpenSearch queries, the server may take longer to execute requests. This can affect the overall data read speed. |
Number of Logstore shards | The write performance of Simple Log Service depends on the number of shards in the destination Logstore. A single shard supports a write throughput of 5 MB/s. If the data volume is large, increase the number of shards for the destination Logstore. For more information, see Manage shards. |
Network | If you use an Alibaba Cloud Elasticsearch/OpenSearch cluster in a VPC environment or a self-managed Elasticsearch/OpenSearch cluster on an ECS instance, you can read data over the VPC network. This saves Internet traffic and provides fast transmission speeds of over 100 MB/s. When you import data over the Internet, network performance and bandwidth are not guaranteed. This can easily cause import latency. |
Impacts on the Elasticsearch/OpenSearch server
Limits | Description |
Excessive search sessions | A data import task uses the Scroll mode to read data from Elasticsearch/OpenSearch. The task creates a Scroll request for each shard of the imported index. The Elasticsearch/OpenSearch server retains session information for each Scroll request, which consumes server-side memory resources. |
High load | If you import many indexes or a large dataset, the overall load on the Elasticsearch/OpenSearch server can become high. This may even affect the overall availability of the Elasticsearch/OpenSearch service. If the load on your Elasticsearch/OpenSearch cluster is high, you can submit a ticket to request an adjustment to the maximum traffic for the import task. |