This topic details the limits on importing data from OSS into Simple Log Service.
Collection limits
| Item | Description |
| Size of a single object |
If an object exceeds the size limit, the import task skips the entire object. |
| Size of a single data record | The maximum size for a single data record is 3 MB. Records exceeding this limit are dropped. The number of dropped records is displayed as write failures on the Data Processing Insight dashboard. For more information, see Related operations. |
| Object update | If you append new content to an already imported OSS object, the import task re-imports the entire object. |
| New object detection latency | The minimum interval for detecting new objects can be set to 1 minute. However, delays may be longer when importing a large number of objects. |
Configuration limits
| Item | Description |
| Number of data import configurations | You can create up to 100 import configurations of all types in a single Project. To request a higher limit, submit a ticket. |
Performance limits
| Item | Description |
| Number of concurrent subtasks | Simple Log Service automatically creates multiple concurrent subtasks based on the number of objects to be imported. By default, each import configuration can generate up to 8 subtasks. Each subtask supports a processing speed of up to 10 MB/s for decompressed data. This results in a total default processing speed of up to 80 MB/s per import task. To request a higher limit, submit a ticket. |
| Number of shards in a Logstore | The number of shards in the destination Logstore determines write performance. A single shard supports a write throughput of 5 MB/s. If you are importing a large volume of data, we recommend increasing the number of shards in the Logstore. For more information, see Manage shards. |
| Reading archive objects | Archive objects must be restored before they can be read. Restoring an object typically takes about 1 minute. |
| Object size | For a given total data volume, read throughput is higher for larger objects and lower for numerous small objects. |
| Network | Placing your OSS bucket and Simple Log Service Project in the same region saves on internet traffic costs and enables faster data transfer. Object read performance is lower for cross-region imports due to network conditions. |
| Import latency for new data | If you have a large number of existing objects and do not enable the OSS Metadata Indexing configuration, the service may not collect new objects within the configured check interval. For example, with approximately 1 million existing objects, the collection latency for new objects is about 2 minutes. The latency scales linearly with the number of existing objects. |