ApsaraDB for MongoDB:Optimization suggestions for parameters

operationProfiling.mode

• Supported major versions: MongoDB 3.0 and later versions.

• Force restart after parameter modification: Required.

• Default value: off.

• Function: Specifies the level of operations that the query profiler profiles.

• Issues:

  • If this parameter is set to all or slowOp and a large number of slow query logs are generated, instance performance is degraded.

  • Some users may be confused due to the presence of the system.profile collection in a database when they forget to disable the query profiler.

  • Some users may also misunderstand that they need to set this parameter to slowOp to generate slow query logs.

• Optimization suggestions:

  We recommend that you keep the default value of this parameter. If you enable the query profiler, instance performance is degraded. Slow query logs generally provide similar analysis information. Enable the query profiler when necessary and disable it in a timely manner after the query analysis is complete. For more information about the database profiler, see Profiler Overhead.

operationProfiling.slowOpThresholdMs

• Supported major versions: MongoDB 3.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 100.

• Function: Specifies the threshold used to identify slow queries.

• Issues:

  • If this parameter is set to a value that is too small, a large number of extraneous slow query logs and audit logs are generated. This extends the time required for slow query analysis.

  • If this parameter is set to a value that is too large, many slow queries are not recorded in logs. This also extends the time required for slow query analysis.

• Optimization suggestions: Increase or decrease the parameter value based on your business requirements. We recommend that you set this parameter to a value larger than the average duration of the main queries for your business. Examples:

  • Assume that the daily query duration in a business that is sensitive to query latency is only approximately 30 milliseconds. To facilitate the analysis of slow queries with transient jitter, decrease the parameter value to 50.

  • Assume that the daily query duration in a business that involves analytical queries approximately ranges from 300 to 400 milliseconds. To reduce the number of slow query logs generated for the daily queries, increase the parameter value to 500.

replication.oplogGlobalIdEnabled

• Supported major versions: MongoDB 4.0 and later versions.

• Force restart after parameter modification: Required.

• Default value: false.

• Function: Specifies whether to enable the global ID (GID) to accelerate two-way synchronization performed by Data Transmission Service (DTS) or mongoShake. This parameter is a self-developed parameter. The GID is engineered to fix the circular synchronization issue that occurs in two-way synchronization.

• Optimization suggestions: Enable this parameter only when two-way synchronization is required. This parameter takes effect only after your instance is restarted. We recommend that you change this parameter during off-peak hours.

replication.oplogSizeMB

• Supported major versions: MongoDB 3.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 10% of the disk size. For example, if the disk size of your instance is 500 GB, the original parameter value is 51200, which is equal to 50 GB.

• Function: Specifies the maximum size of the oplog collection that stores logical synchronization logs.

• Issues: If this parameter is set to a value that is too small, secondary nodes cannot keep up with other nodes and enter the RECOVERING state. In addition, log backups cannot store all oplog entries, which results in data loss in the oplog entries and failed point-in-time restoration.

• Optimization suggestions: We recommend that you keep the default value of this parameter. Do not decrease the parameter value. Increase the parameter value when necessary. Increase the parameter value in the scenario in which a small amount of data with frequent updates is processed and a large number of oplog entries are generated. This way, the oplog can store oplog records for a longer period of time, which avoids data loss in the oplog entries. The best practice is to set the oplog size to a value that allows the oplog collection to retain at least one hour of oplog entries.

setParameter.cursorTimeoutMillis

• Supported major versions: MongoDB 3.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 600000, which is equal to 10 min.

• Function: Specifies the expiration threshold of an idle cursor. Unit: milliseconds. If the idle time of a cursor exceeds this threshold, ApsaraDB for MongoDB automatically deletes the cursor.

• Issues: If you attempt to access an expired cursor that is deleted, your client receives an error in the following format:

  Message: "cursor id xxxxxxx not found"
  ErrorCode: CursorNotFound(43)

• Optimization suggestions: We recommend that you do not increase the parameter value. To reduce the resource overhead of idle cursors, decrease the parameter value. For example, you can decrease the parameter value to 300000. Refrain from cursors that are idle for a long period of time in all scenarios.

setParameter.flowControlTargetLagSeconds

• Supported major versions: MongoDB 4.2 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 10.

• Function: Specifies the threshold for triggering the flowControl mechanism. The mechanism is engineered to ensure that a majority of data can be committed within the specified number of seconds. For more information, see Replication Lag and Flow Control.

• Issues: The slow query logs similar to the following one are generated, and the response time of requests is increased. In the following log, the value of the durationMillis parameter is basically equal to that of the flowControl.timeAcquiringMicros parameter, which indicates that slow requests are mainly caused by the flowControl mechanism.

  {
    "t": {
      "$date": "2024-04-25T13:28:45.840+08:00"
    },
    "s": "I",
    "c": "WRITE",
    "id": 51803,
    "ctx": "conn199253",
    "msg": "Slow query",
    "attr": {
      "type": "update",
      "ns": "xxx.xxxxx",
      "command": ...,
      "planSummary": "IDHACK",
      "totalOplogSlotDurationMicros": 61,
      "keysExamined": 1,
      "docsExamined": 1,
      "nMatched": 1,
      "nModified": 1,
      "nUpserted": 0,
      "keysInserted": 0,
      "keysDeleted": 0,
      "numYields": 0,
      "locks": ...,
      "flowControl": {
        "acquireCount": 1,
        "acquireWaitCount": 1,
        "timeAcquiringMicros": 959000
      },
      "readConcern": {
        "level": "local",
        "provenance": "implicitDefault"
      },
      "storage": {},
      "cpuNanos": 258845,
      "remote": "172.16.6.38:52368",
      "durationMillis": 959
    }
  }
  

• Optimization suggestions: Increase the parameter value to reduce the sensitivity of the flowControl mechanism. If requests are still frequently throttled after you increase the parameter value, your instance encounters performance bottlenecks in terms of data synchronization between primary and secondary instances. In this case, you need to further analyze this issue and adopt other methods to fix the preceding issue. For example, you can upgrade the configurations of the instance or set the write concern to majority.

setParameter.oplogFetcherUsesExhaust

• Supported major versions: MongoDB 4.4 and later versions.

• Force restart after parameter modification: Required.

• Default value: true.

• Function: Specifies whether to enable stream replication. For more information, see Streaming Replication. If you disable stream replication, data synchronization between primary and secondary instances reverts to the same data pulling method as that of the previous version. This means that secondary nodes send requests to the synchronization source to obtain a batch of oplog entries and then wait for responses. This way, each batch of oplog entries requires a round-trip interaction over the network.

• Issues: Stream replication may generate additional performance and network bandwidth overhead in some scenarios.

• Optimization suggestions: We recommend that you do not adjust this parameter value. Stream replication can reduce replication latency in high-load and high-latency network environments. However, it also creates the risk of written data loss when the primary node whose write concern is {w:1} crashes, and write latency when the write concern is set to a level depending on primary/secondary replication, such as {w:majority or {w:>1}.

setParameter.maxTransactionLockRequestTimeoutMillis

• Supported major versions: MongoDB 4.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 5.

• Function: Specifies the timeout period for the transaction to wait to acquire locks required by the operations in the transaction. Unit: milliseconds. If operations in a transaction cannot acquire the required locks within a period of time, the system automatically aborts the transaction.

• Issues: The following error message indicating the timeout for acquiring locks is displayed in logs or your client. Drivers of later versions automatically retry when they receive the TransientTransactionError error. Therefore, the error may appear only in logs and is imperceptible to your client.

  Message: "Unable to acquire lock '{8442595743001781021: Database, 1525066715360699165}' within a max lock request timeout of '5ms' milliseconds."
  ErrorCode: LockTimeout(24)

• Optimization suggestions: If your client often receives such errors, you can increase the parameter value. This reduces the frequency of transaction abort caused by the failure to instantly acquire concurrent locks. However, this delays the abort of deadlocked transactions. If the preceding issue persists after you increase the parameter value, we recommend that you do not increase the parameter value again. You need to optimize the business logic instead. For example, you need to avoid concurrent modifications to the same document in a transaction, review the operations in the transaction, and check whether the transaction contains operations occupying locks for a long period of time, such as DDL operations and queries to be optimized.

setParameter.replWriterThreadCount

• Supported major versions: MongoDB 3.2 and later versions.

• Force restart after parameter modification: Required.

• Default value: 16.

• Function: Specifies the maximum number of threads used to perform replication operations in parallel when synchronizing between primary and secondary instances. The maximum number of threads used is capped at twice the number of CPU cores.

• Issues: Latency on secondary nodes increases due to delayed data synchronization between primary and secondary instances in extreme scenarios.

• Optimization suggestions: In most cases, we recommend that you do not adjust this parameter value. In special cases, we recommend that you make adjustments based on the suggestions of Alibaba Cloud R&D engineers.

setParameter.tcmallocAggressiveMemoryDecommit

• Supported major versions: MongoDB 4.2 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 0., which indicates that the aggressive reclaim strategy of TCMalloc is disabled.

• Function: Specifies whether to enable the aggressive reclaim strategy of TCMalloc. ApsaraDB for MongoDB uses TCMalloc as the memory allocator. After you enable this strategy, ApsaraDB for MongoDB attempts to merge adjacent free memory blocks and return a portion of memory to the operating system.

• Issues:

  • An out-of memory (OOM) error is returned because the memory of mongod cannot be reclaimed in a timely manner due to the excessive memory consumption of query requests.

  • With continuous memory use, heap memory fragmentation increases. In this case, memory usage exceeds 80% and increases in a slow and steady manner.

• Optimization suggestions: In most cases, we recommend that you do not adjust this parameter value. If you encounter memory-related issues, adjust this parameter value during off-peak hours.

setParameter.transactionLifetimeLimitSeconds

• Supported major versions: MongoDB 4.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 60.

• Function: Specifies the lifecycle of a transaction. Unit: seconds. If the execution duration of a transaction exceeds this upper limit, the transaction is marked as expired and aborted by a periodic cleanup thread in the background.

• Issues: Your client receives an error in the following format:

  Message: "Aborting transaction with txnNumber xxx on session with lsid xxxxxxxxxx because it has been running for longer than 'transactionLifetimeLimitSeconds'"

• Optimization suggestions: Decrease the parameter value. For example, you can decrease the parameter value to 30. We recommend that you do not increase the parameter value. A uncommitted long transaction may pose an excessive workload on the WiredTiger cache. If the cache is overloaded, many issues occur, such as database stalling, significant increase in response latency, and high CPU utilization. This results in business damage. Refrain from long transactions in all scenarios. To fix timeout issues, split a transaction into smaller parts so that the transaction can be completed within a period of time. Make sure that query statements are optimized and provide appropriate index coverage to ensure quick data access in transactions.

For more information about the best practices for using transactions, see Transactions and Read/Write Concern.

storage.oplogMinRetentionHours

• Supported major versions: MongoDB 4.4 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 0, which indicates that this parameter does not take effect and the oplog size is specified by the replication.oplogSizeMB parameter.

• Function: Specifies the minimum retention period of the oplog collection that stores logical synchronization logs.

• Issues:

  • If this parameter is set to a value that is too large, the oplog collection occupies a large amount of disk space.

  • Some users may forget specifying this parameter and then be confused by the fluctuation of the disk size.

• Optimization suggestions: If your workloads are stable, we recommend that you keep the default value of this parameter. If your write operations experience significant changes, we recommend that you set this parameter to a floating point number larger than 1.0. When you configure this parameter, evaluate the disk space that can be occupied to avoid the scenario where your instance is locked due to exhausted disk space.

storage.wiredTiger.collectionConfig.blockCompressor

• Supported major versions: MongoDB 3.0 and later versions.

• Force restart after parameter modification: Required.

• Default value: snappy.

• Function: Specifies the default storage compression algorithm for collection data. If you change the default storage compression algorithm, only all new collections use the specified new algorithm. Existing collections are not affected. You can set this parameter to none, snappy, zlib, or zstd. The value zstd is available only for instances that run MongoDB 4.2 and later versions.

• Optimization suggestions: Modify the parameter value based on your business requirements. Different compression algorithms have distinct performance characteristics. Several compression algorithms provide high compression rates but bring high CPU overhead due to compression and decompression. Refer to corresponding test results to understand the actual performance of different compression algorithms. If your instance mainly stores cold data, you can modify the parameter value to zstd to achieve a higher compression ratio.

  Note

  If you want to use different compression algorithms for different collections, use the explicit createCollection command that provides the relevant options. For more information, see Specify Storage Engine Options.

setParameter.minSnapshotHistoryWindowInSeconds/setParameter.maxTargetSnapshotHistoryWindowInSeconds

• Supported major versions: MongoDB 4.4 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 300. Unit: seconds. The default value is equal to 5 minutes.

• Function: Specifies the window during which snapshots in WiredTiger are retained. The value 0 indicates that the window is closed. This parameter supports data reading in a time point specified by the atClusterTime parameter. For more information, see Read Concern and atClusterTime.

• Issues: This parameter poses some workloads on the WiredTiger cache, especially when a document is frequently updated.

• Optimization suggestions: Do not need to adjust this parameter value in most cases.

  • If you do not want to use the read atClusterTime feature, you can set this parameter to 0 for improved performance.

  • If you want to read snapshots that are generated more than 5 minutes ago, you can set this parameter to a larger value. However, you also need to pay attention to the additional memory consumption and performance overhead caused by the increase in the parameter value.

  Note

  If you set this parameter to a small value and specify an earlier time point for reading snapshots, you receive the SnapshotTooOld error.

  rsconf.chainingAllowed

  • Supported major versions: MongoDB 4.0 and later versions.

  • Force restart after parameter modification: Not required.

  • Default value: true.

  • Function: Specifies whether to allow chained replication for a replica set instance.

  • Issues:

    • If chained replication is disabled for a replica set instance, the primary node in the instance may carry a higher load, such as the higher CPU utilization and network traffic.

    • If chained replication is enabled for a replica set instance, data lag is likely to occur on the secondary nodes of the instance.

• Optimization suggestions:

  • If a replica set instance contains a maximum of four nodes, you can enable or disable chained replication for the instance based on you rbusiness requirement.

  • If a replica set instance contains at least five nodes and the writeConcern of the instance is set to{w:majority}, you must balance the load of the primary node and instance performance. If you disable chained replication, the write performance of the instance is improved. However, the load of the primary node significantly increases.

setParameter.migrateCloneInsertionBatchSize

• Supported major versions: MongoDB 4.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 0, which indicates that batches can contain up to 16 MB of documents.

• Function: Specifies the maximum number of documents to insert in a single batch during the cloning stage of the chunk migration process.

• Issues: Performance fluctuations occur on shards during the chunk migration process in some scenarios.

• Optimization suggestions: Do not need to adjust this parameter value in most cases. If performance fluctuations appear in your sharded cluster instance during load balancing due to the chunk migration, adjust this parameter value to a fixed batch size.

setParameter.rangeDeleterBatchDelayMS

• Supported major versions: MongoDB 4.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 20.

• Function: Specifies the interval to wait before the next batch of deletion during the cleanup stage of the chunk migration process. Unit: milliseconds. This parameter affects the cleanupOrphaned command that is used to clean up orphaned documents.

• Issues:

  • In some scenarios, the CPU utilization may spike due to the asynchronous deletion of documents after chunk migration.

  • If this parameter is set to a value that is too large, documents may be orphaned because they are not deleted in a timely manner. In addition, the following error message may be returned because operations to delete a large number of documents timed out:

    Message: "OperationFailed: Data transfer error: ExceededTimeLimit: Failed to delete orphaned <db>.<collection> range [xxxxxx,xxxxx] :: caused by :: operation exceeded time limit"

• Optimization suggestions: Do not need to adjust this parameter value in most cases. If the CPU utilization of your sharded cluster instance spikes due to the asynchronous deletion of documents during load balancing, you can increase the parameter value. For example, increase the parameter value to 200.

setParameter.rangeDeleterBatchSize

• Supported major versions: MongoDB 4.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 0, which indicates that the system uses an appropriate value. In most cases, the value is 128.

• Function: Specifies the maximum number of documents in a single batch of asynchronous deletion during the cleanup stage of the chunk migration process.

• Issues: The CPU utilization may spike due to the asynchronous deletion of documents after chunk migration in some scenarios.

• Optimization suggestions: Do not need to adjust this parameter value in most cases. If the CPU utilization of your sharded cluster instance spikes due to the asynchronous deletion of documents during load balancing, adjust this parameter value to a fixed batch size.

setParameter.chunkMigrationConcurrency

• Supported major versions: MongoDB 5.0 and 7.0.

• Force restart after parameter modification: Not required.

• Default value: 1.

• Function: Specifies the number of concurrent threads used to migrate chunks.

• Issues: Chunks may be slowly migrated by the balancer and acceleration is required in some scenarios, such as when shards are added or removed.

• Optimization suggestions: Do not need to adjust this parameter value in most cases. If you want to accelerate the balancing process, you can increase the parameter value. After the adjustment, you must pay attention to the loads of your instance and the impacts on your business. If issues occur, you must roll back the parameter value in a timely manner.

setParameter.receiveChunkWaitForRangeDeleterTimeoutMS

• Supported major versions: MongoDB 4.4 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 10000. Unit: milliseconds. The default value is equal to 10 seconds.

• Function: Specifies the timeout period required to wait for deleting orphaned documents before chunk migration.

• Issues: When the balancer is running, the following log indicating a timeout error is generated:

  ExceededTimeLimit: Failed to delete orphaned <db.collection> range [{ <shard_key>: MinKey }, { <shard_key>: -9186000910690368367 }) :: caused by :: operation exceeded time limit

• Optimization suggestions: Do not need to adjust this parameter value in most cases. If the preceding error occurs, you can set this parameter to a larger value. This way, the moveChunk operation can wait for a longer period of time to delete orphaned documents, which avoids similar timeout errors.

setParameter.minSnapshotHistoryWindowInSeconds/setParameter.maxTargetSnapshotHistoryWindowInSeconds

• Supported major versions: MongoDB 4.4 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 300. Unit: seconds. The default value is equal to 5 minutes.

• Function: Specifies the window during which snapshots in WiredTiger are retained. The value 0 indicates that the window is closed. This parameter supports data reading in a time point specified by the atClusterTime parameter. For more information, see Read Concern and atClusterTime.

• Issues: This parameter poses some workloads on the WiredTiger cache, especially when a document is frequently updated.

• Optimization suggestions: Do not need to adjust this parameter value in most cases.

  • If you do not want to use the read atClusterTime feature, you can set this parameter to 0 for improved performance.

  • If you want to read snapshots that are generated more than 5 minutes ago, you can set this parameter to a larger value. However, you also need to pay attention to the additional memory consumption and performance overhead caused by the increase in the parameter value.

  Note

  If you set this parameter to a small value and specify an earlier time point for reading snapshots, you receive the SnapshotTooOld error.

  rsconf.chainingAllowed

  • Supported major versions: MongoDB 4.0 and later versions.

  • Force restart after parameter modification: Not required.

  • Default value: true.

  • Function: Specifies whether to allow chained replication for a replica set instance.

  • Issues:

    • If chained replication is disabled for a replica set instance, the primary node in the instance may carry a higher load, such as the higher CPU utilization and network traffic.

    • If chained replication is enabled for a replica set instance, data lag is likely to occur on the secondary nodes of the instance.

• Optimization suggestions:

  • If a replica set instance contains a maximum of four nodes, you can enable or disable chained replication for the instance based on you rbusiness requirement.

  • If a replica set instance contains at least five nodes and the writeConcern of the instance is set to{w:majority}, you must balance the load of the primary node and instance performance. If you disable chained replication, the write performance of the instance is improved. However, the load of the primary node significantly increases.

operationProfiling.slowOpThresholdMs

• Supported major versions: MongoDB 3.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 100.

• Function: Specifies the threshold used to identify slow queries.

• Issues:

  • If this parameter is set to a value that is too small, a large number of extraneous slow query logs and audit logs are generated. This extends the time required for slow query analysis.

  • If this parameter is set to a value that is too large, many slow queries are not recorded in logs. This also extends the time required for slow query analysis.

• Optimization suggestions: Increase or decrease the parameter value based on your business requirements. We recommend that you set this parameter to a value larger than the average duration of the main queries for your business. Examples:

  • Assume that the daily query duration in a business that is sensitive to query latency is only approximately 30 milliseconds. To facilitate the analysis of slow queries with transient jitter, decrease the parameter value to 50.

  • Assume that the daily query duration in a business that involves analytical queries approximately ranges from 300 to 400 milliseconds. To reduce the number of slow query logs generated for the daily queries, increase the parameter value to 500.

setParameter.ShardingTaskExecutorPoolMaxConnecting

• Supported major versions: MongoDB 3.6 and later versions.

• Force restart after parameter modification:

  • For MongoDB 3.6 and MongoDB 4.0: Required.

  • For MongoDB 4.2 and later versions: Not required.

• Default value: 2.

• Function: Specifies the maximum number of connections that can be established in the TaskExecutor connection pool of a mongos node in a sharded cluster instance during connection initialization. This parameter specifies the speed of connection establishment between mongos and mongod.

• Issues: If this parameter is set to a large value, the CPU utilization of a mongos node may spike when a large number of connections are established.

• Optimization suggestions: We recommend that you do not adjust this parameter value.

setParameter.ShardingTaskExecutorPoolMaxSize

• Supported major versions: MongoDB 3.6 and later versions.

• Force restart after parameter modification:

  • For MongoDB 3.6 and MongoDB 4.0: Required.

  • For MongoDB 4.2 and later versions: Not required.

• Default value: 2^64-1, which indicates the maximum value of the INT64 data type.

• Function: Specifies the maximum number of connections in each TaskExecutor connection pool of a mongos node in a sharded cluster instance.

• Optimization suggestions: Do not need to adjust this parameter value. Configure the parameter to limit the maximum number of connections that can be established between mongos and shards. We recommend that you do not set the parameter to a small value. If this parameter is set to a small value, requests on mongos are blocked when all connection are exhausted.

setParameter.ShardingTaskExecutorPoolMinSize

• Supported major versions: MongoDB 3.6 and later versions.

• Force restart after parameter modification:

  • For MongoDB 3.6 and MongoDB 4.0: Required.

  • For MongoDB 4.2 and later versions: Not required.

• Default value: 1.

• Function: Specifies the minimum number of connections in each TaskExecutor connection pool of a mongos node in a sharded cluster instance.

• Issues: In some scenarios, a large number of additional connections need to be established on TaskExecutor connection pools due to burst requests on mongos, which causes a spike CPU utilization and other issues on mongos.

• Optimization suggestions: We recommend that you set the parameter to a value that ranges from 10 to 50, depending on the number of shards and the number of nodes in each shard. Note that a low resource overhead is generated when mongos maintain idle connections to shards.

setParameter.cursorTimeoutMillis

• Supported major versions: MongoDB 3.0 and later versions.

• Force restart after parameter modification: Not required.

• Default value: 600000, which is equal to 10 min.

• Function: Specifies the expiration threshold of an idle cursor. Unit: milliseconds. If the idle time of a cursor exceeds this threshold, ApsaraDB for MongoDB automatically deletes the cursor.

• Issues: If you attempt to access an expired cursor that is deleted, your client receives an error in the following format:

  Message: "cursor id xxxxxxx not found"
  ErrorCode: CursorNotFound(43)

• Optimization suggestions: We recommend that you do not increase the parameter value. To reduce the resource overhead of idle cursors, decrease the parameter value. For example, you can decrease the parameter value to 300000. Refrain from cursors that are idle for a long period of time in all scenarios.