ApsaraDB for MongoDB:Imbalanced chunk count across shards after a MongoDB upgrade

Behavior before MongoDB 6.0 minor 7.0.1

Before ApsaraDB for MongoDB 6.0 minor 7.0.1, the sharded cluster balanced data as follows:

1. The default chunk size was 64 MB.

2. When a chunk grew beyond 64 MB, it was automatically split (AutoSplit) into smaller chunks.

3. The balancer periodically inspected the chunk count on each shard and migrated chunks (moveChunk) once the difference exceeded the migration threshold.

4. Eventually, each shard held many chunks of roughly equal count.

Under this model, sh.status() showed multiple chunks per shard, which gave you an intuitive sense that data was evenly distributed.

Changes in MongoDB 6.0 from minor 7.0.1 (community 6.0.3)

Starting from ApsaraDB for MongoDB 6.0 minor 7.0.1, which corresponds to community MongoDB 6.0.3, the balancer introduced these changes:

• AutoSplit removed: Chunks are no longer split automatically when they fill up. Splitting only happens when needed before a migration.

• New balancing criterion: From chunk count to actual data size.

• Adjusted migration threshold: Migration is triggered when the size of the same collection differs by more than three times the chunk size across shards. (The default chunk size from ApsaraDB for MongoDB 6.0 is 128 MB, which means the threshold is 384 MB.)

• Terminology change: The chunk concept is de-emphasized in favor of "range". moveChunk is replaced by moveRange.

• You no longer need to run sh.enableSharding() before sharding a collection.

Effect: Because AutoSplit is removed and chunk count is no longer the balancing criterion, the chunk counts across shards may be uneven, which is normal. For more information, see sh.enableAutoSplit().

New in MongoDB 7.0: AutoMerger

MongoDB 7.0 introduces the AutoMerger feature:

• The balancer automatically merges contiguous, mergeable chunks on the same shard.

• AutoMerger runs in the background as part of balancing operations and is enabled by default.

• It runs every autoMergerIntervalSecs seconds.

• If a balancing window is configured, AutoMerger only runs within that window.

Effect: In MongoDB 7.0 and later, AutoMerger may eventually consolidate the chunks on each shard into a single chunk. This is also normal. For more information, see AutoMerger.

Why these changes were made

The chunk-count-based balancer in older versions had several drawbacks:

Using actual data size as the balancing criterion gives a more accurate picture of data distribution and reduces unnecessary chunk splits and migrations.

MongoDB 4.4 (legacy behavior)

Run sh.status(). Focus on the chunkMetadata field in the output. Each shard has 435 chunks, and the counts are equal. This is the result of the legacy AutoSplit mechanism:

mongos> sh.status()
shardingVersion
{ _id: 1, clusterId: ObjectId('69fbfda6dec76b373f5eda81') }
---
shards
[
  {
    _id: 'd-uf6xxxx1',
    host: 'mgset-1xxxxx0/192.168.XX.2:3005,192.168.XX.11:3002',
    state: 1
  },
  {
    _id: 'd-uf6xxxx2',
    host: 'mgset-1xxxxx1/192.168.XX.4:3002,192.168.XX.5:3002',
    state: 1
  }
]
---
active mongoses
[ { '4.4.30': 2 } ]
---
autosplit
{ 'Currently enabled': 'yes' }
---
balancer
{
  'Currently enabled': 'yes',
  'Failed balancer rounds in last 5 attempts': 0,
  'Currently running': 'no',
  'Migration Results for the last 24 hours': { '947': 'Success' }
}
---
databases
[
  {
    database: {
      _id: 'ycsb',
      primary: 'd-uf6xxxx2',
      partitioned: true
    },
    collections: {
      'ycsb.test1': {
        shardKey: { _id: 'hashed' },
        unique: false,
        balancing: true,
        chunkMetadata: [
          { shard: 'd-uf6xxxx1', nChunks: 435 },
          { shard: 'd-uf6xxxx2', nChunks: 435 }
        ],
        chunks: [
          'too many chunks to print, use verbose if you want to force print'
        ],
        tags: []
      }
    }
  }
]

Run db.test1.getShardDistribution(). Focus on the data (data size) and chunks (chunk count) fields for each shard:

mongos> use ycsb
switched to db ycsb
mongos> db.test1.getShardDistribution()
Shard d-uf6xxxx1 at mgset-1xxxxx0/192.168.XX.2:3005,192.168.XX.11:3002
{
  data: '27.2GiB',
  docs: 25014674,
  chunks: 435,
  'estimated data per chunk': '64.04MiB',
  'estimated docs per chunk': 57504
}
---
Shard d-uf6xxxx2 at mgset-1xxxxx1/192.168.XX.4:3002,192.168.XX.5:3002
{
  data: '36.23GiB',
  docs: 33313293,
  chunks: 435,
  'estimated data per chunk': '85.29MiB',
  'estimated docs per chunk': 76582
}
---
Totals
{
  data: '63.44GiB',
  docs: 58327967,
  chunks: 870,
  'Shard d-uf6xxxx1': [
    '42.88 % data',
    '42.88 % docs in cluster',
    '1KiB avg obj size on shard'
  ],
  'Shard d-uf6xxxx2': [
    '57.11 % data',
    '57.11 % docs in cluster',
    '1KiB avg obj size on shard'
  ]
}

MongoDB 6.0 (minor 7.0.1 and later)

Run sh.status(). Focus on the chunkMetadata field. The chunk counts are noticeably uneven across shards (216 vs. 1):

mongos> sh.status()
shardingVersion
{ _id: 1, clusterId: ObjectId('69fbfdc420578638757c1d06') }
---
shards
[
  {
    _id: 'd-uf6xxxx3',
    host: 'mgset-1xxxxx2/192.168.XX.7:3002,192.168.XX.8:3002',
    state: 1,
    topologyTime: Timestamp({ t: 1778122310, i: 2 })
  },
  {
    _id: 'd-uf6xxxx4',
    host: 'mgset-1xxxxx3/192.168.XX.9:3002,192.168.XX.10:3002',
    state: 1,
    topologyTime: Timestamp({ t: 1778122310, i: 9 })
  }
]
---
active mongoses
[ { '6.0.27': 2 } ]
---
autosplit
{ 'Currently enabled': 'yes' }
---
balancer
{
  'Currently running': 'no',
  'Currently enabled': 'yes',
  'Failed balancer rounds in last 5 attempts': 0,
  'Migration Results for the last 24 hours': { '216': 'Success' }
}
---
databases
[
  {
    database: {
      _id: 'ycsb',
      primary: 'd-uf6xxxx4',
      partitioned: false,
      version: {
        uuid: UUID('4ee45231-9585-4dba-8f34-f624860e1a79'),
        timestamp: Timestamp({ t: 1778122930, i: 1 }),
        lastMod: 1
      }
    },
    collections: {
      'ycsb.test2': {
        shardKey: { _id: 'hashed' },
        unique: false,
        balancing: true,
        allowMigrations: true,
        chunkMetadata: [
          { shard: 'd-uf6xxxx3', nChunks: 216 },
          { shard: 'd-uf6xxxx4', nChunks: 1 }
        ],
        chunks: [
          'too many chunks to print, use verbose if you want to force print'
        ],
        tags: []
      }
    }
  }
]

Run db.test2.getShardDistribution(). Focus on the data percentage for each shard. Despite the huge difference in chunk counts (216 vs. 1), the data is still evenly distributed (50.31% vs. 49.68%):

mongos> use ycsb
switched to db ycsb
mongos> db.test2.getShardDistribution()
Shard d-uf6xxxx4 at mgset-1xxxxx3/192.168.XX.9:3002,192.168.XX.10:3002
{
  data: '27.36GiB',
  docs: 32285618,
  chunks: 1,
  'estimated data per chunk': '27.36GiB',
  'estimated docs per chunk': 32285618
}
---
Shard d-uf6xxxx3 at mgset-1xxxxx2/192.168.XX.7:3002,192.168.XX.8:3002
{
  data: '27.01GiB',
  docs: 24841944,
  chunks: 216,
  'estimated data per chunk': '128.09MiB',
  'estimated docs per chunk': 115009
}
---
Totals
{
  data: '54.38GiB',
  docs: 57127562,
  chunks: 217,
  'Shard d-uf6xxxx4': [
    '50.31 % data',
    '56.51 % docs in cluster',
    '1KiB avg obj size on shard'
  ],
  'Shard d-uf6xxxx3': [
    '49.68 % data',
    '43.48 % docs in cluster',
    '1KiB avg obj size on shard'
  ]
}

MongoDB 8.0 (with AutoMerger)

Right after migration completes, run sh.status(). Focus on the chunkMetadata field. Each shard still has multiple chunks (55 vs. 1):

mongos> sh.status()
shardingVersion
{ _id: 1, clusterId: ObjectId('69fbfdbdd7b0e3d9551dd833') }
---
shards
[
  {
    _id: 'd-uf6xxxx5',
    host: 'mgset-1xxxxx4/192.168.XX.1:3000,192.168.XX.3:3002',
    state: 1,
    topologyTime: Timestamp({ t: 1778122320, i: 8 }),
    replSetConfigVersion: Long('-1')
  },
  {
    _id: 'd-uf6xxxx6',
    host: 'mgset-1xxxxx5/192.168.XX.5:3003,192.168.XX.6:3003',
    state: 1,
    topologyTime: Timestamp({ t: 1778122319, i: 10 }),
    replSetConfigVersion: Long('-1')
  }
]
---
active mongoses
[ { '8.0.18': 2 } ]
---
autosplit
{ 'Currently enabled': 'yes' }
---
balancer
{
  'Currently enabled': 'yes',
  'Failed balancer rounds in last 5 attempts': 0,
  'Currently running': 'no',
  'Migration Results for the last 24 hours': { '216': 'Success' }
}
---
databases
[
  {
    database: {
      _id: 'ycsb',
      primary: 'd-uf6xxxx6',
      version: {
        uuid: UUID('98799357-048e-45bc-a472-387165e0fe74'),
        timestamp: Timestamp({ t: 1778122931, i: 2 }),
        lastMod: 1
      }
    },
    collections: {
      'ycsb.test2': {
        shardKey: { _id: 'hashed' },
        unique: false,
        balancing: true,
        allowMigrations: true,
        chunkMetadata: [
          { shard: 'd-uf6xxxx5', nChunks: 55 },
          { shard: 'd-uf6xxxx6', nChunks: 1 }
        ],
        chunks: [
          'too many chunks to print, use verbose if you want to force print'
        ],
        tags: []
      }
    }
  }
]

After AutoMerger finishes, run sh.status() again. Focus on the chunkMetadata field. Each shard now has only one chunk:

mongos> sh.status()
shardingVersion
{ _id: 1, clusterId: ObjectId('69fbfdbdd7b0e3d9551dd833') }
---
shards
[
  {
    _id: 'd-uf6xxxx5',
    host: 'mgset-1xxxxx4/192.168.XX.1:3000,192.168.XX.3:3002',
    state: 1,
    topologyTime: Timestamp({ t: 1778122320, i: 8 }),
    replSetConfigVersion: Long('-1')
  },
  {
    _id: 'd-uf6xxxx6',
    host: 'mgset-1xxxxx5/192.168.XX.5:3003,192.168.XX.6:3003',
    state: 1,
    topologyTime: Timestamp({ t: 1778122319, i: 10 }),
    replSetConfigVersion: Long('-1')
  }
]
---
active mongoses
[ { '8.0.18': 2 } ]
---
autosplit
{ 'Currently enabled': 'yes' }
---
balancer
{
  'Currently enabled': 'yes',
  'Failed balancer rounds in last 5 attempts': 0,
  'Currently running': 'no',
  'Migration Results for the last 24 hours': { '216': 'Success' }
}
---
databases
[
  {
    database: {
      _id: 'ycsb',
      primary: 'd-uf6xxxx6',
      version: {
        uuid: UUID('98799357-048e-45bc-a472-387165e0fe74'),
        timestamp: Timestamp({ t: 1778122931, i: 2 }),
        lastMod: 1
      }
    },
    collections: {
      'ycsb.test2': {
        shardKey: { _id: 'hashed' },
        unique: false,
        balancing: true,
        allowMigrations: true,
        chunkMetadata: [
          { shard: 'd-uf6xxxx5', nChunks: 1 },
          { shard: 'd-uf6xxxx6', nChunks: 1 }
        ],
        chunks: [
          { min: { _id: MinKey() }, max: { _id: Long('-58615400192952365') }, 'on shard': 'd-uf6xxxx5', 'last modified': Timestamp({ t: 217, i: 2 }) },
          { min: { _id: Long('-58615400192952365') }, max: { _id: MaxKey() }, 'on shard': 'd-uf6xxxx6', 'last modified': Timestamp({ t: 217, i: 1 }) }
        ],
        tags: []
      }
    }
  }
]

Run db.test2.getShardDistribution(). Focus on the data percentage. Even though each shard has only one chunk, the data is still evenly distributed (49.67% vs. 50.32%):

mongos> use ycsb
switched to db ycsb
mongos> db.test2.getShardDistribution()
Shard d-uf6xxxx5 at mgset-1xxxxx4/192.168.XX.1:3000,192.168.XX.3:3002
{
  data: '27.01GiB',
  docs: 24841944,
  chunks: 1,
  'estimated data per chunk': '27.01GiB',
  'estimated docs per chunk': 24841944
}
---
Shard d-uf6xxxx6 at mgset-1xxxxx5/192.168.XX.5:3003,192.168.XX.6:3003
{
  data: '27.37GiB',
  docs: 35898246,
  chunks: 1,
  'estimated data per chunk': '27.37GiB',
  'estimated docs per chunk': 35898246
}
---
Totals
{
  data: '54.39GiB',
  docs: 60740190,
  chunks: 2,
  'Shard d-uf6xxxx5': [
    '49.67 % data',
    '40.89 % docs in cluster',
    '1KiB avg obj size on shard'
  ],
  'Shard d-uf6xxxx6': [
    '50.32 % data',
    '59.1 % docs in cluster',
    '1KiB avg obj size on shard'
  ]
}

Recommended: getShardDistribution()

Connect to mongos and run:

use <your_database>
db.<your_collection>.getShardDistribution()

This command shows the data size, document count, and percentage of the collection on each shard, providing an accurate view of distribution.

Helper: db.stats()

use <your_database>
db.stats()

Returns the overall storage usage of the database across shards.

No longer recommended: sh.status() chunk counts

sh.status() still works, but the chunk counts no longer reflect data balance. Uneven chunk counts or a single chunk per shard are normal and do not indicate a problem with data distribution.