All Products
Search
Document Center

Solution to Full Instance Space Caused by Binlog Files of RDS MySQL Instances

Last Updated: May 19, 2022

Problem description

ApsaraDB for RDS MySQL instances are automatically locked because the binary log file is full of disk space. The running status on the Instance Details page is locked. In addition, confirm that the log space usage is high in the disk space graph on the Monitoring and Alarm page of the instance. The specific values are subject to the actual environment.

Cause

A MySQL instance may quickly generate binlog files due to large transactions, resulting in full instance space. To avoid data loss, the instance is automatically locked. After the disk is locked, the write operation cannot be performed.

Solution

Take note of the following items:

  • Before you perform high-risk operations such as modifying the specifications or data of an Alibaba Cloud instance, we recommend that you check the disaster recovery and fault tolerance capabilities of the instance to ensure data security.
  • Before you modify the specifications or data of an Alibaba Cloud instance, such as an Elastic Compute Service (ECS) instance or an ApsaraDB RDS instance, we recommend that you create snapshots or enable backups for the instance. For example, you can enable log backups for an ApsaraDB RDS instance.
  • If you have granted specific users the permissions on sensitive information, such as usernames and passwords, or submitted sensitive information in the Alibaba Cloud Management Console, we recommend that you modify the sensitive information at the earliest opportunity.

Prerequisites

For ApsaraDB RDS for MySQL 5.5, 5.6, 5.7, or 8.0, we recommend that you resize the disks in advance to ensure that the disk usage does not exceed 80%. For more information, see Change configurations. Second, we recommend that you clean up useless logs or data.

Limits

Before you perform the steps to clean up useless local logs, check the following precautions:

  • Binlog files record the transaction information of an RDS instance. They are the basis for high availability and recoverability of RDS instances. We recommend that you do not disable them. You can use the one-click binary log upload feature to OSS to release disk space or modify local binary log settings. For more information, see Local binary log settings.
  • There is a delay in clearing binlog files. Please wait patiently for the log space usage of the instance to decrease.
  • One-click upload of binlogs will asynchronously submit the cleanup task in the background, and the cleanup task will upload the written binlogs to OSS (not the OSS purchased by the user), and then delete the binlog files from the instance space. The binlog files currently being written cannot be cleaned because they have not been written. Therefore, there will be a certain delay in the cleanup process. We recommend that you wait patiently after you click Upload Binlog. Do not click this button multiple times. You can check whether the disk space is reduced on the Basic Information page.
  • DML operations, such as DML operations that involve large fields, quickly generate binlogs. In this case, you may upload binlog files to the backup space and delete binlog files from the instance space. In this case, we recommend that you upgrade the disk space and troubleshoot the reasons for the rapid generation of binlog files.

Upload binlogs

  1. Log on to the RDS console. In the upper-left corner of the page, select the region where the instance is located.
  2. On the page that appears, click the ID of the instance that you want to update.
  3. In the left-side navigation pane, click Backup and Restoration.
  4. In the upper-right corner, click Upload Binlog. In the dialog box that appears, click OK.
    Note:
    • One-click upload of binlogs will be delayed. Please wait for about 15 minutes.
    • This operation deletes all log files except the latest two local log files. If there are only two log files, this operation does not delete any files.
    • Basic Edition instances do not support the binary log upload feature. We recommend that you delete local logs by following the instructions in Modify the binary log file settings.

Modify local log (Binlog) file settings

  1. Log on to the RDS console. In the upper-left corner of the page, select the region where the instance is located.
  2. On the page that appears, click the ID of the instance that you want to update.
  3. In the left-side navigation pane, click Backup and Restoration.
  4. Click the Local Log Settings tab, click Local Log retention policy, and set the corresponding value based on your actual situation.
    Note: If the retention period is set to 0, binary logs are not stored locally and are directly uploaded to OSS.

What to do next

If the lock problem is resolved, we recommend that you enable the automatic expansion of the storage space of the instance when resources are insufficient. For more information, see Configure automatic expansion of storage space.

References

For more information about the solution to full instance space, see the following link:

Applicable scope

  • ApsaraDB RDS for MySQL