:Image-related issues for ECS instances

• "This copy of Windows is not genuine" error

  You must activate Windows. For more information, see Activate a genuine Windows Server system for an ECS instance by using a KMS domain name.

• DNS resolution timeouts in CentOS

  Cause

  Due to a change in the DNS resolution mechanism of CentOS 6 and CentOS 7, DNS resolution timeouts may occur on CentOS 6 and CentOS 7 instances created before February 22, 2017, or on instances created from custom images that were created before February 22, 2017.

  Solution

  To resolve this issue, perform the following steps:

  1. Download the fix_dns.sh script.

  2. Place the downloaded script in the /tmp directory of your CentOS system.

  3. Run the bash /tmp/fix_dns.sh command to execute the script.

  The following section describes the purpose and logic of the script:

  • Script purpose

    The script checks whether the /etc/resolv.conf file contains the options single-request-reopen setting. For more information, see the resolv.conf file description.

    In CentOS 6 and CentOS 7, the DNS resolution mechanism uses the same network 5-tuple to send both IPv4 and IPv6 DNS requests. You must enable the single-request-reopen configuration to accommodate this behavior. When this configuration is enabled, if two requests are sent from the same socket, the resolver closes the socket after sending the first request and opens a new socket before sending the second request. The configuration takes effect immediately, and you do not need to restart the instance.

  • Script logic

    1. Checks whether the instance is running CentOS.

       • If the instance is not running a CentOS system (for example, Ubuntu or Debian), the script stops.

       • If the instance is running a CentOS system, the script continues.

    2. Queries the /etc/resolv.conf file for the options configuration.

       • If no options configuration exists:

         The default Alibaba Cloud options configuration options timeout:2 attempts:3 rotate single-request-reopen is used.

       • If an options configuration exists:

         • If the single-request-reopen option does not exist, the script appends it to the options configuration.

         • If the single-request-reopen option exists, the script stops and does not change the DNS nameserver configuration.

• Missing IP addresses in CentOS 7 and Windows

  For information about the cause and solution, see Check for and fix missing IP addresses on CentOS 7 and Windows instances.

• Cannot generate crash dump on CentOS 7.9 Arm

  Symptoms

  After a CentOS 7.9 Arm system experiences downtime, you run the ls /var/crash command to query for dump files, but no vmcore file is generated.

  

  Cause

  The kernel of the CentOS 7.9 Arm system has the CONFIG_ARM64_USER_VA_BITS_52=y feature. The version of the native makedumpfile software is incompatible with the kernel version. As a result, a dump file cannot be generated.

  Solution

  Important

  This solution applies only to systems where the kdump service is properly enabled. If the kdump service is not enabled and you follow the steps in this topic to fix the issue, you must manually configure the crashkernel parameter in the proc/cmdline file.

  1. Run the following command to download the kexec-tools package:

     wget http://mirrors.aliyun.com/centos-vault/7.9.2009/os/Source/SPackages/kexec-tools-2.0.15-51.el7.src.rpm

  2. Run the following command to install the RPM package:

     rpm -ivh kexec-tools-2.0.15-51.el7.src.rpm

  3. Run the following commands to download the patch file:

     cd /root/rpmbuild/SOURCES
     wget https://ecs-image-tools.oss-cn-hangzhou.aliyuncs.com/patch/rhelonly-kexec-tools-2.0.20-makedumpfile-arm64-Add-support-for-ARMv8.2-LVA-52-bi.patch

  4. Modify the kexec-tools.spec file.

     1. Open the kexec-tools.spec file.

        cd /root/rpmbuild/SPECS/
        vi kexec-tools.spec

     2. Press the i key to enter Insert mode and add the following two lines to the file:

        Patch999: rhelonly-kexec-tools-2.0.20-makedumpfile-arm64-Add-support-for-ARMv8.2-LVA-52-bi.patch
        %patch999 -p1

        The following figures show the positions where the content must be added.

        

        

     3. Press the Esc key to exit Insert mode, and then enter :wq to save and exit.

  5. Run the following command to check for installation dependencies:

     yum-builddep kexec-tools.spec 

  6. Run the following commands to build the RPM package:

     yum -y install rpm-build
     rpmbuild -ba kexec-tools.spec

  7. Run the following commands to install the modified RPM package:

     cd /root/rpmbuild/RPMS/aarch64
     rpm -ivh kexec-tools-2.0.15-51.el7.aarch64.rpm

  If downtime occurs again, you can run the ls -lh /var/crash command to check for dump files. If a vmcore file is generated, the issue is resolved.

  

• Convert CentOS 7 to RHEL 7

  CentOS 7 will reach its end of life (EOL) on June 30, 2024, and Alibaba Cloud will stop supporting it on the same date. To avoid disruptions from the OS reaching EOL, you can convert CentOS 7 to RHEL 7. The following steps describe how to convert CentOS 7 to RHEL 7 on Alibaba Cloud. You can also refer to the official Red Hat documentation for the conversion.

  Important

  Before the conversion, we recommend that you stop critical applications, database services, and data storage services and create a snapshot to back up important data. This helps prevent data loss from accidental operations.

  1. (If applicable) If Server Guard is installed on your Alibaba Cloud server, you must first uninstall Server Guard.

     For more information, see Uninstall the agent.

     Note

     Server Guard is the default security enhancement tool for CentOS, whereas RHEL 7 uses security tools provided by Red Hat. Server Guard may have incompatibilities and conflicts with the tools in RHEL 7. Therefore, you must uninstall Server Guard during the conversion to ensure system stability and compatibility.

  2. Run the following commands to upgrade the system packages to the latest version:

     sudo wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
     sudo wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo

  3. Run the following commands to update the system packages and restart the system:

     sudo yum -y update
     sudo reboot

  4. Run the following commands to download and install the convert2rhel tool from the official Red Hat website:

     sudo curl -o /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release https://www.redhat.com/security/data/fd431d51.txt
     sudo curl --create-dirs -o /etc/rhsm/ca/redhat-uep.pem https://ftp.redhat.com/redhat/convert2rhel/redhat-uep.pem
     sudo curl -o /etc/yum.repos.d/convert2rhel.repo https://ftp.redhat.com/redhat/convert2rhel/7/convert2rhel.repo
     sudo yum -y install convert2rhel

  5. Purchase an RHEL subscription on Alibaba Cloud and obtain the URL of the RHEL 7 repo RPM package.

     For assistance, submit a ticket.

  6. Run the following commands to install the RHEL 7 repo source package:

     sudo rpm -ivh --replacefiles <repo_rpm_package_url> 
     sudo sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/rh-cloud.repo

     Replace <repo_rpm_package_url> with the actual URL of the RHEL 7 repo RPM package, which you can obtain when you purchase the RHEL subscription.

  7. Run the following command to convert CentOS 7 to RHEL 7:

     sudo convert2rhel -y  --no-rhsm --enablerepo rhui-rhel-7-server-rhui-rpms --enablerepo rhui-rhel-7-server-rhui-extras-rpms --enablerepo rhui-rhel-7-server-rhui-optional-rpms

     The conversion process takes some time. When the process is complete, an output similar to the following one appears, indicating a successful conversion.

     

  8. Run the following command to restart the system:

     After the conversion, the system prompts you to restart. Restarting boots the new RHEL kernel, and you can then verify that the OS was successfully converted.

     sudo reboot

     Note

     After you convert CentOS 7 to RHEL 7, if you need to upgrade RHEL 7 to RHEL 8, see Upgrade Red Hat Enterprise Linux.

• Slow startup of Red Hat 8.1/8.2 on bare metal instances

  On an ECS Bare Metal Instance, a Red Hat 8.1 or 8.2 image takes one to two minutes longer to start than a Red Hat 7 image. To resolve this issue, in the /boot/grub2/grubenv file of your Red Hat 8.1/8.2 system, change the kernel startup parameter from console=ttyS0 console=ttyS0,115200n8 to console=tty0 console=ttyS0,115200n8, and then restart the server for the change to take effect.

High system load from Server Guard on Ubuntu

On an ECS instance that runs a specific version of Ubuntu, such as Ubuntu 18.04, the average system load becomes high after the Server Guard (AliYunDun) process is started.

• Patch and compile the FreeBSD kernel

  The Alibaba Cloud FreeBSD public images have been patched to meet the startup requirements of instances that belong to series-V or later instance families. You can call the DescribeInstanceTypeFamilies operation and check the value of the Generation parameter to query the instance families.

  In the following cases, the system may fail to start. You can apply a patch to the FreeBSD kernel source code and compile the kernel to prevent or resolve startup failures.

  • If you use a FreeBSD image not provided by Alibaba Cloud, or a custom image based on it, to create a series-V or later ECS instance, the instance may fail to start.

  • When you use a FreeBSD public image to create an ECS instance and use freebsd-update or a similar tool to update the kernel patch, an ECS instance that belongs to a series-V or later instance family may fail to start.

  You do not need to apply a patch for FreeBSD 13 or later. This section uses FreeBSD 12.3 as an example to describe how to use the FreeBSD kernel source code to apply a patch and compile the kernel.

  1. Download and decompress the FreeBSD kernel source code.

     wget https://mirrors.aliyun.com/freebsd/releases/amd64/12.3-RELEASE/src.txz -O /src.txz
     cd /
     tar -zxvf /src.txz

  2. Download the patch package.

     In this example, the patch package 0001-virtio.patch is applied to the virtio driver.

     cd /usr/src/sys/dev/virtio/
     wget https://ecs-image-tools.oss-cn-hangzhou.aliyuncs.com/0001-virtio.patch
     patch -p4 < 0001-virtio.patch

  3. Copy the kernel files, and then compile and install the kernel.

     make -j<N> specifies the number of parallel compilation jobs, which should be determined based on your compilation environment's configuration. For example, for a 1 vCPU environment, we recommend that you set -j2. This means the ratio of vCPU cores to the variable N is 1:2.

     cd /usr/src/
     cp ./sys/amd64/conf/GENERIC .
     make -j2 buildworld KERNCONF=GENERIC
     make -j2 buildkernel KERNCONF=GENERIC
     make -j2 installkernel KERNCONF=GENERIC

  4. After the compilation is complete, delete the source code.

     rm -rf /usr/src/*
     rm -rf /usr/src/.*

• System disk not found for FreeBSD in KVM

  Symptoms

  When you use VNC to log on to a FreeBSD system in a KVM virtualized environment, the system disk cannot be found and you cannot enter the system, as shown in the following figure.

  Solution

  1. In the VNC window, enter ? to view the ufsid of the related root file system (rootfs).

  2. Enter ufs:/dev/ufsid/5565b5a09045**** and press Enter to access the operating system.

  3. Enter the username and password to log on to the system.

  4. Run the following command to view the /etc/fstab configuration:

     cat /etc/fstab

     As shown in the following figure, the /etc/fstab configuration uses the UUID-based mounting method. However, FreeBSD systems do not support this mounting method. You must change it to the ufsid method.

  5. Change the mounting method of the FreeBSD system to ufsid.

     1. Run the following command to open the /etc/fstab file:

        vi /etc/fstab

     2. Press the i key to enter Insert mode.

     3. Change UUID=5565b5a09045**** to /dev/ufsid/5565b5a09045****.

     4. After you make the change, press the Esc key, enter :wq, and then press the Enter key to save and exit.

  6. Run the following command to restart the system for the configuration to take effect:

     reboot 

• Cannot connect to Fedora 33 with ssh-rsa keys

  When you use an ECS instance that runs the Fedora 33 64-bit operating system, if your logon credential is an SSH key pair with the ssh-rsa signature algorithm, you may fail to remotely connect to the instance over SSH. You can use one of the following methods to resolve this issue:

  • Replace the SSH key pair that uses the ssh-rsa signature algorithm with an SSH key pair that uses another signature algorithm, such as the Elliptic Curve Digital Signature Algorithm (ECDSA).

  • Run the update-crypto-policies --set LEGACY command on the system to change the cryptographic POLICY to LEGACY. This allows you to continue using the SSH key pair with the ssh-rsa signature algorithm.

• CPU count halved on Fedora CoreOS instances

  After you create an instance of a specific instance family, such as g5, from a Fedora CoreOS image, you can run the lscpu command to view CPU information. The total number of online CPUs in the On-line CPU(s) list is only half of the actual CPU specification of the instance. For example, if you select 2 vCPUs when you create the instance, the On-line CPU(s) list shows only one CPU, as shown in the following figure.

  Note

  The value of the On-line CPU(s) list parameter represents the CPU number. In the example figure, only CPU 0 is available.

  This is because the kernel of the Fedora CoreOS image is configured with the mitigations=auto,nosmt boot parameter by default. This parameter automatically disables Simultaneous Multi-Threading (SMT) for vulnerable systems, which halves the number of available CPUs. You can run the cat /proc/cmdline command to view the mitigations=auto,nosmt parameter.

  For more information about SMT, see Automatically disable SMT when needed to address vulnerabilities and Policy for disabling SMT.

• Crash dump failure on bare metal instances

  For information about the cause and solution, see What to do if some ECS instances fail to generate crash dump files?

• Softlockup error during kernel writeback

  In some earlier versions of the Linux operating system, a softlockup error may occur when the kernel writes back file caches. For the solution, see Solution to the softlockup error that occurs during kernel writeback in a Linux operating system.

• Softlockup error when deleting a cgroup

  For the solution, see Solution to the softlockup error that occurs when a cgroup is deleted within an ECS instance.

• ECS instance downtime

  • What to do if an ECS instance experiences downtime and reports the "Out of memory and no killable processes" error

  • What to do if an ECS instance experiences downtime and generates an alert log with the "Objects remaining in kmalloc" message

  • What to do if an ECS instance experiences downtime and generates a log with the "RIP:get_target_pstate_use_performance" message

  • What to do if an ECS instance experiences downtime and generates a log with the "VFS: Unable to mount root fs on unknown-block" message

• FTP service in public images

  No. You must install and configure the FTP service yourself. For more information, see Build an FTP site on a Windows instance and Build an FTP site on a Linux instance.

• Virtual memory and swap partitions disabled by default

  A swap partition or virtual memory file is a mechanism used by the system memory manager. When the physical memory is insufficient, the memory manager temporarily saves memory data that has been inactive for a long time to the swap partition or virtual memory file to increase the available memory.

  However, this mechanism can have the opposite effect if memory usage is already high and I/O performance is poor. Alibaba Cloud ECS cloud disks use a distributed file system for storage and create multiple, strongly consistent replicas of each piece of data. While this mechanism ensures data security, the resulting threefold increase in I/O operations can degrade storage and I/O performance compared to local disks.

  To avoid degrading the I/O performance of ECS cloud disks when system resources are low, virtual memory is disabled by default on Windows instances, and swap partitions are not configured by default on Linux instances.

• Enable kdump in public images

  By default, the kdump service is disabled in public images. If you want an instance to generate a core file during downtime for root cause analysis, perform the following steps to enable the kdump service. This section uses the CentOS 7.2 public image as an example. The actual steps may vary based on your operating system.

  1. Set the directory for generating the core file.

     1. Run vim /etc/kdump.conf to open the kdump configuration file.

     2. Set path to the directory where the core file is generated. In this example, the core file is generated in the /var/crash directory. Therefore, the path is set as follows:

        path /var/crash

     3. Save and close the /etc/kdump.conf file.

  2. Enable the kdump service.

     Select a method to enable the service based on the commands supported by your operating system.

     • Method 1: Run the following commands in sequence to enable the kdump service.

       systemctl enable kdump.service

       systemctl start kdump.service

     • Method 2: Run the following commands in sequence to enable the kdump service.

       chkconfig kdump on

       service kdump start

     • Method 3: If Cloud Assistant is installed on your server, see How to resolve downtime issues after instance migration to enable the kdump service.

• NTP service fails to sync after configuring IPv6

  Symptoms

  When you run the ntpq -p command on the server to synchronize time, a timeout is returned, as shown in the following figure.

  Solution

  Note

  This method is applicable to operating systems such as CentOS 7 and earlier, Ubuntu 20.04 and earlier, Anolis OS (ANCK/RHCK), Alibaba Cloud Linux, and Debian.

  1. Remotely connect to the Linux instance.

     For more information, see Connect to a Linux instance by using Workbench.

  2. Run the following command to modify the /etc/ntp.conf configuration file:

     vi /etc/ntp.conf

  3. Press the i key to enter Insert mode.

  4. Add the restrict -6 ::1 content to the file, as shown in the following figure.

  5. After you make the change, press the Esc key, enter :wq, and then press the Enter key to save and exit.

  6. Run the following command to restart the NTP service:

     systemctl restart ntp 

• Hot-swapping failure on instances from custom images

  Symptoms

  Hot-swapping a cloud disk means attaching or detaching a cloud disk while an instance is in the Running state. Hot-swapping a network card means attaching or detaching an elastic network interface while an instance is in the Running state.

  Alibaba Cloud supports hot-swapping for cloud disks and elastic network interfaces, but the operation's success depends on operating system kernel support. If the kernel does not support hot-swapping, the following issues may occur:

  • After you attach a cloud disk or bind an elastic network interface, the corresponding device is not visible within the operating system.

  • You fail to detach a cloud disk or unbind an elastic network interface.

  Solution

  The kernel features required for hot-swapping differ between standard ECS instances and ECS Bare Metal Instances. We recommend that the kernel supports both Peripheral Component Interconnect (PCI) and Advanced Configuration and Power Management Interface (ACPI) hot-swap features. These are typically enabled by default on most systems, except for older versions such as CentOS 5. You can perform the following steps to check whether the PCI and ACPI hot-swap features are enabled in the kernel.

  1. Remotely connect to the Linux instance.

     For more information, see Connect to a Linux instance by using Workbench.

  2. Run the following command to check the current kernel version of the instance:

     uname -r

     The following response indicates that the current kernel version is 3.10.0-1127.19.1.el7.x86_64.

     

  3. Run the following command to view the files in the /boot directory:

     ll /boot

     The following response shows that config-3.10.0-1127.19.1.el7.x86_64 is the kernel configuration file of the system.

     

  4. Run the following command to view the kernel configuration file of the system:

     cat /boot/config-3.10.0-1127.19.1.el7.x86_64

     • If the following configuration items are all set to y, the features are compiled into the kernel, and the operating system supports the corresponding hot-swap operations.

       CONFIG_HOTPLUG_PCI_PCIE=y
       CONFIG_HOTPLUG_PCI=y
       CONFIG_HOTPLUG_PCI_ACPI=y

     • If a configuration item is marked as is not set, the kernel was not compiled with this feature. You must recompile the kernel to add support.

     • If a configuration item is set to m, it is compiled as a module. For example, if CONFIG_HOTPLUG_PCI_ACPI is compiled as a module, you need to load the corresponding module.

       CONFIG_HOTPLUG_PCI_PCIE=y
       CONFIG_HOTPLUG_PCI=y
       CONFIG_HOTPLUG_PCI_ACPI=m

       For a CentOS 5.x operating system with a 2.6 kernel, the module corresponding to CONFIG_HOTPLUG_PCI_ACPI is acpiphp.ko. To load it, run the modprobe acpiphp command. If the module fails to load, you can upgrade to a higher kernel version or perform a cold-swap by stopping the instance.

       Important

       We do not recommend that you arbitrarily upgrade the kernel or operating system of your server. If you need to upgrade the kernel, see Methods to prevent system startup failures after a kernel upgrade on a Linux instance.

• Instance shutdown after a kernel panic

  Symptoms

  When an unexpected kernel error (kernel panic) occurs in the operating system, a second kernel (capture kernel) is loaded to perform a memory dump and generate a Kdump log. However, due to compatibility issues with certain ECS Bare Metal Instance families, the second kernel fails to recognize the disk during startup. This failure prevents Kdump log collection, causes the second kernel to fail, and stops the instance. You must then restart the instance from the console.

  For more information about ECS Bare Metal Instance families, see Instance families.

  Cause

  An ECS Bare Metal Instance may fail to generate a dump file by using the Kdump service that comes with the operating system.

  • For ebm*6-generation series ECS Bare Metal Instances, this issue occurs when the following images are used:

    • CentOS 8.3 and earlier CentOS versions

    • Ubuntu 16.04 and 18.04

    • Debian 10

    • Alibaba Cloud Linux 2 with a kernel version earlier than 4.19.91-24.al7 (The issue is fixed in version 4.19.91-24.al7 and later.)

  • For ebm*7-generation series ECS Bare Metal Instances, this issue occurs when a Debian 10 image is used.

  Solution

  • For CentOS and other images

    We recommend that you switch to a later version of the operating system. For more information, see Change the operating system (system disk).

  • For Alibaba Cloud Linux 2 images

    We recommend that you upgrade the kernel to version 4.19.91-24.al7 or later by following these steps.

    1. Remotely log on to the ECS instance.

       For more information, see Connect to a Linux instance by using Workbench.

    2. Run the following command to query the kernel version:

       uname -r

    3. Run the following command to upgrade the kernel version:

       sudo yum update kernel

    4. Run the following command to restart the ECS instance for the new kernel version to take effect:

       sudo reboot