This topic describes the known issues of Alibaba Cloud images for different operating systems, the scope of these issues, and the solutions to these issues.

A call trace may occur on some instance types when instances of operating systems that run more recent kernel versions are started.

  • Problem description: If an instance of a specific instance type such as ecs.i2.4xlarge runs an operating system that has a more recent kernel version, such as RHEL 8.3 or CentOS 8.3 with the 4.18.0-240.1.1.el8_3.x86_64 kernel version, a call trace may occur when the instance is started. Call trace example:
    Dec 28 17:43:45 localhost SELinux:  Initializing.
    Dec 28 17:43:45 localhost kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
    Dec 28 17:43:45 localhost kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
    Dec 28 17:43:45 localhost kernel: Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes)
    Dec 28 17:43:45 localhost kernel: Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes)
    Dec 28 17:43:45 localhost kernel: unchecked MSR access error: WRMSR to 0x3a (tried to write 0x000000000000****) at rIP: 0xffffffff8f26**** (native_write_msr+0x4/0x20)
    Dec 28 17:43:45 localhost kernel: Call Trace:
    Dec 28 17:43:45 localhost kernel:  init_ia32_feat_ctl+0x73/0x28b
    Dec 28 17:43:45 localhost kernel:  init_intel+0xdf/0x400
    Dec 28 17:43:45 localhost kernel:  identify_cpu+0x1f1/0x510
    Dec 28 17:43:45 localhost kernel:  identify_boot_cpu+0xc/0x77
    Dec 28 17:43:45 localhost kernel:  check_bugs+0x28/0xa9a
    Dec 28 17:43:45 localhost kernel:  ? __slab_alloc+0x29/0x30
    Dec 28 17:43:45 localhost kernel:  ? kmem_cache_alloc+0x1aa/0x1b0
    Dec 28 17:43:45 localhost kernel:  start_kernel+0x4fa/0x53e
    Dec 28 17:43:45 localhost kernel:  secondary_startup_64+0xb7/0xc0
    Dec 28 17:43:45 localhost kernel: Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
    Dec 28 17:43:45 localhost kernel: Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
    Dec 28 17:43:45 localhost kernel: FEATURE SPEC_CTRL Present
    Dec 28 17:43:45 localhost kernel: FEATURE IBPB_SUPPORT Present
  • Cause: The kernel version is updated by using the latest community update package to include the patches for writes to MSRs. However, some instance types such as ecs.i2.4xlarge do not support writes to MSRs due to the limits of virtualization technology.
  • Solution: The call trace does not affect the operation and stability of the system. You can ignore this issue.

Compatibility issues between the hfg6 general purpose instance family with high clock speeds and some Linux kernel versions may cause kernel panic

  • Problem description: When kernels of some open source Linux distributions such as CentOS 8, SUSE Linux Enterprise Server (SLES) 15 SP2, and openSUSE 15.2 are updated to the latest versions in hfg6 instances, a kernel panic error may occur. The following figure shows an example of the call trace debugging method:kernel panic
  • Cause: Some Linux kernel versions are incompatible with the hfg6 general purpose instance family with high clock speeds.
  • Solution:
    • The compatibility issue is fixed for the latest kernel versions of SLES 15 SP2 and openSUSE 15.2. The following code shows the information of the change commit. If your latest kernel version contains this information, the kernel version is compatible with the hfg6 instance family.
      commit 1e33d5975b49472e286bd7002ad0f689af33fab8
      Author: Giovanni Gherdovich <ggherdovich@suse.cz>
      Date:   Thu Sep 24 16:51:09 2020 +0200
      
          x86, sched: Bail out of frequency invariance if
          turbo_freq/base_freq gives 0 (bsc#1176925).
      
          suse-commit: a66109f44265ff3f3278fb34646152bc2b3224a5
          
          
      commit dafb858aa4c0e6b0ce6a7ebec5e206f4b3cfc11c
      Author: Giovanni Gherdovich <ggherdovich@suse.cz>
      Date:   Thu Sep 24 16:16:50 2020 +0200
      
          x86, sched: Bail out of frequency invariance if turbo frequency
          is unknown (bsc#1176925).
      
          suse-commit: 53cd83ab2b10e7a524cb5a287cd61f38ce06aab7
      
      commit 22d60a7b159c7851c33c45ada126be8139d68b87
      Author: Giovanni Gherdovich <ggherdovich@suse.cz>
      Date:   Thu Sep 24 16:10:30 2020 +0200
      
          x86, sched: check for counters overflow in frequency invariant
          accounting (bsc#1176925).
    • If you run the yum update command to update the kernel of CentOS 8 to kernel-4.18.0-240 or later in hfg6 instances, a kernel panic error may occur. If this issue occurs, roll back the kernel to the previous version.

SLES 12 SP5: Kernel updates may cause the system to freeze during startup

  • When an earlier kernel version is updated to SLES 12 SP5, or when you update the kernel of SLES 12 SP5, instances that have some CPU types may freeze during startup. These known CPU types are Intel® Xeon® CPU E5-2682 v4 @ 2.50GHz and Intel® Xeon® CPU E7-8880 v4 @ 2.20GHz. The following code describes the call trace debugging result:
    [    0.901281] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [    0.901281] CR2: ffffc90000d68000 CR3: 000000000200a001 CR4: 00000000003606e0
    [    0.901281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [    0.901281] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    [    0.901281] Call Trace:
    [    0.901281]  cpuidle_enter_state+0x6f/0x2e0
    [    0.901281]  do_idle+0x183/0x1e0
    [    0.901281]  cpu_startup_entry+0x5d/0x60
    [    0.901281]  start_secondary+0x1b0/0x200
    [    0.901281]  secondary_startup_64+0xa5/0xb0
    [    0.901281] Code: 6c 01 00 0f ae 38 0f ae f0 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00 90 31 d2 65 48 8b 34 25 40 6c 01 00 48 89 d1 48 89 f0 <0f> 01 c8 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00 ** **
  • Cause: The new kernel version is incompatible with the CPU microcode.
  • Solution: In the /boot/grub2/grub.cfg file, add the idle kernel parameter to the row that starts with linux and set this parameter to nomwait. The following example shows how to modify the file:
    menuentry 'SLES 12-SP5'  --class sles --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-fd7bda55-42d3-4fe9-a2b0-45efdced****' {
            load_video
            set gfxpayload=keep
            insmod gzio
            insmod part_msdos
            insmod ext2
            set root='hd0,msdos1'
            if [ x$feature_platform_search_hint = xy ]; then
              search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  fd7bda55-42d3-4fe9-a2b0-45efdced****
            else
              search --no-floppy --fs-uuid --set=root fd7bda55-42d3-4fe9-a2b0-45efdced****
            fi
            echo    'Loading Linux 4.12.14-122.26-default ...'
            linux   /boot/vmlinuz-4.12.14-122.26-default root=UUID=fd7bda55-42d3-4fe9-a2b0-45efdced****  net.ifnames=0 console=tty0 console=ttyS0,115200n8 mitigations=auto splash=silent quiet showopts idle=nomwait
            echo    'Loading initial ramdisk ...'
            initrd  /boot/initrd-4.12.14-122.26-default
    }

openSUSE 15: Kernel updates may cause the system to freeze during startup

  • Problem description: When openSUSE kernel versions are updated to 4.12.14-lp151.28.52-default, instances that have some CPU types may freeze during startup. The known CPU type is Intel® Xeon® CPU E5-2682 v4 @ 2.50GHz. The following code describes the call trace debugging result:
    [    0.901281] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [    0.901281] CR2: ffffc90000d68000 CR3: 000000000200a001 CR4: 00000000003606e0
    [    0.901281] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [    0.901281] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    [    0.901281] Call Trace:
    [    0.901281]  cpuidle_enter_state+0x6f/0x2e0
    [    0.901281]  do_idle+0x183/0x1e0
    [    0.901281]  cpu_startup_entry+0x5d/0x60
    [    0.901281]  start_secondary+0x1b0/0x200
    [    0.901281]  secondary_startup_64+0xa5/0xb0
    [    0.901281] Code: 6c 01 00 0f ae 38 0f ae f0 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00 90 31 d2 65 48 8b 34 25 40 6c 01 00 48 89 d1 48 89 f0 <0f> 01 c8 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00 00 ** **
  • Cause: The new kernel version is incompatible with the CPU microcode. For more information, visit Issues of freezing during startup.
  • Affected image: opensuse_15_1_x64_20G_alibase_20200520.vhd
  • Solution: In the /boot/grub2/grub.cfg file, add the idle=nomwait option to the row that starts with linux. The following example shows how to modify the file:
    menuentry 'openSUSE Leap 15.1'  --class opensuse --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-20f5f35a-fbab-4c9c-8532-bb6c66ce****' {
            load_video
            set gfxpayload=keep
            insmod gzio
            insmod part_msdos
            insmod ext2
            set root='hd0,msdos1'
            if [ x$feature_platform_search_hint = xy ]; then
              search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  20f5f35a-fbab-4c9c-8532-bb6c66ce****
            else
              search --no-floppy --fs-uuid --set=root 20f5f35a-fbab-4c9c-8532-bb6c66ce****
            fi
            echo    'Loading Linux 4.12.14-lp151.28.52-default ...'
            linux   /boot/vmlinuz-4.12.14-lp151.28.52-default root=UUID=20f5f35a-fbab-4c9c-8532-bb6c66ce****  net.ifnames=0 console=tty0 console=ttyS0,115200n8 splash=silent mitigations=auto quiet idle=nomwait
            echo    'Loading initial ramdisk ...'
            initrd  /boot/initrd-4.12.14-lp151.28.52-default
    }

CentOS 8.0: The image version numbers of created instances change after the public image is updated

  • Problem description: After you connect to an instance created from the centos_8_0_x64_20G_alibase_20200218.vhd public image, you find that the system version of the instance is CentOS 8.1.
    root@ecshost:~$ lsb_release -a
    LSB Version:    :core-4.1-amd64:core-4.1-noarch
    Distributor ID:    CentOS
    Description:    CentOS Linux release 8.1.1911 (Core)
    Release:    8.1.1911
    Codename:    Core
  • Cause: The centos_8_0_x64_20G_alibase_20200218.vhd image is in the public image list and was updated by using the latest community update package. CentOS in the image is upgraded to version 8.1. Therefore, the actual system version is CentOS 8.1.centos_8_0_x64_20G_alibase_20200218.vhd
  • Affected image: centos_8_0_x64_20G_alibase_20200218.vhd.
  • Solution: You can call operations such as RunInstances and set ImageId to centos_8_0_x64_20G_alibase_20191225.vhd to create an instance whose system version is CentOS 8.0.

Debian 9.6: Classic network-type instances have network configuration issues

  • Problem description: Classic network-type instances created from Debian 9 public images cannot be pinged.
  • Cause: The systemd-networkd service is disabled by default in Debian 9. Classic network-type instances created from Debian 9 public images cannot be automatically assigned IP addresses based on the Dynamic Host Configuration Protocol (DHCP).
  • Affected image: debian_9_06_64_20G_alibase_20181212.vhd.
  • Solution: Run the following commands in sequence:
    systemctl enable systemd-networkd 
    systemctl start systemd-networkd

CentOS 6.8: An instance installed with the NFS client does not respond

  • Problem description: A CentOS 6.8 instance installed with the NFS client does not respond and must be restarted.
  • Cause: When you use the NFS service on instances whose operating system kernel versions range from 2.6.32-696 to 2.6.32-696.10, the NFS client attempts to end a TCP connection if a glitch occurs due to communication latency. Specifically, if the NFS server is slow to respond to NFS requests, the connection initiated by the NFS client may be in the FIN_WAIT2 state for an extended period of time. In normal cases, the connection times out and closes one minute after the connection enters the FIN_WAIT2 state, and the NFS client can initiate a new connection. However, kernel versions 2.6.32-696 to 2.6.32-696.10 have issues with establishing TCP connections. As a result, the connection remains in the FIN_WAIT2 state, the NFS client is unable to recover the TCP connection, and a new TCP connection cannot be initiated. This causes the requests to freeze, and the only way to fix the issue is to restart the instance.
  • Affected images: centos_6_08_32_40G_alibase_20170710.vhd and centos_6_08_64_20G_alibase_20170824.vhd.
  • Solution: Run the yum update command to update the kernel to 2.6.32-696.11 or later.
    Notice Before you perform operations on the instance, you must create a snapshot to back up your data. For more information, see Create a snapshot for a disk.

CentOS 7: The hostname changes from uppercase letters to lowercase letters after an instance restarts

  • Problem description: When some instances that run CentOS 7 are restarted for the first time, the hostnames of these instances change from uppercase letters to lowercase letters. The following table describes some examples.
    Hostname Hostname after the instance is restarted for the first time The hostname remains in lowercase after the instance restart
    iZm5e1qe*****sxx1ps5zX izm5e1qe*****sxx1ps5zx Yes
    ZZHost zzhost Yes
    NetworkNode networknode Yes
  • Affected images: the following CentOS public images and custom images derived from these public images:
    • centos_7_2_64_40G_base_20170222.vhd
    • centos_7_3_64_40G_base_20170322.vhd
    • centos_7_03_64_40G_alibase_20170503.vhd
    • centos_7_03_64_40G_alibase_20170523.vhd
    • centos_7_03_64_40G_alibase_20170625.vhd
    • centos_7_03_64_40G_alibase_20170710.vhd
    • centos_7_02_64_20G_alibase_20170818.vhd
    • centos_7_03_64_20G_alibase_20170818.vhd
    • centos_7_04_64_20G_alibase_201701015.vhd
  • Affected hostnames: If the hostnames of your applications are case-sensitive, services may be affected when you restart such instances. The following table describes whether the hostname changes after an instance is restarted.
    Current state of hostname The hostname changes after an instance restart Time when the hostname changes Continue reading this section
    The hostname contains uppercase letters when you create the instance by using the ECS console or by calling ECS API operations. Yes When the instance is restarted for the first time Yes
    The hostname contains only lowercase letters when you create the instance by using the ECS console or by calling ECS API operations. No N/A No
    The hostname contains uppercase letters, and you modify the hostname after you log on to the instance. No N/A Yes
  • Solution: To retain uppercase letters in the hostname of an instance after you restart the instance, perform the following operations:
    1. Connect to your instance. For more information, see Connection methods.
    2. View the existing hostname.
      [root@izbp193*****3i161uynzzx ~]# hostname
      izbp193*****3i161uynzzx
    3. Run the following command to staticize the hostname:
      hostnamectl set-hostname --static iZbp193*****3i161uynzzX
    4. Run the following commands to view the updated hostname:
      [root@izbp193*****3i161uynzzx ~]# hostname
      iZbp193*****3i161uynzzX
  • What to do next: If you are using an affected custom image, we recommend that you update cloud-init to the latest version and create another custom image. This prevents the previous issue from occurring on the instances that are created from the affected custom image. For more information, see Install cloud-init and Create a custom image from an instance.

Linux: Pip requests time out

  • Problem description: Pip requests occasionally time out or fail.
  • Affected images: CentOS, Debian, Ubuntu, SUSE, openSUSE, and Alibaba Cloud Linux.
  • Cause: Alibaba Cloud provides three pip source addresses. The default address is mirrors.aliyun.com. To access this address, instances must be able to access the Internet. If your instance is not assigned a public IP address, pip requests time out.
    • The public source address (default): mirrors.aliyun.com
    • The internal source address in VPCs: mirrors.cloud.aliyuncs.com
    • The internal source address in the classic network: mirrors.aliyuncs.com
  • Solution: You can solve the problem by using one of the following methods:
    • Method 1

      Assign a public IP address to your instance by associating an elastic IP address (EIP) with your instance. For more information, see Overview.

      You can also reassign a public IP address to a subscription instance by upgrading or downgrading the instance configurations. For more information, see Upgrade the instance types of subscription instances.

    • Method 2
      If a pip request fails, you can run the fix_pypi.sh script in your ECS instance and retry the pip operation. Perform the following operations:
      1. Connect to your instance. For more information, see Connect to a Linux instance by using password authentication.
      2. Run the following command to obtain the script file:
        wget http://image-offline.oss-cn-hangzhou.aliyuncs.com/fix/fix_pypi.sh
      3. Run one of the following scripts based on the network type of the instance:
        • If your instance is a VPC-type instance, run the bash fix_pypi.sh "mirrors.cloud.aliyuncs.com" script.
        • If your instance is a classic network-type instance, run the bash fix_pypi.sh "mirrors.aliyuncs.com" script.
      4. Retry the pip operation.
      The following section describes the fix_pypi.sh script:
      #! /bin/bash
      
      function config_pip() {
          pypi_source=$1
      
          if [[ ! -f ~/.pydistutils.cfg ]]; then
      cat > ~/.pydistutils.cfg << EOF
      [easy_install]
      index-url=http://$pypi_source/pypi/simple/
      EOF
          else
              sed -i "s#index-url.*#index-url=http://$pypi_source/pypi/simple/#" ~/.pydistutils.cfg
          fi
      
          if [[ ! -f ~/.pip/pip.conf ]]; then
          mkdir -p ~/.pip
      cat > ~/.pip/pip.conf << EOF
      [global]
      index-url=http://$pypi_source/pypi/simple/
      [install]
      trusted-host=$pypi_source
      EOF
          else
              sed -i "s#index-url.*#index-url=http://$pypi_source/pypi/simple/#" ~/.pip/pip.conf
              sed -i "s#trusted-host.*#trusted-host=$pypi_source#" ~/.pip/pip.conf
          fi
      }
      
      config_pip $1