All Products
Search
Document Center

:When you remotely connect to an ECS instance running Windows, a message indicating a protocol error is displayed

Last Updated:Dec 17, 2020

Disclaimer: This article may contain information about third-party products. Such information is for reference only. Alibaba Cloud does not make any guarantee, express or implied, with respect to the performance and reliability of third-party products, as well as potential impacts of operations on the products.

Problem description

When you connect to an ECS instance that runs the Windows system, a protocol error is returned. The specific error is as follows:

The session will be disconnected due to a protocol error. Please reconnect to the remote computer.

Possible cause

Some of the possible reasons for this problem are as follows:

  • Reason one
    The remote session configuration for the target instance is incorrect. When the remote session is configured incorrectly, a protocol error message will appear during the remote connection.
  • Reason Two
    The remote connection port of the destination instance is occupied by another application. In Windows, the default Remote Desktop Connection port number is 3389. If this port is occupied by another application, the remote connection service cannot run properly.

Solution

Alibaba Cloud reminds you that:

  • Before you perform operations that may cause risks, such as modifying instance configurations or data, we recommend that you check the disaster recovery and fault tolerance capabilities of the instances to ensure data security.
  • If you modify the configurations and data of instances including but not limited to ECS and RDS instances, we recommend that you create snapshots or enable RDS log backup.
  • If you have authorized or submitted security information such as the logon account and password in the Alibaba Cloud Management console, we recommend that you modify such information in a timely manner.

Different causes of the problem correspond to different solutions. You can select the corresponding causes and solutions based on the actual conditions on site.

Note: the procedure in this topic takes Windows Server 2016 as an example.

Cause 1: The remote session configuration of the target instance is incorrect

To troubleshoot this failure, perform the following steps:

  1. Log on to the Windows instance through VNC.
  2. Click start, and enter gedit.msc. Click the Enter button to open the local group policy editor.
  3. In the local group policy editor window, click computer configuration>Manage templates>Windows components>Remote Desktop Services>Remote desktop session host>Security computer configuration.
  4. In the security catalog, double-click require for remote (RDP) connections to use the specified security layer.
  5. Select enabled. Select security layer from the RDP drop-down list and click OK.
  6. Click start, cmd, click Enter, open the command line.
  7. Run the following command to update the policy so that the preceding settings can take effect.
    gpupdate
    The system displays the following figure, and the computer policy update is complete.
  8. Retry the remote connection and confirm that the problem has been fixed.

Cause 2: The remote connection port of the destination instance is occupied by another application.

If the remote connection port of the target instance is occupied by another application, you can choose one of the following solutions:

Solution 1: modify the remote connection port number

You can modify the remote connection port number and retry the remote connection by following these steps:

  1. Log on to the Windows instance through VNC.
  2. Click start, Enter regedit, and click Enter to open the registry editor.
  3. Go to the following directories:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp
  4. Double-click the registry key called RDP-Tcp in the PortNumber directory.


  5. On the edit page, select cardinality in the decimal, and modify the value in the value data. This value is the port number used by the Remote Desktop.
    Note: when modifying the default port of a remote desktop, select an appropriate TCP port based on the on-site conditions.
    Note: port 3389 in the figure is the default port number for remote connection, only as an example.
  6. Restart Windows or the ECS instance.

Solution 2: disable the applications that occupy remote connection ports

Close the application processes that are using the remote connection port, release the port number, and retry the remote connection by following these steps:

  1. Log on to the Windows instance through VNC.
  2. Click start, cmd, click Enter, open the command line.
  3. Run the following command to obtain the PID that uses the port used for remote connection:
    netstat -ano |findstr 3389
    Note: The remote connection uses the default port number 3389 as an example. The actual port number on the site must be the same.
  4. Run the following command to view the application that runs in the specified process based on the PID: Normally, the remote connection port is occupied by the svchost.exe process.
    tasklist |findstr [$PID]
    Note:[$PID] indicates the PID obtained in the previous step.
  5. If the remote connection port is used by a process other than svchost.exe, it indicates that the remote connection port is occupied by other applications. Run the following command to disable a specific process based on the PID to release the remote connection port that is in use.
    Warning: it is a risk operation to shut down the process. Make sure that the shutdown operation does not affect the business.
    taskkill /f /pid [$PID]

Application scope

  • Elastic Compute Service