:Resolve the RDP error "Because of a protocol error,the session will be disconnected" when connecting to a Windows instance

Step 1: Check for and resolve port conflicts

Another application may be using the default RDP port, 3389.

1. Log on to an ECS instance using a VNC connection.

   1. Go to ECS console - Instance. In the top navigation bar, select the target region and resource group.

   2. Go to the details page of the target instance. Click Connect and select VNC. Enter the username and password to log on to the ECS instance.

2. Check the port status.

   1. Search for and open PowerShell from the taskbar. Run the following command to find the Process ID (PID) of the process using the remote port (default 3389).

      netstat -ano | findstr :3389

   2. Replace <PID> with the Process ID from the previous step to identify the corresponding application.

      tasklist |findstr <PID>

3. Analyze the results and take action.

   • Normal: The process is svchost.exe. This is the expected behavior, and the port is not occupied by another application. Skip to Step 2: Check and adjust the security protocol.

   • Abnormal: The process is another application. You must change the default RDP port.

4. Change the default RDP port.

   1. Choose a new port: We recommend selecting an unused port in the range of 10000 to 65535. You can run the netstat -ano | findstr :<PortNumber> command in PowerShell to check if the target port is in use.

   2. Modify the registry:

      1. In the taskbar search box, type regedit and open the Registry Editor.

      2. In the left navigation pane, navigate to HKEY_LOCAL_MACHINE > SYSTEM > CurrentControlSet > Control > Terminal Server > WinStations > RDP-Tcp. In the right pane, find and double-click PortNumber.

      3. For the Base option, select Decimal. Change the Value Data to the new port number you selected.

   3. Restart and test:

      1. On the instance details page, click Restart to apply the changes. After the instance restarts, click Connect and select Workbench.

      2. In the Instance Login window, select Terminal, click More Options, and enter the new port number in the Port field.

         Important

         You must specify the new port each time you connect after changing the default RDP port. Alternatively, if the conflicting process can be safely terminated, run the taskkill /f /pid <PID> command to end it. This frees the default port, eliminating the need to specify a new one for each connection.

Step 2: Check and adjust the security protocol

1. Search for gpedit.msc and open the Local Group Policy Editor.

2. Navigate to Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Security. In the right pane, find and double-click Require use Of specific security layer for remote (RDP) connections.

3. In the Require use Of specific security layer for remote (RDP) connections dialog box, select Enabled. From the Security Layer drop-down list, select RDP.

4. Click OK to save the settings and try to connect again.