Elastic Compute Service:Reduce public network exposure with PrivateLink

Use PrivateLink for cross-VPC access within the same zone

Description: A network service (such as a web service) is deployed on an ECS instance in VPC2. A PrivateLink connection between VPC1 and VPC2 allows the service to be accessed from VPC1 through a private IP address (endpoint).

: The endpoint (service consumer) and the endpoint service (service provider) must be in the same zone. The endpoint zones must be a subset of the endpoint service zones. For example, if an endpoint service is deployed in Zone A and Zone B, the endpoint can only be created in Zone A or Zone B.

Core steps:

1. Server-side (service provider) configuration

   1. Create a load balancer: In VPC2, create an SLB instance that supports PrivateLink, such as a private-facing CLB or NLB. Configure a backend server group and a listener.

   2. Create an endpoint service: In the PrivateLink console, create an endpoint service and associate it with the SLB instance created in the previous step.

   3. Configure a service whitelist (cross-account access only): If VPC1 belongs to a different Alibaba Cloud account, add the account's UID to the service whitelist of the endpoint service.

   4. Accept the connection request: After the client initiates a connection, go to the Connections tab of the endpoint service and accept the pending request. Alternatively, configure the service to automatically accept connections.

2. Client-side (service consumer) configuration

   1. Create an endpoint: In VPC1, create an endpoint and select the endpoint service created on the server side.

   2. Configure the network: Select a vSwitch and a security group in VPC1 for the endpoint to control which resources in VPC1 can access it.

   3. Obtain the endpoint details: After creation, a private domain name and IP address are generated.

   4. Initiate access: After the service provider accepts the connection, applications in VPC1 can access the service in VPC2 through the generated domain name or IP address.

Use Proxy Protocol to control accessible VPCs and their private IP addresses

When you provide services through PrivateLink, the server can identify the source endpoint or VPC of each request and implement security rules accordingly. For example, you can restrict an account to accept access only from a legitimate VPC. Even if credentials are leaked, unauthorized networks cannot access the service.

Building on Use PrivateLink for cross-VPC access within the same zone, you can enable the server-side application to identify the source VPC and implement access control.

1. Enable Proxy Protocol v2: Enable Proxy Protocol v2 on the NLB listener and subscribe to the VPC ID and endpoint ID.

   # Use the Cloud Assistant CLI to enable Proxy Protocol v2 for a specified listener and subscribe to the VPC ID and endpoint ID.
   aliyun nlb UpdateListenerAttribute \
        --ListenerId lsn-xxxxxxxxxxxxxxxx \
        --ProxyProtocolEnabled true \
        --ProxyProtocolV2Config '{"Ppv2VpcIdEnabled":true,"Ppv2PrivateLinkEpIdEnabled":true}' \
        --RegionId cn-hangzhou

   --ListenerId: The listener ID created for the web service.

   --ProxyProtocolEnabled: Set to true to enable Proxy Protocol. This protocol passes client connection information in the first packet after the TCP handshake. The backend service must support this protocol. See Step 2 for details. Otherwise, the service will be interrupted.

   --ProxyProtocolV2Config: Adds Alibaba Cloud-defined Type-Length-Value (TLV) fields. This example subscribes to the source VPC ID and PrivateLink endpoint ID.

   17 7.005726    10.0.0.186    10.0.0.184    TCP    74 3059 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=128
   18 7.005753    10.0.0.184    10.0.0.186    TCP    62 80 → 3059 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 WS=128
   19 7.006781    10.0.0.186    10.0.0.184    TCP    54 3059 → 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0
   20 7.006804    10.0.0.186    10.0.0.184    HTTP    154 Continuation
   21 7.006814    10.0.0.184    10.0.0.186    TCP    54 80 → 3059 [ACK] Seq=1 Ack=101 Win=64256 Len=0
   22 7.006805    10.0.0.186    10.0.0.184    HTTP    210 GET /login HTTP/1.1
   23 7.006821    10.0.0.184    10.0.0.186    TCP    54 80 → 3059 [ACK] Seq=1 Ack=257 Win=64128 Len=0

   The following table describes how to parse the TLV.

   	Description/Content	Original content
   Protocol header	Fixed signature of the Proxy Protocol	0d0a0d0a000d0a515549540a
   	Version 2, PROXY command	21
   	TCP/IPv4	11
   	The following data is 84 bytes long	0054
   Address information	Source IP: 10.0.0.14	0a00000e
   	Destination IP: 10.0.0.15	0a00000f
   	Source port: 59074	e6c2
   	Destination port: 80	0050
   TLV1	Type 03	03
   	4 bytes	0004
   	0764b56b	0764b56b
   TLV2	Type e1 (custom type)	e1
   	24 bytes	0018
   	PrivateLink endpoint ID: ep-bp1i288487e586152d4b	0265702d6270316932383834383765353836313532643462
   TLV3	Type e1 (custom type)	e1
   	26 bytes	001a
   	VPC ID: vpc-bp179qeke0wzo1mr8bxhl	017670632d627031373971656b6530777a6f316d72386278686c
   TLV4	04 (PP2_TYPE_NETNS), used to pass the network namespace	04
   	6 bytes	0006
   	000000000000	000000000000

2. If your web service uses Nginx, configure it to support Proxy Protocol:

   (Requires NGINX Plus R16 or later, or open source Nginx 1.13.11 or later.)

   server {
       listen 80 proxy_protocol;   # Enable Proxy Protocol v1/v2.
       # Other configurations...
   }

   Nginx automatically parses Proxy Protocol TLV on port 80, but the standard version only supports fetching the IP address from the TLV.

   Use a Lua extension to handle Alibaba Cloud-customized VPC IDs and PrivateLink endpoint IDs.

   The following Lua plugin in nginx.conf handles TCP connections, reads packets from the socket, and parses the TLV buffer.

   The parsed data is stored in the request context. When a /login request is made, the code verifies the PrivateLink endpoint ID:

   -- Define shared memory before the http block.
   lua_shared_dict tlv_cache 10m;
   
   stream {
     server {
       listen 80;  # Do not enable proxy_protocol. Let the Lua code below handle it.
       preread_by_lua_block {
          -- 1. Try to read the fixed 16-byte protocol header.
          local sock = ngx.req.socket(true)
          local header, err = sock:receive(16)
   
          if not header then
             ngx.log(ngx.ERR, "failed to receive proxy protocol header: ", err)
             return
          end
   
          -- 2. Check the protocol signature (12 bytes).
          if string.sub(header, 1, 12) ~= "\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A" then
              -- If it is not the Proxy Protocol, put the read data back into the buffer to avoid corrupting the original request.
               sock:setreused(header)
                     return
         end
   
         -- 3. Parse the rest of the protocol header.
         local ver_cmd = string.byte(header, 13) -- Version and command
         local family = string.byte(header, 14) -- Address family and protocol
         local len = string.byte(header, 15) * 256 + string.byte(header, 16) -- Total length of the variable part
         if len == 0 then
            -- No address or TLV information. Return directly.
            return
         end
   
         -- 4. Read the variable-length part (contains address and TLV).
         local variable_part, err = sock:receive(len)
         if not variable_part or #variable_part < len then
            ngx.log(ngx.ERR, "failed to read variable part (address + tlvs): ", err)
            return
         end
   
         -- 5. Determine the length of the address information based on the address family.
         local addr_len = 0
         local protocol = family & 0x0F
         local address_family = family >> 4
   
         if address_family == 1 then -- AF_INET (IPv4)
             addr_len = 12
         elseif address_family == 2 then -- AF_INET6 (IPv6)
             addr_len = 36
         elseif address_family == 3 then -- AF_UNIX
             addr_len = 216
         end
   
         -- 6. Separate the address information and TLV data.
         local address_block = string.sub(variable_part, 1, addr_len) -- You can parse the address information if needed.
         local tlv_string = string.sub(variable_part, addr_len + 1)
   
         -- 7. Parse the TLV data.
         local tlvs = {}
         local pos = 1
         while pos <= #tlv_string do
             -- Make sure there are at least 3 bytes (Type, Length).
             if pos + 2 > #tlv_string then
                 ngx.log(ngx.ERR, "malformed TLV data: not enough bytes for type and length")
                 break
             end
   
             local tlv_type = string.byte(tlv_string, pos)
             local tlv_length = string.byte(tlv_string, pos + 1) * 256 + string.byte(tlv_string, pos + 2)
         
             -- Make sure the length of the value does not exceed the boundary.
             if pos + 2 + tlv_length > #tlv_string then
                 ngx.log(ngx.ERR, "malformed TLV data: length exceeds available data")
                 break
             end
         
             local tlv_value = string.sub(tlv_string, pos + 3, pos + 2 + tlv_length)
                         
             table.insert(tlvs, {
                 type = tlv_type,
                 length = tlv_length,
                 -- Encode the binary value in Base64 for easy storage and viewing.
                 value = ngx.encode_base64(tlv_value) 
              })
              pos = pos + 3 + tlv_length
        end
   
        -- 8. Cache the parsed TLV in shared memory.
        if #tlvs > 0 then
           local cjson = require "cjson"
           local tlv_cache = ngx.shared.tlv_cache
           -- Use the client IP address as the key to store the TLV data.
           local client_ip = ngx.var.remote_addr
           local tlv_json = cjson.encode(tlvs)
           tlv_cache:set(client_ip, tlv_json, 60)  -- Cache for 60 seconds.
           -- Optional: Also store in ngx.ctx for the stream layer to use.
           ngx.ctx.proxy_protocol_tlvs = tlvs
        end
       }
       proxy_pass localhost:8080;   # Pass subsequent HTTP requests to the HTTP server for processing.
     }
   }
   
   http {
       server {
           listen 8080;
   
           # Dedicated location to handle /login requests.
           location /login {
               access_by_lua_block {
                   local cjson = require "cjson"
                   local tlv_cache = ngx.shared.tlv_cache
                   -- Get TLV data from shared memory.
                   local client_ip = ngx.var.remote_addr
                   local tlv_json = tlv_cache:get(client_ip)
                   if not tlv_json then
                      ngx.log(ngx.ERR, "No TLV data found for client: ", client_ip)
                      return ngx.exit(ngx.HTTP_FORBIDDEN)
                   end
                   local tlvs = cjson.decode(tlv_json)
                   -- Extract the value of the second TLV (PrivateLink endpoint ID).
                   local privateLinkEndId = nil
                   if tlvs[2] then
                      privateLinkEndId = tlvs[2].value  -- Base64-encoded value.
                   end
                   -- Define the expected endpoint ID (Base64-encoded).
                   local expected_value = "ZXAtYnAxaTI4ODQ4N2U1ODYxNTJkNGI="  -- Example value. Modify it as needed.
                   -- Verify the endpoint ID.
                   if privateLinkEndId ~= expected_value then
                      ngx.log(ngx.ERR, "Access to /login denied: TLV value mismatch. Expected '", expected_value, "', got '", tostring(privateLinkEndId), "'")
                      return ngx.exit(ngx.HTTP_FORBIDDEN)
                   end
                   -- If the verification is successful, the request continues to be processed.
               }
               proxy_pass http://your_python_backend;
           }
   
           # Handle all other requests.
           location / {
               proxy_pass http://your_python_backend;
           }
       }
   
       # Your Python backend service upstream definition.
       upstream your_python_backend {
           server 127.0.0.1:5000; # Assume your backend program runs on port 5000.
       }
   }