Fix SSL Certificate Errors on HTTPS Services Accessed via IP - MaxCompute

When a MaxCompute Spark or UDF task accesses a remote service in a Virtual Private Cloud (VPC) — such as Key Management Service (KMS) or Object Storage Service (OSS) — using an IP address instead of a domain name, SSL certificate verification fails. The server certificate does not include an IP-based Subject Alternative Name, so the SSL handshake is rejected with:

SSL: no alternative certificate subject name matches target host name '47.116.XX.XX'
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it.

This topic describes how to work around the issue by injecting the domain name into the HTTPS request so that SSL verification succeeds while the TCP connection still routes to the resolved IP.

If your VPC supports private DNS, configuring DNS resolution is a more permanent fix that requires no code changes. Use the code-based workaround in this topic when you cannot modify the network DNS configuration.

How it works

SSL certificate verification matches the domain name in the Host header against the names listed in the server certificate. When you connect by IP, there is no domain name in the request, so verification fails.

The fix has two steps:

Resolve the domain name to its IP address from inside the same network environment where your task runs.
Use a custom Python HTTP adapter that routes the TCP connection to the resolved IP while keeping the domain name in the Host header and the SSL SNI (Server Name Indication) field, so certificate verification succeeds.

Prerequisites

Before you begin, make sure you have:

A MaxCompute Spark or UDF task that needs to access a remote HTTPS service inside a VPC
Network access from the task's runtime environment to the remote service
Python 3 (recommended) or Python 2

Step 1: Get the IP address of the remote service

Run one of the following commands from inside the network environment where your task runs — for example, from within the VPC. The resolved IP differs between VPC and public network environments, so always retrieve it from the correct environment.

Use ping

Run the following command in a Windows or Linux console:

ping service.cn-shanghai-vpc.maxcompute.aliyun-inc.com

Windows result:
Linux result:

Use dig

dig provides more detailed DNS output. Install bind-utils first if it is not already available.

Windows

Download BIND9.17.12.x64.zip, extract it to a directory such as D:\install\BIND9.17.12.x64, and add that path to the Windows Path environment variable.

Linux (CentOS)

sudo yum install bind-utils

After installing, run:

dig service.cn-shanghai-vpc.maxcompute.aliyun-inc.com

Windows result:

ppp

Linux result:

123h

Step 2: Configure the HTTP adapter

Pass the resolved IP to the access_url function. The custom HostHeaderSSLAdapter replaces the hostname in the URL with the IP address, then restores the domain name in the Host header and the SSL SNI field so that certificate verification succeeds.

Choose the snippet that matches your Python version. Python 3 is recommended.

Python 3

# _*_ coding: utf-8 _*_
import requests
from urllib.parse import urlparse

class HostHeaderSSLAdapter(requests.adapters.HTTPAdapter):
    def __init__(self, resolved_ip):
        super().__init__()
        self.resolved_ip = resolved_ip

    def send(self, request, **kwargs):
        connection_pool_kwargs = self.poolmanager.connection_pool_kw
        result = urlparse(request.url)

        if result.scheme == 'https' and self.resolved_ip:
            request.url = request.url.replace(
                'https://' + result.hostname,
                'https://' + self.resolved_ip,
            )
            connection_pool_kwargs['server_hostname'] = result.hostname
            # Overwrite the Host header with the domain name
            request.headers['Host'] = result.hostname
        else:
            # Clear any server_hostname left from a previous request
            connection_pool_kwargs.pop('server_hostname', None)
        return super().send(request, **kwargs)

def access_url(url, resolved_ip):
    session = requests.Session()
    parsed_url = urlparse(url)
    hostname = parsed_url.hostname

    session.mount('https://' + hostname, HostHeaderSSLAdapter(resolved_ip))
    try:
        r = session.get(url)
    except Exception as e:
        print("err : " + str(e))
    else:
        if r.status_code != 200:
            print("not 200, resp : " + r.text)
        else:
            print("success, resp : " + r.text)

if __name__ == "__main__":
    # Retrieve the IP using dig from inside the VPC before running this test
    # VPC address test
    # access_url("https://service.cn-shanghai-vpc.maxcompute.aliyun-inc.com", "100.103.104.45")
    # Public network address test
    access_url("https://service.cn-shanghai.maxcompute.aliyun.com", "47.116.XX.XX")

Python 2

# _*_ coding: utf-8 _*_
# Only for Python 2
import requests
from urlparse import urlparse

class HostHeaderSSLAdapter(requests.adapters.HTTPAdapter):
    def __init__(self, resolved_ip):
        super(HostHeaderSSLAdapter, self).__init__()
        self.resolved_ip = resolved_ip

    def send(self, request, **kwargs):
        connection_pool_kwargs = self.poolmanager.connection_pool_kw
        result = urlparse(request.url)

        if result.scheme == 'https' and self.resolved_ip:
            request.url = request.url.replace(
                'https://' + result.hostname,
                'https://' + self.resolved_ip,
            )
            connection_pool_kwargs['assert_hostname'] = result.hostname
            request.headers['Host'] = result.hostname
        else:
            connection_pool_kwargs.pop('assert_hostname', None)

        return super(HostHeaderSSLAdapter, self).send(request, **kwargs)

def access_url(url, resolved_ip):
    session = requests.Session()
    parsed_url = urlparse(url)
    hostname = parsed_url.hostname

    session.mount('https://' + hostname, HostHeaderSSLAdapter(resolved_ip))
    try:
        r = session.get(url)
    except Exception as e:
        print("err : " + str(e))
    else:
        if r.status_code != 200:
            print("not 200, resp : " + r.text)
        else:
            print("success, resp : " + r.text)

if __name__ == "__main__":
    # Retrieve the IP using dig from inside the VPC before running this test
    # VPC address test
    # access_url("https://service.cn-shanghai-vpc.maxcompute.aliyun-inc.com", "100.103.104.45")
    # Public network address test
    access_url("https://service.cn-shanghai.maxcompute.aliyun.com", "47.116.XX.XX")

Important

Run the test from the same network environment where your task runs. For VPC access, set up the Python environment inside the VPC and use the VPC service URL and the IP address resolved from within that VPC. The IP resolved from a public network differs from the IP resolved inside a VPC.

Test results

The following screenshots show successful connections from two environments.

Local machine accessing MaxCompute through the public network:
Elastic Compute Service (ECS) instance running Linux accessing MaxCompute through the public network: