遠程VPC服務僅支援使用網域名稱訪問,直接使用IP訪問會導致錯誤使請求失敗。本文將介紹通過將網域名稱添加到請求Host中,以便通過IP地址訪問HTTPS服務,來滿足Spark或UDF等任務的遠端存取需求。
直接使用IP訪問HTTPS服務失敗
報錯資訊
SSL: no alternative certificate subject name matches target host name '47.116.XX.XX'
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it.
To learn more about this situation and how to fix it, please visit the web page mentioned above.
問題描述
Spark或UDF等任務,訪問遠程VPC服務,比如KMS、OSS等。在此過程中需要使用HTTPS功能,如果直接通過IP來訪問對端服務會報上述錯誤資訊。
解決方案
將網域名稱添加到請求的host中,以解決通過IP直接存取HTTPS服務時所遇到的IP無法通過SSL認證驗證的問題。
1. 擷取遠程服務的IP資訊。
ping命令擷取
在Windows或Linux的控制台,直接輸入如下指令,並擷取遠程服務IP。
ping service.cn-shanghai-vpc.maxcompute.aliyun-inc.com
-
Windows環境返回結果如下:
PS C:\Users\xxx> ping service.cn-shanghai-vpc.maxcompute.aliyun-inc.com Ping service.cn-shanghai-vpc.maxcompute.aliyun-inc.com [100.103.104.45] xxx -
Linux環境返回結果如下:
[root@iZbxxx ~]# ping service.cn-shanghai-vpc.maxcompute.aliyun-inc.com PING service.cn-shanghai-vpc.maxcompute.aliyun-inc.com (100.103.104.45) 56(84) bytes of data.
dig命令擷取
-
在Windows環境或Linux環境下安裝Bind-utils。
-
Windows環境
下載BIND9.17.12.x64.zip並解壓至指定目錄,例如
D:\install\BIND9.17.12.x64。在Windows環境變數Path中添加此路徑即可。 -
Linux環境(CentOS)
直接在LinuxOperations 主控台輸入指令
sudo yum install bind-utils完成安裝即可。
-
-
在各自環境的控制台執行如下指令:
dig service.cn-shanghai-vpc.maxcompute.aliyun-inc.comWindows環境
PS C:\Users\xxx> dig service.cn-shanghai-vpc.maxcompute.aliyun-inc.com ; <<>> DiG 9.17.12 <<>> service.cn-shanghai-vpc.maxcompute.aliyun-inc.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49974 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;service.cn-shanghai-vpc.maxcompute.aliyun-inc.com. IN A ;; ANSWER SECTION: service.cn-shanghai-vpc.maxcompute.aliyun-inc.com. 1 IN A 100.103.104.45 ;; Query time: 4 msec ;; SERVER: 10.61.150.xxx ;; WHEN: Wed Jan 08 14xxxLinux環境
[root@iZbxxx ~]# dig service.cn-shanghai-vpc.maxcompute.aliyun-inc.com ; <<>> DiG 9.11.4-P2-RedHat-9.11.4 <<>> service.cn-shanghai-vpc.maxcompute.aliyun-inc.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36725 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;service.cn-shanghai-vpc.maxcompute.aliyun-inc.com. IN A ;; ANSWER SECTION: service.cn-shanghai-vpc.maxcompute.aliyun-inc.com. 1 IN A 100.103.104.45 ;; Query time: 2 msec ;; SERVER: 100.100.2.xxx ;; WHEN: Wed Jan 08 14xxx
2. 配置IP資訊。
您可以根據Python不同版本,參考如下代碼,在發送請求的access_url中添加IP資訊。在正式發布任務之前,可先在對應網路環境進行遠端存取測試。
-
python2
# _*_ coding: utf-8 _*_ # only for python2 import requests from urlparse import urlparse class HostHeaderSSLAdapter(requests.adapters.HTTPAdapter): def __init__(self, resolved_ip): super(HostHeaderSSLAdapter,self).__init__() self.resolved_ip = resolved_ip def send(self, request, **kwargs): connection_pool_kwargs = self.poolmanager.connection_pool_kw result = urlparse(request.url) if result.scheme == 'https' and self.resolved_ip: request.url = request.url.replace( 'https://' + result.hostname, 'https://' + self.resolved_ip, ) connection_pool_kwargs['assert_hostname'] = result.hostname request.headers['Host'] = result.hostname else: connection_pool_kwargs.pop('assert_hostname', None) return super(HostHeaderSSLAdapter, self).send(request, **kwargs) def access_url(url, resolved_ip): session = requests.Session() # 從url中擷取網域名稱部分 parsed_url = urlparse(url) hostname = parsed_url.hostname session.mount('https://'+hostname, HostHeaderSSLAdapter(resolved_ip)) try: r = session.get(url) except Exception as e: print("err : "+ str(e)) else: if r.status_code != 200: print("not 200 " + ",resp : "+ r.text) else: print("success" + ",resp : "+ r.text) if __name__ == "__main__": # ip通過dig網域名稱擷取,需要在vpc環境內dig # vpc地址測試 #access_url("https://service.cn-shanghai-vpc.maxcompute.aliyun-inc.com", "100.103.104.45") # 公網地址測試 access_url("https://service.cn-shanghai.maxcompute.aliyun.com", "47.116.XX.XX") -
python3
# _*_ coding: utf-8 _*_ import requests from urllib.parse import urlparse class HostHeaderSSLAdapter(requests.adapters.HTTPAdapter): def __init__(self, resolved_ip): super().__init__() self.resolved_ip = resolved_ip def send(self, request, **kwargs): connection_pool_kwargs = self.poolmanager.connection_pool_kw result = urlparse(request.url) if result.scheme == 'https' and self.resolved_ip: request.url = request.url.replace( 'https://' + result.hostname, 'https://' + self.resolved_ip, ) connection_pool_kwargs['server_hostname'] = result.hostname # overwrite the host header request.headers['Host'] = result.hostname else: # theses headers from a previous request may have been left connection_pool_kwargs.pop('server_hostname', None) return super().send(request, **kwargs) def access_url(url, resolved_ip): session = requests.Session() # 從url中擷取網域名稱部分 parsed_url = urlparse(url) hostname = parsed_url.hostname session.mount('https://'+hostname, HostHeaderSSLAdapter(resolved_ip)) try: r = session.get(url) except Exception as e: print("err : "+ str(e)) else: if r.status_code != 200: print("not 200 " + ",resp : "+ r.text) else: print("success" + ",resp : "+ r.text) if __name__ == "__main__": # ip通過dig網域名稱擷取,需要在vpc環境內dig # vpc地址測試 #access_url("https://service.cn-shanghai-vpc.maxcompute.aliyun-inc.com", "100.103.104.45") # 公網地址測試 access_url("https://service.cn-shanghai.maxcompute.aliyun.com", "47.116.XX.XX")
測試結果
若您需要訪問VPC網路的相關執行個體服務,請在VPC網路環境下,配置Python環境,替換執行個體服務所在VPC網路連結和IP地址進行測試。
-
在本地通過公網訪問MaxCompute服務。
if __name__ == "__main__": access_url( url="https://service.cn-shanghai.maxcompute.aliyun.com", resolved_ip="47.116.XX.XX") "D:\Program Files\Python311\python.exe" D:\ProgramData\PycharmProjects\pythontest1\text.py success,resp : <!DOCTYPE html> <html> <head> <title>Welcome to tengine!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to tengine!</h1> <p>If you see this page, the tengine web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://tengine.taobao.org/">tengine.taobao.org</a>.</p> <p><em>Thank you for using tengine.</em></p> </body> </html> -
在Linux雲端服務器上通過公網訪問MaxCompute服務。
[root@iZbp1ehm6ky76ig8n1jd8dZ opt]# python3 text.py success,resp : <!DOCTYPE html> <html> <head> <title>Welcome to tengine!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to tengine!</h1> <p>If you see this page, the tengine web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://tengine.taobao.org/">tengine.taobao.org</a>.</p> <p><em>Thank you for using tengine.</em></p> </body> </html>