Http Redirect port loss in K8S Ingress environment

Recently, I found a problem: the application lost the original port when returning Http Redirect. For example, we visit http://IP-A:Port -A/app/delete, this url will respond to 302, but the response header Location returned by it has lost the port. The correct result should be as follows: http://IP-A:Port -A/app/index returns: http://IP-A/app/index , lost the port.

Basic information

Our deployment is as follows:

Deployed Nginx Ingress and exposed the Nginx Ingress Service by NodePort

Ingress with App configured

In fact, the above are not all servers. Two K8S services are not servers but VIPs. Please refer to the article K8S - Using Source IP for details http://IP-A:Port -When A/app/delete, the request runs through these servers from left to right.

By the way, the above NAT Server is an ordinary server. We use it to make PAT so that our Nginx Ingress can be accessed by the Internet.

observation

We use the Echo Server mentioned earlier to observe the request header passed to the Echo Server when accessing the Echo Server through Ingress: http://IP-A:Port -A/echo server gets these interesting Request header:

host=IP-A:Port-A
x-original-uri=/echo-server
x-forwarded-for=IP-B
x-forwarded-host=IP-A:Port-A
x-forwarded-port=80 x-forwarded-proto=http

Then directly access the Echo Server Svc and find that there is no x - * Request header mentioned above. So it is suspected that the problem lies in these headers.

Terminology

Tell me what these heads mean.

X-forwarded-for, when the client accesses the proxy, the client's IP address.

The reason why this is the IP address of the K8S node is that Nginx Ingress thinks that the request is from the K8S node (take a good look at the previous K8S - Using Source IP article). Before this, NAT was unknown.

X-forwarded-host refers to the original host that the client accesses when accessing the proxy.

X-forwarded-proto: When the client accesses the proxy, it accesses the original http scheme.

X-forwarded-port: the port that the client accesses when accessing the proxy.

X-original uri, no authoritative data can be found.

Note that the first three are factual standards, which are included in MDN. x-forwarded-port and x-original uri seem to be private extensions.

experiment

Find a handy Http Request tool (I use Postman), remember to turn off Follow Redirect, and then simulate Nginx requests (that is, bring/remove/modify the value of the x - * header mentioned above) to directly request the App Svc.

It is found that x-forwarded-port is the key to the response header location, that is, if x-forwarded-port=Port-A, the location will bring the correct port.

Analysis

How to construct the Redirect url

It can be inferred that the app uses the host and x-forwarded - * headers to construct the redirect url.

In the Java Servlet API, when describing HttpServletResponse # sendRedirect, it is mentioned that the returned URL must be Absolute URL.

Tomcat's org. apache. catalina. connector The toAbsolute method of the Response is responsible for constructing the Absolute URL.

How does it know which port to use? This is related to RemoteIPVave. If you are interested, you can consult the relevant documents.

The above only describes how Tomcat constructs the redirect url, but this method is not standard. Different containers have their own implementations. After all, the Java Servlet API does not specify how to construct the Absolute URL.

I also wrote an article on related topics before, "The reverse proxy uses the https protocol, the background tomcat uses http, and the solution to using the wrong protocol when redirecting". You can have a look.

Why x-forwarded-port is 80

Then the question arises. I clearly visited IP-A: Port-A. Why does Nginx get a value of 80?

This is because in the front section of the entire request link: NAT Server>K8S Node>Nginx Ingress Svc all work in Layer 4. It can be considered that everything they do is NAT. The Nginx Ingress Pod does not know the ports of these servers/network nodes, so it can only give its own port 80 (the port in the container) to the x-forwarding port.

You can see the Nginx Ingress configuration file about this logic:

kubectl -n kube-system exec -it -- cat /etc/nginx/nginx.conf

terms of settlement

Take x-forwarded-port when requesting (unreliable)

View the Nginx Ingress configuration file. It is found that if the initial request is made with an x-forward port, it can be changed to the following values. However, there are two problems:

You can't add this header when accessing through the browser

This header is generally added by a reverse proxy, that is, there must be a reverse proxy before our Nginx Ingress

So this method is not good.

Modify the code of tomcat (unreliable)

Although you can modify the code of tomcat to get the port from the x-forward-host/host header, this is unrealistic.

Modify the port of the NAT Server to 80 (reliable)

This method is reliable. As long as the port of the NAT Server is changed to 80, there is no problem.

In fact, if you directly access the K8S node (NodePort mode), you also need to set the NodePort to 80. Remember that? Nginx Ingress cannot know the port of the upper NAT.

In a word, the URL you originally requested cannot be a port other than 80, it must be http://some-ip/app Can only.