Container Service for Kubernetes:Advanced NGINX Ingress configurations

Configuration description

The NGINX Ingress Controller configuration method used by Container Service for Kubernetes (ACK) is fully compatible with the community version. For a complete list of configurations, see NGINX Configuration.

Configure a routing service for URL redirection

When you use the NGINX Ingress Controller, NGINX forwards the full path to the backend. For example, a request to /service1/api through the Ingress is forwarded to the /service1/api path of the backend pod. If the backend service path is /api, a path mismatch occurs and a 404 error is returned. To fix this, you can configure the nginx.ingress.kubernetes.io/rewrite-target annotation to rewrite the path to the correct directory.

Create an NGINX Ingress based on your cluster version.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo.bar.com
  namespace: default
  annotations:
    # URL redirection.
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
    # For Ingress Controller versions 0.22.0 or later, you must use a regular expression to define the path and use it with a capturing group in rewrite-target.
      - path: /svc(/|$)(.*)
        backend:
          service: 
            name: web1-service
            port: 
              number: 80
        pathType: ImplementationSpecific

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: foo.bar.com
  namespace: default
  annotations:
    # URL redirection.
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      # For Ingress Controller versions 0.22.0 or later, you must use a regular expression to define the path and use it with a capturing group in rewrite-target.
      - path: /svc(/|$)(.*)
        backend:
          serviceName: web1-service
          servicePort: 80

1. Access the NGINX service.

   1. Run the following command to obtain the ADDRESS.

      kubectl  get  ingress

      Expected output:

      NAME           CLASS   HOSTS                ADDRESS          PORTS   AGE
      foo.bar.com    nginx   foo.bar.com        172.16.XX.XX       80      46m

   2. Run the following command. Replace ADDRESS with the IP address of the Ingress.

      curl -k -H "Host: foo.bar.com"  http://<ADDRESS>/svc/foo

      Expected output:

      web1: /foo

Rewrite configurations

The nginx.ingress.kubernetes.io/rewrite-target annotation supports basic rewrite configurations. For more information, see Configure a routing service for URL redirection.

For complex or advanced rewrite requirements, you can use the following annotations:

• nginx.ingress.kubernetes.io/server-snippet: Adds custom configurations to the server block.

• nginx.ingress.kubernetes.io/configuration-snippet: Adds custom configurations to the location block.

These two annotations allow you to add custom code snippets to the NGINX server block of the Ingress component. This provides the flexibility to extend and customize NGINX configurations for different scenarios.

Configuration example:

annotations:
     nginx.ingress.kubernetes.io/server-snippet: |
         rewrite ^/v4/(.*)/card/query http://foo.bar.com/v5/#!/card/query permanent;
     nginx.ingress.kubernetes.io/configuration-snippet: |
         rewrite ^/v6/(.*)/card/query http://foo.bar.com/v7/#!/card/query permanent;

Run the following command to view the NGINX configuration file in the NGINX Ingress Controller component.

kubectl exec nginx-ingress-controller-xxxxx --namespace kube-system -- cat /etc/nginx/nginx.conf   # Replace the pod name with the actual one in your environment.

The following nginx.conf file is generated from the example configuration.

# start server foo.bar.com
    server {
        server_name foo.bar.com ;
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";
    # server-snippet configuration.
        rewrite ^/v4/(.*)/card/query http://foo.bar.com/v5/#!/card/query permanent;
        ...
    # configuration-snippet configuration.
      rewrite ^/v6/(.*)/card/query http://foo.bar.com/v7/#!/card/query permanent;
      ...
    }
    # end server foo.bar.com

The snippet feature also supports some global configurations. For more information, see server-snippet.

For more information about how to use the rewrite instruction, see the official NGINX documentation.

Configure an HTTPS certificate for a routing rule

You can use the native semantics of Ingress to configure an HTTPS certificate for your website.

1. Prepare your service certificate.

   Note

   The domain name must be the same as the host that you configure. Otherwise, the NGINX Ingress Controller cannot load it correctly.

   1. Run the following command to generate a certificate file named tls.crt and a private key file named tls.key.

      openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=foo.bar.com/O=foo.bar.com"

   2. Run the following command to create a Secret.

      Create a Kubernetes Secret named tls-test-ingress from the certificate and private key. You must reference this Secret when you create the Ingress.

      kubectl create secret tls tls-test-ingress --key tls.key --cert tls.crt

2. Run the following command to deploy the template and create an Ingress resource. You must reference the Secret that you created in the previous step in the tls field.

   Clusters of version 1.19 or later

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: test-test-ingress
   spec:
     # Reference the TLS certificate.
     tls:
     - hosts:
       - foo.bar.com # The domain name corresponding to the certificate. 
       secretName: tls-test-ingress
     rules:
     - host: tls-test-ingress.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: web1-svc
               port:
                 number: 80
           pathType: ImplementationSpecific

   Clusters of a version earlier than 1.19

   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: test-test-ingress
   spec:
     # Reference the TLS certificate.
     tls:
     - hosts:
       - foo.bar.com # The domain name corresponding to the certificate.
       secretName: tls-test-ingress
     rules:
     - host: tls-test-ingress.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: web1-svc
             servicePort: 80

3. Configure the hosts file or set a real domain name to access the TLS service.

   You can access the web1-svc service at https://tls-test-ingress.com/foo.

Configure HTTPS mutual authentication

In some scenarios, you may need to configure mutual HTTPS authentication between the server and the client to ensure connection security. The NGINX Ingress Controller lets you configure this feature using annotations.

1. Run the following command to create a self-signed CA certificate.

   openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Fern Cert Authority'

   Expected output:

   Generating a 4096 bit RSA private key
   .............................................................................................................++
   .....................................................................................++
   writing new private key to 'ca.key'

2. Run the following command to create a server-side certificate.

   1. Run the following command to generate a server-side certificate request file.

      openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=foo.bar.com'

      Expected output:

      Generating a 4096 bit RSA private key
      ................................................................................................................................++
      .................................................................++
      writing new private key to 'server.key'

   2. Run the following command to sign the server-side certificate request file with the root certificate and generate the server-side certificate.

      openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

      Expected output:

      Signature ok
      subject=/CN=foo.bar.com
      Getting CA Private Key

3. Run the following command to create a client certificate.

   1. Generate a request file for the client certificate.

      openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=Fern'

      Expected output:

      Generating a 4096 bit RSA private key
      .......................................................................................................................................................................................++
      ..............................................++
      writing new private key to 'client.key'
      -----

   2. Run the following command to sign the client certificate request file with the root certificate and generate the client certificate.

      openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt

      Expected output:

      Signature ok
      subject=/CN=Fern
      Getting CA Private Key

4. Run the following command to verify the certificates.

   ls

   The following output is expected:

   ca.crt  ca.key  client.crt  client.csr  client.key  server.crt  server.csr  server.key

5. Run the following command to create a Secret for the CA certificate.

   kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt

   The following output is expected:

   secret/ca-secret created

6. Run the following command to create a Secret for the server certificate.

   kubectl create secret generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key

   Expected output:

   secret/tls-secret created

7. Deploy the following template to create a test NGINX Ingress use case.

   Clusters of version 1.19 or later

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
       nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret"
       nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
       nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
     name: nginx-test
     namespace: default
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - backend:
             service:
               name: http-svc
               port: 
                 number: 80
           path: /
           pathType: ImplementationSpecific
     tls:
     - hosts:
       - foo.bar.com
       secretName: tls-secret

   Clusters of a version earlier than 1.19

   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     annotations:
       nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
       nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret"
       nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
       nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
     name: nginx-test
     namespace: default
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - backend:
             serviceName: http-svc
             servicePort: 80
           path: /
     tls:
     - hosts:
       - foo.bar.com
       secretName: tls-secret

   Expected output:

   ingress.networking.k8s.io/nginx-test configured

8. Run the following command to view the IP address of the Ingress.

   kubectl get ing

   The expected output is as follows. The IP address of the Ingress is displayed in the ADDRESS field.

   NAME         HOSTS                    ADDRESS         PORTS     AGE
   nginx-test   foo.bar.com              39.102.XX.XX    80, 443   4h42m

9. Run the following command to update the hosts file. Replace the IP address with the actual IP address of the Ingress.

   echo "39.102.XX.XX  foo.bar.com" | sudo tee -a /etc/hosts

   Verify the results:

   • Access the service from a client without a certificate

     curl --cacert ./ca.crt  https://foo.bar.com

     Expected output:

     <html>
     <head><title>400 No required SSL certificate was sent</title></head>
     <body>
     <center><h1>400 Bad Request</h1></center>
     <center>No required SSL certificate was sent</center>
     <hr><center>nginx/1.19.0</center>
     </body>
     </html>

   • Access the service from a client with a certificate

     curl --cacert ./ca.crt --cert ./client.crt --key ./client.key https://foo.bar.com

     Expected output:

     <!DOCTYPE html>
     <html>
     <head>
     <title>Welcome to nginx!</title>
     <style>
         body {
             width: 35em;
             margin: 0 auto;
             font-family: Tahoma, Verdana, Arial, sans-serif;
         }
     </style>
     </head>
     <body>
     <h1>Welcome to nginx!</h1>
     <p>If you see this page, the nginx web server is successfully installed and
     working. Further configuration is required.</p>
     
     <p>For online documentation and support please refer to
     <a href="http://nginx.org/">nginx.org</a>.<br/>
     Commercial support is available at
     <a href="http://nginx.com/">nginx.com</a>.</p>
     
     <p>Thank you for using nginx.</p>
     </body>
     </html>

Configure an HTTPS service to forward requests to a backend container over HTTPS

By default, the NGINX Ingress Controller forwards requests to backend application containers over HTTP. If your application container uses HTTPS, you can use the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation to configure the NGINX Ingress Controller to forward requests to the backend application container over HTTPS.

The following is an NGINX Ingress configuration example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backend-https
  annotations:
    # Note: You must specify that the backend service is an HTTPS service.
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - <YOUR-HOST-NAME>
    secretName: <YOUR-SECRET-CERT-NAME>
  rules:
  - host: <YOUR-HOST-NAME>
    http:
      paths:
      - path: /
        backend:
          service:
            name: <YOUR-SERVICE-NAME>
            port: 
              number: <YOUR-SERVICE-PORT>
        pathType: ImplementationSpecific

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: backend-https
  annotations:
    # Note: You must specify that the backend service is an HTTPS service.
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - <YOUR-HOST-NAME>
    secretName: <YOUR-SECRET-CERT-NAME>
  rules:
  - host: <YOUR-HOST-NAME>
    http:
      paths:
      - path: /
        backend:
          serviceName: <YOUR-SERVICE-NAME>
          servicePort: <YOUR-SERVICE-PORT>

Configure regular expression support for domain names

In a Kubernetes cluster, Ingress resources do not support regular expressions for domain names. However, you can use the nginx.ingress.kubernetes.io/server-alias annotation to enable this feature.

1. Deploy the following template. This example uses the regular expression ~^www\.\d+\.example\.com.

   Clusters of version 1.19 or later

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
     annotations:
       nginx.ingress.kubernetes.io/server-alias: '~^www\.\d+\.example\.com$, abc.example.com'
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: http-svc1
               port:
                 number: 80
           pathType: ImplementationSpecific

   Clusters of a version earlier than 1.19

   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
     annotations:
       nginx.ingress.kubernetes.io/server-alias: '~^www\.\d+\.example\.com$, abc.example.com'
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: http-svc1
             servicePort: 80

2. View the configuration of the corresponding NGINX Ingress Controller.

   1. Run the following command to view the pods where the NGINX Ingress Controller service is deployed.

      kubectl get pods -n kube-system | grep nginx-ingress-controller

      Expected output:

      nginx-ingress-controller-77cd987c4c-c****         1/1     Running   0          1h
      nginx-ingress-controller-77cd987c4c-x****         1/1     Running   0          1h

   2. Run the following command to view the configuration of the NGINX Ingress Controller. You can find the effective configuration in the server_name field.

      kubectl exec -n kube-system nginx-ingress-controller-77cd987c4c-c**** cat /etc/nginx/nginx.conf | grep -C3 "foo.bar.com"

      Expected output:

        # start server foo.bar.com
        server {
      --
        server {
          server_name foo.bar.com abc.example.com ~^www\.\d+\.example\.com$ ;
          listen 80  ;
          listen 443  ssl http2 ;
      --
      --
          }
        }
        # end server foo.bar.com

3. Run the following command to retrieve the IP address of the Ingress.

   kubectl get ing

   Expected output:

   NAME            HOSTS         ADDRESS          PORTS     AGE
   ingress-regex   foo.bar.com   101.37.XX.XX     80        11s

4. Run the following commands to test service access under different rules.

   Set IP_ADDRESS to the IP address that you retrieved in the previous step.

   • Run the following command to access the service using Host: foo.bar.com.

     curl -H "Host: foo.bar.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the service using Host: www.123.example.com.

     curl -H "Host: www.123.example.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the service using Host: www.321.example.com.

     curl -H "Host: www.321.example.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

Configure wildcard support for domain names

In a Kubernetes cluster, NGINX Ingress resources support wildcard domain names. For example, you can configure the wildcard domain name *.ingress-regex.com.

1. Deploy the following template to create an NGINX Ingress.

   Clusters of version 1.19 or later

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
   spec:
     rules:
     - host: *.ingress-regex.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: http-svc1
               port:
                 number: 80
           pathType: ImplementationSpecific

   Clusters of a version earlier than 1.19

   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
   spec:
     rules:
     - host: *.ingress-regex.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: http-svc1
             servicePort: 80

2. Run the following command to view the configuration of the NGINX Ingress Controller. You can find the effective configuration in the server_name field.

   kubectl exec -n kube-system <nginx-ingress-pod-name> cat /etc/nginx/nginx.conf | grep -C3 "*.ingress-regex.com"

   Note

   Replace nginx-ingress-pod-name with the name of the nginx-ingress pod in your environment.

   Expected output:

   # start server *.ingress-regex.com
     server {
       server_name *.ingress-regex.com ;
       listen 80;
       listen [::]:80;
   ...
     }
     # end server *.ingress-regex.com

   Expected output in a later version of the NGINX Ingress Controller:

   ## start server *.ingress-regex.com
     server {
       server_name ~^(?<subdomain>[\w-]+)\.ingress-regex\.com$ ;
       listen 80;
       listen [::]:80;
   ...
     }
     ## end server *.ingress-regex.com

3. Run the following command to retrieve the IP address of the Ingress.

   kubectl get ing

   Expected output:

   NAME            HOSTS                 ADDRESS           PORTS     AGE
   ingress-regex   *.ingress-regex.com   101.37.XX.XX      80        11s

4. Run the following commands to test service access under different rules.

   Set IP_ADDRESS to the IP address that you retrieved in the previous step.

   • Run the following command to access the service using Host: abc.ingress-regex.com.

     curl -H "Host: abc.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the service using Host: 123.ingress-regex.com.

     curl -H "Host: 123.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the service using Host: a1b1.ingress-regex.com.

     curl -H "Host: a1b1.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

Implement a phased release using annotations

Use cert-manager to request a free HTTPS certificate

cert-manager is an open source certificate management tool that provisions and automatically renews HTTPS certificates in a cluster. The following example describes how to use cert-manager to request and automatically renew a free certificate.

1. Run the following command to deploy cert-manager.

   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

2. Run the following command to view the pod status.

   kubectl get pods -n cert-manager

   Expected output:

   NAME                     READY   STATUS    RESTARTS   AGE
   cert-manager-1           1/1     Running   0          2m11s
   cert-manager-cainjector  1/1     Running   0          2m11s
   cert-manager-webhook     1/1     Running   0          2m10s

3. Use the following template to create a ClusterIssuer.

   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-prod-http01
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       email: <your_email_n***@gmail.com>  # Replace this with your mailbox.
       privateKeySecretRef:
         name: letsencrypt-http01
       solvers:
       - http01: 
           ingress:
             class: nginx

4. Run the following command to view the ClusterIssuer.

   kubectl get clusterissuer

   Expected output:

   NAME                         READY   AGE
   letsencrypt-prod-http01      True    17s

5. Create an NGINX Ingress resource object.

   Clusters of version 1.19 or later

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-tls
     annotations:
       kubernetes.io/ingress.class: "nginx"
       cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
   spec:
     tls:
     - hosts:
       - <YOUR_DOMAIN_NAME>        # Replace this with your domain name.
       secretName: ingress-tls   
     rules:
     - host: <YOUR_DOMAIN_NAME>    # Replace this with your domain name.
       http:
         paths:
         - path: /
           backend:
             service:
               name: <YOUR_SERVICE_NAME>  # Replace this with your backend service name.
               port: 
                 number: <YOUR_SERVICE_PORT>  # Replace this with your service port.
           pathType: ImplementationSpecific

   Clusters of a version earlier than 1.19

   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     name: ingress-tls
     annotations:
       kubernetes.io/ingress.class: "nginx"
       cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
   spec:
     tls:
     - hosts:
       - <YOUR_DOMAIN_NAME>        # Replace this with your domain name.
       secretName: ingress-tls   
     rules:
     - host: <YOUR_DOMAIN_NAME>    # Replace this with your domain name.
       http:
         paths:
         - path: /
           backend:
             serviceName: <YOUR_SERVICE_NAME>  # Replace this with your backend service name.
             servicePort: <YOUR_SERVICE_PORT>  # Replace this with your service port.

   Note

   The domain name that you use to replace your_domain_name must meet the following conditions:

   • The domain name cannot exceed 64 characters.

   • Wildcard domain names are not supported.

   • It is accessible over the public network via the HTTP protocol.

6. Run the following command to view the certificate.

   kubectl get cert

   Expected output:

   NAME          READY   SECRET        AGE
   ingress-tls   True    ingress-tls   52m

   Note

   If the READY status is not True, run kubectl describe cert ingress-tls to view the certificate processing procedure.

7. Run the following command to view the Secret.

   kubectl get secret  ingress-tls

   Expected output:

   NAME          TYPE                DATA   AGE
   ingress-tls   kubernetes.io/tls   2      2m

8. In a web browser, enter https://[your_domain_name] to access the configured domain name.

HTTP-to-HTTPS redirect

You can use the nginx.ingress.kubernetes.io/ssl-redirect annotation for NGINX Ingress to force HTTP to HTTPS redirection. The following is an example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true" # Force HTTP traffic to be redirected to HTTPS.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true" # Force HTTP traffic to be redirected to HTTPS.