Container Service for Kubernetes:Advanced NGINX Ingress configurations

Configuration details

The methods used to configure the NGINX Ingress controller in ACK are fully compatible with the open source version of the component. For more information about all configuration options, see NGINX Configuration.

Configure a routing service for URL redirection

When you use the NGINX Ingress controller, NGINX forwards the full path to the backend. For example, a request to the Ingress path /service1/api is forwarded to the backend pod at /service1/api/. If your backend service path is /api, a path mismatch occurs and a 404 error is returned. In this case, you can use the nginx.ingress.kubernetes.io/rewrite-target annotation to rewrite the path to the correct directory.

Create an NGINX Ingress based on the cluster version.

cat <<-EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo.bar.com
  namespace: default
  annotations:
    # URL redirection.
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
    # For Ingress controller versions 0.22.0 and later, you must use a regular expression to define the path and use it with a capturing group in the rewrite-target annotation.
      - path: /svc(/|$)(.*)
        backend:
          service: 
            name: web1-service
            port: 
              number: 80
        pathType: ImplementationSpecific
EOF

cat <<-EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: foo.bar.com
  namespace: default
  annotations:
    # URL redirection.
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
    # For Ingress controller versions 0.22.0 and later, you must use a regular expression to define the path and use it with a capturing group in the rewrite-target annotation.
      - path: /svc(/|$)(.*)
        backend:
          serviceName: web1-service
          servicePort: 80
EOF

1. Access the NGINX service.

   1. Run the following command to retrieve the value of ADDRESS.

      kubectl  get  ingress

      Expected output:

      NAME           CLASS   HOSTS                ADDRESS          PORTS   AGE
      foo.bar.com    nginx   foo.bar.com        172.16.XX.XX       80      46m

   2. Run the following command. Replace ADDRESS with the IP address of the Ingress.

      curl -k -H "Host: foo.bar.com"  http://<ADDRESS>/svc/foo

      Expected output:

      web1: /foo

Rewrite configuration

You can use the nginx.ingress.kubernetes.io/rewrite-target annotation for basic rewrite configurations. For more information, see Configure a routing service for URL redirection.

For complex and advanced rewrite requirements, you can use the following annotations:

• nginx.ingress.kubernetes.io/server-snippet: Extends the configuration to the `server` block.

• nginx.ingress.kubernetes.io/configuration-snippet: Extends the configuration to the `location` block.

These two annotations add custom code snippets to the NGINX configuration of the Ingress component. This provides the flexibility to extend and customize NGINX configurations for different scenarios.

Example configuration:

annotations:
     nginx.ingress.kubernetes.io/server-snippet: |
         rewrite ^/v4/(.*)/card/query http://foo.bar.com/v5/#!/card/query permanent;
     nginx.ingress.kubernetes.io/configuration-snippet: |
         rewrite ^/v6/(.*)/card/query http://foo.bar.com/v7/#!/card/query permanent;

Run the following command to view the NGINX configuration file in the NGINX Ingress controller component.

kubectl exec nginx-ingress-controller-xxxxx --namespace kube-system -- cat /etc/nginx/nginx.conf   # Modify the pod name based on your environment.

The example configuration generates the following nginx.conf file.

# start server foo.bar.com
    server {
        server_name foo.bar.com ;
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";
    # server-snippet configuration.
        rewrite ^/v4/(.*)/card/query http://foo.bar.com/v5/#!/card/query permanent;
        ...
    # configuration-snippet configuration.
      rewrite ^/v6/(.*)/card/query http://foo.bar.com/v7/#!/card/query permanent;
      ...
    }
    # end server foo.bar.com

The snippet also supports some global configurations. For more information, see server-snippet.

For more information about how to use the rewrite directive, see the NGINX official documentation for the directive.

Configure an HTTPS certificate for a routing rule

You can use the native semantics of Ingress to configure an HTTPS certificate for a website.

1. Prepare your service certificate.

   Note

   The domain name must match the host that you configure. Otherwise, the NGINX Ingress controller cannot correctly load the certificate.

   1. Run the following command to generate a certificate file named tls.crt and a private key file named tls.key.

      openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=foo.bar.com/O=foo.bar.com"

   2. Run the following command to create a Secret.

      Use the certificate and private key to create a Kubernetes Secret named `tls-test-ingress`. You must reference this Secret when you create the Ingress.

      kubectl create secret tls tls-test-ingress --key tls.key --cert tls.crt

2. Create an Ingress resource using the following template. The `tls` field is used to reference the Secret that you created in the previous step.

   Clusters that run Kubernetes 1.19 or later

   cat <<EOF | kubectl create -f - 
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: test-test-ingress
   spec:
     # Reference the TLS certificate.
     tls:
     - hosts:
       - foo.bar.com # The domain name that corresponds to the certificate. 
       secretName: tls-test-ingress
     rules:
     - host: tls-test-ingress.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: web1-svc
               port:
                 number: 80
           pathType: ImplementationSpecific
   EOF

   Clusters that run a Kubernetes version earlier than 1.19

   cat <<EOF | kubectl create -f - 
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: test-test-ingress
   spec:
     # Reference the TLS certificate.
     tls:
     - hosts:
       - foo.bar.com # The domain name that corresponds to the certificate.
       secretName: tls-test-ingress
     rules:
     - host: tls-test-ingress.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: web1-svc
             servicePort: 80
   EOF

3. Configure the hosts file or set a real domain name to access the TLS service.

   You can access the web1-svc service at https://tls-test-ingress.com/foo.

Configure HTTPS mutual authentication

In some cases, you may need to configure mutual HTTPS authentication between the server and client to ensure connection security. The NGINX Ingress controller supports this feature through annotations.

1. Run the following command to create a self-signed certificate authority (CA) certificate.

   openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Fern Cert Authority'

   Expected output:

   Generating a 4096 bit RSA private key
   .............................................................................................................++
   .....................................................................................++
   writing new private key to 'ca.key'

2. Run the following commands to create a server-side certificate.

   1. Run the following command to generate a request file for the server-side certificate.

      openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=foo.bar.com'

      Expected output:

      Generating a 4096 bit RSA private key
      ................................................................................................................................++
      .................................................................++
      writing new private key to 'server.key'

   2. Run the following command to use the root certificate to sign the server-side request file and generate the server-side certificate.

      openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

      Expected output:

      Signature ok
      subject=/CN=foo.bar.com
      Getting CA Private Key

3. Run the following commands to create a client-side certificate.

   1. Run the following command to generate a request file for the client-side certificate.

      openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=Fern'

      Expected output:

      Generating a 4096 bit RSA private key
      .......................................................................................................................................................................................++
      ..............................................++
      writing new private key to 'client.key'
      -----

   2. Run the following command to use the root certificate to sign the client-side request file and generate the client-side certificate.

      openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt

      Expected output:

      Signature ok
      subject=/CN=Fern
      Getting CA Private Key

4. Run the following command to verify the created certificates.

   ls

   Expected output:

   ca.crt  ca.key  client.crt  client.csr  client.key  server.crt  server.csr  server.key

5. Run the following command to create a Secret for the CA certificate.

   kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt

   Expected output:

   secret/ca-secret created

6. Run the following command to create a Secret for the server certificate.

   kubectl create secret generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key

   Expected output:

   secret/tls-secret created

7. Create a test NGINX Ingress using the following template.

   Clusters that run Kubernetes 1.19 or later

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
       nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret"
       nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
       nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
     name: nginx-test
     namespace: default
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - backend:
             service:
               name: http-svc
               port: 
                 number: 80
           path: /
           pathType: ImplementationSpecific
     tls:
     - hosts:
       - foo.bar.com
       secretName: tls-secret
   EOF

   Clusters that run a Kubernetes version earlier than 1.19

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     annotations:
       nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
       nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret"
       nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
       nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
     name: nginx-test
     namespace: default
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - backend:
             serviceName: http-svc
             servicePort: 80
           path: /
     tls:
     - hosts:
       - foo.bar.com
       secretName: tls-secret
   EOF

   Expected output:

   ingress.networking.k8s.io/nginx-test configured

8. Run the following command to retrieve the IP address of the Ingress.

   kubectl get ing

   The value in the ADDRESS column is the IP address of the Ingress.

   NAME         HOSTS                    ADDRESS         PORTS     AGE
   nginx-test   foo.bar.com              39.102.XX.XX    80, 443   4h42m

9. Run the following command to update the hosts file. Replace the sample IP address with the actual IP address of the Ingress.

   echo "39.102.XX.XX  foo.bar.com" | sudo tee -a /etc/hosts

   Verification:

   • Access without a client certificate

     curl --cacert ./ca.crt  https://foo.bar.com

     Expected output:

     <html>
     <head><title>400 No required SSL certificate was sent</title></head>
     <body>
     <center><h1>400 Bad Request</h1></center>
     <center>No required SSL certificate was sent</center>
     <hr><center>nginx/1.19.0</center>
     </body>
     </html>

   • Access with a client certificate

     curl --cacert ./ca.crt --cert ./client.crt --key ./client.key https://foo.bar.com

     Expected output:

     <!DOCTYPE html>
     <html>
     <head>
     <title>Welcome to nginx!</title>
     <style>
         body {
             width: 35em;
             margin: 0 auto;
             font-family: Tahoma, Verdana, Arial, sans-serif;
         }
     </style>
     </head>
     <body>
     <h1>Welcome to nginx!</h1>
     <p>If you see this page, the nginx web server is successfully installed and
     working. Further configuration is required.</p>
     
     <p>For online documentation and support please refer to
     <a href="http://nginx.org/">nginx.org</a>.<br/>
     Commercial support is available at
     <a href="http://nginx.com/">nginx.com</a>.</p>
     
     <p>Thank you for using nginx.

Configure HTTPS services to forward requests to backend containers over HTTPS

By default, the NGINX Ingress controller forwards requests to backend application containers over HTTP. If your application containers use HTTPS, you can use the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation to configure the NGINX Ingress controller to forward requests to backend application containers over HTTPS.

The following is an example NGINX Ingress configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backend-https
  annotations:
    # Note: You must specify that the backend service is an HTTPS service.
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - <your-host-name>
    secretName: <your-secret-cert-name>
  rules:
  - host: <your-host-name>
    http:
      paths:
      - path: /
        backend:
          service:
            name: <your-service-name>
            port: 
              number: <your-service-port>
        pathType: ImplementationSpecific

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: backend-https
  annotations:
    # Note: You must specify that the backend service is an HTTPS service.
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - <your-host-name>
    secretName: <your-secret-cert-name>
  rules:
  - host: <your-host-name>
    http:
      paths:
      - path: /
        backend:
          serviceName: <your-service-name>
          servicePort: <your-service-port>

Configure domain names to support regular expressions

In Kubernetes clusters, Ingress resources do not support regular expressions for domain names. However, you can use the nginx.ingress.kubernetes.io/server-alias annotation to enable this feature.

1. Deploy the following template. This example uses the regular expression ~^www\.\d+\.example\.com.

   Clusters that run Kubernetes 1.19 or later

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
     annotations:
       nginx.ingress.kubernetes.io/server-alias: '~^www\.\d+\.example\.com$, abc.example.com'
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: http-svc1
               port:
                 number: 80
           pathType: ImplementationSpecific
   EOF

   Clusters that run a Kubernetes version earlier than 1.19

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
     annotations:
       nginx.ingress.kubernetes.io/server-alias: '~^www\.\d+\.example\.com$, abc.example.com'
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: http-svc1
             servicePort: 80
    EOF

2. View the configuration of the corresponding NGINX Ingress controller.

   1. Run the following command to list the pods of the NGINX Ingress controller service.

      kubectl get pods -n kube-system | grep nginx-ingress-controller

      Expected output:

      nginx-ingress-controller-77cd987c4c-c****         1/1     Running   0          1h
      nginx-ingress-controller-77cd987c4c-x****         1/1     Running   0          1h

   2. Run the following command to retrieve the configuration of the NGINX Ingress controller. You can find the effective configuration in the `Server_Name` field.

      kubectl exec -n kube-system nginx-ingress-controller-77cd987c4c-c**** cat /etc/nginx/nginx.conf | grep -C3 "foo.bar.com"

      Expected output:

        # start server foo.bar.com
        server {
      --
        server {
          server_name foo.bar.com abc.example.com ~^www\.\d+\.example\.com$ ;
          listen 80  ;
          listen 443  ssl http2 ;
      --
      --
          }
        }
        # end server foo.bar.com

3. Run the following command to retrieve the IP address of the Ingress.

   kubectl get ing

   Expected output:

   NAME            HOSTS         ADDRESS          PORTS     AGE
   ingress-regex   foo.bar.com   101.37.XX.XX     80        11s

4. Run the following commands to test service access with different rules.

   Set IP_ADDRESS to the IP address obtained in the previous step.

   • Run the following command to access the service using Host: foo.bar.com.

     curl -H "Host: foo.bar.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the service using Host: www.123.example.com.

     curl -H "Host: www.123.example.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the service using Host: www.321.example.com.

     curl -H "Host: www.321.example.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

Configure domain names to support wildcards

In Kubernetes clusters, NGINX Ingress resources support wildcard domain names. For example, you can configure the wildcard domain name *.ingress-regex.com.

1. Deploy the following template to create an NGINX Ingress.

   Clusters that run Kubernetes 1.19 or later

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
   spec:
     rules:
   - host: *.ingress-regex.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: http-svc1
               port:
                 number: 80
           pathType: ImplementationSpecific
   EOF

   Clusters that run a Kubernetes version earlier than 1.19

   $ cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
   spec:
     rules:
    - host: *.ingress-regex.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: http-svc1
             servicePort: 80
    EOF

2. Run the following command to retrieve the configuration of the NGINX Ingress controller. You can find the effective configuration in the `Server_Name` field.

   kubectl exec -n kube-system <nginx-ingress-pod-name> cat /etc/nginx/nginx.conf | grep -C3 "*.ingress-regex.com"

   Note

   Replace nginx-ingress-pod-name with the actual NGINX Ingress pod name in your environment.

   Expected output:

   # start server *.ingress-regex.com
     server {
       server_name *.ingress-regex.com ;
       listen 80;
       listen [::]:80;
   ...
     }
     # end server *.ingress-regex.com

   Expected output in newer versions of the NGINX Ingress controller:

   ## start server *.ingress-regex.com
     server {
       server_name ~^(?<subdomain>[\w-]+)\.ingress-regex\.com$ ;
       listen 80;
       listen [::]:80;
   ...
     }
     ## end server *.ingress-regex.com

3. Run the following command to retrieve the IP address of the Ingress.

   kubectl get ing

   Expected output:

   NAME            HOSTS                 ADDRESS           PORTS     AGE
   ingress-regex   *.ingress-regex.com   101.37.XX.XX      80        11s

4. Run the following commands to test service access with different rules.

   Set IP_ADDRESS to the IP address obtained in the previous step.

   • Run the following command to access the service using Host: abc.ingress-regex.com.

     curl -H "Host: abc.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the service using Host: 123.ingress-regex.com.

     curl -H "Host: 123.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the service using Host: a1b1.ingress-regex.com.

     curl -H "Host: a1b1.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

Implement canary releases using annotations

Use cert-manager to apply for a free HTTPS certificate

cert-manager is an open source certificate management tool that provides and automatically renews HTTPS certificates in a cluster. The following example describes how to use cert-manager to obtain a free certificate and enable auto-renewal.

1. Run the following command to deploy cert-manager.

   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

2. Run the following command to check the pod status.

   kubectl get pods -n cert-manager

   Expected output:

   NAME                     READY   STATUS    RESTARTS   AGE
   cert-manager-1           1/1     Running   0          2m11s
   cert-manager-cainjector  1/1     Running   0          2m11s
   cert-manager-webhook     1/1     Running   0          2m10s

3. Run the following command to create a ClusterIssuer.

   cat <<-EOF | kubectl apply -f -
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-prod-http01
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       email: <your_email_name@gmail.com>  # Replace this with your email address.
       privateKeySecretRef:
         name: letsencrypt-http01
       solvers:
       - http01: 
           ingress:
             class: nginx
   EOF

4. Run the following command to check the status of the ClusterIssuer.

   kubectl get clusterissuer

   Expected output:

   NAME                         READY   AGE
   letsencrypt-prod-http01      True    17s

5. Run the following command to create an NGINX Ingress resource.

   Clusters that run Kubernetes 1.19 or later

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-tls
     annotations:
       kubernetes.io/ingress.class: "nginx"
       cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
   spec:
     tls:
     - hosts:
       - <your_domain_name>        # Replace this with your domain name.
       secretName: ingress-tls   
     rules:
     - host: <your_domain_name>    # Replace this with your domain name.
       http:
         paths:
         - path: /
           backend:
             service:
               name: <your_service_name>  # Replace this with your backend service name.
               port: 
                 number: <your_service_port>  # Replace this with your service port.
           pathType: ImplementationSpecific
   EOF

   Clusters that run a Kubernetes version earlier than 1.19

   cat <<-EOF | kubectl apply -f -
   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     name: ingress-tls
     annotations:
       kubernetes.io/ingress.class: "nginx"
       cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
   spec:
     tls:
     - hosts:
       - <your_domain_name>        # Replace this with your domain name.
       secretName: ingress-tls   
     rules:
     - host: <your_domain_name>    # Replace this with your domain name.
       http:
         paths:
         - path: /
           backend:
             serviceName: <your_service_name>  # Replace this with your backend service name.
             servicePort: <your_service_port>  # Replace this with your service port.
   EOF

   Note

   The domain name that you use to replace your_domain_name must meet the following conditions:

   • The domain name cannot exceed 64 characters.

   • Wildcard domain names are not supported.

   • The domain name must be accessible from the Internet over HTTP.

6. Run the following command to check the status of the certificate.

   kubectl get cert

   Expected output:

   NAME          READY   SECRET        AGE
   ingress-tls   True    ingress-tls   52m

   Note

   If the READY status is not True, you can run kubectl describe cert ingress-tls to check the certificate processing procedure.

7. Run the following command to check the Secret.

   kubectl get secret  ingress-tls

   Expected output:

   NAME          TYPE                DATA   AGE
   ingress-tls   kubernetes.io/tls   2      2m

8. Enter https://[website_domain_name] in a web browser to access the configured domain name.

HTTP-to-HTTPS redirect

The nginx.ingress.kubernetes.io/ssl-redirect annotation for NGINX Ingress redirects HTTP traffic to HTTPS. The following example shows the configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true" # Force HTTP traffic to be redirected to HTTPS.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true" # Force HTTP traffic to be redirected to HTTPS.