Container Service for Kubernetes:Advanced Ingress configurations

NGINX configuration

The methods that are used to configure the NGINX Ingress controller in ACK are fully compatible with the open source version of the component. For more information about the configuration methods, see NGINX Configuration.

Configure URL redirection

After you configure the NGINX Ingress controller, the NGINX Ingress controller forwards requests based on the full paths of the requests. For example, the NGINX Ingress controller forwards requests destined for /service1/api to /service1/api/ of backend pods. If the path of your backend service is /api, a 404 status code is returned because the path of the backend service is different from the requested path. In this case, you can configure the nginx.ingress.kubernetes.io/rewrite-target annotation to rewrite the requested path to the path that you want to use.

Create an NGINX Ingress by using the following template:

cat <<-EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo.bar.com
  namespace: default
  annotations:
    # URL redirection. 
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
    # If the version of the Ingress controller is 0.22.0 or later, you must use regular expressions to specify paths and use the regular expressions together with capture groups in the rewrite-target annotation. 
      - path: /svc(/|$)(.*)
        backend:
          service: 
            name: web1-service
            port: 
              number: 80
        pathType: ImplementationSpecific
EOF

cat <<-EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: foo.bar.com
  namespace: default
  annotations:
    # URL redirection. 
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
    # If the version of the Ingress controller is 0.22.0 or later, you must use regular expressions to specify paths and use the regular expressions together with capture groups in the rewrite-target annotation. 
      - path: /svc(/|$)(.*)
        backend:
          serviceName: web1-service
          servicePort: 80
EOF

1. Access the NGINX service.

   1. Run the following command to obtain the value of ADDRESS:

      kubectl  get  ingress

      Expected output:

      NAME           CLASS   HOSTS                ADDRESS          PORTS   AGE
      foo.bar.com    nginx   foo.bar.com        172.16.XX.XX       80      46m

   2. Run the following command. Replace ADDRESS with the IP address of the Ingress that you obtained:

      curl -k -H "Host: foo.bar.com"  http://<ADDRESS>/svc/foo

      Expected output:

      web1: /foo

Configure rewrite rules

The nginx.ingress.kubernetes.io/rewrite-target annotation can be used to configure basic rewrite rules. For more information, see Configure URL redirection.

To configure advanced rewrite rules, use the following annotations:

• nginx.ingress.kubernetes.io/server-snippet: This annotation adds custom configurations to the Server configuration block.

• nginx.ingress.kubernetes.io/configuration-snippet: This annotation adds custom configurations to the Location configuration block.

These annotations add custom code snippets to the NGINX server module of an Ingress. This allows you to extend and customize NGINX configurations to meet the business requirements in various scenarios.

Examples:

annotations:
     nginx.ingress.kubernetes.io/server-snippet: |
         rewrite ^/v4/(.*)/card/query http://foo.bar.com/v5/#!/card/query permanent;
     nginx.ingress.kubernetes.io/configuration-snippet: |
         rewrite ^/v6/(.*)/card/query http://foo.bar.com/v7/#!/card/query permanent;

Run the following command to query the NGINX configuration file of the NGINX Ingress controller:

kubectl exec nginx-ingress-controller-xxxxx --namespace kube-system -- cat /etc/nginx/nginx.conf /# Modify the pod name based on your business requirements

The following example shows the configuration of the nginx.conf file:

# start server foo.bar.com
    server {
        server_name foo.bar.com ;
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";
    # The server-snippet configuration. 
        rewrite ^/v4/(.*)/card/query http://foo.bar.com/v5/#!/card/query permanent;
        ...
    # The configuration-snippet configuration. 
      rewrite ^/v6/(.*)/card/query http://foo.bar.com/v7/#!/card/query permanent;
      ...
    }
    # end server foo.bar.com

In addition, snippet supports some global configurations. For more information, see server-snippet.

For more information about how to use rewrite rules, see Description of rewrite rules in the NGINX official document.

Configure a TLS certificate for Ingress rules

You can configure NGINX Ingress configurations to specify a TLS certificate for a website.

1. Prepare your certificate.

   Note

   The domain name associated with the certificate must be the same as the host that you specified. Otherwise, the NGINX Ingress controller cannot load the TLS certificate.

   1. Run the following command to generate a certificate file named tls.crt and a private key file named tls.key:

      openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=foo.bar.com/O=foo.bar.com"

   2. Run the following command to create a Kubernetes Secret.

      Use the certificate and private key to create a Secret named tls-test-ingress. The Secret is referenced when you create an Ingress.

      kubectl create secret tls tls-test-ingress --key tls.key --cert tls.crt

2. Run the following command to create an Ingress and use the tls field to reference the Secret that is created in the preceding step.

   Clusters that run Kubernetes 1.19 or later

   cat <<EOF | kubectl create -f - 
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: test-test-ingress
   spec:
     # Reference the TLS certificate. 
     tls:
     - hosts:
       - foo.bar.com # The domain name associated with the TLS certificate.  
       secretName: tls-test-ingress
     rules:
     - host: tls-test-ingress.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: web1-svc
               port:
                 number: 80
           pathType: ImplementationSpecific
   EOF

   Clusters that run Kubernetes versions earlier than 1.19

   cat <<EOF | kubectl create -f - 
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: test-test-ingress
   spec:
     # Reference the TLS certificate. 
     tls:
     - hosts:
       - foo.bar.com # The domain name associated with the TLS certificate. 
       secretName: tls-test-ingress
     rules:
     - host: tls-test-ingress.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: web1-svc
             servicePort: 80
   EOF

3. After the Ingress is created, you must modify the hosts file or specify the actual domain name before you can use TLS to secure data transmission.

   You can access the web1-svc Service by using https://tls-test-ingress.com/foo.

Configure mutual TLS authentication

The NGINX Ingress controller supports mutual TLS authentication. You may want to configure mutual TLS authentication between servers and clients to ensure the security of connections in specific scenarios.

1. Run the following command to create a self-signed certificate authority (CA) certificate:

   openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=Fern Cert Authority'

   Expected output:

   Generating a 4096 bit RSA private key
   .............................................................................................................++
   .....................................................................................++
   writing new private key to 'ca.key'

2. Create a server certificate.

   1. Run the following command to generate a file that is used to request a server certificate:

      openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=foo.bar.com'

      Expected output:

      Generating a 4096 bit RSA private key
      ................................................................................................................................++
      .................................................................++
      writing new private key to 'server.key'

   2. Run the following command to use the root certificate to sign the file and generate a server certificate:

      openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt

      Expected output:

      Signature ok
      subject=/CN=foo.bar.com
      Getting CA Private Key

3. Create a client certificate.

   1. Run the following command to generate a file that is used to request a client certificate:

      openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=Fern'

      Expected output:

      Generating a 4096 bit RSA private key
      .......................................................................................................................................................................................++
      ..............................................++
      writing new private key to 'client.key'
      -----

   2. Run the following command to use the root certificate to sign the file and generate a client certificate:

      openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt

      Expected output:

      Signature ok
      subject=/CN=Fern
      Getting CA Private Key

4. Run the following command to query the created certificate files:

   ls

   Expected output:

   ca.crt  ca.key  client.crt  client.csr  client.key  server.crt  server.csr  server.key

5. Run the following command to create a Secret based on the created CA certificate:

   kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt

   Expected output:

   secret/ca-secret created

6. Run the following command to create a Secret based on the created server certificate:

   kubectl create secret generic tls-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key

   Expected output:

   secret/tls-secret created

7. Run the following command to create an NGINX Ingress for testing purposes.

   Clusters that run Kubernetes 1.19 or later

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     annotations:
       nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
       nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret"
       nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
       nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
     name: nginx-test
     namespace: default
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - backend:
             service:
               name: http-svc
               port: 
                 number: 80
           path: /
           pathType: ImplementationSpecific
     tls:
     - hosts:
       - foo.bar.com
       secretName: tls-secret
   EOF

   Clusters that run Kubernetes versions earlier than 1.19

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     annotations:
       nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
       nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret"
       nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
       nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
     name: nginx-test
     namespace: default
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - backend:
             serviceName: http-svc
             servicePort: 80
           path: /
     tls:
     - hosts:
       - foo.bar.com
       secretName: tls-secret
   EOF

   Expected output:

   ingress.networking.k8s.io/nginx-test configured

8. Run the following command to query the IP address of the created Ingress:

   kubectl get ing

   The IP address in the ADDRESS field is the IP address of the Ingress, as shown in the following output:

   NAME         HOSTS                    ADDRESS         PORTS     AGE
   nginx-test   foo.bar.com              39.102.XX.XX    80, 443   4h42m

9. Run the following command to replace the IP address in the hosts file with the obtained IP address:

   echo "39.102.XX.XX  foo.bar.com" | sudo tee -a /etc/hosts

   Verify the configuration:

   • The client does not provide the client certificate when it accesses the server

     curl --cacert ./ca.crt  https://foo.bar.com

     Expected output:

     <html>
     <head><title>400 No required SSL certificate was sent</title></head>
     <body>
     <center><h1>400 Bad Request</h1></center>
     <center>No required SSL certificate was sent</center>
     <hr><center>nginx/1.19.0</center>
     </body>
     </html>

   • The client provides the client certificate when it accesses the server

     curl --cacert ./ca.crt --cert ./client.crt --key ./client.key https://foo.bar.com

     Expected output:

     <!DOCTYPE html>
     <html>
     <head>
     <title>Welcome to nginx!</title>
     <style>
         body {
             width: 35em;
             margin: 0 auto;
             font-family: Tahoma, Verdana, Arial, sans-serif;
         }
     </style>
     </head>
     <body>
     <h1>Welcome to nginx!</h1>
     <p>If you see this page, the nginx web server is successfully installed and
     working. Further configuration is required.</p>
     
     <p>For online documentation and support please refer to
     <a href="http://nginx.org/">nginx.org</a>.<br/>
     Commercial support is available at
     <a href="http://nginx.com/">nginx.com</a>.</p>
     
     <p><em>Thank you for using nginx.</em></p>
     </body>
     </html>

Forward HTTPS requests from Services to backend containers

By default, the NGINX Ingress controller forwards HTTP requests to backend containers. If your backend service uses HTTPS, you can configure the NGINX Ingress controller to forward HTTPS requests to backend containers by adding the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation.

The following template shows the configuration of the NGINX Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: backend-https
  annotations:
    # Note: You must set the backend protocol to HTTPS. 
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - <your-host-name>
    secretName: <your-secret-cert-name>
  rules:
  - host: <your-host-name>
    http:
      paths:
      - path: /
        backend:
          service:
            name: <your-service-name>
            port: 
              number: <your-service-port>
        pathType: ImplementationSpecific

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: backend-https
  annotations:
    # Note: You must set the backend protocol to HTTPS. 
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  tls:
  - hosts:
    - <your-host-name>
    secretName: <your-secret-cert-name>
  rules:
  - host: <your-host-name>
    http:
      paths:
      - path: /
        backend:
          serviceName: <your-service-name>
          servicePort: <your-service-port>

Use regular expressions to specify domain names

By default, you cannot use regular expressions to specify domain names when you configure Ingresses for Kubernetes clusters. However, you can enable Ingresses to support regular expressions by adding the nginx.ingress.kubernetes.io/server-alias annotation.

1. Create an NGINX Ingress. In the following example, the domain name is set to the regular expression ~^www\.\d+\.example\.com.

   Clusters that run Kubernetes 1.19 or later

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
     annotations:
       nginx.ingress.kubernetes.io/server-alias: '~^www\.\d+\.example\.com$, abc.example.com'
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: http-svc1
               port:
                 number: 80
           pathType: ImplementationSpecific
   EOF

   Clusters that run Kubernetes versions earlier than 1.19

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
     annotations:
       nginx.ingress.kubernetes.io/server-alias: '~^www\.\d+\.example\.com$, abc.example.com'
   spec:
     rules:
     - host: foo.bar.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: http-svc1
             servicePort: 80
    EOF

2. Query the configuration of the NGINX Ingress controller.

   1. Run the following command to query the pods that are provisioned for the NGINX Ingress controller:

      kubectl get pods -n kube-system | grep nginx-ingress-controller

      Expected output:

      nginx-ingress-controller-77cd987c4c-c****         1/1     Running   0          1h
      nginx-ingress-controller-77cd987c4c-x****         1/1     Running   0          1h

   2. Run the following command to query the configuration of the NGINX Ingress controller. The output shows the effective configurations, such as the Server_Name field in this example.

      kubectl exec -n kube-system nginx-ingress-controller-77cd987c4c-c**** cat /etc/nginx/nginx.conf | grep -C3 "foo.bar.com"

      Expected output:

        # start server foo.bar.com
        server {
      --
        server {
          server_name foo.bar.com abc.example.com ~^www\.\d+\.example\.com$ ;
          listen 80  ;
          listen 443  ssl http2 ;
      --
      --
          }
        }
        # end server foo.bar.com

3. Run the following command to query the IP address of the Ingress:

   kubectl get ing

   Expected output:

   NAME            HOSTS         ADDRESS          PORTS     AGE
   ingress-regex   foo.bar.com   101.37.XX.XX     80        11s

4. Run the following commands to access a Service by using different Ingress rules.

   Replace IP_ADDRESS with the IP address obtained in the previous step.

   • Run the following command to access the Service through Host: foo.bar.com:

     curl -H "Host: foo.bar.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the Service through Host: www.123.example.com:

     curl -H "Host: www.123.example.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the Service through Host: www.321.example.com:

     curl -H "Host: www.321.example.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

Specify wildcard domain names

In Kubernetes clusters, NGINX Ingresses support wildcard domain names. For example, you can specify the wildcard domain name *. ingress-regex.com for an Ingress.

1. Create an NGINX Ingress by using the following template.

   Clusters that run Kubernetes 1.19 or later

   cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
   spec:
     rules:
   - host: *.ingress-regex.com
       http:
         paths:
         - path: /foo
           backend:
             service:
               name: http-svc1
               port:
                 number: 80
           pathType: ImplementationSpecific
   EOF

   Clusters that run Kubernetes versions earlier than 1.19

   $ cat <<-EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: ingress-regex
     namespace: default
   spec:
     rules:
    - host: *.ingress-regex.com
       http:
         paths:
         - path: /foo
           backend:
             serviceName: http-svc1
             servicePort: 80
    EOF

2. Run the following command to query the configuration of the NGINX Ingress controller. The output shows the effective configurations, such as the Server_Name field in this example.

   kubectl exec -n kube-system <nginx-ingress-pod-name> cat /etc/nginx/nginx.conf | grep -C3 "*.ingress-regex.com"

   Note

   Replace nginx-ingress-pod-name with the name of the pod that is provisioned for the NGINX Ingress controller.

   Expected output:

   # start server *.ingress-regex.com
     server {
       server_name *.ingress-regex.com ;
       listen 80;
       listen [::]:80;
   ...
     }
     # end server *.ingress-regex.com

   Expected output when the latest version of the NGINX Ingress controller is used:

   ## start server *.ingress-regex.com
     server {
       server_name ~^(?<subdomain>[\w-]+)\.ingress-regex\.com$ ;
       listen 80;
       listen [::]:80;
   ...
     }
     ## end server *.ingress-regex.com

3. Run the following command to query the IP address of the Ingress:

   kubectl get ing

   Expected output:

   NAME            HOSTS                 ADDRESS           PORTS     AGE
   ingress-regex   *.ingress-regex.com   101.37.XX.XX      80        11s

4. Run the following commands to access a Service by using different Ingress rules.

   Replace IP_ADDRESS with the IP address obtained in the previous step.

   • Run the following command to access the Service through Host: abc.ingress-regex.com:

     curl -H "Host: abc.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the Service through Host: 123.ingress-regex.com:

     curl -H "Host: 123.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

   • Run the following command to access the Service through Host: a1b1.ingress-regex.com:

     curl -H "Host: a1b1.ingress-regex.com" <IP_ADDRESS>/foo

     Expected output:

     /foo

Use annotations to perform canary releases

Use cert-manager to apply for a free TLS certificate

cert-manager is an open source tool used to manage cloud-native certificates. You can use cert-manager in Kubernetes clusters to apply for TLS certificates and enable auto-renewal for the certificates. This section describes how to apply for a free certificate and enable auto-renewal for the certificate by using cert-manager.

1. Run the following command to deploy cert-manager:

   kubectl apply -f https://raw.githubusercontent.com/AliyunContainerService/serverless-k8s-examples/master/cert-manager/ask-cert-manager.yaml

   Note

   The YAML template referenced in the preceding command can be used to deploy cert-manager only in ACK clusters. For more information about how to deploy cert-manager in ACK clusters, see Use cert-manager to apply for a free TLS certificate.

2. Run the following command to query the status of the pods that are created for cert-manager:

   kubectl get pods -n cert-manager

   Expected output:

   NAME                     READY   STATUS    RESTARTS   AGE
   cert-manager-1           1/1     Running   0          2m11s
   cert-manager-cainjector  1/1     Running   0          2m11s
   cert-manager-webhook     1/1     Running   0          2m10s

3. Run the following command to create a ClusterIssuer:

   cat <<EOF | kubectl apply -f -
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-prod-http01
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       email: <your_email_name@gmail.com>  # Replace the value with your email address. 
       privateKeySecretRef:
         name: letsencrypt-http01
       solvers:
       - http01: 
           ingress:
             class: nginx
   EOF

4. Run the following command to query the created ClusterIssuer:

   kubectl get clusterissuer

   Expected output:

   NAME                         READY   AGE
   letsencrypt-prod-http01      True    17s

5. Run the following command to create an NGINX Ingress:

   Clusters that run Kubernetes 1.19 or later

   cat <<EOF | kubectl apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ingress-tls
     annotations:
       kubernetes.io/ingress.class: "nginx"
       cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
   spec:
     tls:
     - hosts:
       - <your_domain_name>        # Replace the value with your domain name. 
       secretName: ingress-tls   
     rules:
     - host: <your_domain_name>    # Replace the value with your domain name. 
       http:
         paths:
         - path: /
           backend:
             service:
               name: <your_service_name>  # Replace the value with the Service name. 
               port: 
                 number: <your_service_port>  # Replace the value with the Service port. 
           pathType: ImplementationSpecific
   EOF

   Clusters that run Kubernetes versions earlier than 1.19

   cat <<EOF | kubectl apply -f -
   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     name: ingress-tls
     annotations:
       kubernetes.io/ingress.class: "nginx"
       cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
   spec:
     tls:
     - hosts:
       - <your_domain_name>        # Replace the value with your domain name. 
       secretName: ingress-tls   
     rules:
     - host: <your_domain_name>    # Replace the value with your domain name. 
       http:
         paths:
         - path: /
           backend:
             serviceName: <your_service_name>  # Replace the value with the Service name. 
             servicePort: <your_service_port>  # Replace the value with the Service port. 
   EOF

   Note

   The domain name that you use to replace your_domain_name in the template must meet the following conditions:

   • The domain name cannot exceed 64 characters in length.

   • The domain name cannot be a wildcard domain name.

   • The domain name is accessible from the Internet over HTTP.

6. Run the following command to query the certificate:

   kubectl get cert

   Expected output:

   NAME          READY   SECRET        AGE
   ingress-tls   True    ingress-tls   52m

   Note

   If the value in the READY field is not True, you can query the status of the certificate by running the kubectl describe cert ingress-tls command.

7. Run the following command to query the Secret of the certificate:

   kubectl get secret  ingress-tls

   Expected output:

   NAME          TYPE                DATA   AGE
   ingress-tls   kubernetes.io/tls   2      2m

8. You can enter https:[website domain name] into the address bar of your browser to access the specified domain name.