All Products
Search
Document Center

Container Service for Kubernetes:Associate an exclusive EIP with a pod

Last Updated:Jun 12, 2024

In most cases, pods in Kubernetes clusters use private IP addresses However, in some scenarios, you may want to assign an exclusive public IP address to a pod to allow the pod to separately communicate with external networks without being affected by the traffic of other pods. You can use the Terway and ack-extend-network-controller plug-ins to directly associate an elastic IP address (EIP) with a pod. This way, the pod can communicate with external networks over an exclusive data egress.

Prerequisites

A Container Service for Kubernetes (ACK) managed cluster or an ACK dedicated cluster is created. The cluster uses Terway as the network plug-in. For more information, see Create an ACK managed cluster or Create an ACK dedicated cluster. For more information about Terway, see Work with Terway.

Important

Since November 2023, Terway has discontinued updates of the EIP feature. This feature was removed in v1.7.0. We recommend that you use the ack-extend-network-controller plug-in to associate an exclusive EIP with a pod. For more information, see Migrate EIPs from Terway to ack-extend-network-controller.

Limits

  • You can associate EIPs with pods that run on Elastic Compute Service (ECS) instances. You cannot associate EIPs with pods that run on elastic container instances.

  • Before you use EIPs, you must understand the limits of EIPs. For more information, see Limits.

Billing rules

You are not charged for enabling the Terway and ack-extend-network-controller plug-ins. However, you may be charged fees for the cloud resources that you use. For more information about the billing of Alibaba Cloud services used by ACK, see Cloud service billing.

Background Information

In most cases, SNAT and EIPs are used to enable pods to communicate with external networks in Terway mode. Public traffic is usually forwarded by a LoadBalancer Service to backend pods. However, you may want to assign an exclusive public IP address to a pod in the following scenarios:

  • The pod exposes random ports to the Internet in the scenario of game servers and audio conferences that use UDP. For example, if Real Time Streaming Protocol (RTSP) is used, different ports are used for different clients.

  • The pod has to compete with other pods for a public IP address to access the Internet. In this case, the pod requires an exclusive public IP address.

Step 1: Configure RAM permissions that are required for associating EIPs

ack-extend-network-controller

  • Grant RAM permissions to an ACK managed cluster or ACK dedicated cluster

    1. Create a custom policy and add the following content to the policy. For more information, see the Step 1: Create a custom policy section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.

           {
            "Effect": "Allow",
            "Action": [
              "vpc:DescribeVSwitches",
              "vpc:AllocateEipAddress",
              "vpc:AllocateEipAddressPro",
              "vpc:DescribeEipAddresses",
              "vpc:AssociateEipAddress",
              "vpc:UnassociateEipAddress",
              "vpc:ReleaseEipAddress",
              "vpc:AddCommonBandwidthPackageIp",
              "vpc:RemoveCommonBandwidthPackageIp",
              "vpc:TagResources",
              "ecs:DescribeNetworkInterfaces"
            ],
            "Resource": [
              "*"
            ],
            "Condition": {}
          }
    2. Grant permissions to the worker Resource Access Management (RAM) role of the cluster. For more information, see Attach the custom policy to the worker RAM role.

  • Grant RAM permissions to an ACK Serverless cluster

    If your cluster is an ACK Serverless cluster, you need to generate an AccessKey pair for the RAM user. For more information, see Create a RAM user and Create custom policies.

Terway

Terway requires EIP management permissions to apply for and associate EIPs. Therefore, the custom policy also provides permissions related to EIPs.

  • Grant RAM permissions to an ACK managed cluster or an ACK Basic cluster created in June 2020 or later.

    1. Create a custom policy and add the following content to the policy. For more information, see the Step 1: Create a custom policy section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.

           {
            "Effect": "Allow",
            "Action": [
              "vpc:DescribeVSwitches",
              "vpc:AllocateEipAddress",
              "vpc:DescribeEipAddresses",
              "vpc:AssociateEipAddress",
              "vpc:UnassociateEipAddress",
              "vpc:ReleaseEipAddress",
              "vpc:AddCommonBandwidthPackageIp",
              "vpc:RemoveCommonBandwidthPackageIp"
            ],
            "Resource": [
              "*"
            ],
            "Condition": {}
          }
    2. Grant permissions to the worker RAM role of the cluster. For more information, see the Step 2: Attach the custom policy to the worker RAM role section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.

  • Grant RAM permissions to an ACK dedicated cluster or an ACK managed cluster created before June 2020.

    1. Create a custom policy and add the following content to the policy. For more information, see the Step 1: Create a custom policy section of the "[Product Changes] Permissions of the worker RAM role of ACK managed clusters are revoked" topic.

           {
            "Effect": "Allow",
            "Action": [
              "vpc:DescribeVSwitches",
              "vpc:AllocateEipAddress",
              "vpc:DescribeEipAddresses",
              "vpc:AssociateEipAddress",
              "vpc:UnassociateEipAddress",
              "vpc:ReleaseEipAddress",
              "vpc:AddCommonBandwidthPackageIp",
              "vpc:RemoveCommonBandwidthPackageIp"
            ],
            "Resource": [
              "*"
            ],
            "Condition": {}
          }
    2. On the Permissions tab of the AliyunCSManagedNetworkRole page, click Grant Permission. In the Grant Permission panel, click Policy and select the custom policy that you created in the previous step.

    3. Click Grant permission.

    4. Click Close.

Step 2: Install or update the plug-in

Warning

You cannot use the ack-extend-network-controller and Terway plug-ins to associate EIPs with pods at the same time. Otherwise, an EIP may be associated with multiple pods, which causes EIP association failures. We recommend that you use the ack-extend-network-controller plug-in.

Compare Terway and ack-extend-network-controller

Item

ack-extend-network-controller

Terway

Supported cluster types

  • ACK managed cluster

  • ACK dedicated cluster

  • ACK Serverless cluster

  • ACK managed cluster

  • ACK dedicated cluster

Static EIPs

Supported

Not supported

Phase

The ack-extend-network-controller plug-in associates EIPs with pods during the pod IP address allocation phase.

The Terway plug-in associates EIPs with pods when the plug-in applies configurations.

Supported versions

v0.2.0 and later

v1.0.10.280-gdc2cb6c-aliyun or later versions that are earlier than v1.7.0

Install or update the plug-in

If your cluster uses ack-extend-network-controller, go to the Marketplace page in the ACK console, install ack-extend-network-controller, and then enable the EIP controller. If your cluster uses Terway, update Terway to v1.0.10.280-gdc2cb6c-aliyun or later.

Install ack-extend-network-controller

  1. Log on to the ACK console. In the left-side navigation pane, choose Marketplace > Marketplace.

  2. On the Marketplace page, search for ack-extend-network-controller and then click the plug-in.

  3. In the upper-right corner of the chart details page, click Deploy. The Deploy panel appears.

  4. In the Basic Information step, select a cluster and a namespace, and click Next.

    1. In the Parameters step, select a version, configure the parameters, and then click OK.

      The following table describes the parameters.

      Parameter

      Type

      Required

      Description

      clusterID

      string

      Yes

      The ID of the cluster.

      regionID

      string

      Yes

      The ID of the region in which the cluster is deployed.

      enableControllers

      []string

      Yes

      The controllers that you want to use. Specify eip to enable the EIP feature.

      networkController.eipController.maxConcurrentReconciles

      int

      No

      The number of concurrent EIP controllers.

      networkController.eipController.garbageCollectionPeriodInMinutes

      int

      No

      The interval at which the EIP controller clears static EIPs.

      customStatefulWorkloadKinds

      []string

      No

      The custom stateful controller.

      credential.accessKey

      string

      Yes

      The AccessKey ID of the account that is used to associate EIPs.

      Important

      This parameter stores sensitive information in a Kubernetes Secret. We recommend that you do not use this parameter.

      credential.accessSecret

      string

      Yes

      The AccessKey secret of the account that is used to associate EIPs.

      Important

      This parameter stores sensitive information in a Kubernetes Secret. We recommend that you do not use this parameter.

      Sample code:

      clusterID: "c11ba338192xxxxxxx"  
      regionID: "cn-hangzhou"      
      vpcID: "vpc-bp1rkq0zxxxxxx" 
      enableControllers:
        - eip              
      
      networkController:
        eipController:
          maxConcurrentReconciles: 1   
          garbageCollectionPeriodInMinutes: 1  
        customStatefulWorkloadKinds:           
        - foo
      
      credential:                             
        accessKey: ""
        accessSecret: ""

For more information about how to update the version and parameters of ack-extend-network-controller, see App Marketplace.

Update Terway

If your cluster uses Terway, update the Terway plug-in (terway-eni or terway-eniip) to a version that supports the EIP feature. For more information about how to update Terway, see Manage components.

Note

Terway must be updated to v1.0.10.280-gdc2cb6c-aliyun or a later version that is earlier than v1.7.0. For more information about Terway, see Terway.

Step 3: Enable the EIP feature

Annotations for enabling the EIP feature

ACK allows you to use annotations to enable the EIP feature. You can use the automatic EIP allocation feature or manually associate static EIPs with pods.

Automatically associate an EIP with a pod

Important

If you use the automatic EIP allocation feature, the system may repeatedly apply for and release EIPs. For example, this issue occurs when the pod is recreated or the Container Network Interface (CNI) plug-in fails to configure the network. To prevent this issue, you can manually associate an EIP with the pod by specifying the ID of the EIP.

Annotation

Description

k8s.aliyun.com/pod-with-eip

Specifies whether to automatically create an EIP and associate the EIP with the pod. Valid values:

  • true

  • false

k8s.aliyun.com/eci-with-eip (compatible)

k8s.aliyun.com/eip-bandwidth

The maximum bandwidth of the EIP. Unit: Mbit/s. For more information, see Apply for an EIP.

k8s.aliyun.com/eip-internet-charge-type

The metering method of the EIP. Valid values:

  • PayByTraffic: Fees are charged based on data transfer.

  • PayByBandwidth: Fees are charged based on bandwidth usage.

For more information, see Apply for an EIP.

Note
  • Default value: PayByTraffic.

k8s.aliyun.com/eip-charge-type (compatible)

k8s.aliyun.com/eip-instance-charge-type

The billing method of the EIP. Valid values:

  • PrePaid: subscription.

  • PostPaid: pay-as-you-go.

For more information, see Apply for an EIP.

Note

Only ack-extend-network-controller supports this annotation.

k8s.aliyun.com/eip-common-bandwidth-package-id

The EIP bandwidth plan that you want to use.

Note

To use this annotation, update Terway to v1.2.3 or later. This annotation can be used with any version of the ack-extend-network-controller plug-in.

k8s.aliyun.com/eip-isp

The line type of the EIP. Valid values:

  • BGP: BGP (Multi-ISP) lines.

  • BGP_PRO: BGP (Multi-ISP) Pro lines.

For more information, see Apply for an EIP.

Note

To use this annotation, update Terway to v1.2.3 or later. This annotation can be used with any version of the ack-extend-network-controller plug-in.

k8s.aliyun.com/eip-public-ip-address-pool-id

The IP address pool. For more information, see Create and manage an IP address pool.

Note

Only ack-extend-network-controller supports this annotation.

k8s.aliyun.com/eip-resource-group-id

The resource group to which the EIP belongs. For more information, see Apply for an EIP.

Note

Only ack-extend-network-controller v0.4.0 and later support this annotation.

k8s.aliyun.com/eip-name

The name of the EIP. For more information, see Apply for an EIP.

Note

Only ack-extend-network-controller v0.6.0 and later support this annotation.

k8s.aliyun.com/eip-description

The description of the EIP. For more information, see Apply for an EIP.

Note

Only ack-extend-network-controller v0.6.0 and later support this annotation.

k8s.aliyun.com/eip-security-protection-types

The protection type of the EIP. Separate multiple protection types with commas (,). For more information, see Apply for an EIP

Note

Only ack-extend-network-controller v0.9.0 and later support this annotation.

Specify the ID of an EIP

You can associate an existing EIP with a pod by specifying the ID of the EIP. The pod annotations do not modify the configurations of the EIP. The annotations only associate the EIP with the specified pod.

Important

This feature is not applicable to controllers that manage multiple replicated pods. Make sure that the EIP is used by only one pod. We recommend that you use this feature only in StatefulSets.

Annotation

Description

k8s.aliyun.com/pod-eip-instanceid

The ID of the EIP, such as eip-bp14qxxxxxxx.

Important

Do not associate an EIP with a pod that uses a name different than the PodEIP resource. The EIP controller disassociates an EIP from a pod when the pod terminates. During the disassociation process, do not use the EIP in other pods. You can determine whether the EIP has been disassociated from the pod by checking whether a PodEIP resource that uses the same name as the pod exists.

k8s.aliyun.com/eci-eip-instanceid (compatible)

Annotation for configuring EIP release policies

If you configure a pod to use a static EIP, the pod still uses the same EIP after the pod is recreated. You can use this method together with the automatic EIP allocation feature to associate static EIPs with the pods of StatefulSets.

Important
  • Only ack-extend-network-controller provides this capability and you must use a stateful pod controller.

  • You can specify an EIP by its ID. The specified EIP will not be released.

Annotation

Description

k8s.aliyun.com/pod-eip-release-strategy

The EIP release policy. Valid values:

  • Follow: follows the lifecycle of the pod that is associated with the EIP. This is the default value.

  • Never: does not release the EIP. You need to manually release the EIP that is associated with the pod if you no longer require the EIP.

You can directly configure the expiration time of the EIP. For example, if you set the expiration time of the EIP to 5m30s, the EIP that is associated with the pod is released 5.5 minutes after the pod is deleted. Time expressions written in Go are supported.

Enable the EIP feature

Enable the EIP feature in a cluster that uses ack-extend-network-controller

Note

ack-extend-network-controller needs to access OpenAPI Explorer to create resources. You need to first grant the required permissions to the worker role of the cluster in the RAM console. Then, install ack-extend-network-controller on the Marketplace page of the ACK console, and add annotations to create and associate an EIP with the specified pod.

Add annotations to create and associate an EIP with the specified pod.

You can add annotations to the configurations of a pod to create or associate an EIP with the pod. For more information about annotations, see the FAQ section of this topic.

  1. Create an application. Then, create and associate an EIP with the specified pod.

    Deployments

    Create a Deployment based on the following sample code. The Deployment allocates an EIP to each pod. The bandwidth of each EIP is 5 Mbit/s.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: example
      labels:
        app: example
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: example
      template:
        metadata:
          labels:
            app: example
          annotations:
            k8s.aliyun.com/pod-with-eip: "true"
            k8s.aliyun.com/eip-bandwidth: "5"
        spec:
          readinessGates:
          - conditionType: "k8s.aliyun.com/eip"
          containers:
          - name: example
            image: nginx

    After the pods are created, run the following command to query the podeips.alibabacloud.com resources that use the same names as the pods to track the EIP allocation information.

    kubectl get podeip -n <namespace> <podname>  -o yaml

    Expected output:

    apiVersion: alibabacloud.com/v1beta1
    kind: PodEIP
    metadata:
      creationTimestamp: "2023-12-15T04:25:37Z"
      finalizers:
      - podeip-controller.alibabacloud.com/finalizer
      generation: 1
      name: example-xxx
      namespace: default
      resourceVersion: "222800"
      uid: 43xxx-f1xx-4xxx-b3xx-969faxxx
    spec:
      allocationID: eip-2ze2qe8zsxxx
      allocationType:
        releaseStrategy: Follow
        type: Auto
    status:
      eipAddress: 39.102.XX.XX
      internetChargeType: PayByTraffic
      isp: BGP
      networkInterfaceID: eni-2zeagv8f3xxxx
      podLastSeen: "2023-12-15T05:18:47Z"
      privateIPAddress: 192.168.XX.XX
      resourceGroupID: rg-acfmwxxxxxsq
      status: InUse

    StatefulSets

    Create a StatefulSet based on the following sample code. The StatefulSet creates two pods and allocates an EIP to each pod. The EIPs are automatically released 10 minutes after the pods are deleted.

    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: example
      labels:
        app: example
    spec:
      serviceName: "example"
      replicas: 2
      selector:
        matchLabels:
          app: example
      template:
        metadata:
          labels:
            app: example
          annotations:
            k8s.aliyun.com/pod-with-eip: "true"
            k8s.aliyun.com/pod-eip-release-strategy: "10m"
        spec:
          containers:
          - name: example
            image: nginx

    After the pods are created, run the following command to query the podeips.alibabacloud.com resources that use the same names as the pods to track the EIP allocation information.

    kubectl get podeip -n <namespace> -o yaml

    Expected output:

    apiVersion: v1
    items:
    - apiVersion: alibabacloud.com/v1beta1
      kind: PodEIP
      metadata:
        creationTimestamp: "2023-12-15T03:28:01Z"
        finalizers
        - podeip-controller.alibabacloud.com/finalizer
        generation: 1
        name: example-0
        namespace: default
        resourceVersion: "227221"
        uid: 79954xx-17xx-4dxx-b7xx-15b84xxx
      spec:
        allocationID: eip-2ze08metxxx
        allocationType:
          releaseAfter: 10m
          releaseStrategy: TTL
          type: Auto
      status:
        eipAddress: 39.105.XX.XX
        internetChargeType: PayByTraffic
        isp: BGP
        networkInterfaceID: eni-2ze4tkg4xxx
        podLastSeen: "2023-12-15T05:31:34Z"
        privateIPAddress: 192.168.XX.XX
        resourceGroupID: rg-acfmwxxx
        status: InUse
    - apiVersion: alibabacloud.com/v1beta1
      kind: PodEIP
      metadata:
        creationTimestamp: "2023-12-15T03:28:03Z"
        finalizers:
        - podeip-controller.alibabacloud.com/finalizer
        generation: 1
        name: example-1
        namespace: default
        resourceVersion: "227222"
        uid: 1339xxxe7-97xx-46xx-9bxx-537690xxx
      spec:
        allocationID: eip-2zetwhffqxxx
        allocationType:
          releaseAfter: 10m
          releaseStrategy: TTL
          type: Auto
      status:
        eipAddress: 39.105.XX.XX
        internetChargeType: PayByTraffic
        isp: BGP
        networkInterfaceID: eni-2zeagv8f3wxxx
        podLastSeen: "2023-12-15T05:31:34Z"
        privateIPAddress: 192.168.XX.XX
        resourceGroupID: rg-acfmwqnwxxx
        status: InUse
    - apiVersion: alibabacloud.com/v1beta1
      kind: PodEIP
      metadata:
        creationTimestamp: "2023-12-15T04:25:37Z"
        finalizers:
        - podeip-controller.alibabacloud.com/finalizer
        generation: 1
        name: example-5bxxx-9xx
        namespace: default
        resourceVersion: "227220"
        uid: 43cdfxxx-f1xx-42xx-b3xx-969fxxx
      spec:
        allocationID: eip-2ze2qe8zsmnxxx
        allocationType:
          releaseStrategy: Follow
          type: Auto
      status:
        eipAddress: 39.102.XX.XX
        internetChargeType: PayByTraffic
        isp: BGP
        networkInterfaceID: eni-2zeagv8f3wxxx
        podLastSeen: "2023-12-15T05:31:34Z"
        privateIPAddress: 192.168.XX.XX
        publicIpAddressPoolID: pippool-2ze498cxxx
        resourceGroupID: rg-acfmwqnxxx
        status: InUse
    kind: List
    metadata:
      resourceVersion: ""

    After the pods of the StatefulSet are deleted, the EIPs are retained for 10 minutes and then released. If you create a pod with the same name during this period, the pod uses the same EIP.

  2. Verify the configuration.

    After the state of a pod changes to Running, you can check the value of the k8s.aliyun.com/allocated-eipAddress annotation of the pod to identify the EIP used by the pod. Then, you can use the EIP to access the pod.

    image.png

Enable the EIP feature in a cluster that uses Terway

  1. Configure Terway to support EIPs.

    1. Run the following command to modify the eni_conf ConfigMap of Terway:

      kubectl edit cm eni-config -n kube-system
    2. Add the following content to the eni_conf ConfigMap:

      "enable_eip_pool": "true"
      Note

      If you want to disassociate the existing EIP before you associate another EIP, add "allow_eip_rob": "true" to the eni_conf ConfigMap.

    3. After you modify the ConfigMap, press the ESC key. Then, enter :wq! and press the ENTER key to save the modified file and exit the edit mode.

    4. Run the following command to redeploy Terway:

      kubectl delete pod -n kube-system -l app=terway-eniip # If the terway-eni plug-in is installed, replace terway-eniip with terway-eni.

  2. Add annotations to create and associate an EIP with a pod.

    You can add annotations to the configurations of a pod to create or associate an EIP with the pod. For more information about the limits of EIPs, see Limits.

    Add annotations based on the scenario.

  3. Automatically create and associate an EIP with a pod

    Add the following annotations to automatically create and associate an EIP with a pod and specify the maximum bandwidth of the EIP:

    • k8s.aliyun.com/pod-with-eip: "true": the ID of an EIP.

    • k8s.aliyun.com/pod-with-eip: "true": the maximum bandwidth of the EIP. The default value is 5 Mbit/s, which equals the default value defined in the EIP service.

    The following sample YAML template provides an example:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment-basic
      labels:
        app: nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          annotations:
            k8s.aliyun.com/pod-with-eip: "true" # Specifies that an EIP is automatically created and associated with each NGINX pod. 
            k8s.aliyun.com/pod-with-eip: "true" # Specifies the maximum bandwidth of the EIP. The default value is 5 Mbit/s, which equals the default value defined in the EIP service. 
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx

    Specify an existing EIP

    Important
    • An EIP cannot be associated with multiple pods. Therefore, you cannot use this annotation for pods that are created by Deployments or StatefulSets.

    • By default, if an EIP is already associated with a pod, you cannot create a new EIP for the pod. If you want to disassociate the existing EIP and create another EIP, add the "allow_eip_rob": "true" setting to the eni_conf ConfigMap as described in the preceding step.

    • You can specify an existing EIP only if the application runs in one replicated pod. For example, when you create a Deployment, make sure that the value of Replicas is 1 so that you can associate an existing EIP.

    Add the following annotation to specify the ID of an existing EIP and associate the EIP with a pod:

    k8s.aliyun.com/pod-eip-instanceid: "<youreipInstanceId>"
  4. Verify the configuration.

    After a pod enters the Running state, you can check the value of the k8s.aliyun.com/allocated-eipAddress annotation of the pod to identify the EIP used by the pod. Then, you can use the EIP to access the pod.

FAQ

Why does the state of a pod change to Ready?

After the controller allocates an EIP to a pod, the state of the pod may change to Ready before the pod is associated with the EIP. You can use one of the following methods to control when the state of the pod changes to Ready.

Configure readiness gates for the pod

Note

Only ack-extend-network-controller supports pod readiness gates.

After you configure readiness gates for the pod, the controller sets pod conditions only after an EIP is associated with the pod. The state of the pod does not change to Ready before the pod is associated with an EIP.

kind: Pod
...
spec:
  readinessGates:
  - conditionType: "k8s.aliyun.com/eip"
status:
  conditions:
  - lastProbeTime: "2022-12-12T03:45:48Z"
    lastTransitionTime: "2022-12-12T03:45:48Z"
    reason: Associate eip succeed
    status: "True"
    type: k8s.aliyun.com/eip
...

Configure an init container for the pod

You can configure an init container for the pod and use the init container to check whether the pod is associated with an EIP. The following sample code shows how to configure an init container.

apiVersion: v1
kind: Pod
metadata:
  name: example
  annotations:
    k8s.aliyun.com/pod-with-eip: "true"
spec:
  containers:
  - name: example
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init
    image: busybox:1.28
    command: ['timeout', '-t' ,'60', 'sh','-c', "until grep -E '^k8s.aliyun.com\\/allocated-eipAddress=\\S?[0-9]+\\S?' /etc/podinfo/annotations; do echo waiting for annotations; sleep 2; done"]
    volumeMounts:
      - name: podinfo
        mountPath: /etc/podinfo
  volumes:
    - name: podinfo
      downwardAPI:
        items:
          - path: "labels"
            fieldRef:
              fieldPath: metadata.labels
          - path: "annotations"
            fieldRef:
              fieldPath: metadata.annotations