All Products
Search
Document Center

Container Service for Kubernetes:Manage multi-cluster Services by using the CLI

Last Updated:Jul 11, 2023

The multi-cluster Services (MCS) feature allows you to access Services across Kubernetes clusters without the need to create load balancers. This topic describes how to create ServiceExports and ServiceImports on a master instance to manage multi-cluster Services. This allows you to access Services across Container Service for Kubernetes (ACK) clusters.

Prerequisites

Overview

The MCS feature allows you to access Services across ACK clusters. The following figure shows the architecture of the MCS feature.

多集群服务
  1. The administrator creates application-related resources, such as namespaces, Deployments, and Services, on the master instance through Connection 0, and creates application propagation policies to distribute the resources to ACK Cluster 1 through Connection 1.

  2. The administrator creates a ServiceExport and a ServiceImport on the master instance through Connection 0, and creates application propagation policies to distribute the ServiceExport to ACK Cluster 1 and the ServiceImport to ACK Cluster 2.

  3. The client pod in ACK Cluster 2 can access Service 1 in ACK Cluster 1.

    Note

    In the preceding Step 1 and Step 2, a master instance is used to distribute namespaces, Deployments, Services, a ServiceExport, and a ServiceImport to ACK Cluster 1 and ACK Cluster 2. You can also directly create these resources in ACK Cluster 1 and ACK Cluster 2 for cross-cluster Service access.

Step 1: Create resources for the Service provider

If you deployed Kubernetes Services and the relevant resources to ACK Cluster 1, skip this step. You need to only modify the names of these resources when you create the ServiceExport and ServiceImport. For more information, see Modify and delete the multi-cluster Service.

  1. Run the following command to create a namespace in ACK Cluster 1.

    In this example, the provider-ns namespace is created.

    kubectl create ns provider-ns
  2. Run the following command to query the associated clusters:

    kubectl amc get managedclusters

    Expected output:

    Name        Alias      HubAccepted
    c984b0****   cluster2   true            # The Service consumer, which is ACK Cluster 2 in this example. 
    cc36f4****   cluster1   true            # The Service provider, which is ACK Cluster 1 in this example.
  3. Create a Kubernetes Service, a Deployment, and other resources in ACK Cluster 1.

    1. Create a file named app-meta.yaml with the following content. The file defines a Service, a Deployment, and a propagation policy.

      apiVersion: v1       # The Service provider. 
      kind: Service
      metadata:
        name: service1
        namespace: provider-ns
      spec:
        ports:
        - port: 80
          protocol: TCP
          targetPort: 8080
        selector:
          app: web-demo
        sessionAffinity: None
        type: ClusterIP
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          app: web-demo
        name: web-demo
        namespace: provider-ns
      spec:
        replicas: 2
        selector:
          matchLabels:
            app: web-demo
        template:
          metadata:
            labels:
              app: web-demo
          spec:
            containers:
            - image: acr-multiple-clusters-registry.cn-hangzhou.cr.aliyuncs.com/ack-multiple-clusters/web-demo:0.4.0
              name: web-demo
              env:
              - name: ENV_NAME
                value: cluster1-beijing
      ---
      apiVersion: core.oam.dev/v1beta1
      kind: Application
      metadata:
        name: web-demo        # The resources of the application and the propagation policy. 
        namespace: provider-ns
        annotations:
          app.oam.dev/publishVersion: version1
      spec:
        components:
          - name: web-demo
            type: ref-objects
            properties:
              objects:
                - resource: deployment
                  name: web-demo
                - resource: service
                  name: service1
        policies:
          - type: topology
            name: cluster1
            properties:
              clusters: ["cc36f4****"]    # The destination cluster to which the resources are distributed. Replace the value with a complete cluster ID.

    2. Run the following command to deploy the resources on the master instance and then distribute them to the associated clusters:

      kubectl apply -f app-meta.yaml
  4. Run the following command to query the deployment progress of the application:

    kubectl get app web-demo -n provider-ns

    Expected output:

    NAME       COMPONENT   TYPE          PHASE     HEALTHY   STATUS      AGE
    web-demo   web-demo    ref-objects   running   true      Ready:2/2   34s
  5. Run the following command to query the status of the associated clusters:

    kubectl amc get deployment,service -n provider-ns -m cc36f4****  # The destination cluster to which the resources are distributed. Replace the value with a complete cluster ID.

    Expected output:

    Run on ManagedCluster cc36f4**** (cluster1)
    NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/web-demo   2/2     2            2           101s
    
    NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
    service/service1   ClusterIP   172.16.39.201   <none>        80/TCP    101s

Step 2: Create a multi-cluster Service

  1. Create a ServiceExport with the following content and specify the Kubernetes Service that you want to export:

    apiVersion: multicluster.x-k8s.io/v1alpha1
    kind: ServiceExport
    metadata:
      name: service1           # The value must be the same as the name of the Kubernetes Service that you want to export. 
      namespace: provider-ns   # The value must be the same as the namespace of the Kubernetes Service that you want to export.

    The ServiceExport is a new type of custom resource that is distributed to the Service provider. The ServiceExport describes the Kubernetes Service that you want to export. The following table describes the parameters.

    Parameter

    Description

    metadata.name

    The name of the Kubernetes Service that you want to export.

    metadata.namespace

    The namespace of the Kubernetes Service that you want to export.

  2. Create a ServiceImport with the following content:

    apiVersion: multicluster.x-k8s.io/v1alpha1
    kind: ServiceImport
    metadata:
      name: service1           
      namespace: provider-ns   
    spec:
      ports:                   # The values in this field must be the same as the values used by the Kubernetes Service that you want to export. 
      - port: 80
        protocol: TCP
      type: ClusterSetIP

    The ServiceImport is a new type of custom resource that is distributed to the Service consumer. Clients can use the ServiceImport to access Services across clusters. The preceding example shows only some of the spec parameters of the ServiceImport. The following table describes all spec parameters.

    Parameter

    Description

    metadata.name

    The name of the ServiceImport must be the same as the name of the Kubernetes Service that you want to export.

    metadata.namespace

    The namespace of the ServiceImport must be the same as the namespace of the Kubernetes Service that you want to export.

    spec. ports. name

    The name of the port. The value must be the same as that of the Kubernetes Service that you want to export.

    spec. ports. protocol

    The protocol. The value must be the same as that of the Kubernetes Service that you want to export.

    spec. ports. appProtocol

    The application protocol. The value must be the same as that of the Kubernetes Service that you want to export.

    spec. ports. port

    The port. The value must be the same as that of the Kubernetes Service that you want to export.

    spec. ips

    The virtual IP address of the Service. You do not need to specify this parameter. The virtual IP address is assigned by the master instance.

    spec. type

    Valid values: ClusterSetIP and Headless. If ClusterIP of the Kubernetes Service that you want to export is None, the Service is a Headless Service. In this case, set type to Headless. Otherwise, set type to ClusterSetIP.

    spec. sessionAffinity

    The session affinity. Valid values: ClientIP and None. The value must be the same as that of the Kubernetes Service that you want to export.

    spec. sessionAffinityConfig

    The session affinity configuration. The value must be the same as that of the Kubernetes Service that you want to export.

  3. Create a propagation policy named mcs-service-policy.yaml with the following content to distribute the ServiceExport and ServiceImport to the associated clusters:

    apiVersion: core.oam.dev/v1beta1
    kind: Application
    metadata:
      name: amcs-export-service1
      namespace: provider-ns
      labels:
        amcs: export
        amcs-service: service1
      annotations:
        app.oam.dev/publishVersion: version1
    spec:
      components:
        - name: export-service
          type: ref-objects
          properties:
            objects:
              - resource: serviceexport  # Reference the ServiceExport. 
                name: service1
      policies:
        - type: topology
          name: export-clusters
          properties:
            clusters: ["cc36f4****"]  # Distribute the ServiceExport to ACK Cluster 1. 
    ---
    apiVersion: core.oam.dev/v1beta1
    kind: Application
    metadata:
      name: amcs-import-service1
      namespace: provider-ns
      labels:
        amcs: import
        amcs-service: service1
      annotations:
        app.oam.dev/publishVersion: version1
    spec:
      components:
        - name: import-service
          type: ref-objects
          properties:
            objects:
              - resource: serviceimport  # Reference the ServiceImport. 
                name: service1
      policies:
        - type: topology
          name: import-clusters
          properties:
            clusters: ["c984b0****"]  # Distribute the ServiceImport to ACK Cluster 2.

  4. Run the following command to deploy the multi-cluster Service:

    kubectl apply -f mcs-service-policy.yaml
  5. Run the following command to query the status of the distributed resources:

    kubectl amc get serviceexport,serviceimport -n provider-ns -m all

    Expected output:

    Run on ManagedCluster c984b0**** (cluster2)
    NAME                                           TYPE           IP                 AGE
    serviceimport.multicluster.x-k8s.io/service1   ClusterSetIP   ["172.xx.xx.xx"]   65s
    
    Run on ManagedCluster cc36f4**** (cluster1)
    NAME                                           AGE
    serviceexport.multicluster.x-k8s.io/service1   65s

    The output indicates that the ServiceExport is distributed to ACK Cluster 1 and the ServiceImport is distributed to ACK Cluster 2.

Step 3: Access the multi-cluster Service

You can access Service 1 in ACK Cluster 1 from the client pod in ACK Cluster 2 by using one of the following methods.

Method 1: Use the name of the Service

After the ServiceImport is created in ACK Cluster 2, the master instance creates a Kubernetes Service that is prefixed with amcs- to represent the Service that is deployed in ACK Cluster 1. The client pod in ACK Cluster 2 can access Service 1 in ACK Cluster 1 by accessing amcs-service1.provider-ns.

  1. Run the following command on the master instance to query the information about the Service:

    kubectl amc get service -n provider-ns -m all

    Expected output:

    Run on ManagedCluster c984b0**** (cluster2)
    NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
    amcs-service1   ClusterIP   172.xx.xx.xx   <none>        80/TCP    26m
    
    Run on ManagedCluster cc36f4**** (cluster1)
    NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
    service1   ClusterIP   172.xx.xx.xx   <none>        80/TCP    169m
  2. Run the following command on the client pod in ACK Cluster 2 to access Service 1 in ACK Cluster 1:

    curl amcs-service1.provider-ns

Method 2: Use the domain name of the Service

  1. Install or update CoreDNS in ACK Cluster 2. The version of CoreDNS must be 1.9.3 or later. For more information, see CoreDNS and Manage components.

  2. Modify the Corefile configurations of CoreDNS.

    1. Run the following command to modify the ConfigMap of CoreDNS:

      kubectl edit configmap coredns -n kube-system
    2. In the Corefile field, add multicluster clusterset.local to enable domain name resolution for multi-cluster Services.

      apiVersion: v1
      data:
        Corefile: |
          .:53 {
              errors
              health {
                 lameduck 15s
              }
              ready
              multicluster clusterset.local    # Add this configuration to enable domain name resolution for multi-cluster Services. 
              kubernetes cluster.local in-addr.arpa ip6.arpa {
      
                pods verified
                ttl 30
                fallthrough in-addr.arpa ip6.arpa
              }
              ...
          }
      kind: ConfigMap
      metadata:
        name: coredns
        namespace: kube-system
                                      
  3. Run the following command on the client pod in ACK Cluster 2 to access Service 1 in ACK Cluster 1:

    After the ServiceImport is created in ACK Cluster 2, ACK Cluster 1 adds the clusterset.local domain name to the multi-cluster Service.

    curl service1.provider-ns.svc.clusterset.local
  4. (Optional) Modify the dnsConfig.searches field of the client pod in ACK Cluster 2 to access the multi-cluster Service.

    1. Add clusterset.local to the dnsConfig.searches field of the client pod.

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: client-pod
        namespace: consumer-ns
      spec:
        ...
        template:
          ...
          spec:
            dnsPolicy: "ClusterFirst"    
            dnsConfig:                   
              searches:                        #dnsConfig. Add clusterset.local. 
                - svc.clusterset.local
                - clusterset.local
                - consumer-ns.svc.cluster.local
                - svc.cluster.local
                - cluster.local
            containers:
            - name: client-pod
              ...
    2. Run the following command to access the multi-cluster Service:

      curl service1.provider-ns

Modify and delete the multi-cluster Service

Modify the multi-cluster Service

  1. Modify the ServiceExport and ServiceImport that are created in Substep 1 and Substep 2.

  2. Modify app.oam.dev/publishVersion in the propagation policy to redistribute the ServiceExport and ServiceImport.

Delete the multi-cluster Service

  1. Run the following command to delete the propagation policy of the multi-cluster Service:

    kubectl delete application amcs-export-service1 -n provider-ns
    kubectl delete application amcs-import-service1 -n provider-ns
  2. Run the following command to delete the ServiceExport:

    kubectl delete serviceexport service1 -n provider-ns
  3. Run the following command to delete the ServiceImport:

    kubectl delete serviceimport service1 -n provider-ns