You can register clusters that are deployed in data centers or clusters that are deployed on a third-party cloud to Alibaba Cloud Distributed Cloud Container Platform (ACK One). This way, you can create clusters in a hybrid cloud environment and manage clusters in a centralized manner. This topic describes the features that are supported by registered clusters and the resources that are involved when you register clusters.

Benefits of registered clusters

Architecture
Your daily operations and maintenance work may involve ACK clusters, self-managed Kubernetes clusters deployed in data centers, and Kubernetes clusters deployed on third-party clouds. If multiple Kubernetes clusters are deployed in a multi-cloud or hybrid cloud environment and you have the following requirements, we recommend that you use clusters that are registered in ACK One.
  • Use the ACK console to centrally manage ACK clusters and self-managed Kubernetes clusters that are deployed in data centers. The ACK console provides consistent O&M experience and security governance capability across the clusters. For example, the ACK console provides a unified logging, monitoring, and alerting system, and supports unified authorization management of Alibaba Cloud accounts, Resource Access Management (RAM) users, and RAM roles. This allows you to centrally manage all your clusters and applications.
  • Use cloud computing resources to scale out self-managed Kubernetes clusters that are deployed in data centers. For example, you can manually or set the system to automatically adjust the number of Elastic Compute Service (ECS) instances or ECS bare metal instances in self-managed Kubernetes clusters that are deployed in data centers.
  • Use the Application Center feature of ACK to centrally manage application lifecycles and GitOps delivery pipelines for all Kubernetes clusters deployed on the cloud and in data centers. To achieve this goal, you must register external Kubernetes clusters in the ACK console so that you can manage all Kubernetes clusters by using the same control plane.
  • Use Alibaba Cloud Service Mesh (ASM) to implement centralized service governance for all Kubernetes clusters deployed on the cloud and in data centers. To achieve this goal, you must register external Kubernetes clusters in the ACK console so that you can manage all Kubernetes clusters by using the same control plane.

Security features

When you register external Kubernetes clusters, you can use the ack-cluster-agent component to manage these clusters. This component also allows you to deploy applications in external Kubernetes clusters and manage the lifecycles of these applications.

Components

Component Description
ACK Console The ACK console for cluster and service management.
ack-cluster-agent ack-cluster-agent is an agent that runs on Deployments within an external Kubernetes cluster. ack-cluster-agent receives requests from ACK Stub and forwards them to the Kubernetes API server. ack-cluster-agent also receives responses from the API server and forwards them to ACK Stub.
ACK Stub ACK Stub is a cluster registration proxy deployed on Alibaba Cloud. ACK launches ACK Stub for all external Kubernetes clusters registered in the ACK console. ACK Stub forwards requests between the ACK console and ACK Register Agent in external Kubernetes clusters.
K8s API Server The API server runs in an external Kubernetes cluster.

Architecture of component connections

In an architecture where ACK Stub and ack-cluster-agent are used, requests from the ACK console are sent to the API server in an external Kubernetes cluster. ack-cluster-agent runs on Deployments that create two pods within the external Kubernetes cluster. ack-cluster-agent is connected to ACK Stub that is deployed in the ACK console. The following figure shows how ack-cluster-agent forwards requests to the API server within an external Kubernetes cluster.

架构

After you register an external Kubernetes cluster in the ACK console, ack-cluster-agent is deployed in the external Kubernetes cluster. Then, a two-way persistent connection is established between ACK Stub and ack-cluster-agent over Transport Layer Security (TLS) 1.2. Requests from authorized users or ACK management services are first sent to ACK Stub through the TLS connection, then forwarded to ack-cluster-agent, and finally delivered to the API server. After the API server receives the requests, the API server first performs authentication, authorization check, and admission control, then audits the requests, and finally returns responses. Responses are returned through the same TLS connection. The responses pass through ack-cluster-agent and ACK Stub, and finally reach the ACK console. All requests sent to the external Kubernetes cluster through the connection are authenticated and verified. This ensures that the external Kubernetes cluster is accessed in a secure way.

Security of intercommunication among components

Authentication and authorization check are required for intercommunication among all components. All components to be accessed must pass security checks. This ensures that data is transmitted among only trusted components.

访问安全

Authentication is based on the credentials of RAM users and two-way TLS certification. Data is encrypted by TLS during transmission. Authorization check is based on RAM and the TLS (x509) certificate whitelist.

  • Security of request transmission

    All credentials contained in requests that are sent through the TLS connection carry the identity information of the users who send the requests. The user identity information includes the credential issued by ACK to access an external cluster and the internal credential required to access ACK components. This ensures that all requests sent to API servers are verified and audited.

  • Security of user request transmission

    All user requests sent to external Kubernetes clusters must carry credentials issued by the ACK console. The credentials contain the user identity information. The following figure shows how a request with a credential is forwarded to an external cluster.

    终端

    Cluster administrators on Alibaba Cloud can use RAM permission policies to control access from users to external clusters. Authorized RAM users can obtain credentials that are required to access external clusters from the ACK console. The credentials are provided to ACK Stub and ack-cluster-agent for authentication. Data transmission among components is encrypted by TLS. After ack-cluster-agent verifies the credential, the user identity carried in the credential is encapsulated into the impersonation headers of the request for the destination API server to authenticate the request. The API server performs authentication, authorization check, and admission control based on the received credential and user identity, and then audits the request.

  • Security of service request transmission

    Requests from ACK and ASM to the external API server are transmitted in the same way as user requests. The ACK console sends a request that carries the credential required for ack-cluster-agent to authenticate the request. Then, the request is authenticated by ACK Stub and ack-cluster-agent and forwarded to the destination API server. During this process, ack-cluster-agent impersonates the user identity in the request and forwards the request to the API server over Layer 7. Finally, the API server performs authorization checks with role-based access control (RBAC) and then audits the request. Data transmission is encrypted by TLS from end to end.

    组件请求

Internal security of clusters

The following figure shows how a request from ack-cluster-agent is processed by an API server. Requests from ack-cluster-agent
After the API server receives the request, the API server first checks whether the credential of the request is valid.
  • If the credential is invalid, the API server returns a 401 error, which indicates that the request failed the authentication.
  • If the request passes the authentication, the API server checks whether the request contains valid impersonation headers. If the request contains valid impersonation headers, the request is passed to the next round with the impersonated user identity.
Then, the API server checks whether the impersonated user identity is granted the required permissions on the cluster.
  • If the request fails the authorization check, the API server returns a 403 error, which indicates that the user identity is unauthorized.
  • If the request passes the authorization check, the API server returns a response after it audits the request.

ack-cluster-agent is a forward proxy and cluster registration component written in the Go programming language. The programming and publishing of ack-cluster-agent are audited by Alibaba Cloud to ensure security. The administrators of registered external clusters must ensure the security of cluster nodes by following the best practices for Kubernetes security. They can ensure the security of ack-cluster-agent and external clusters by using security configurations related to the infrastructure and applications.

Comparison of features supported by registered clusters and ACK clusters

Item Feature ACK cluster Registered cluster
Access to cluster Access clusters by using kubectl and kubeconfig files ✔️ ✔️
Access clusters by using the ACK console ✔️ ✔️
Core Kubernetes concepts Namespaces management ✔️ ✔️
Pods management ✔️ ✔️
Workloads management ✔️ ✔️
Persistent volumes (PVs) and StorageClasses management ✔️

Supports Alibaba Cloud disks, local disks, Apsara File Storage NAS (NAS) file systems, Cloud Paralleled File System (CPFS) file systems, and Object Storage Service (OSS) buckets.

✔️

The supported storage types vary based on the environment in which the registered cluster is deployed.

Services management ✔️ ✔️
Ingresses management ✔️ ✔️
Network policies management ✔️

Supports the Terway network plug-in.

✔️

The supported network policies vary based on the environment in which the registered cluster is deployed.

ConfigMaps management ✔️ ✔️
Secrets management ✔️ ✔️
RBAC management ✔️ ✔️
Security policies of pods management ✔️ ✔️
Resource quotas management ✔️ ✔️
CustomResourceDefinitions (CRDs) management ✔️ ✔️
Horizontal Pod Autoscaler (HPA) ✔️ ✔️
Pod RuntimeClasses ✔️

Supports only sandboxed containers.

No
Elastic container instances ✔️ ✔️

For more information, see Scale out elastic container instances.

Application management Applications deployment and management by using Helm charts ✔️ ✔️
GitOps application delivery ✔️ No
Knative add-on ✔️ ✔️
Istio add-on ✔️ ✔️
ASM ✔️ ✔️

For more information, see Use ASM to manage applications in registered external Kubernetes clusters.

Security governance RAM-based authentication and RBAC-based authorization ✔️ ✔️
Cluster auditing by using Log Service ✔️ ✔️
Security Center ✔️ No
Inspections ✔️ ✔️

For more information, see Use the inspection feature to check for security risks in the workloads of a registered Kubernetes cluster.

Observability Event Center (node problem detection) ✔️ ✔️

For more information, see Create and use an event center.

Ingress charts ✔️ ✔️

For more information, see Analyze and monitor the access log of nginx-ingress.

Log collection ✔️ ✔️

For more information, see Enable Log Service for an external Kubernetes cluster.

Application Real-Time Monitoring Service (ARMS) ✔️ ✔️

For more information, see Enable ARMS for a registered Kubernetes cluster.

ARMS Prometheus monitoring ✔️ ✔️

For more information, see Enable ARMS Prometheus for a registered Kubernetes cluster.

Architecture characteristics discovery provided by Application High Availability Service (AHAS) ✔️ ✔️

No

Application throttling provided by AHAS ✔️ ✔️

No

Node Problem Detector (NPD) ✔️ ✔️

For more information, see Create a Kubernetes event center for an external Kubernetes cluster.

Metrics Adapter ✔️ ✔️

For more information, see Deploy alibaba-cloud-metrics-adapter in a registered cluster.

Integration with CloudMonitor ✔️ No
Integration with Key Management Service (KMS) ✔️ No
Cluster lifecycle management Nodes management ✔️ ✔️
Manage node pools ✔️ No
Cluster auto scaling ✔️ ✔️

The supported network policies vary based on the environment in which the registered cluster is deployed.

Certificate rotation ✔️ No
Cluster upgrades ✔️ No
System components management ✔️ No
Cluster health checks ✔️ No

Resources that are involved when you register clusters

When you register an external cluster in the ACK console, the following resources are involved:
  • A virtual private cloud (VPC) and multiple vSwitches. The VPC must have access to the Internet. You can associate the cluster with an elastic IP address (EIP) or configure SNAT rules. You are charged for the VPC and vSwitches.
  • An elastic network interface (ENI). You are not charged for the ENI.
  • An internal-facing Server Load Balancer (SLB) instance. You are charged based on the specification of the SLB instance that you use. For more information about the pricing of different instance specifications, see Instance specifications.
  • Optional:An EIP. You are charged for data transfer. For more information about data transfer fees, see Pay-as-you-go.
If you want to automatically scale self-managed Kubernetes clusters by adding or removing cloud computing resources, the following resources are also involved:
  • ECS instances, ECS bare metal instances, and elastic container instances. You are charged for these resources based on your actual resource usage.
  • Auto Scaling, which is free of charge.

You are charged other fees based on the features that you use.