Container Service for Kubernetes (ACK) Open Nexus Everywhere (ONE) provides registered clusters, which allow you to register external Kubernetes clusters in the ACK console and build hybrid cloud clusters. You can register and then centrally manage the following external Kubernetes clusters in the ACK console: self-managed Kubernetes clusters deployed in data centers and Kubernetes clusters deployed on platforms of other cloud service providers. This topic describes the features that are supported by registered clusters and the resources that are involved when you register clusters.
Benefits of registered clusters in ACK
- Use the ACK console to centrally manage ACK clusters and self-managed Kubernetes clusters that are deployed in data centers. The ACK console provides consistent O&M experience and security governance capability across the clusters. For example, the ACK console provides a unified logging, monitoring, and alerting system, and supports unified authorization management of Alibaba Cloud accounts and Resource Access Management (RAM) users. This allows you to centrally manage all your clusters and applications.
- Use cloud computing resources to scale out self-managed Kubernetes clusters that are deployed in data centers. For example, manually or set the system to automatically adjust the number of Elastic Compute Service (ECS) instances or ECS bare metal instances in self-managed Kubernetes clusters that are deployed in data centers.
- Use the Application Center feature of ACK to centrally manage application lifecycles and GitOps delivery pipelines for all Kubernetes clusters deployed on the cloud and in data centers. To achieve this goal, you must register external Kubernetes clusters in the ACK console so that you can manage all Kubernetes clusters by using the same control plane.
- Use Alibaba Cloud Service Mesh (ASM) to implement centralized service governance for all Kubernetes clusters deployed on the cloud and in data centers. To achieve this goal, you must register external Kubernetes clusters in the ACK console so that you can manage all Kubernetes clusters by using the same control plane.
When you register external Kubernetes clusters, you can use the ACK Register Agent component to manage these clusters. This component also allows you to deploy applications in external Kubernetes clusters and manage the lifecycles of these applications.
|ACK Console||The ACK console for cluster and service management.|
|ACK Register Agent||ACK Register Agent is an agent that runs on Deployments within an external Kubernetes cluster. ACK Register Agent receives requests from ACK Stub and forwards them to the Kubernetes API server. ACK Register Agent also receives responses from the API server and forwards them to ACK Stub.|
|ACK Stub||ACK Stub is a cluster registration proxy deployed on Alibaba Cloud. ACK launches ACK Stub for all external Kubernetes clusters registered in the ACK console. ACK Stub forwards requests between the ACK console and ACK Register Agent in external Kubernetes clusters.|
|K8s API Server||The API server runs in an external Kubernetes cluster.|
Architecture of component connections
After you register an external Kubernetes cluster in the ACK console, ACK Register Agent is deployed in the external Kubernetes cluster. Then, a two-way persistent connection is established between ACK Stub and ACK Register Agent over Transport Layer Security (TLS) 1.2. Requests from authorized users or ACK management services are first sent to ACK Stub through the TLS connection, then forwarded to ACK Register Agent, and finally delivered to the API server. After the API server receives the requests, the API server first performs authentication, authorization check, and admission control, then audits the requests, and finally returns responses. Responses are returned through the same TLS connection. The responses pass through ACK Register Agent and ACK Stub, and finally reach the ACK console. All requests sent to the external Kubernetes cluster through the connection are authenticated and verified. This ensures that the external Kubernetes cluster is accessed in a secure way.
Security of intercommunication among components
Authentication is based on the credentials of Resource Access Management (RAM) users and two-way TLS certification. Data is encrypted by TLS during transmission. Authorization check is based on RAM and the TLS (x509) certificate whitelist.
- Security of request transmission
All credentials contained in requests that are sent through the TLS connection carry the identity information of the users who send the requests. The user identity information includes the credential issued by ACK to access an external cluster and the internal credential required to access ACK components. This ensures that all requests sent to API servers are verified and audited.
- Security of user request transmission
All user requests sent to external Kubernetes clusters must carry credentials issued by the ACK console. The credentials contain the user identity information. The following figure shows how a request with a credential is forwarded to an external cluster.
Cluster administrators on Alibaba Cloud can use RAM permission policies to control access from users to external clusters. Authorized RAM users can obtain credentials that are required to access external clusters from the ACK console. The credentials are provided to ACK Stub and ACK Register Agent for authentication. Data transmission among components is encrypted by TLS. After ACK Register Agent verifies the credential, the user identity carried in the credential is encapsulated into the impersonation headers of the request for the destination API server to authenticate the request. The API server performs authentication, authorization check, and admission control based on the received credential and user identity, and then audits the request.
- Security of service request transmission
Requests from ACK and ASM to the external API server are transmitted in the same way as user requests. The ACK console sends a request that carries the credential required for ACK Register Agent to authenticate the request. Then, the request is authenticated by ACK Stub and ACK Register Agent and forwarded to the destination API server. During this process, ACK Register Agent impersonates the user identity in the request and forwards the request to the API server over Layer 7. Finally, the API server performs authorization checks with role-based access control (RBAC) and then audits the request. Data transmission is encrypted by TLS from end to end.
Internal security of clusters
- If the credential is invalid, the API server returns a 401 error, which indicates that the request failed the authentication.
- If the request passes the authentication, the API server checks whether the request contains valid impersonation headers. If the request contains valid impersonation headers, the request is passed to the next round with the impersonated user identity.
- If the request fails the authorization check, the API server returns a 403 error, which indicates that the user identity is unauthorized.
- If the request passes the authorization check, the API server returns a response after it audits the request.
ACK Register Agent is a forward proxy and cluster registration component written in the Go programming language. The programming and publishing of ACK Register Agent are audited by Alibaba Cloud to ensure security. The administrators of registered external clusters must ensure the security of cluster nodes by following the best practices for Kubernetes security. They can ensure the security of ACK Register Agent and external clusters by using security configurations related to the infrastructure and applications.
Comparison of features supported by registered clusters and ACK clusters
|Item||Feature||ACK cluster||Registered cluster|
|Access to cluster||Access clusters by using kubectl and kubeconfig files||✔️||✔️|
|Access clusters by using the ACK console||✔️||✔️|
|Core Kubernetes concepts||Namespaces management||✔️||✔️|
|Persistent volumes (PVs) and StorageClasses management||✔️
Supports Alibaba Cloud disks, local disks, Apsara File Storage NAS (NAS) file systems, Cloud Paralleled File System (CPFS) file systems, and Object Storage Service (OSS) buckets.
The supported storage types vary based on the environment in which the registered cluster is deployed.
|Network policies management||✔️
Supports the Terway network plug-in.
The supported network policies vary based on the environment in which the registered cluster is deployed.
|Security policies of pods management||✔️||✔️|
|Resource quotas management||✔️||✔️|
|CustomResourceDefinitions (CRDs) management||✔️||✔️|
|Horizontal Pod Autoscaler (HPA)||✔️||✔️|
Supports only sandboxed containers.
|Elastic container instances||✔️||✔️
For more information, see Scale out elastic container instances.
|Application management||Applications deployment and management by using Helm charts||✔️||✔️|
|GitOps application delivery||✔️||N/A|
|Alibaba Cloud Service Mesh (ASM)||✔️||✔️
For more information, see Use ASM to manage applications in registered external Kubernetes clusters.
|Security governance||RAM-based authentication and RBAC-based authorization||✔️||✔️|
|Cluster auditing by using Log Service||✔️||✔️|
For more information, see Use the inspection feature to check for security risks in the workloads of a registered Kubernetes cluster.
|Observability||Event Center (node problem detection)||✔️||✔️
For more information, see Create and use a Kubernetes event center.
For more information, see Monitor nginx-ingress and analyze the access log of nginx-ingress.
For more information, see Enable Log Service for an external Kubernetes cluster.
|Application Real-Time Monitoring Service (ARMS)||✔️||✔️
For more information, see Enable ARMS for an external Kubernetes cluster.
|ARMS Prometheus monitoring||✔️||✔️
For more information, see Enable ARMS Prometheus for an external Kubernetes cluster.
|Architecture characteristics discovery provided by Application High Availability Service (AHAS)||✔️||✔️
|Application throttling provided by AHAS||✔️||✔️
|Node Problem Detector (NPD)||✔️||✔️
For more information, see Create a Kubernetes event center for an external Kubernetes cluster.
For more information, see Deploy alibaba-cloud-metrics-adapter in an external Kubernetes cluster.
|Integration with CloudMonitor||✔️||N/A|
|Integration with Key Management Service (KMS)||✔️||N/A|
|Cluster lifecycle management||Nodes management||✔️||✔️|
|Node pools management||✔️||N/A|
|Cluster auto scaling||✔️||✔️
The support for this feature varies based on the environment in which the registered cluster is deployed.
|System components management||✔️||N/A|
|Cluster health checks||✔️||N/A|
Resources that are involved when you register clusters
- A virtual private cloud (VPC) and vSwitches. The VPC must have access to the Internet. You can associate the cluster with an elastic IP address (EIP) or configure SNAT rules. You are charged for the VPC and vSwitches.
- An elastic network interface (ENI). You are not charged for the ENI.
- An internal-facing Server Load Balancer (SLB) instance. You are charged based on the specification of the SLB instance that you use. For more information about the pricing of different instance specifications, see Instance types and specifications.
- Optional:An EIP. You are charged for data transfer. For more information about data transfer fees, see Pay-as-you-go.
- ECS instances, ECS bare metal instances, and elastic container instances. You are charged for these resources based on your actual resource usage.
- Auto Scaling, which is free of charge.
You are charged other fees based on the features that you use.