Container Compute Service:Kubernetes 1.28 release notes

Component versions

The following table lists the versions of the key components for an ACS cluster.

Upgrade considerations

Key concepts

Before learning about feature changes and deprecations in this Kubernetes version, you may need to understand the following concepts.

Major changes

• Kubernetes v1.28 optimizes the scheduling logic to reduce unnecessary retries, improving overall scheduler performance.

  If you use a custom scheduler plugin, update it to improve performance. For more information, see Changes in the scheduling framework.

• For CSI migration, the Kubernetes community continues to migrate in-tree storage plugins to out-of-tree drivers that implement the CSI standard. CSI migration reached GA in v1.25. In v1.27, the storage.k8s.io/v1beta1 API and the in-tree EBS storage plugin were removed. In v1.28, the in-tree CephFS volume plugin code is removed, the kubernetes.io/rbd plugin is deprecated, and you must switch to the CephFS CSI driver.

• The following CVE vulnerabilities are fixed in version 1.28.9-aliyun.1:

  • CVE-2023-45288

  • CVE-2024-3177

  • CVE-2024-24786

New features

In Kubernetes v1.28

• The non-graceful node shutdown feature is now Generally Available (GA). If a node shuts down unexpectedly (for example, due to a power outage), stateful applications are restarted on a healthy node, ensuring service continuity.

• The NodeOutOfServiceVolumeDetach feature gate is now GA. This allows volumes to be detached immediately from Pods terminated on an abnormal node, so they can recover quickly on other nodes.

• Retroactive default StorageClass assignment is now generally available. Previously, if a PVC without a specified storageClassName existed before a default StorageClass was created, the PVC would remain in a pending state indefinitely. With this update, after a default StorageClass is created, any PVCs without a specified storageClassName are automatically updated to use the default StorageClass.

• Two new features are introduced to handle Job failures:

  • The JobPodReplacementPolicy (Alpha feature gate) ensures that replacement Pods are created only after the original Pods enter the Failed phase (status.phase: Failed). This is different from the previous behavior where a Pod with a deletionTimestamp would be replaced immediately, which could lead to two Pods competing for the same index and node resources.

  • The JobBackoffLimitPerIndex (Alpha feature gate) lets you set a retry limit for specific indexes in an Indexed Job by using the .spec.backoffLimitPerIndex field. This prevents the entire Job from failing when a single index continuously fails and hits the global .spec.backoffLimit.

• If an Indexed Job is configured with a completion count (completion) of more than 100,000 and a parallelism (parallelism) of more than 10,000, you may not be able to track the termination status of the Job when a large number of Pods fail. Therefore, a warning message is displayed if the parameters are set too high when you create a Job.

• CRD validation rules now include reason and fieldPath fields, allowing validation failures to return a specific reason and field path. For more information, see CRD Validation Expression Language.

• Webhook request filtering now supports CEL expressions, with a maximum of 64 match conditions. For more information, see Matching requests: matchConditions.

• The new SidecarContainers feature gate adds native support for sidecar containers and allows you to optimize the container startup order. For example, a logging sidecar container can be started before other containers to improve log collection reliability. For more information, see Kubernetes v1.28: Introducing native sidecar containers. This feature is in Alpha in v1.28 and is disabled by default.

• The .status.resizeStatus field in a PVC is replaced by the .status.allocatedResourceStatus map, which indicates the status of a PVC resize operation. For more information, see PersistentVolumeClaimStatus.

• Pods managed by both Indexed Jobs and StatefulSets now include a Pod index label.

• The ValidatingAdmissionPolicy (in Beta) provides a declarative way to validate resource requests as an alternative to deploying validating admission webhooks. It supports complex validation rules written in CEL expressions. The API server validates requests against the CEL expressions and only approves requests that pass the evaluation.

• The Kube Controller Manager adds the --concurrent-cron-job-syncs flag to configure the concurrency of the CronJob controller and the --concurrent-job-syncs flag for the Job controller. For more information, see --concurrent-cron-job-syncs and --concurrent-job-syncs.

• The API server includes the following optimizations:

  • Memory consumption for GetList operations from the cache is improved. For more information, see GetList test data.

  • During a graceful shutdown, Kubernetes Service Endpoints are now properly removed, even if only one API server replica remains.

  • The OpenAPI v2 controller is set to lazily aggregate CRD information, and the OpenAPI v2 specification is significantly reduced. If no clients request OpenAPI v2, this reduces the CPU and memory consumption of the API server and improves performance when you install a large number of CRDs. However, the first client request will be slower. We recommend that you upgrade clients to support OpenAPI v3.

  • The new Consistent Reads from Cache feature gate allows List requests to be served from the watch cache with full consistency.

  • More monitoring metrics are available through the metrics endpoint.

Deprecated features

In Kubernetes v1.28

• The in-tree CephFS volume plugin code has been removed.

  We recommend using the CephFS CSI driver instead.

• The RBD volume plugin (kubernetes.io/rbd) is deprecated and will be removed in future versions.

  We recommend that you use the CephFS CSI driver instead.

• KMSv1 is deprecated. To continue using KMSv1, you must set the --feature-gates=KMSv1=true flag. For more information, see Mark KMS v1beta1 as deprecated.

  We recommend using KMSv2.

• The Kube Controller Manager startup flags --volume-host-cidr-denylist and --volume-host-allow-local-loopback are deprecated.

• The --azure-container-registry-config kubelet flag is deprecated.

  We recommend using the --image-credential-provider-config and --image-credential-provider-bin-dir flags instead.

• Creating Windows node pools is no longer supported.

Deprecated APIs

The CSIStorageCapacity API exposes available storage capacity to ensure that Pods are scheduled on nodes with sufficient storage. The storage.k8s.io/v1beta1 API version of CSIStorageCapacity was deprecated in v1.24 and removed in v1.27.

We recommend that you use storage.k8s.io/v1, which has been available since v1.24. For more information, see Storage Capacity Constraints for Pod Scheduling KEP.

Feature gates

This section lists only the major changes. For more information, see Feature Gates.

In Kubernetes v1.28

• NodeOutOfServiceVolumeDetach: When a node is marked out-of-service with the node.kubernetes.io/out-of-service taint, Pods that do not tolerate this taint are forcefully deleted, and their volumes are immediately detached. This feature gate is GA in v1.28 and is always enabled.

• AdmissionWebhookMatchCondition: Enabled by default, this allows webhooks to use CEL expressions in their match conditions.

• UnknownVersionInteroperabilityProxy: This feature is in Alpha. It proxies requests to the correct API server when multiple API server versions are present. For more information, see Mixed Version Proxy.

• IPTablesOwnershipCleanup: This feature is GA. Kubelet no longer creates the KUBE-MARK-DROP and KUBE-MARK-MASQ iptables chains.

• ConsistentListFromCache: This feature is in Alpha. It allows the API server to serve List requests from the watch cache with full consistency.

• ProbeTerminationGracePeriod: This feature is GA and enabled by default. It allows you to set probe-level terminationGracePeriodSeconds.

• Several feature gates that graduated to GA have been removed, including DelegateFSGroupToCSIDriver, DevicePlugins, KubeletCredentialProviders, MixedProtocolLBService, ServiceInternalTrafficPolicy, ServiceIPStaticSubrange, and EndpointSliceTerminatingCondition.

References

For the complete changelogs for Kubernetes v1.27 and v1.28, see CHANGELOG-1.27 and CHANGELOG-1.28.