Container Compute Service:Kubernetes 1.28 release notes

Non-graceful node shutdown (GA)

The non-graceful node shutdown feature has reached GA. When a node shuts down abnormally — for example, due to a power failure — StatefulSets can create pods with the same name on another node to prevent service interruption.

NodeOutOfServiceVolumeDetach (GA)

The NodeOutOfServiceVolumeDetach feature gate has reached GA. When a node shuts down abnormally, volume detach operations are immediately performed for terminated pods on that node, allowing pods to recover quickly on other nodes.

Retroactive default StorageClass assignment (GA)

The retroactive default StorageClass assignment feature has reached GA. Previously, a PersistentVolumeClaim (PVC) created without a storageClassName field remained in the Pending state if no default StorageClass existed. When a default StorageClass is created, such PVCs now automatically use it.

Sidecar containers (Alpha)

The SidecarContainers feature gate is introduced in Alpha and is disabled by default. It lets you specify when sidecar containers start — for example, starting log collection containers before other containers to improve log collection reliability. See Kubernetes v1.28: Introducing native sidecar containers.

Job improvements

JobPodReplacementPolicy (Alpha)

The JobPodReplacementPolicy feature gate is in Alpha. Unlike the deletionTimestamp approach that creates replacement pods immediately after deletion, this policy creates replacement pods only after pods reach the Failed phase (status.phase: Failed). This prevents two pods from simultaneously using the same index and node resources.

JobBackoffLimitPerIndex (Alpha)

The JobBackoffLimitPerIndex feature gate is in Alpha. Set .spec.backoffLimitPerIndex to limit the maximum number of retries for pod failures per index. Previously, if consecutive pod failures for an index reached .spec.backoffLimit, the entire indexed Job failed.

Job scale warnings

If the completion field of an indexed Job exceeds 100,000, or the parallelism field exceeds 10,000, and large numbers of pods fail, pod terminal phase tracing may fail. Kubernetes now displays warnings when you set these fields to excessively large values.

Pod indexes as labels

Pod indexes (sequence numbers) are added as labels to pods created by indexed Jobs and StatefulSets.

ValidatingAdmissionPolicy (Beta)

The ValidatingAdmissionPolicy feature gate is in Beta. It provides a declarative alternative to admission webhooks for validating resource requests. Write complex validation rules using Common Expression Language (CEL) expressions. The API server validates resource requests against these CEL expressions.

CRD validation enhancements

The reason and fieldPath fields are added to CustomResourceDefinition (CRD) validation rules. When CRD validation fails, the API server returns the reason and field path of the failure. See CRD Validation Expression Language.

CEL webhook matching

CEL expressions can now be used in webhook matching conditions. Up to 64 matching conditions are supported per webhook. See Matching requests: matchConditions.

PVC resize status field update

The .status.resizeStatus field of a PVC is replaced with the .status.allocatedResourceStatus map field to store the resize states of resources. See PersistentVolumeClaimStatus.

Controller manager concurrency flags

Two new flags are added to the Kubernetes controller manager:

• --concurrent-cron-job-syncs: Sets the concurrency of the CronJob controller. See --concurrent-cron-job-syncs.

• --concurrent-job-syncs: Sets the concurrency of the Job controller. See --concurrent-job-syncs.

API server optimizations

• Reduced memory usage for GetList: Memory usage when getting a list (GetList) from the cache is reduced. See GetList test data.

• Endpoint cleanup on graceful shutdown: The issue where a Kubernetes Service endpoint was not removed when only one replicated API server existed is fixed. Endpoints are now correctly removed during graceful shutdown.

• Lazy OpenAPI v2 controller: The OpenAPI v2 controller is made lazy and OpenAPI v2 specs are reduced. When no clients request OpenAPI v2, the API server's CPU and memory usage is reduced and installing large numbers of CRDs is more efficient. Note that this slows down processing of first-time requests. Update your client to a version that supports OpenAPI v3.

• ConsistentListFromCache (Alpha): The Consistent Reads from Cache feature gate is in Alpha. It allows the API server to use the watch cache to guarantee consistent reads for LIST requests.

• Expanded metrics collection: A wider range of metrics can be collected by calling the metrics API.

Kubernetes 1.28

The following features and plugins are deprecated or removed in Kubernetes 1.28.

In-tree CephFS plugin removed

The in-tree CephFS plugin code is removed in Kubernetes 1.28. Use the CephFS CSI driver.

Ceph RBD migration deprecated

Support for migrating Ceph RBD volumes to an out-of-tree CSI driver is deprecated and will be removed in a later version. Complete the migration before the in-tree plugin code is removed.

`kubernetes.io/rbd` RBD volume plugin deprecated

The kubernetes.io/rbd RBD volume plugin is deprecated and will be removed in a later version. Use the CephFS CSI driver.

KMSv1 deprecated

KMSv1 is deprecated and will only receive security updates. To continue using KMSv1, set --feature-gates=KMSv1=true. Use KMSv2 instead. See Mark KMS v1beta1 as deprecated.

Deprecated controller manager flags

The following flags are deprecated in Kubernetes controller manager commands:

• --volume-host-cidr-denylist

• --volume-host-allow-local-loopback

Deprecated kubelet flag

The --azure-container-registry-config flag is deprecated in kubelet commands. Use the image-credential-provider-config and --image-credential-provider-bin-dir flags instead.

Windows node pools

Creating Windows node pools is no longer supported.