Container Service for Kubernetes:(Discontinued) ACK release notes for Kubernetes 1.28

Kubernetes 1.28

Non-graceful node shutdown (GA)

When a node shuts down unexpectedly—for example, due to a power failure—Kubernetes can now automatically reschedule StatefulSet pods with the same name on another node, preventing service interruptions. This feature is now Generally Available (GA).

Retroactive default StorageClass assignment (GA)

Previously, a PersistentVolumeClaim (PVC) created without storageClassName would remain in Pending indefinitely if no default StorageClass existed at the time. Now, when a default StorageClass is created later, Kubernetes automatically assigns it to any unbound PVC with no storageClassName. This feature is now GA.

Scheduler performance

The scheduler's retry logic is optimized to reduce invalid scheduling attempts, improving overall performance. If your cluster uses a custom scheduler plug-in, update the plug-in to take advantage of the new scheduling framework. See Scheduling framework changes for details.

Job failure handling

Two new Alpha feature gates improve how failed Jobs are handled:

• `JobPodReplacementPolicy`: A replacement pod is created only when the original reaches the Failed phase (status.phase: Failed), not when it has a deletionTimestamp. This prevents two pods from simultaneously occupying the same index and consuming node resources.

• `JobBackoffLimitPerIndex`: Set .spec.backoffLimitPerIndex on an Indexed Job to cap the number of retries per index, rather than counting all failures toward a single shared .spec.backoffLimit. A persistently failing index no longer causes the entire Job to fail.

Large Indexed Job warning

If a Job has completion set above 100,000 and parallelism set above 10,000, and many pods fail, Kubernetes now displays a warning—termination status may not be tracked reliably at that scale.

CRD validation improvements

reason and fieldPath fields are added to CustomResourceDefinition (CRD) validation rules. When a Common Expression Language (CEL) expression fails, the API server returns a specific reason code and the path to the failing field. See CRD Validation Expression Language.

Webhook matching with CEL expressions

Webhook matchConditions now support CEL expressions, with up to 64 matching conditions per webhook. This gives you fine-grained control over which requests trigger a webhook without writing a full admission controller. See Matching requests: matchConditions.

ValidatingAdmissionPolicy (Beta)

ValidatingAdmissionPolicy provides a declarative, in-cluster alternative to validating admission webhooks. Write CEL expressions directly in the policy object; the API server evaluates them on every matching request—no webhook infrastructure needed. See ValidatingAdmissionPolicy.

PVC resize status field replaced

The .status.resizeStatus field on PVCs is replaced by .status.allocatedResourceStatus, a map that tracks the resize state of each resource independently. Update any tooling that reads .status.resizeStatus. See PersistentVolumeClaimStatus.

Pod index labels for Indexed Jobs and StatefulSets

Pods created by Indexed Jobs and StatefulSets now have their ordinal index added as a label, making it easier to select or filter pods by position.

Kube Controller Manager concurrency flags

Two new flags control controller concurrency:

• --concurrent-cron-job-syncs: Sets the number of concurrent CronJob reconciliation workers.

• --concurrent-job-syncs: Sets the number of concurrent Job reconciliation workers.

See --concurrent-cron-job-syncs and --concurrent-job-syncs.

API server optimizations

• Reduced GetList memory: Memory usage for serving list requests from the cache is reduced. See GetList test data.

• Endpoint cleanup fix: Fixed a bug where the endpoint for a Kubernetes Service was not removed when only one API server replica remained during a graceful shutdown.

• OpenAPI v2 lazy aggregation: The OpenAPI v2 controller now aggregates CRD information only when a client requests it, reducing CPU and memory overhead when no client polls the OpenAPI v2 endpoint. This also improves install time for large numbers of CRDs but may slow the first request to the endpoint. Migrate clients to OpenAPI v3.

• ConsistentListFromCache (Alpha): The new ConsistentListFromCache feature gate lets the API server use the watch cache to serve LIST requests with consistency guarantees, reducing load on etcd.

• Additional monitoring metrics: More metrics are now available through the metrics endpoint.

Kubernetes 1.27

Pod termination status corrected

• Pods deleted in the Pending phase are now set to Failed.

• Pods deleted in the Running phase are set to Succeeded or Failed based on the container exit code.

This fixes a bug where pods with a configured failure policy could remain stuck in Pending. If a pod has RestartPolicy=Always, it may now show Succeeded after deletion—review your controllers if this behavior matters for your workloads. See Set the termination status for pods that do not require a restart.

ReadWriteOncePod for PersistentVolumes (Beta)

The ReadWriteOncePod access mode limits a volume to a single pod at a time—more restrictive than ReadWriteOnce, which allows multiple pods on the same node. See Single Pod Access Mode for PersistentVolumes graduates to Beta.

Pod topology spread constraints (Beta)

Several topology spread constraint options are now Beta and enabled by default: minDomains (minimum number of eligible domains), nodeTaintsPolicy (whether to count tainted nodes), nodeAffinityPolicy (whether to count nodes excluded by affinity rules), and whenUnsatisfiable (behavior during rolling updates when constraints cannot be met). See More fine-grained pod topology spread policies.

Server-side field validation (GA)

The API server now validates resource fields server-side. kubectl automatically uses Strict mode and skips client-side validation, returning an error if the server rejects a field. See Server Side Field Validation and OpenAPI V3 move to GA.

OpenAPI v3 (GA)

OpenAPI v3, introduced in Kubernetes 1.23, is now GA. See Server-side field validation and OpenAPI v3 move to GA.

HPA ContainerResource metric type (Beta)

Horizontal Pod Autoscaler (HPA) now supports the ContainerResource metric type, which scales based on the resource usage of individual containers rather than the pod average. This addresses a common issue where a pod fails to scale out because a low-usage sidecar container pulls the average below the threshold, even though the application container is resource-constrained. See Container resource metrics.

StatefulSet enhancements (Beta)

Two StatefulSet features reach Beta:

• `StatefulSetStartOrdinal`: Start pod ordinals at a number other than zero using the ordinals.start field.

• `StatefulSetAutoDeletePVC`: Control whether StatefulSets delete PVCs created from volumeClaimTemplate when pods are deleted or the StatefulSet is scaled in.

In-place pod vertical scaling (Alpha)

The InPlacePodVerticalScaling feature gate (Alpha, disabled by default) lets you change the CPU and memory requests and limits for a running container without restarting the pod. See Resize CPU and memory resources assigned to containers.

Parallel image pulls

Set serializeImagePulls: false in the kubelet configuration to pull container images in parallel instead of serially. Use maxParallelImagePulls (added in 1.27) to cap concurrent pulls and avoid saturating network bandwidth or disk I/O. See Serial and parallel image pulls.

Volume group snapshots (Alpha)

A new volume group snapshot API lets you create crash-consistent snapshots of multiple PersistentVolumes at a single point in time. See Introducing an API for volume group snapshots.

Kubernetes 1.28

Kubernetes 1.27

Deprecated APIs

The storage.k8s.io/v1beta1 version of the CSIStorageCapacity API was deprecated in Kubernetes 1.24 and removed in 1.27. Use storage.k8s.io/v1, available from Kubernetes 1.24. See Storage Capacity Constraints for Pod Scheduling KEP.

Promoted to GA

The following feature gates reached GA and are permanently enabled. The feature gates themselves are removed.

In Kubernetes 1.28:

In Kubernetes 1.27:

Promoted to Beta (enabled by default)

In Kubernetes 1.28:

In Kubernetes 1.27:

New in Alpha (disabled by default)

In Kubernetes 1.28:

In Kubernetes 1.27:

Removed (no longer needed to configure)

In Kubernetes 1.27:

The IPv6DualStack feature gate was removed after reaching GA in v1.23. If you manually set this in your cluster configuration, remove it before upgrading.