Kubernetes is constantly evolving to meet and adapt to the growth of AI, increasing workloads, and even security concerns. If you’ve followed Kubernetes for a while, you’ve felt the ground move under your feet, sometimes because of technology, sometimes because of people.
SoftwarePlaza conducted a webinar with Abdel (long-time Googler, CNCF ambassador, and co-host of the Kubernetes Podcast) where a few clear patterns were discussed.
This blog post explores the most significant technical and cultural shifts currently impacting the ecosystem, and their implications for platform teams, SREs, and builders deploying on Kubernetes.
1. Gen AI rewrote the assumptions about “cloud-native”
Kubernetes thrived by standardizing stateless, microservice-heavy architectures. GenAI, particularly LLM inference, involves workloads that resemble large, stateful monoliths, requiring tight locality, high bandwidth, and GPU/accelerator density. As a result, GPUs don’t have enough memory for frontier models on a single card, forcing tensor/parameter sharding across multiple GPUs, and then, often, across multiple nodes because of PCIe slot limits per server.
Consequently, the long-standing mental model, “a pod fits on one node”, starts to wobble. Teams increasingly think in node groups (or “islands”) as a single logical placement unit for one model instance.
The takeaway: Kubernetes remains the control plane of choice for many, but inference-heavy AI pushes it into new territory where scheduling, networking, and storage assumptions are being reworked.
2. Kubernetes is evolving to meet accelerator-hungry workloads
The open source platform is reshaped to accommodate the increasing workload of its users. To that end, there are two significant changes.
The first being Dynamic Resource Allocation (DRA). This newer API approach lets workloads ask the cluster for specific resources (e.g., a family of GPUs, IPs, storage classes) and have the platform satisfy those requests dynamically. This is critical when accelerators are scarce or spread across heterogeneous pools.
There is also a significant shift from single nodes to node “groups”. While not a formal API today, the operational pattern is emerging: treat a tightly connected set of nodes as the unit of scheduling and operations for one LLM instance. Expect more tooling and controllers that abstract these groups as first-class constructs for placement, updates, and failure handling.
The takeaway: Declarative, API-driven resource negotiation and multi-node placement abstractions will define the next phase of “Kubernetes for AI.”
3. The ops reality: hardware scarcity and heavyweight artifacts
Apart from design patterns, other parameters that matter to Kubernetes users must also be addressed. Whether you’re on-prem or in a major cloud, accelerators are scarce and often locked behind reservations or enterprise commitments. This impacts capacity planning, DR strategies, and regional redundancy.
Similarly, years of work made container images minimal and secure. AI has reintroduced 10–100+ GB images (inference servers + models), slowing rollouts, increasing storage pressure, and complicating supply-chain scanning.
The takeaway: Lead times and logistics now matter as much as YAML. Invest in artifact strategies (OCI model registries, layered model formats, image streaming) and capacity brokering.
4. Multicloud is real, and operationally divergent
Many organizations now deploy across at least two providers. Kubernetes smooths some edges, but cloud-specific quirks still leak into networking, security, and identity. Add GPUs and specialized storage to the mix, and drift grows:
- Different CNI defaults, service/load-balancing behaviors, and IAM envelopes.
- Different accelerator SKUs, quotas, and provisioning latencies.
- Different managed-K8s opinions (upgrade cadence, node images, autoscaling internals).
The takeaway: Treat multicloud as an engineering product, with a compatibility matrix, golden paths per cloud, and strong conformance tests, rather than hoping “Kubernetes makes it all the same.”
5. What matters at KubeCon isn’t just code, it’s community
There is also a definite culture shift that was witnessed in KubeCon. The cloud-native community is now more receptive to diversity scholarships, accessibility support (including interpreters for Deaf/hard-of-hearing contributors), and programs to bring new voices on stage have become part of the KubeCon fabric. This inclusivity is the first step in broadening the group of maintainers, reviewers, and users, which makes the tech itself more robust.
The takeaway: The CNCF community has matured from “IRC-thread sharpness” to mentorship, inclusion, and psychological safety, and that’s accelerating the ecosystem’s velocity.
6. Hype cycles come and go; enduring concerns persist
Every KubeCon has a season’s buzzword, Service Mesh, eBPF, Security, FinOps, MCP, but the signal beneath the noise remains consistent:
- FinOps never left: As clusters scaled, so did bills. Engineering needs clear showback/chargeback, granular cost visibility, and policy guardrails. Environmental sustainability often rides alongside resource efficiency, which is both a cost and a carbon issue.
- Pragmatism over complexity: Hot take from the field, “Most teams probably don’t need a service mesh.” If you do, be crisp about why (mTLS at scale, consistent L7 policy, traffic shaping for complex topologies). If not, keep it simple.
- Security remains table stakes: As images balloon with AI tooling and models, software supply chain risk and runtime isolation deserve renewed focus.
The takeaway: Be skeptical of new acronyms; double down on cost, simplicity, and security, they compound.
7. Handling inference operations in AI
For ML engineers, training and fine-tuning are part of their domain. The unsolved issue lies in getting models into production, keeping them healthy, and doing it economically. This often involves placing model shards to scale across nodes with the right accelerators, without starving other tenants.
Another method consists of rolling a 60–200 GB image without hours of pull time, safely updating quantization or tokenizer versions, and performing blue/green on GPU pools. To improve observability, include latency profiles (prefill vs. decode), token-throughput, cache hit rates, GPU memory fragmentation, and power/cost per request. For those looking to establish SLOs for AI, move beyond p95 latency to quality-aware SLOs (guardrail violations, rejection/deferral rates, cost per successful call).
The takeaway: Treat inference like any high-stakes production service, with capacity planning, SRE budgets, error budgets, and product-level governance.
Where is this headed?
Kubernetes won’t morph into an AI-only scheduler, but it will keep absorbing lessons from AI workloads: richer resource negotiation, better topology-aware scheduling, and more ergonomic abstractions for “big-thing-on-many-nodes.” Meanwhile, the community’s cultural choices, inclusion, mentorship, and pragmatic teaching will continue to attract new talent to the project.
If you’re investing in Kubernetes now, don’t wait for a perfect “AI edition.” Start building clear golden paths, embrace DRA-like models, proactively manage hardware scarcity, and keep your culture open and inviting. The teams that blend sound engineering with inclusive community practices are the ones shipping resilient platforms and enjoying the ride.
This blog is based on an interview with Abdel Sghiouar. You can watch the full video here.
Expert weighs in on technical and cultural shifts happening in the Kubernetes ecosystem while also discussing the future.
