KubeCon + CloudNativeCon India 2024

It all started when CoreDNS pods entered a crash loop backoff state soon after the cluster was deployed. The only change was an unprivileged execution desired in the latest coreDNS image. The issue occurred only in worker nodes, not in control plane nodes, and refreshing the image by deleting and pulling it again resolved the issue. It happened in one OS distribution flavor but not in another. We investigated Docker build, setcap, and libc, and found that the required capability was missing in nodes during the issue. The stack was complex, involving FIPS/CGO, build system, runtime configuration, and binary packaging in OVA. Debugging included CoreDNS binary, containerd, runc, SELinux, AppArmor, Photon OS, and the kernel... lots of learning which could be useful for many developers/admins or cluster operators to debug unprivileged pod execution. Finally, we discovered something interesting in the bootstrapping scripts. Let's debug.

As organizations scale, robust disaster recovery becomes critical. As clusters multiply and complexities grow, maintaining consistent RPO and RTO becomes challenging, requiring new techniques for efficient large-scale data backup and recovery. While GitOps excels at maintaining the desired state of infrastructure using Git as the single source of truth, it falls short for stateful workloads like databases and S3 buckets. Tools like Crossplane help deploy these resources in a GitOps model, but disaster recovery remains complex. You can't just spin up a new K8s cluster, point it to a Git repo, and expect a seamless recovery of critical stateful data Disaster recovery isn't 'one size fits all', each scenario requires a different approach depending on the scale of the disaster. In this talk, we’ll explore different strategies for handling different levels of disasters, from losing a single K8s cluster to facing a full region outage, helping folks to be better prepared for the unexpected.

As Large Language Models gain prominence, the discussion around AI fairness and ethics has never been more pressing. Cloud Native is emerging as the standard infrastructure for training, deploying, and serving AI workloads, but its role doesn't stop there—it can also play a crucial part in minimizing bias in AI models. This session will delve into: - Mapping Cloud Native principles like attestation and supply chain fundamentals to AI workloads, enhancing explainability and providing actionable recourse. - Leveraging Kubernetes' reconciliation loop to offer feedback that corrects false positives in AI output. - Utilizing the distributed nature of cloud computing to decentralize AI power, promoting more responsible automation. This session is designed for AI practitioners seeking insights from the Cloud Native tech stack, infrastructure beginners looking to understand its extensibility, and experts and maintainers aiming to extend the boundaries of current Cloud Native infrastructure.

Imagine adding all your favorite products to your shopping cart, only to have the page reload and—poof—everything’s gone. Frustrating, right?. For mission-critical apps, seamless upgrades in Kubernetes demand a solid grasp of how to tackle these challenges effectively. This session equips you with practical strategies for performing zero-downtime upgrades in Kubernetes. We will cover the role of readiness and liveness probes, explore rolling update strategies and Blue/Green deployments, and compare their benefits. We will also discuss handling SIGTERM signals, managing sidecar containers, and the importance of PodDisruptionBudgets and TopologySpreadConstraints. Finally, we will examine how Priority Classes and Pod Eviction impact stability during Kubernetes upgrades. By the end, you'll be equipped to confidently handle upgrades without downtime, ensuring your applications stay online and reliable, even during the most challenging updates.

As AI becomes more popular, it's bringing some new headaches when it comes to managing models, from building them to getting them into production. The usual DevOps tools don't always cut it, especially when you need everything to be consistent and handle large datasets without a hitch. As ML projects grow, trying to manage everything manually just gets messy and prone to mistakes. In this talk, I'll break down how GitOps can make life easier by automating and simplifying ML operations. With Git as the go-to source for everything, GitOps keeps your ML models consistent and reproducible across different environments. I'll share some real-world examples, talk about the challenges, and give you practical tips on how to bring GitOps into your ML workflow. The goal is to help you deploy models faster and with fewer headaches.

gRPC is often chosen for its high-performance communication, making it crucial to minimize latency. Distributed tracing helps identify and diagnose latency bottlenecks in gRPC calls, ensuring that the performance benefits of gRPC are fully realized. Setting up tracing from scratch for gRPC applications is complex and warrants significant maintenance overhead. Moreover, there is not much literature on how to do it properly, resulting in bespoke and suboptimal solutions. In this talk, Purnesh Dixit from the gRPC team will talk about the best practices for utilizing distributed tracing effectively in gRPC environments by configuring the new OpenTelemetry plug-in, created by the gRPC team in google.