KubeCon + CloudNativeCon India 2024

It all started when CoreDNS pods entered a crash loop backoff state soon after the cluster was deployed. The only change was an unprivileged execution desired in the latest coreDNS image. The issue occurred only in worker nodes, not in control plane nodes, and refreshing the image by deleting and pulling it again resolved the issue. It happened in one OS distribution flavor but not in another. We investigated Docker build, setcap, and libc, and found that the required capability was missing in nodes during the issue. The stack was complex, involving FIPS/CGO, build system, runtime configuration, and binary packaging in OVA. Debugging included CoreDNS binary, containerd, runc, SELinux, AppArmor, Photon OS, and the kernel... lots of learning which could be useful for many developers/admins or cluster operators to debug unprivileged pod execution. Finally, we discovered something interesting in the bootstrapping scripts. Let's debug.

As organizations scale, robust disaster recovery becomes critical. As clusters multiply and complexities grow, maintaining consistent RPO and RTO becomes challenging, requiring new techniques for efficient large-scale data backup and recovery. While GitOps excels at maintaining the desired state of infrastructure using Git as the single source of truth, it falls short for stateful workloads like databases and S3 buckets. Tools like Crossplane help deploy these resources in a GitOps model, but disaster recovery remains complex. You can't just spin up a new K8s cluster, point it to a Git repo, and expect a seamless recovery of critical stateful data Disaster recovery isn't 'one size fits all', each scenario requires a different approach depending on the scale of the disaster. In this talk, we’ll explore different strategies for handling different levels of disasters, from losing a single K8s cluster to facing a full region outage, helping folks to be better prepared for the unexpected.

As Large Language Models gain prominence, the discussion around AI fairness and ethics has never been more pressing. Cloud Native is emerging as the standard infrastructure for training, deploying, and serving AI workloads, but its role doesn't stop there—it can also play a crucial part in minimizing bias in AI models. This session will delve into: - Mapping Cloud Native principles like attestation and supply chain fundamentals to AI workloads, enhancing explainability and providing actionable recourse. - Leveraging Kubernetes' reconciliation loop to offer feedback that corrects false positives in AI output. - Utilizing the distributed nature of cloud computing to decentralize AI power, promoting more responsible automation. This session is designed for AI practitioners seeking insights from the Cloud Native tech stack, infrastructure beginners looking to understand its extensibility, and experts and maintainers aiming to extend the boundaries of current Cloud Native infrastructure.

Imagine adding all your favorite products to your shopping cart, only to have the page reload and—poof—everything’s gone. Frustrating, right?. For mission-critical apps, seamless upgrades in Kubernetes demand a solid grasp of how to tackle these challenges effectively. This session equips you with practical strategies for performing zero-downtime upgrades in Kubernetes. We will cover the role of readiness and liveness probes, explore rolling update strategies and Blue/Green deployments, and compare their benefits. We will also discuss handling SIGTERM signals, managing sidecar containers, and the importance of PodDisruptionBudgets and TopologySpreadConstraints. Finally, we will examine how Priority Classes and Pod Eviction impact stability during Kubernetes upgrades. By the end, you'll be equipped to confidently handle upgrades without downtime, ensuring your applications stay online and reliable, even during the most challenging updates.

Picture this: Your startup just landed a big client, and as your business starts to grow, so does your Kubernetes cluster. The number of microservices and the traffic that they serve has started increasing. You configure auto-scaling of Nodes and Pods. Do you think that is enough, or do you need to look into other areas? What other components need scaling? What strategies should be devised? How do we verify whether there will not be any impact during scaling? In this session, we answer all these questions and come up with a blueprint for scaling a Kubernetes cluster from our experience with scaling our production cluster from merely 20-25 nodes to around 250 nodes with more than 5000 pods. We will touch base on the issues we faced and their resolution that will form a good prescription for those who want to scale their Kubernetes infrastructure.

gRPC is often chosen for its high-performance communication, making it crucial to minimize latency. Distributed tracing helps identify and diagnose latency bottlenecks in gRPC calls, ensuring that the performance benefits of gRPC are fully realized. Setting up tracing from scratch for gRPC applications is complex and warrants significant maintenance overhead. Moreover, there is not much literature on how to do it properly, resulting in bespoke and suboptimal solutions. In this talk, Purnesh Dixit from the gRPC team will talk about the best practices for utilizing distributed tracing effectively in gRPC environments by configuring the new OpenTelemetry plug-in, created by the gRPC team in google.