KubeCon + CloudNativeCon India 2024

Applying modern deployment strategies, such as incremental releases, to complex ML architectures presents unique challenges, including managing resource constraints and ensuring cost efficiency. In the GPU environment, new replicas can take up to an hour to become operational due to extended image pull times. Despite these challenges, implementing advanced deployment strategies is crucial to minimise the impact of faulty versions, much like in traditional CPU workloads. In this session we will present on how to overcome these hurdles in GPU-based workloads. We will: - Start with traditional deployment approaches. - Progressively address advanced scenarios, such as implementing cost-efficient progressive delivery for high CPU-consuming workloads on GPU nodes and orchestrating sequential multi-resource deployments. - Reference our recent experience designing CI/CD for the GenAI applications. - Conclude by sharing cost comparison and best practices for GenAI model deployments.

Inference engines are used to generate prediction or deduce new information based on certain rules and data. With rise in the number of applications that benefit from inferencing, there is a definitive need to build Inferencing Services that are robust, performant (latency optimized) and can seamlessly scale on demand. In this session, we will discuss a proven set of procedures and guidelines for building and managing inference pipelines on Kubernetes. We will discuss details of the underlying hardware (GPU/CPU/memory) and K8s configuration requirements of some of the well-known Inference engines. We will demonstrate how robust and fault tolerant pipelines for LLM and RAG can be built using basic K8s constructs like operators, statefulsets and persistent volumes; along with enabling automated monitoring, triaging and remediation of the failed hardware and software components.

Traditionally, search engines and vector DBs (the boundaries got blurry lately) are deployed as monoliths in a cluster, managed by a StatefulSet and a lot of technology-specific logic. Newer use-cases (e.g. RAG, semantic search in general) implicitly push for more stateless work: e.g. generate embeddings on the fly, re-rank, local LLMs. Decoupling this work and managing it separately makes more sense now, but is it worth the complexity? This session takes a deep dive into the architectures of search engines (mainly Vespa, but Elasticsearch, Solr and others will be in the picture, too) to highlight both pros and cons of separating concerns in information retrieval. And how much separation is good for which scenario. In the end, you should have a better idea about both the technology and the deployment type that fits your use-case best.

As Large Language Models gain prominence, the discussion around AI fairness and ethics has never been more pressing. Cloud Native is emerging as the standard infrastructure for training, deploying, and serving AI workloads, but its role doesn't stop there—it can also play a crucial part in minimizing bias in AI models. This session will delve into: - Mapping Cloud Native principles like attestation and supply chain fundamentals to AI workloads, enhancing explainability and providing actionable recourse. - Leveraging Kubernetes' reconciliation loop to offer feedback that corrects false positives in AI output. - Utilizing the distributed nature of cloud computing to decentralize AI power, promoting more responsible automation. This session is designed for AI practitioners seeking insights from the Cloud Native tech stack, infrastructure beginners looking to understand its extensibility, and experts and maintainers aiming to extend the boundaries of current Cloud Native infrastructure.

As AI becomes more popular, it's bringing some new headaches when it comes to managing models, from building them to getting them into production. The usual DevOps tools don't always cut it, especially when you need everything to be consistent and handle large datasets without a hitch. As ML projects grow, trying to manage everything manually just gets messy and prone to mistakes. In this talk, I'll break down how GitOps can make life easier by automating and simplifying ML operations. With Git as the go-to source for everything, GitOps keeps your ML models consistent and reproducible across different environments. I'll share some real-world examples, talk about the challenges, and give you practical tips on how to bring GitOps into your ML workflow. The goal is to help you deploy models faster and with fewer headaches.

Explore AI innovation with the "Generative AI Demo using Open Data Hub," an open-source AI/ML platform tailored for Kubernetes. This demo features two Mistral-based LLMs, one utilizing Retrieval-Augmented Generation (RAG) with PostgreSQL vector extensions, and another without. Users interact through a Gradio chatbot, demonstrating how RAG improves accuracy. Two persona-based pipelines (Admin & Data Scientist) highlight Open Data Hub’s seamless integration with Tekton-powered OpenShift Pipelines, showcasing the Hybrid Cloud Strategy. Key components include the Kubernetes Platform, Node Feature Discovery, NVIDIA GPU Operators, Software Defined Storage, Tekton, Argo CD (for GitOps), Gitea Operator, Knative and Istio. Join us to see how GPU-accelerated AI and scalable, secure infrastructure power next-gen applications.