KubeCon + CloudNativeCon India 2024

Inference engines are used to generate prediction or deduce new information based on certain rules and data. With rise in the number of applications that benefit from inferencing, there is a definitive need to build Inferencing Services that are robust, performant (latency optimized) and can seamlessly scale on demand. In this session, we will discuss a proven set of procedures and guidelines for building and managing inference pipelines on Kubernetes. We will discuss details of the underlying hardware (GPU/CPU/memory) and K8s configuration requirements of some of the well-known Inference engines. We will demonstrate how robust and fault tolerant pipelines for LLM and RAG can be built using basic K8s constructs like operators, statefulsets and persistent volumes; along with enabling automated monitoring, triaging and remediation of the failed hardware and software components.