· ai-devtools
Podcast: Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM
InfoQ Homepage Podcasts Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes.
Edoardo explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization. Play episode Play Related Sponsors Transcript Olimpiu Pop: Hello everybody, I'm Olimpiu Pop an InfoQ editor, and I have in front of me today, Edoardo Vacchi, one of the two Italians that usually talk about boxes inside of boxes.
The other one is Andrea Peruffo. Eduardo worked for a long period of time in the WebAssembly ecosystem, he did some spotting around other types of VMs, and today he's back at Red Hat and he's trying to make things easier in terms of Kubernetes and inference.
