· ai-devtools
Presentation: Serving LLMs at Scale: The Hidden KV Cache Advantage
InfoQ Homepage Presentations Serving LLMs at Scale: The Hidden KV Cache Advantage Khawaja Shams shares a distributed systems perspective on scaling LLM inference stacks efficiently.
He discusses practical strategies for optimizing GPU utilization, managing KV cache memory cliffs, leveraging tiered storage, reducing tail latency, and implementing disaggregated prefill and decode architectures to improve overall fleet performance, cost economics, and availability. Khawaja Shams is Co-Founder and CEO of Momento, a startup focused on high-performance data infrastructure, and a contributor to Valkey open source. He received the NASA Early Career Medal for his work on Mars Rovers, from onboard cameras to image processing pipelines.
At AWS, he led DynamoDB and served as VP of Engineering for AWS Elemental, bringing deep scale expertise to LLM inference. QCon AI is a practitioner-led event focused entirely on the engineering discipline required to scale these workloads safely. It provides direct access to the architectural playbooks and failure metrics that peer organizations use in production. Khawaja Shams: How many of you find yourselves to be experts in inference?
