· ai-devtools
Presentation: Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads
He discusses distilling a 7B model down to 400M parameters, compressing job contexts into single-token embeddings for a 10x throughput leap, and eliminating hidden system bottlenecks - from custom flash attention masks to Python GIL workarounds, gc.freeze, and non-blocking CUDA memory copies.
Sundara Raman Ramachandran is a Lead Engineer on LinkedIn’s LLM Inference team, where he plays a key role in designing and scaling the infrastructure powering LLM-based ranking systems for search and recommendation. His work centers on building latency-critical, high-throughput LLM serving platforms that operate reliably at global scale.
QCon AI is a practitioner-led event focused entirely on the engineering discipline required to scale these workloads safely. It provides direct access to the architectural playbooks and failure metrics that peer organizations use in production.
My current focus is on LLM inference and using LLMs for ranking and recommendation workloads. I'm also an open-source contributor in SGLang where I have mainly contributed to the scoring API where we use LLMs for ranking. I have close to 10 years of industry experience.




