· ai-devtools

Google Open-Sources TPU Sync for KV Cache and Weight Transfer on TPUs

GCP also unveiled the TPU 8tx and TPU 8i Max systems, the project targets two workloads where accelerators often wait on data: long-context and disaggregated LLM serving, and reinforcement learning loops that push updated weights from trainers to rollout samplers.


For prefill/decode disaggregation, Google says median KV transfer latency for Qwen3 235B fell from several seconds to about 230 ms, with average TTFT reductions of 60% for that model and 90% for a 480B model at low concurrency. For prefix caching, it reports up to 90% lower average TTFT on 64K-context workloads.

Developers get Python wheels for JAX and PyTorch, with KV cache manager, KV store, and weight synchronizer APIs. TPU Sync acts as the KV connector for vLLM-TorchTPU through vLLM's KVConnector interface, as the disaggregation and hierarchical-cache backend for SGLang-TorchTPU, and pairs with llm-d 1.0 on GKE for prefix-cache-aware routing.

Earlier coverage

Related coverage

Also worth reading

  1. next.js 16.5.0-canary.50 released
  2. OpenAI Daybreak brings AI agents for security testing, vulnerability investigation, and fixes
  3. Podcast: Doing The Right Thing: Accountability In The Age Of AI Coding Summary
  4. Is there a 'right amount' of AI?
  5. Your pinned MCP server may not be as pinned as you think