Google Launches LiteRT Tensor API for Building, Customizing, and Optimizing ML Models for On-Device

The traditional approach, Google says, is for developers to write computations manually in C++ for maximum control, which comes with a major drawback: Implementations are tightly coupled to specific backends (such as Metal, OpenCL, or Vulkan) and must be rewritten for each new hardware target. Furthermore, managing zero-copy buffer interoperability and synchronization across CPU, GPU, and NPU, while gluing together pre-processing, inference, and post-processing, demands substantial manual engineering effort. The LiteRT Tensor API aims to address these challenges by allowing developers to build computation graphs operation by operation in C++.

This provided greater control over model architecture, optimization, and accelerator execution, while enabling the same graphs to run through LiteRT on CPU, GPU, and NPU hardware without requiring a rewrite for each new target. Google used a coding agent to create a real-world app demonstrating how to use the LiteRT Tensor API for multi-object segmentation and tracking.