Get ready for a game-changer in the world of AI! Google's new LiteRT accelerator is about to revolutionize on-device AI performance on Snapdragon-powered Android devices.
Introducing the Qualcomm AI Engine Direct (QNN), a powerful tool developed by Google in collaboration with Qualcomm. This accelerator is designed to enhance AI capabilities on Android devices equipped with Snapdragon 8 SoCs, offering an incredible boost in performance.
While GPUs are commonly used for AI tasks, Google software engineers Lu Wang, Wiyi Wanf, and Andrew Wang highlight a potential bottleneck. They explain that relying solely on GPUs for complex AI operations, like text-to-image generation and live camera feed processing, can lead to a less-than-ideal user experience with potential frame drops and jitter.
But here's where it gets interesting: many mobile devices now come with Neural Processing Units (NPUs) - custom-designed AI accelerators. These NPUs can significantly speed up AI workloads compared to GPUs, all while consuming less power. It's a win-win situation!
QNN is a replacement for the previous TFLite QNN delegate, and it's a game-changer. It provides developers with a streamlined workflow, integrating various SoC compilers and runtimes into a simple API. With support for 90 LiteRT operations, QNN aims to enable full model delegation, a key factor in achieving optimal performance. And it doesn't stop there; QNN includes specialized kernels and optimizations to further boost the performance of LLMs like Gemma and FastLVM.
Google put QNN to the test, benchmarking it across 72 ML models. The results speak for themselves: 64 models achieved full NPU delegation, resulting in performance gains of up to 100 times compared to CPU execution and 10 times compared to GPU. That's a massive improvement!
On the latest Snapdragon 8 Elite Gen 5 SoC, the performance benefits are even more impressive. Over 56 models run in under 5ms with the NPU, while only 13 models achieve that on the CPU. This opens up a whole new world of live AI experiences that were previously out of reach.
Google engineers also developed a concept app using an optimized version of Apple's FastVLM-0.5B vision-encoding model. This app can interpret live camera scenes almost instantly. On the Snapdragon 8 Elite Gen 5 NPU, it achieves an incredible time-to-first-token (TTFT) of just 0.12 seconds on 1024×1024 images, with over 11,000 tokens/sec for prefill and more than 100 tokens/sec for decoding. The secret? Int8 weight quantization and int16 activation quantization, unlocking the NPU's high-speed int16 kernels.
QNN is currently supported on a limited subset of Android hardware, primarily devices powered by Snapdragon 8 and Snapdragon 8+ SoCs. To get started, check out the NPU acceleration guide and download LiteRT from GitHub.
So, what do you think about this new development? Is this a game-changer for on-device AI performance? Let's discuss in the comments and share our thoughts on this exciting advancement!
