Do I Need an AI Chip in my Laptop?

I recently did a deep dive researching into NPUs (Neural Processing Units), the chips that power Copilot+ PCs, specifically trying to find out just *what they actually do* in a computer, and why anyone would need them. And it turns out that the answer is interesting but also confusing!

CPUs and GPUs and NPUs, oh my!

Some people might be familiar with a CPU (Central Processing Unit), the main brain of your computer. Many people are familiar with GPUs (Graphical Processing Unit), which are often used in gaming for rendering complicated 3D graphics, video editing for compressing and handling video, and recently running AI computations. The new kid on the block is the NPU (Neural Processing Unit), which aims to specialize in AI tasks.

But why couldn't you just use the CPU or GPU (or both)? The answer comes down to how quickly those tasks can be accomplished. CPUs are designed to run tasks one at a time (though with cores they can run several tasks at a time), using whole numbers. GPUs are very good at doing a ton of calculations on numbers with decimal fractions. NPU bridge the gap, being good at doing many calculations on whole numbers quickly and with low power. They run with a simpler design than a CPU, so you couldn't run a whole computer off them, and they work best on whole numbers, so they wouldn't be good at the graphical tasks that GPUs do.

A product in search of a solution

These chips seem to fit a great niche, so what do you get today if you buy a laptop with one? The answer is, sadly, not much (at least not yet). A recent thorough test of several devices with NPUs and applications boasting about AI showed that Copilot, ChatGPT, and other "Chatbot" AIs don't use any part of your computer and run all queries in the cloud. Adobe uses the GPU. Some things do use the NPU though: Apple's notification summary seemed to use the NPU, and the results are underwhelming; Windows Recall but only supported browser windows; and Zoom, which uses the NPU for blurring and background replacements. So I guess if you're going to be on a Zoom call all day and are worried about your battery, you can rest easy knowing it's probably giving you an extra few minutes of life!

So why do these exist?

While the idea is sound: as models get more efficient, the ability to run some tasks locally and offline becomes more appealing, especially to cloud providers who are spending billions on a technology that has yet to be profitable. The irony is, Microsoft is OpenAI's biggest investor, and if NPUs take off, they would likely see their cloud service usage plummet as people would start using cheaper, open-source models. The true answer is that the companies pushing NPUs (Microsoft, Apple, Intel, Qualcomm, Samsung, Nvidia) are all companies that either make NPUs or make the OS/software that uses them. It's very likely that we're in an AI bubble, and NPUs are the next way to continue to press hype for generative AI.

So, do you need a laptop with an NPU? The answer is a resounding no. There's no "killer app" even on the horizon, and the apps that exist don't use it, or are underwhelming in terms of the benefits that NPUs provide.