Are you interested in laptop AI capabilities?
757 votes
What is an NPU?
NPU is an acronym for Neural Processing Unit. NPUs are dedicated to running mathematical functions associated with neural network/machine learning/AI tasks. While these can be standalone chips, they are increasingly integrated directly into a system-on-chip (SoC) alongside more familiar CPU and GPU components.
NPUs are dedicated to accelerating machine learning, aka AI tasks.
The idea is now taking off in the laptop and PC space, too. For instance, there’s the Neural Engine inside the latest Apple M4, Qualcomm’s Hexagon features in the Snapdragon X Elite platform, and AMD and Intel have begun integrating NPUs into their latest chipsets. While not quite the same, NVIDIA’s GPUs blur the lines, given their impressive number-crunching capabilities. NPUs are increasingly everywhere.
Why do gadgets need an NPU?
As we mentioned, NPUs are purpose-built to handle machine learning workloads (along with other math-heavy tasks). In layman’s terms, an NPU is a very useful, perhaps even essential, component for running AI on-device rather than in the cloud. As you’ve no doubt spotted, AI seems to be everywhere these days, and incorporating support directly into products is a key step in that journey.
A lot of today’s AI processing is done in the cloud, but this isn’t ideal for several reasons. First is latency and network requirements; you can’t access tools when offline or might have to wait for long processing times during peak hours. Sending data over the internet is also less secure, which is a very important factor when using AI that has access to your personal information, such as Microsoft’s Recall.
Put simply, running on-device is preferable. However, AI tasks are very compute-heavy and don’t run well on traditional hardware. You might have noticed this if you’ve tried to generate images via Stable Diffusion on your laptop. It can be painfully slow for more advanced tasks, although CPUs can run a number of “simpler” AI tasks just fine.
NPUs enable AI tasks to run on-device, without the need of an internet connection.
Talking of TOPS, smartphone and early laptop NPUs are rated in the tens of TOPS. Broadly speaking, this means they can accelerate basic AI tasks, such as camera object detection to apply bokeh blur or summarize text. If you want to run a large language model or use generative AI to produce media quickly, you’ll want a more powerful accelerator/GPU in the hundreds or thousands of TOPS range.
Is an NPU different from a CPU?
However, machine learning workloads are different and don’t need quite so much flexibility. They’re much more math-heavy for a start, often requiring repetitive computationally expensive instructions like matrix-multiply and very quick access to large pools of memory. They also often operate on unusual data formats, such as sixteen-, eight- or even four-bit integers. By comparison, your typical CPU is built around 64-bit integer and floating-point math (often with additional instructions added on).
An NPU is faster and more power efficient at running AI tasks compared to a CPU.
As a backup, CPUs can run machine-learning tasks but are often much slower. Modern CPUs from Arm, Apple, Intel, and AMD support the necessary mathematical instructions and some of the smaller quantization levels. Their bottleneck is often just how many of these functions they can run in parallel and how quickly they can move data in and out of memory, which is what NPUs are specifically designed to do.
Should I buy a laptop with an NPU?
While far from essential, especially if you don’t care about the AI trend, NPUs are required for some of the latest features you’ll find in the mobile and PC space.
Microsoft’s Copilot Plus, for example, specifies an NPU with 40TOPS of performance as its minimum requirement, which you’ll need to use Windows Recall. Unfortunately, Intel’s Meteor Lake and AMD’s Ryzen 8000 chips found in current laptops (at the time of writing) don’t meet that requirement. However, AMD’s newly announced Stix Point Ryzen chips are compatible. You won’t have to wait long for an x64 alternative to Arm-based Snapdragon X Elite laptops, as Stix Point-powered laptops are expected in H1 2024.
Popular PC-class tools like Audacity, DaVinci Resolve, Zoom, and many others are increasingly experimenting with more demanding on-device AI capabilities. While not essential for core workloads, these features are becoming increasingly popular, and AI capabilities should factor into your next purchase if you’re regularly using these tools.
CoPilot Plus will only be supported on laptops with a sufficiently powerful NPU.
Ultimately, AI is here, and NPUs are the key to enjoying on-device features that can’t run on older hardware. That said, we’re still in the early days of AI workloads, especially in the laptop space. Software requirements and hardware capabilities will only grow in the coming years. In that sense, waiting until the dust settles before leaping in won’t hurt.
You might like
Discover more from TechPros: Innovate, Learn & Connect
Subscribe to get the latest posts sent to your email.