The first time I saw AWS benchmark numbers that showed near-bare-metal network performance for EC2 instances, I did not believe them. It was around 2017, shortly after the Nitro system launched. My instinct from twenty years of infrastructure work was that virtualization always costs something. You run a virtual switch in the hypervisor, you eat CPU cycles. You enforce tenant isolation, you eat more. There is no free lunch in software networking.
Nitro was not a free lunch. It was a hardware lunch. AWS had moved all that overhead off the host CPU and onto a dedicated piece of silicon sitting on the PCI bus. That card was running its own operating system, its own network stack, its own storage controller. From the guest’s perspective, the hypervisor had vanished.
That was the moment I understood what DPUs and SmartNICs actually are: not a clever software trick, but a structural redistribution of compute. The “third processor” model. You have your application CPUs. You have your GPUs for compute-heavy workloads. And now you have a separate programmable processor whose only job is to run the infrastructure plumbing so the other two can focus on what they were actually sold to do.
This article is about what that hardware is, how the major hyperscalers implement it, what specifically gets offloaded, and why every serious cloud architect needs a mental model of this layer to reason correctly about cloud performance, cost, and security isolation.
The Problem That DPUs Solve
Before we get into the hardware, let us be precise about the problem.
A cloud server running twenty tenant VMs needs to enforce a lot of guarantees that have nothing to do with any of those tenants’ actual workloads. It needs to tag and route traffic between VMs on the same host. It needs to enforce security group rules. It needs to encrypt storage I/O in transit. It needs to rate-limit network bandwidth per tenant. It needs to run the vSwitch that handles east-west traffic. It needs to process NVMe commands over the network fabric and present them as local block devices.
All of that costs CPU cycles. In the pre-DPU world, every cloud provider was essentially taxing their tenants silently: a percentage of every physical CPU core was consumed by hypervisor overhead that tenants paid for but never got to use. For compute-intensive workloads, this was infuriating. For I/O-intensive workloads, it was catastrophic. A busy storage node could spend 30-40% of its CPU budget on network stack processing before it even touched user data.
The noisy neighbor problem is the other side of this coin. If one tenant generates a burst of network traffic or storage I/O, the kernel interrupts required to handle it steal CPU time from adjacent tenants. Isolation in software is hard and expensive. The only reliable isolation is physical separation.
DPUs solve both problems at once: move all the infrastructure tax off the host CPU onto dedicated silicon, and enforce isolation at hardware boundaries where software cannot reach.
What a DPU Actually Is
A DPU (Data Processing Unit) or SmartNIC is a PCIe card that plugs into a server’s PCIe bus and connects to the network fabric. Inside, it contains several subsystems working together:
Network interface hardware. This is the NIC part. It handles physical layer, link-level framing, and at high-end cards, hardware offloads for TCP segmentation, checksum computation, and receive-side scaling.
Programmable packet processing pipeline. This is where the “smart” comes from. Unlike a traditional NIC that just moves packets, a SmartNIC runs a programmable data plane. The pipeline can inspect packet headers, apply match-action rules, forward or drop traffic, and modify headers, all at line rate without involving the host CPU. Many cards implement this using P4 (Programming Protocol-independent Packet Processors), a domain-specific language for describing data plane behavior.
General-purpose ARM cores. Most DPUs include several ARM Cortex cores running a full OS (usually Linux). This is where more complex control-plane logic runs: DHCP clients, routing daemon neighbors, storage protocol terminators, health checkers, and the management interfaces.
Hardware accelerators. On top of the ARM cores, DPUs include dedicated cryptographic accelerators for line-rate encryption and decryption, compression engines, and in newer generations, inline RDMA processing for zero-copy data transfers.
High-speed uplinks. The card has direct connections to the network fabric, typically 25GbE, 100GbE, or 400GbE depending on the generation, bypassing the host CPU entirely.
The critical architectural point is that the DPU is not visible as a network device to the host OS. The host sees virtual functions presented by the DPU, but the actual packet processing happens on the card. The host CPU never touches most traffic.
AWS Nitro: The Blueprint That Changed Everything
AWS’s Nitro system is the most mature and well-documented DPU deployment in production. Understanding it gives you the mental model for everything else.
The Nitro system is actually a family of cards, not a single piece of hardware. There are Nitro cards for networking, for storage (EBS), and for security (the Nitro Security Chip that handles hardware root of trust). The entire hypervisor control plane runs on these cards, not on the host CPU at all. When a Nitro-based EC2 instance boots, the host CPU cores are fully available to the tenant from day one, with zero hypervisor overhead.
For networking, the Nitro card implements the full AWS VPC data plane: security groups, network ACLs, VPC routing, and the GENEVE encapsulation that underlies VPC networking. This is the same work that a software vSwitch would do, running entirely on the card at hardware speeds. For EBS, a separate Nitro storage card terminates the NVMe-oF (NVMe over Fabrics) protocol, presenting local block devices to instances while the actual storage lives on distributed storage nodes across the fleet. The instance sees a local NVMe device. The Nitro card is translating every I/O over the network fabric transparently.
The security posture this enables is worth noting. Because all infrastructure functions run on Nitro cards with their own isolated firmware, AWS can make a meaningful claim that even a fully compromised host OS cannot escape the tenant’s security boundary. The hypervisor code that could theoretically access tenant memory is not running on the same CPU. This is a fundamentally different security model from software-based hypervisors.
This architectural separation is also why bare metal instances on AWS still get Nitro capabilities like VPC networking and EBS, even though there is no hypervisor. The infrastructure stack runs on the card, not on the host.
Azure Boost and Google Titanium
Microsoft’s Azure Boost program is their equivalent, announced in 2023 and now deployed at scale. Azure Boost moves the software-defined networking (SDN) stack, storage stack, and live migration traffic off the host CPU and onto dedicated hardware. Microsoft has been public about the performance gains: Boost-enabled VMs deliver higher and more consistent network and storage throughput than their predecessors because the host CPU is no longer contending with infrastructure overhead.
Azure Boost’s hardware accelerator cards use FPGA-derived logic for the critical path and ARM cores for control plane functions, similar to the Nitro architecture. Microsoft has also integrated CXL (Compute Express Link) work into this stack, pointing toward future memory tiering capabilities.
Google’s equivalent is the Titanium system, which includes purpose-built silicon that Google calls “Titanium chips” or Titanium MLUs (Machine Learning Units in the context of TPUs, but Titanium for networking). Google has less public documentation than AWS, but the architectural pattern is identical: dedicated infrastructure processors running the virtual networking layer and NVMe-oF storage termination, completely isolated from tenant CPUs.
NVIDIA’s BlueField DPU line (now in the BlueField-3 generation) is the commercially available version of this technology that you can deploy in your own data center or colocated infrastructure. BlueField-3 is a 400GbE DPU with 16 ARM Cortex-A78 cores running Ubuntu, hardware accelerators for cryptography and compression, and NVIDIA’s DOCA (Data Center Infrastructure-on-a-Chip Architecture) SDK for programming it. When I have worked with organizations running their own OpenStack or Kubernetes infrastructure at scale, BlueField is what comes up in the conversation about solving the vSwitch overhead problem.
What Actually Gets Offloaded
Let me make this concrete. Here is the set of functions that DPUs handle in production deployments:
Virtual switching. The Open vSwitch (OVS) that runs in software to handle VM-to-VM traffic on the same host can consume 2-4 cores on a busy server. DPUs implement OVS offload through kernel bypass and hardware accelerators, eliminating most of that overhead. This is the single biggest CPU recovery in most deployments.
Security enforcement. Stateful firewall rules, security groups, and network ACLs can be implemented in the DPU’s programmable packet processing pipeline. Every packet gets inspected at line rate without host CPU involvement. For high-connection-rate workloads like DDoS scrubbing or rate-limited API services, this is the difference between surviving a traffic spike and browning out.
Encryption and decryption. TLS termination, IPsec tunnels, and storage encryption are cryptographically intensive operations. DPUs include dedicated AES-GCM and SHA accelerators that can sustain 100Gbps of inline encryption at a fraction of the power cost of doing this on general-purpose cores. This is relevant to post-quantum cryptography migration as well: offloading ML-KEM key encapsulation to hardware accelerators on DPUs is the only practical way to sustain line-rate encryption with the new NIST algorithms.
Storage protocol termination. NVMe-oF (NVMe over Fabrics, typically over RDMA or TCP) allows remote storage to appear as local block devices. The DPU terminates the NVMe-oF protocol, handles the RDMA transport, and presents virtual NVMe devices to the host. This is exactly how EBS on AWS works: the block device in your instance is a Nitro card translating I/O across a fabric.
Live migration. Moving a running VM between physical hosts requires copying memory pages over the network without the tenant noticing. This is bandwidth-intensive and requires careful rate limiting to avoid affecting tenant networking. Running live migration traffic through the DPU’s dedicated uplink means it does not compete with tenant traffic or host CPU cycles.
Telemetry and flow collection. At 400Gbps line rate, you cannot afford software-based packet sampling. DPUs implement flow records and telemetry export in hardware, feeding into observability systems without touching the host.
DPDK: The Software Path That DPUs Replaced (and Extended)
Before DPUs, the standard answer to kernel networking overhead was DPDK (Data Plane Development Kit): a framework that bypasses the kernel network stack entirely, polls NICs directly from user space, and processes packets without the interrupt overhead and memory copies that make the kernel path expensive.
DPDK is still relevant, but the relationship with DPUs is interesting. For workloads that run directly on bare metal (network appliances, high-frequency trading infrastructure, telecom packet processing), DPDK is the right tool. You write your forwarding logic in C, run it in user space, get close to line rate on commodity CPUs.
For cloud infrastructure operators managing thousands of tenant VMs, DPDK on the host CPU is insufficient because the problem is not just performance, it is isolation. DPDK forwarding processes run in the host kernel’s user space, sharing CPU cores with other infrastructure processes. DPUs move the entire data plane off the host CPU, which is a qualitatively different answer.
The practical overlap is that many DPU programming models, including NVIDIA’s DOCA and Pensando’s P4-based pipeline, expose DPDK-compatible APIs. Code written for DPDK on CPU can often be ported to DPU execution with moderate effort. This matters if you are building your own SDN stack on top of commodity DPU hardware.
P4 and Programmable Pipelines
P4 deserves its own section because it is becoming the standard way to describe what a DPU’s packet processing pipeline does. P4 is a domain-specific language for programming match-action pipelines in network hardware. You describe your headers (Ethernet, IP, TCP, VXLAN, whatever you need), your tables (match on these fields, execute this action), and your parser/deparser logic. The P4 compiler generates configurations for the target hardware.
The reason P4 matters for cloud architects is that it makes the DPU’s data plane programmable without firmware changes. Cisco Pensando (now AMD Pensando), NVIDIA BlueField, Intel’s Mount Evans IPU, and Marvell’s LiquidIO all support P4-like programming to varying degrees. In a hyperscaler context, this means the team that manages the virtual networking layer can deploy new forwarding behaviors (a new encapsulation protocol, an additional ACL matching strategy) without waiting for hardware revisions.
For the operators running their own infrastructure, P4 is also how you express custom routing behaviors that would otherwise require a specialized network appliance. I have seen P4-programmed DPUs replace purpose-built load balancers in front of microservices clusters, because the match-action logic needed was simple enough to express in P4 and the throughput requirements were high enough that a software load balancer on a sidecar would have been problematic. The connection to Envoy proxy’s filter chain model is conceptually similar: describe what you want to happen to traffic, let the runtime enforce it efficiently.
RDMA, RoCE, and Storage Networking
The storage offload story is inseparable from RDMA (Remote Direct Memory Access). RDMA allows a DPU to transfer data directly between application memory on two machines, bypassing both machines’ CPUs entirely for the data path. Only the initiation and completion signals touch the CPU.
This is the technology underneath NVMe-oF-over-RDMA (NVMe-oF/RDMA), which is how AWS EBS actually moves data between the storage backend and the Nitro cards in compute instances. It is also the networking stack that makes GPU cluster training viable at scale: all-reduce operations across 1000-GPU clusters require memory copies between GPUs on different hosts, and RDMA is the only mechanism that can do this at the bandwidth and latency required without the host CPU becoming the bottleneck.
RoCE (RDMA over Converged Ethernet) is the version of RDMA that runs over standard Ethernet fabrics rather than specialized InfiniBand. Most hyperscaler storage networking and AI cluster fabrics use RoCEv2. The DPU handles both the RoCE transport layer and the NVMe-oF protocol layer, presenting remote storage devices as local to the host.
Understanding this stack matters when you are designing storage-heavy workloads in the cloud. The “local NVMe” you see inside an EC2 instance or a Google Cloud VM is almost always RDMA-backed NVMe-oF to a networked storage backend. The DPU is doing the translation work. This is why storage I/O latency on cloud instances has improved so dramatically over the past five years: software-based storage virtualization has been replaced by hardware-accelerated protocol termination.
What This Means for Cloud Architects
Twenty years in this field has taught me that understanding one layer below the abstraction you work at gives you a huge advantage. Most engineers who deploy workloads on EC2 or GCE think about vCPUs, memory, and network bandwidth. The DPU layer below gives you a better framework for several practical questions:
Why do “enhanced networking” instance types matter? Enhanced networking on AWS means the DPU is in the path for your instance’s network traffic, with SR-IOV (Single Root I/O Virtualization) presenting a virtual function directly to your instance with kernel-bypass semantics. Standard networking means more software switching overhead. The difference is significant for high-bandwidth workloads.
Why are bare metal instances sometimes the right answer for non-performance reasons? On AWS, a bare metal instance still sits behind a Nitro card. You get full VPC networking, security groups, and EBS without a software hypervisor. This means bare metal is useful not just for workloads with hypervisor overhead sensitivity, but for workloads that need to run their own hypervisor (nested virtualization) or have strict security requirements around hypervisor surface area.
Why does cloud storage latency have a floor that you cannot optimize past? The NVMe-oF path adds a round-trip over the fabric. For EBS in AWS, this is typically 100-200 microseconds added to every I/O, regardless of how fast your code is. No amount of application tuning removes this. It is inherent in the architecture. Instance store volumes bypass NVMe-oF and connect directly to local NVMe drives, which is why they have sub-millisecond latency.
How does zero-trust networking actually scale? In a large multi-tenant environment, enforcing per-flow mTLS and microsegmentation in software requires significant CPU overhead. DPUs change this equation: cryptographic enforcement at line rate in hardware means zero-trust enforcement does not degrade application performance. This is the architecture that makes aggressive microsegmentation practical at hyperscale.
Why do multi-region architectures behave differently from single-region ones? Cross-region traffic exits through the provider’s backbone network, bypassing the DPU-to-DPU fast paths that work within a region. This is why cross-region latency and cost behave differently from intra-region communication, regardless of bandwidth.
The BYOD DPU Future
An emerging trend worth watching is DPU adoption in enterprise data centers. NVIDIA BlueField-3, Marvell OCTEON, Pensando Elba, and Intel Mount Evans are all commercially available hardware that enterprises can buy and deploy. The use cases are similar to hyperscalers: offload the vSwitch overhead from a large Kubernetes cluster, enforce network policies in hardware, accelerate storage I/O for database workloads.
The operational complexity is real. You are now managing three classes of processor in your infrastructure: CPUs, GPUs, and DPU/SmartNICs. Each has its own firmware lifecycle, debugging tools, and failure modes. A misconfigured P4 pipeline on a DPU can silently drop traffic in ways that are genuinely hard to debug without good telemetry.
The eBPF ecosystem has helped here. Modern DPUs support XDP (eXpress Data Path) programs that can be loaded from the host, making it easier to express packet processing logic in a familiar programming model. Projects like Cilium’s DPU offload work and Open vSwitch’s hardware acceleration layer abstract the vendor-specific differences, giving operators a more portable programming interface.
The convergence I expect to see over the next three years: DPUs become standard infrastructure in large on-premises Kubernetes clusters, the way GPUs became standard for AI workloads. The platform engineering team that builds the internal developer platform will manage DPU firmware and P4 policies alongside Kubernetes network policies, treating them as different layers of the same networking control plane. For background on how this fits into the wider platform story, the Platform Engineering article covers the organizational patterns well.
A Final Observation
The DPU trend represents something bigger than a performance optimization. It is the hardware industry’s acknowledgment that the original hypervisor model, where the host CPU runs both tenant workloads and all infrastructure plumbing, was fundamentally flawed for multi-tenant cloud economics.
When I started in this industry, the hardware was dumb and the software was clever. All the interesting work happened in software running on general-purpose CPUs. Over twenty years, I have watched the pendulum swing back toward purpose-built hardware for the things that need to be fast, predictable, and isolated. FPGAs in CDN edge nodes, ASICs in network switches, TPUs for matrix math, and now DPUs for the infrastructure tax.
The lesson is not that software is dead or that hardware beats software. The lesson is that the right decomposition matters: put the right computations on the right hardware, with clean interfaces between them. The cloud providers who figured this out earliest built the most defensible performance moats. For architects and engineers, understanding where those hardware boundaries are and what they imply is becoming as fundamental as understanding TCP/IP was twenty years ago.
If you are not thinking about DPUs today, you will be thinking about them in three years, when the question of why some workloads perform well in the cloud and others do not will increasingly trace back to whether the DPU is in the fast path.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.