Pull up a chair. If you are reading this, there is a high probability you are staring at a monitoring dashboard right now, watching your database latency spike while your CPU cores burn cycles doing absolutely nothing useful. You recently repatriated a critical workload from the cloud back to high-spec bare-metal infrastructure. You bought state-of-the-art enterprise U.3 or E3.S NVMe drives capable of millions of IOPS, wired them up to your SAN storage array using your battle-tested iSCSI infrastructure, and then hit an invisible wall.

Your drives are idling at 5% capacity, yet your host storage latency is hovering between 200 and 300 microseconds. Worse still, your dual-socket database hosts are spending 25% to 40% of their total CPU cycles in system space, thrashing through kernel interrupt handlers and lock contention. You are not alone in this frustration. This is the classic architectural trap that hits teams transitioning to ultra-fast flash storage on legacy network block storage protocols.

Why Does Legacy iSCSI Create a CPU and Latency Bottleneck on Bare-Metal Systems?

To understand why iSCSI chokes modern flash storage, we have to look back at the technological landscape of 1998. The Internet Small Computer System Interface (iSCSI) was designed in an era dominated by spinning mechanical hard disk drives (HDDs). A high-end 15,000 RPM SAS drive of that era could deliver roughly 200 IOPS with an access latency of 3 to 5 milliseconds. The underlying SCSI storage protocol was architected around a single hardware command queue capable of holding just 32 commands (Queue Depth = 32).

When you encapsulate SCSI commands inside TCP/IP packets over an Ethernet network—which is precisely what iSCSI does—you carry every piece of legacy SCSI technical debt straight into your modern infrastructure. The operating system kernel’s storage stack must route every single read and write request through the centralized SCSI mid-layer (drivers/scsi in Linux).

Here is what happens inside your host system every time an application issues an I/O request over iSCSI:

1. Global Driver Lock Contention: Because the legacy SCSI architecture relies on single-queue semantics, multi-core processors cannot issue I/O in parallel without synchronization. When 64 or 128 CPU cores attempt to submit I/O simultaneously, they collide at the SCSI driver spinlocks (e.g., spin_lock_irqsave). Cores spend precious clock cycles waiting for permission to submit an I/O request.
2. Deep Kernel Stack Traversals: An I/O request must travel from the application’s user space down through the Virtual File System (VFS), into the generic block layer, down into the SCSI mid-layer, into the iSCSI initiator driver, and finally down through the kernel’s TCP/IP network stack (net/ipv4/tcp.c). This journey involves multiple context switches and socket buffer memory allocations (sk_buff).
3. Interrupt Saturation: As network interface cards (NICs) receive iSCSI responses, they trigger hardware interrupts. The kernel must handle these interrupts, wake up the waiting storage processes, and copy payload data from kernel network buffers into application memory space. At 200,000 IOPS, these software interrupts (softirq) saturate multiple CPU cores completely.

The result is a hard protocol ceiling. A single host core handling iSCSI I/O saturates at roughly 150,000 to 200,000 IOPS while introducing 100 to 300 microseconds of pure software latency. Installing enterprise Gen4 or Gen5 NVMe drives—which natively feature sub-10-microsecond media latencies—behind an iSCSI network target is like mounting high-performance racing tires on a horse-drawn carriage. The carriage frame simply cannot handle the transfer of power.

How Does NVMe over Fabrics Bypass Kernel Locks to Deliver Sub-Microsecond Performance?

NVM Express (NVMe) was not designed for spinning disks; it was built from the ground up for solid-state non-volatile memory and highly parallel multi-core CPU architectures. The contrast between SCSI and NVMe architectural capabilities is stark:

• SCSI Architecture: 1 command queue, max 32 commands per queue. Requires global locking.
• NVMe Architecture: Up to 64,000 parallel command queues, each supporting up to 64,000 commands per queue. Zero global locking.

NVMe over Fabrics (NVMe-oF) extends this native NVMe protocol across an Ethernet or InfiniBand network fabric without altering the underlying command set. NVMe-oF maps CPU cores directly to dedicated hardware I/O queue pairs (Submission Queue and Completion Queue). If your bare-metal server has 128 logical CPU cores, the host NVMe driver establishes 128 independent pairs of I/O queues directly to the target storage array.

When implementing NVMe-oF, enterprise architects generally select between two primary transport protocols: NVMe/RoCEv2 (RDMA over Converged Ethernet) and NVMe/TCP.

1. NVMe/RoCEv2 (Remote Direct Memory Access)

RoCEv2 bypasses the host host kernel operating system completely during data transfer phase. Utilizing specialized network adapters (SmartNICs or RNICs), the storage controller performs Direct Memory Access (DMA) operations straight across the network network. Data moves directly from the target storage array’s memory into the host application’s user-space memory buffer. Kernel context switches, socket allocations, and CPU software interrupts are entirely bypassed. End-to-end network protocol latency drops to 10–15 microseconds—virtually indistinguishable from a drive plugged directly into a local motherboard PCIe slot.

2. NVMe/TCP (Standard Ethernet Transport)

NVMe/TCP encapsulates NVMe command structures directly inside standard TCP segments. Unlike RoCEv2, NVMe/TCP utilizes the kernel’s existing networking stack, meaning standard network interface cards can be utilized without hardware replacement. However, because it retains NVMe’s multi-queue lockless model (mapping CPU cores to independent TCP socket connections), it avoids the SCSI lock contention of iSCSI. NVMe/TCP delivers end-to-end latencies between 40 and 70 microseconds—cutting legacy iSCSI latency by over 70% while requiring zero network infrastructure modifications.

What Network Infrastructure Do You Actually Need for NVMe/TCP vs NVMe/RoCEv2?

The single biggest mistake infrastructure engineering teams make when evaluating NVMe-oF is assuming all network fabrics are created equal. Upgrading to NVMe-oF requires a clear-eyed assessment of your physical network switching topology.

The RoCEv2 Infrastructure Reality Check

RoCEv2 is an unacknowledged, hardware-level memory transport running over UDP (Destination Port 4791). Because RDMA memory engines assume a zero-loss physical medium, dropping even a single packet forces RoCEv2 to initiate expensive, hardware-level retransmissions that destroy performance. Therefore, RoCEv2 requires a Lossless Ethernet Fabric.

To run RoCEv2 reliably in production, your network engineering team must configure and maintain three complex network standards across every switch leaf and spine in the data path:

1. Priority-Based Flow Control (PFC – IEEE 802.1Qbb): Operates at Data Link Layer (Layer 2) to pause traffic on specific Class-of-Service (CoS) queues when switch buffer thresholds are breached, preventing frame drops.
2. Explicit Congestion Notification (ECN – RFC 3168): Operates at Layer 3. Switches mark packet headers when queues fill up. The receiving host NIC sees these marks and instructs the sending target to throttle back its transmission rate before PFC pause frames are triggered.
3. Data Center Bridging Exchange (DCBX): Automatically negotiates PFC and ECN settings between host RNICs and physical switch ports.

If your team lacks deep experience in data center network engineering, PFC can introduce severe operational hazards. A misconfigured switch or a flapping host adapter can trigger a PFC Storm, where pause frames propagate backward through the entire switch fabric, halting all network traffic across un-related systems in a classic Head-of-Line (HoL) blocking cascade.

The NVMe/TCP Operational Advantage

In contrast, NVMe/TCP runs over standard IP networks. It relies on standard TCP windowing and congestion control mechanisms (such as CUBIC or BBR) to handle packet drops gracefully. You do not need PFC. You do not need ECN. You do not need specialized switches like Arista Spectrum or Cisco Nexus 9000 enterprise series; standard top-of-rack commodity 25GbE or 100GbE switches will handle NVMe/TCP out of the box.

Which Protocol Wins? The Protocol Nuance Matrix

When selecting a storage protocol for high-performance bare-metal repatriation, evaluating pure speed is insufficient. You must weigh hardware requirements, software maintenance costs, and operational risk factors.