Pull up a chair, grab a fresh cup of coffee, and let’s have an honest talk about private cloud storage. If you have ever watched your database latency spike to 15 milliseconds while your brand-new PCIe Gen4 enterprise NVMe drives sit at 5% utilization, you are not alone. I’ve lost count of how many infrastructure audits I’ve conducted where brilliant engineering teams spent six figures upgrading host NICs to 100GbE and buying all-flash arrays, only to watch their storage throughput choke on the network transport protocol.

For twenty-five years, our industry built block storage on SCSI-based paradigms: iSCSI over IP networks or Fibre Channel over dedicated SAN fabrics. Those protocols were masterfully designed—for an era of spinning magnetic platters and single-core CPUs. When you run iSCSI over modern networks, you are essentially placing a fast-food drive-thru ordering queue in front of a Formula 1 pit stop. The storage media isn’t your bottleneck anymore; the protocol wrapper, the single command queue per LUN, and kernel lock contention are destroying your IOPS.

NVMe over Fabrics (NVMe-oF) breaks this bottleneck by extending native PCIe controller queues directly across your network. But moving to high-IOPS storage over IP comes with real architectural trade-offs that vendor marketing brochures love to gloss over. Today, we are going to unpack the mechanics, contrast NVMe/TCP against NVMe/RoCEv2, evaluate user-space drivers against the kernel, and build a production-ready deployment plan for your hypervisors.

Why Is Your Legacy iSCSI Architecture Throttling Modern Flash Drives?

To understand why NVMe-oF is revolutionary, we must examine why legacy iSCSI collapses under modern workloads. The fundamental architectural failure of iSCSI is rooted in the SCSI architecture model itself. SCSI was conceived around a single submission queue model. Even with modern SCSI tagged command queuing (TCQ), an iSCSI Target LUN typically exposes a maximum queue depth of 32 to 1024 commands per session. Worse, all IOPS funnel through a central software lock within the operating system kernel.

When an application on Hypervisor A writes to an iSCSI target over TCP/IP, the kernel must:

1. Translate the POSIX block request into a SCSI Command Descriptor Block (CDB).
2. Wrap the SCSI CDB inside an iSCSI Protocol Data Unit (PDU).
3. Pass the PDU through the Linux TCP/IP socket layer, triggering software interrupts and kernel context switches.
4. Process the packet on the storage target by stripping TCP headers, unwrapping the iSCSI PDU, translating SCSI commands to local controller commands, and executing the write.

This path introduces massive serialization delays. Modern multi-core servers running virtualized workloads generate thousands of parallel I/O threads. With iSCSI, those multi-core CPUs fight for access to serial hardware queues and kernel socket locks. You end up with CPU core pinning, high inter-processor interrupts (IPIs), and tail latencies that ruin database performance.

Local NVMe changed this paradigm entirely. A single NVMe drive natively supports up to 64,000 submission and completion queues, with each queue capable of holding up to 64,000 concurrent commands. NVMe matches host multi-core CPUs 1:1; CPU Core 0 submits I/O to NVMe Queue 0 without needing to lock or communicate with CPU Core 1. NVMe-oF simply takes those native PCIe submission and completion queues and projects them over a network fabric without SCSI translation.

Should You Deploy NVMe over TCP or RoCEv2 in Your Network?

Once you decide to deploy NVMe-oF over IP, you face the primary architectural split: NVMe/TCP versus NVMe/RoCEv2 (RDMA over Converged Ethernet). The decision here determines whether your deployment is an easy software upgrade or a complete physical network re-engineering project.

NVMe/TCP: High-Speed Simplicity on Standard IP Fabrics

NVMe/TCP (standardized in NVM Express specification 1.4) encapsulates native NVMe Command PDUs directly within standard TCP/IP frames. It requires no specialized Network Interface Cards (NICs), no proprietary switch ASICs, and no specialized network configuration. If your switches can route standard IPv4/IPv6 traffic over 25GbE or 100GbE, you can run NVMe/TCP right now.

Because it leverages the standard kernel TCP stack (or user-space TCP implementations), host CPUs handle TCP packet encapsulation and window management. While this introduces a small CPU overhead (typically 10-15% per 100k IOPS) and adds ~20 to 50 microseconds of transport latency compared to local PCIe, it provides sub-200 microsecond end-to-end latency reliably across standard leaf-spine networks. For 90% of private cloud environments, NVMe/TCP delivers incredible performance improvements over iSCSI without touching network infrastructure.

NVMe/RoCEv2: Sub-15 Microsecond RDMA Latency with Network Complexity

If your workload demands bare-metal PCIe speeds—such as high-frequency trading platforms, real-time analytics engines, or massive AI training nodes—NVMe/RoCEv2 is the ultimate transport. RoCEv2 uses Remote Direct Memory Access (RDMA) to bypass the host kernel entirely. Storage read/write operations move data directly between the host system memory (RAM) and the storage array’s target memory via specialized RDMA-capable NICs (RNICs) like Mellanox ConnectX or Broadcom Thor adapters.

Host CPU overhead drops to near zero because network processing occurs in hardware on the SmartNIC ASIC. End-to-end network overhead drops to under 10–15 microseconds. However, RoCEv2 requires a lossless Ethernet fabric. Standard TCP handles packet loss through retransmissions and window scaling; RDMA over UDP (RoCEv2) cannot handle dropped packets without severe performance degradation (known as catastrophic priority degradation).

To run RoCEv2 safely, your network team must configure and maintain:

• Priority Flow Control (PFC – IEEE 802.1Qbb): Operates at Layer 2 to send pause frames per traffic class, preventing switch buffer overflows.
• Explicit Congestion Notification (ECN – IEEE 802.1Qau / RFC 3168): Operates at Layer 3 to mark packets when buffer queues build up, instructing the sending NIC to throttle transmission before drops occur.
• Data Center Bridging Capability Exchange (DCBX): Dynamically negotiates lossless parameters between host adapters and switch ports.

Misconfigured PFC can trigger “PFC Deadlocks,” where pause frames ripple backward across your leaf-spine topology, freezing your entire network. Unless your team possesses deep network engineering expertise and dedicated monitoring for lossless fabrics, NVMe/TCP is the safer, more resilient architecture.

Kernel Drivers vs. SPDK: Where Does the Real Speed Live?

When implementing NVMe-oF target arrays or ultra-high performance nodes, you must decide where I/O operations are processed: within the traditional Linux kernel driver stack, or using the Storage Performance Development Kit (SPDK).

The standard Linux kernel driver stack (via nvme-tcp and nvmet-tcp) is robust and built directly into modern distributions like RHEL, Ubuntu Server, and Debian (Proxmox). Context switches and interrupt processing still occur, but because the kernel uses block multi-queue (blk-mq) architectures, it easily scales to hundreds of thousands of IOPS per node. For host initiators (hypervisors running virtual machines), the kernel driver is almost always the right choice due to native integration with hypervisor storage drivers, multipathing, and block devices.

However, on the target array side—where you aggregate hundreds of NVMe SSDs to serve an entire cluster—the standard kernel stack can become CPU-bound at around 1.5 million to 2 million IOPS. This is where SPDK shines.

SPDK is an open-source suite of drivers and libraries developed primarily by Intel. It moves the entire storage protocol stack out of kernel space and into user-space. SPDK uses Polling Mode Drivers (PMDs) rather than hardware interrupts. Instead of a CPU core going to sleep and waking up when an I/O completes (causing context switches), an SPDK dedicated core continuously polls the NVMe controller queues and network interface cards.

The architectural advantages of SPDK are striking:

• Zero Lock Contention: Each CPU core allocated to SPDK manages its own discrete queues and network sockets with zero inter-core synchronization locks.
• Zero-Copy Data Transfer: Data moves directly from network buffers to drive memory without landing in kernel page caches or requiring intermediate buffer copies.
• Extreme Core Efficiency: While kernel drivers might require 4 to 8 CPU cores to process 2 million IOPS, SPDK can drive 5+ million IOPS on a single dedicated CPU core.

The operational trade-off is clear: SPDK reserves entire CPU cores exclusively for storage processing (100% CPU usage will be reported on those cores in tools like top), and it bypasses traditional Linux administration utilities like lsblk or fdisk. Use SPDK for dedicated software-defined storage target nodes, and use Linux kernel drivers for host hypervisors.

How Do You Configure NVMe/TCP on Proxmox and Linux Hypervisors Today?

Let’s move from architecture to execution. The following 4-step playbook builds a resilient, highly available NVMe/TCP storage connection between a Linux/Proxmox VE hypervisor (initiator) and an NVMe-oF target array using dual 25GbE interfaces and kernel-native tools.

Step 1: Configure the Network Fabric and Kernel Modules

First, verify that jumbo frames (MTU 9000) are configured across your storage interfaces. Large MTU settings significantly reduce TCP packet framing overhead and interrupt frequencies during large sequential reads and writes.

On your hypervisor initiator node, load the standard NVMe over TCP kernel modules and set them to load at boot:

# Load kernel drivers immediately
modprobe nvme-tcp

# Ensure drivers persist across system reboots
echo "nvme-tcp" >> /etc/modules-load.d/nvme-tcp.conf

Verify system tuning settings in /etc/sysctl.d/99-nvme-tcp.conf to optimize TCP socket buffers for high-bandwidth storage networks:

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 100000

Apply these network tunings using sysctl -p /etc/sysctl.d/99-nvme-tcp.conf.

Step 2: Install CLI Tools and Discover Storage Targets

Install the official NVMe management utility nvme-cli. On Proxmox VE or Debian-based distributions:

apt-get update && apt-get install -y nvme-cli

Every NVMe-oF target exposes a unique string known as an NVMe Qualified Name (NQN). To query your storage array over your dual redundant storage networks (e.g., Target IP 1: 192.168.50.10 and Target IP 2: 192.168.51.10 on port 4420), execute the discovery command:

nvme discover -t tcp -a 192.168.50.10 -s 4420

The system will query the target’s Discovery Log Page and return available storage subsystems:

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  1
trsvcid: 4420
subnqn:  nqn.2026-03.com.infraforensics:storage:nvme-pool-01
traddr:  192.168.50.10

Step 3: Connect to the Target Subsystem and Verify Native Multipathing

Unlike legacy iSCSI setups that require user-space tools like multipathd and complex multipath.conf configurations, the Linux kernel NVMe driver contains native multipathing support (nvme_core.multipath=Y, enabled by default in modern kernels).

Connect to the storage subsystem over both physical storage interfaces to establish multi-path redundancy:

# Connect over Storage Network A
nvme connect -t tcp -n nqn.2026-03.com.infraforensics:storage:nvme-pool-01 -a 192.168.50.10 -s 4420

# Connect over Storage Network B
nvme connect -t tcp -n nqn.2026-03.com.infraforensics:storage:nvme-pool-01 -a 192.168.51.10 -s 4420

To inspect your active connection topology and multipath mapping, run:

nvme list-subsys

You will see a clean, multi-path subsystem block mapping output:

nvme-subsys0 - NQN=nqn.2026-03.com.infraforensics:storage:nvme-pool-01
\
 +- nvme0 tcp traddr=192.168.50.10 trsvcid=4420 live
 +- nvme1 tcp traddr=192.168.51.10 trsvcid=4420 live

This creates a virtual controller block device at /dev/nvme0n1. The Linux kernel will automatically perform round-robin load balancing across both network links and handle failover instantly if a network path drops, without stalling virtual machine I/O.

Step 4: Integrate Block Devices into Proxmox VE Storage Architecture

Now that your system exposes a high-performance, resilient block device at /dev/nvme0n1, you can attach it to Proxmox VE as a shared storage pool.

Create an LVM Volume Group on top of the NVMe-oF block device:

pvcreate /dev/nvme0n1
vgcreate vg_nvme_cloud /dev/nvme0n1

Next, add this volume group directly to your Proxmox VE storage configuration using the CLI (or via the Proxmox Web GUI under Cluster -> Storage -> Add -> LVM):

pvesm add lvm nvme-shared-storage --vgname vg_nvme_cloud --content images,rootdir --shared 1

Finally, to guarantee that your hypervisor automatically reconnects to your NVMe-oF target array upon system boot before Proxmox brings up virtual machines, enable systemic JSON/CLI persistence via /etc/nvme/discovery.conf:

# Append discovery endpoints to persistent config




High-Throughput NVMe-oF & Storage Networking
Local pool redundancy designs form the foundational storage layers exposed over high-speed IP fabrics. For NVMe-oF protocol benchmarks, consult our report on ZFS Mirror vs RAIDZ2: High-Density IOPS & Rebuilds.