You’ve just unboxed a pair of Mellanox ConnectX-6 cards or perhaps the Intel E810s. You’ve spent the better part of a week ensuring your fiber runs are clean, your transceivers are seated, and your PCIe lanes are negotiated at Gen4 speeds. You fire up iperf3, expecting to see that glorious 90+ Gbps throughput, but instead, the screen mocks you with a stubborn 12 Gbps. It feels like buying a Ferrari and finding out it’s been electronically governed to school-zone speeds. I’ve sat in that chair. I’ve felt that specific brand of frustration where the hardware is screaming “ready” but the software is whispering “wait.” The reality is that the Linux kernel, for all its brilliance, is tuned by default for the “average” use case. In the kernel’s eyes, an average use case is a web server or a laptop, not a high-performance compute node pushing 100 billion bits per second. The default TCP window sizes, the backlog queues, and the congestion control algorithms are all designed to prevent a modest system from drowning in its own traffic. When you step into the 100GbE arena, those safety nets become nooses. We are going to fix that today. We aren’t just going to “turn things up to eleven”; we are going to understand the Bandwidth-Delay Product (BDP) and why the kernel’s conservative memory management is the primary bottleneck in your high-speed fabric. By the time we’re done, your sysctl configuration will be the definitive blueprint for your production environment.

Why does my 100GbE network perform like a 10GbE link?

The most common trap I see engineers fall into is the “Hardware Fallacy.” They assume that because the NIC is rated for 100Gbps, the OS will naturally utilize it. But Linux is a general-purpose kernel. Its default net.core.rmem_default is often as low as 212KB. Think about the math for a moment. At 100Gbps, a single millisecond of round-trip time (RTT) requires a buffer of about 12.5 megabytes just to keep the pipe full. If your buffer is capped at 212KB, the TCP window closes almost immediately, the sender stops, and your throughput collapses. This is the Bandwidth-Delay Product in action. If your buffers are smaller than your BDP, you will never, ever hit line rate. Furthermore, the default congestion control algorithm, CUBIC, is designed to be “fair” to other streams. On a dedicated 100GbE backbone, we don’t necessarily want “fair” in the traditional sense; we want “efficient.” CUBIC reacts aggressively to packet loss, often slashing throughput by 50% at the first sign of a hiccup. In a high-speed environment, we need an algorithm like BBR (Bottleneck Bandwidth and Round-trip propagation time) that looks at actual throughput rather than just packet loss.

How do I tune the Linux kernel for 100GbE throughput?

This is the actionable playbook. We are going to move from the hardware layer up to the protocol layer. Do not skip steps; tuning is a cumulative process.

1. Expand the Hardware Ring Buffers

Before we touch the kernel, we have to talk to the NIC. The ring buffer is the memory on the NIC itself where packets sit before the CPU can process them. At 100GbE, the default buffer (often 512 or 1024) will overflow in microseconds during a burst. Check your current settings: ethtool -g eth0 Set them to the maximum (usually 4096): ethtool -G eth0 rx 4096 tx 4096

2. Implement the Master Sysctl Configuration

Create a file at /etc/sysctl.d/99-network-tuning.conf. This configuration targets the three pillars of high-speed networking: buffer depth, queue management, and congestion control. “`bash

Increase the maximum number of open files

fs.file-max = 2097152

Expand the core network stack buffers

net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.core.rmem_default = 67108864 net.core.wmem_default = 67108864 net.core.optmem_max = 65536

Increase the maximum number of packets in the receive queue

net.core.netdev_max_backlog = 250000

Increase the limit for the number of connections

net.core.somaxconn = 65535

TCP Memory Tuning (min, default, max in bytes)

Max is set to 128MB to accommodate high BDP

net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728

Enable TCP Window Scaling (Mandatory for >64KB windows)

net.ipv4.tcp_window_scaling = 1

Enable BBR Congestion Control

Requires kernel 4.9+ and fq qdisc

net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr

Fast Open and Low Latency settings

net.ipv4.tcp_fastopen = 3 net.ipv4.tcp_low_latency = 1

Increase the ephemeral port range for high-concurrency

net.ipv4.ip_local_port_range = 1024 65535 “` Apply these changes immediately with sysctl -p /etc/sysctl.d/99-network-tuning.conf.

3. Enable Jumbo Frames (MTU 9000)

Standard Ethernet frames are 1500 bytes. At 100Gbps, the CPU has to process roughly 8.3 million packets per second. That is a massive interrupt load. By switching to Jumbo Frames (9000 bytes), you reduce the packet count by a factor of six, significantly lowering CPU utilization and overhead. Warning: Every device in the path (switches, routers, other servers) must support and be configured for MTU 9000, or packets will be dropped. ip link set dev eth0 mtu 9000

4. Address NUMA Node Affinity

On multi-socket servers, your 100GbE NIC is physically wired to one specific CPU socket (NUMA node). If your application is running on CPU 0 but the NIC is on CPU 1, every packet must cross the QPI/UPI interconnect, adding latency and jitter. Find your NIC’s NUMA node: cat /sys/class/net/eth0/device/numa_node If it returns `1`, ensure your high-performance applications are pinned to that node using numactl --cpunodebind=1 --membind=1 [command].

Is BBR really better than CUBIC for 100GbE?

In my experience, the answer is a resounding yes, but with a caveat. BBR (Bottleneck Bandwidth and Round-trip propagation time) is a model-based congestion control algorithm. Unlike CUBIC, which interprets packet loss as a signal to slow down, BBR measures how fast the network is actually delivering data. On a 100GbE link, even a tiny amount of packet loss (0.001%)—which can happen due to minor congestion or optical imperfections—will cause CUBIC to tank your throughput. BBR ignores this “noise” and continues to push until it sees an actual increase in RTT, which indicates the buffers are truly full. In real-world 100GbE deployments, especially those spanning data centers, BBR often yields a 2x to 4x improvement in throughput over CUBIC.

Frequently Asked Questions

Does increasing sysctl buffers consume more RAM immediately?

No. The values in tcp_rmem and tcp_wmem are limits, not allocations. The kernel only allocates the memory it needs for active connections. However, if you have 100,000 active connections and set your max buffer to 128MB, you could theoretically exhaust your system memory. Monitor your `Slab` and `PageTables` in /proc/meminfo.

Why do I need to set net.core.default_qdisc to fq?

Google’s BBR algorithm requires the Fair Queuing (fq) scheduler to function correctly. `fq` handles the pacing of packets, ensuring that BBR doesn’t send bursts that overwhelm the local NIC or the first-hop switch. Without `fq`, BBR’s performance is significantly degraded.

Can I achieve 100GbE with a single TCP stream?

It is extremely difficult. Even with perfect tuning, a single TCP stream is often limited by the clock speed of a single CPU core due to the serial nature of TCP processing. To truly saturate a 100GbE link, you should use multi-stream tools (e.g., iperf3 -P 8 or nuttcp) or ensure your application is designed for parallel I/O.