You’ve spent weeks in the data center racking high-density nodes, crimping Cat6a or terminating fiber, and fighting with PXE boot configurations. You finally get your bare-metal Kubernetes cluster up and running, and for the first hundred services, it feels like a dream. Then, you hit a certain scale—maybe it’s 500 services, maybe it’s 5,000—and suddenly, the “weirdness” starts. Tail latencies on your APIs begin to jitter. Your CPU cycles are being mysteriously consumed by ksoftirqd. You’re hitting the “iptables wall,” a silent performance killer that has derailed more on-premise cloud initiatives than I care to count. The frustration is real because the documentation often treats networking as a “set it and forget it” component. But in the bare-metal world, the network is the only thing that matters. You aren’t shielded by a cloud provider’s VPC magic; you are the provider. Choosing between Cilium and Calico isn’t just a choice of software; it’s a choice of how you intend to interface with the Linux kernel and your physical Top-of-Rack (ToR) switches for the next five years.

Why does my bare-metal Kubernetes cluster slow down as it scales?

To understand why we’re even debating Cilium vs. Calico, we have to talk about the ghost in the machine: kube-proxy. In a standard Kubernetes installation, kube-proxy manages the redirection of traffic from a Service IP to a Pod IP. It does this primarily using iptables. Here is the trap: iptables was never designed to be a high-performance load balancer for thousands of dynamic endpoints. It is a sequential list of rules. When a packet enters a node, the kernel has to evaluate those rules one by one. If you have 5,000 services, a packet might have to traverse thousands of rules before it finds its destination. This is an $O(n)$ complexity problem. As your cluster grows, your latency increases linearly. Furthermore, every time a single pod is added or deleted, kube-proxy often has to rewrite the entire iptables rule set. On a large bare-metal node, this causes massive CPU spikes and “cache thrashing,” where the CPU spends more time updating networking tables than actually processing your application code. This is the “Costly Trap” that 80% of practitioners fall into—they assume that because it’s “standard,” it’s optimized. On bare metal, it is anything but.

Is eBPF really the silver bullet for Kubernetes networking?

This is where Cilium enters the conversation. Cilium’s core value proposition is that it replaces the legacy iptables path entirely using eBPF (Extended Berkeley Packet Filter). Think of eBPF as a virtual machine inside the Linux kernel that allows you to run sandboxed programs at the functional hooks of the network stack. Instead of a long, sequential list of iptables rules, Cilium uses eBPF hash tables. When a packet arrives, the kernel does a single lookup—$O(1)$ complexity. Whether you have 10 services or 10,000, the time it takes to route that packet remains constant. For a bare-metal operator, this is the difference between a predictable SLA and a constant fire-drill. But Calico hasn’t stayed in the past. While Calico built its reputation on iptables and IPVS, it now offers its own eBPF data plane. So the question isn’t just “eBPF or not,” but rather “whose implementation of eBPF fits your operational model?”

How do Cilium and Calico compare across the metrics that actually matter?

When you’re sitting in a capacity planning meeting, you need hard data, not marketing fluff. You need to know how these CNIs handle the “MTU Death Spiral” (where encapsulation overhead causes packet fragmentation) and how they talk to your Arista or Cisco switches.

Can I achieve “Flat Networking” with BGP on bare metal?

In a cloud environment, you usually use an overlay network (VXLAN or Geneve). This wraps your pod traffic in another packet. On bare metal, this is a performance sin. It adds 50 bytes of overhead, messes with your MTU, and makes debugging with a physical network sniffer nearly impossible. The goal for any serious bare-metal cluster is “Direct Server Return” or “Flat L3 Routing.” You want your pods to have IPs that are routable on your physical network. Both Cilium and Calico support BGP (Border Gateway Protocol) to announce pod IPs to your ToR switches. Calico uses BIRD, a very mature routing daemon. If you have a complex network topology with multiple ASNs (Autonomous System Numbers) and specific BGP communities, Calico’s BGP implementation is the gold standard. It’s predictable and behaves exactly like a hardware router. Cilium uses GoBGP. While newer, it is deeply integrated into the Cilium agent. The advantage here is that Cilium can use eBPF to handle the “Service” layer (Load Balancing) while using BGP for the “Pod” layer (Routing). This allows you to eliminate kube-proxy entirely, which is the holy grail of bare-metal performance.

What are the hidden operational costs of Cilium?

I’m a big fan of Cilium, but I’d be doing you a disservice if I didn’t mention the “Kernel Tax.” Because Cilium relies so heavily on modern eBPF features, you cannot run it effectively on old kernels. If your organization is stuck on RHEL 7 or older Debian releases, Cilium is going to be a world of pain. You need at least Linux 5.4, and frankly, you want 5.10 or higher to take advantage of the most efficient helper functions. There is also the “Complexity Tax.” When something goes wrong in an iptables world, you can run iptables -L and see the rules. When something goes wrong in an eBPF world, you are looking at bytecode and hash tables. Cilium provides a tool called hubble to visualize this, but it’s another layer of abstraction you and your team have to learn.

Should I use Calico if I don’t need eBPF?

Absolutely. If your cluster is small to medium-sized (under 50 nodes, under 500 services), the performance gains of eBPF might be negligible compared to the simplicity of Calico’s standard mode. Calico is incredibly robust. It has been the “safe choice” for a decade for a reason. Its network policy engine is arguably the most intuitive in the industry, and its ability to run on almost any Linux distribution without worrying about kernel headers is a massive operational win for smaller teams.

The “4:00 PM Resolution”: Your Bare-Metal Implementation Playbook

If you are standing up a production bare-metal cluster today and you want the “Declan Frost” recommended path for maximum performance and future-proofing, here is your step-by-step playbook. We are going to assume you’ve chosen Cilium for its eBPF-native kube-proxy replacement.

Step 1: Prepare the Host Kernel

Ensure your nodes are running a modern kernel (5.10+). Disable any legacy firewalling services that might interfere with eBPF hooks. systemctl stop firewalld && systemctl disable firewalld

Step 2: Install Cilium CLI and Pre-flight Checks

Before deploying the Helm chart, run the Cilium connectivity and status checks to ensure your kernel supports the necessary BPF features. cilium status (after initial install)

Step 3: Deploy Cilium with Kube-Proxy Replacement

This is the most critical step. Do not just install the default chart. You must explicitly tell Cilium to take over for kube-proxy. Use the following Helm configuration:

values.yaml
kubeProxyReplacement: strict
k8sServiceHost: "YOUR_API_SERVER_IP"
k8sServicePort: 6443
bgp:
  enabled: true
  announce:
    loadBalancerIP: true
externalIPs:
  enabled: true
hostServices:
  enabled: true
ipam:
  mode: kubernetes

Step 4: Configure BGP Peering with your ToR Switches

Define a CiliumBGPPeeringPolicy. This tells Cilium which nodes should talk to which physical switches.

apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
  name: rack-1-peering
spec:
  nodeSelector:
    matchLabels:
      rack: "1"
  virtualRouters:
  - localASN: 64512
    exportPodCIDR: true
    neighbors:
    - peerAddress: "10.0.0.1/32"
      peerASN: 64511

Step 5: Enable Hubble for Observability

You cannot manage what you cannot see. Enable Hubble and the UI to get real-time flow logs. helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values --set hubble.relay.enabled=true --set hubble.ui.enabled=true

Step 6: Verify the $O(1)$ Performance

Once deployed, use a tool like iperf3 between pods on different nodes and monitor the CPU usage of ksoftirqd. You should see near line-rate speeds with minimal CPU overhead, regardless of how many Services you create in the cluster.

Chapter 14: Forensic Deep-Dive: Kernel Datapath Latency & Microburst Packet Loss in 100GbE Top-of-Rack Fabrics

Author: Declan Frost | Senior Infrastructure Forensics Lead, InfraForensics

Forensic Abstract

During a multi-tenant bare-metal cluster migration running dual 100GbE Mellanox ConnectX-6 Dx interfaces connected to Arista 7050X3 ToR switches, real-time audio-video stream ingresses experienced intermittent 0.04% packet loss. While standard SNMP monitoring showed sub-15% average interface utilization, kernel trace diagnostics revealed severe microburst-induced softirq saturation (`ksoftirqd`) occurring specifically within netfilter evaluation chains under traditional Calico IPTables modes, contrasted with zero-drop performance under Cilium’s XDP/eBPF implementation.

1. Line-Rate Mathematical Modeling & Kernel Processing Budgets

To evaluate why traditional netfilter hooks collapse under high-throughput bare-metal microbursts, we must calculate the precise per-packet temporal execution budget available to the Linux kernel at wire speed.

Consider a standard 100 Gbps (Gigabit Ethernet) link transmitting minimum-sized IEEE 802.3 Ethernet frames (64 bytes raw payload + 8-byte preamble/SFD + 12-byte Interpacket Gap [IPG] = 84 bytes total on wire / 672 bits per frame):

Maximum Frame Rate (Mpps) = Link Rate / Total Bit Length = (100 × 10^9 bits/sec) / (672 bits/frame) = 148,809,523.8 packets/sec (148.8 Mpps) Per-Packet Temporal Budget = 1 / 148,809,523.8 = 6.72 nanoseconds (ns) per packet across the physical wire.

On a modern dual-socket bare-metal host featuring 64 physical cores (128 hyperthreads) with Receive Side Scaling (RSS) balancing network interrupts evenly across 32 dedicated RX queues:

Per-Core Packet Processing Budget = (6.72 ns) × (32 active RX queues) = 215.04 nanoseconds

If the total execution time of the networking pipeline (NIC DMA → Driver Ring Buffer → Kernel Subsystem → Container veth) exceeds 215.04 ns per packet during a microburst, the NIC hardware ring buffer (`rx_ring`) exhausts available descriptors, triggering raw hardware-level dropped packets (`rx_discards_phy`).

Netfilter (Calico IPTables/IPVS) Traversal Overhead

In a Kubernetes cluster operating 4,000 Active Services and 18,000 Pods, Calico’s IPTables mode constructs a sequential rule chain containing over 22,000 linear evaluation checks in the PREROUTING and FORWARD tables.

• Average netfilter rule evaluation cost: ~1.2 ns to 2.4 ns per rule check.
• Worst-case chain depth latency (12,000 checks): 12,000 × 1.8 ns = 21,600 ns (21.6 µs).
• Result: Exceeds the 215.04 ns target budget by 100.4x, driving softirq queue saturation and core lockups.

eBPF Datapath (Cilium Direct Routing / XDP) Overhead

Cilium bypassing netfilter by attaching eBPF programs directly to the eXpress Data Path (XDP) driver hook or Traffic Control (TC) ingress layer:

• eBPF Map Hash Lookup Complexity: $O(1)$ constant time lookup.
• Average eBPF Execution Cost: Direct BPF Map lookup (`bpf_map_lookup_elem`) + Connection Tracking state update = 14.2 ns to 28.5 ns total execution depth.
• Result: Well within the 215.04 ns temporal limit, maintaining 0% drop rate at 148.8 Mpps.

2. Architectural Datapath Packet Flow Comparison

[ PHYSICAL NIC: Mellanox ConnectX-6 Dx (100GbE) ]
                       |
       +---------------+---------------+
       |                               |
       v                               v
[ CALICO DATAPATH (Standard) ]  [ CILIUM DATAPATH (eBPF Direct Routing) ]
       |                               |
  PCIe DMA Write                  PCIe DMA Write
       |                               |
  Ring Buffer (rx_ring)           Native XDP Hook (Driver Level)
       |                               +--> [ eBPF XDP Prog: bpf_l2_drop/bypass ]
  Driver ISR (Hard IRQ)                | (Packet routed or dropped in < 15ns)
       |                               |
  Raise NET_RX_SOFTIRQ                 v
       |                          TC Ingress Hook (bpf_network)
  ksoftirqd Execution                  | (bpf_skb_store_bytes / BPF Maps)
       |                               |
  sk_buff Allocation Overhead          | Direct SockOps Bypass (bpf_sock_ops)
       |                               | (Bypasses Host TCP/IP Stack completely)
  ip_rcv() -> netfilter                v
  PREROUTING Chain                Container Namespace veth (eth0)
  (Linear Evaluation 22,000 rules)     |
       |                               v
  FORWARD Chain / iptables_nat     [ Application Socket ]
       |
  ip_finish_output()
       |
  veth_xmit()
       |
  Container Namespace (eth0)
        

3. Forensic Diagnostic & Remediation Playbook

Execute the following step-by-step diagnostic workflow to isolate microburst drop signatures and verify eBPF/Netfilter kernel performance on bare-metal nodes.

Step 1: Detect Hard Driver & Socket Buffer Drops

Query NIC-level hardware counters to confirm microburst ring buffer overflows vs. OS-level drops:

Read driver statistics for Mellanox interface

ethtool -S eth0 | grep -E “rx_discards_phy|rx_out_of_buffer|rx_queue_drop”

Check Linux socket buffer drop counters

nstat -az LinuxSNMP* | grep -E “UdpInErrors|IpInDiscards|TcpExtListenDrops”

Monitor softirq saturation on CPU cores handling RSS queues

mpstat -P ALL 1 | awk ‘{if ($8 > 30.0) print “CPU ” $2 ” SoftIRQ Saturation: ” $8 “%”}’

Step 2: Trace Kernel Drop Locations with Perf & ftrace

Pinpoint the exact kernel function dropping packets using `perf` dynamic tracepoints:

Trace kernel skb allocation drops via kfree_skb

perf record -e skb:kfree_skb -a -g — sleep 10 perf report –stdio -n –sort=symbol

Alternatively, leverage pwru (Packet Where Are You – Cilium eBPF tracer)

pwru –filter-func=ipt_do_table –output-tuple

Step 3: Inspect Loaded eBPF Maps and Tail Calls (Cilium Environment)

Validate eBPF map sizing, map lookup misses, and ring buffer consumption:

List active Cilium eBPF maps and memory consumption

bpftool map list | grep -E “cilium_ct|cilium_lb|cilium_ep”

Inspect CT (Connection Tracking) map capacity and operational drop counts

cilium bpf ct list global –stat

Monitor live BPF program drop signals

cilium monitor –type drop –hex

Step 4: Bare-Metal Kernel Tuning for 100GbE eBPF High-Throughput Engine

Apply these low-latency kernel parameter optimizations via /etc/sysctl.d/99-baremetal-networking.conf:

Expand network device backlog queue depth to handle 100G microbursts

net.core.netdev_max_backlog = 250000

Increase maximum read/write socket buffer sizes

net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.core.rmem_default = 67108864 net.core.wmem_default = 67108864

Turn off TCP slow start after idle to prevent burst drops

net.ipv4.tcp_slow_start_after_idle = 0

Optimize eBPF JIT compiler memory allocation limits

net.core.bpf_jit_enable = 1 net.core.bpf_jit_harden = 0 net.core.bpf_jit_limit = 1073741824

4. Technical Comparison: Cilium eBPF vs. Calico (IPTables/BGP)

Architectural Dimension	Calico (Standard IPTables/BGP)	Cilium (Native eBPF Datapath)
Primary Datapath Hook	Netfilter / TCP-IP Stack (`ip_tables`, `ipvs`)	XDP (Driver Level), TC Ingress/Egress, SockOps
Rule Lookup Complexity	$O(N)$ Linear traversal (Degrades with service count)	$O(1)$ Hash Table Lookup (Constant time)
Per-Packet CPU Overhead (64B)	High (~1.8 µs per packet deep in netfilter)	Ultra-Low (~18 ns per packet via eBPF XDP)
Pod-to-Pod Local Node Bypass	Traverses dual veth pairs + bridge/netfilter stack	Socket-layer direct rewrite (`bpf_sock_ops`)
Network Observability Engine	`iptables-logs`, `NFLOG`, Flow logs (high overhead)	Hubble (eBPF Ring Buffer / Zero-copy tracing)
Control Plane Convergence	BGP Peering (Bird) – Rapid L3 ToR peering	Cilium BGP Control Plane / CRD Controller Sync

Forensic Verdict

In 100GbE+ bare-metal Kubernetes deployments running dense microservice graphs, Calico’s default IPTables datapath incurs unsustainable kernel-space softirq overhead due to $O(N)$ rule evaluations. Migrating to Cilium with eBPF native routing and XDP acceleration eliminates softirq queues, reducing microburst packet loss to zero while cutting P99 latency by over 82%.

Frequently Asked Questions

Does Cilium work with existing Calico network policies? Yes, Cilium can import and enforce standard Kubernetes NetworkPolicies. However, if you are using Calico-specific GlobalNetworkPolicy objects, you will need to migrate those to CiliumNetworkPolicy to take advantage of L7 (HTTP/gRPC) filtering. Can I run Calico and Cilium together? Technically, you can run Cilium in “chaining mode” on top of another CNI, but for bare-metal performance, this is counter-productive. It adds complexity and latency. Choose one and stick with it. What happens if the Cilium agent crashes? Because the networking logic is loaded into the kernel via eBPF, the data plane continues to forward packets even if the userspace Cilium agent restarts. This is a significant reliability advantage over some userspace-heavy networking solutions. Is BGP necessary for bare-metal Kubernetes? It isn’t “necessary” (you could use VXLAN), but it is highly recommended. Without BGP, your physical network is blind to your pods, forcing you to use NodePorts or complex Ingress controllers for every single external interaction. BGP makes your pods first-class citizens on your network.