When engineering teams repatriate containerized microservices from cloud-managed Kubernetes (such as AWS EKS or Google GKE) to owned physical servers, they encounter an immediate networking challenge: how do you expose Kubernetes `Type: LoadBalancer` services without a cloud provider’s managed load balancing API? In AWS, creating a LoadBalancer service automatically deploys an external ALB or NLB. On bare metal, the Kubernetes control plane leaves the external IP in `` status unless an ingress controller or load-balancing daemon is explicitly configured.

Historically, bare-metal clusters relied on legacy Layer 2 solutions like MetalLB in ARP mode. In Layer 2 ARP mode, a single worker node is elected leader via leader election and responds to ARP requests for the service VIP. Every incoming packet in the entire cluster is funneled through that single node’s physical network card before being forwarded across the overlay network to backend pods. This creates an acute single-node bandwidth bottleneck (choking 100Gbps traffic down to a single 10G/25G link) and introduces 3 to 10 seconds of complete packet drop during node failure failover.

The enterprise standard for bare-metal ingress is Cilium BGP Control Plane with native eBPF routing. By peering Cilium directly with physical Top-of-Rack (ToR) spine-leaf switches via the Border Gateway Protocol (BGP), Kubernetes nodes announce service Virtual IPs (VIPs) directly into the data center hardware routing table. Top-of-Rack switches then distribute traffic across all healthy worker nodes using Equal-Cost Multi-Path (ECMP) hardware switching at wire speed. This forensic architectural guide details how to configure Cilium BGP peering, audits eBPF packet acceleration, and models high-availability failover mechanics.

How Does Cilium BGP Peering Work in Bare-Metal Kubernetes?

In traditional networking, physical data center switches are managed strictly by network operations teams, while Kubernetes manages software overlays. Cilium BGP bridges this divide through the **CiliumBGPPeeringPolicy** Custom Resource Definition (CRD). Rather than running heavy Quagga/FRR daemons in separate privileged sidecar pods, Cilium’s native GoBGP engine runs inside the unified Cilium CNI agent.

When an engineer deploys an ingress controller (such as Envoy, Traefik, or Ingress-Nginx) backed by a Kubernetes Service with `spec.type: LoadBalancer`, Cilium’s IPAM allocates a routable IP from a predefined CIDR pool (e.g., `10.200.50.0/24` or a public `/24` block). The Cilium agent on each node hosting an active pod immediately advertises a `/32` BGP route to the upstream Top-of-Rack switches (such as Arista 7050X, Dell PowerSwitch, or Juniper QFX):

Route Announcement: 10.200.50.10/32 via Next-Hop 192.168.10.21 (Node 1)
Route Announcement: 10.200.50.10/32 via Next-Hop 192.168.10.22 (Node 2)
Route Announcement: 10.200.50.10/32 via Next-Hop 192.168.10.23 (Node 3)

The upstream switch sees three equal-cost routes to the same destination IP. Using its internal packet-forwarding ASIC, the switch hashes the incoming TCP connection’s 5-tuple (source IP, destination IP, source port, destination port, protocol) and forwards packets directly down the appropriate physical 25GbE/100GbE cable. Total cluster ingress capacity scales linearly with node count, easily achieving aggregate throughput of 200 Gbps to 800 Gbps.

Architectural Comparison: Cilium BGP vs Legacy MetalLB ARP

The eBPF XDP Advantage: Bypassing the Kernel Network Stack

Where Cilium’s architecture pulls decisively ahead of all traditional software routing solutions is in its utilization of eBPF (Extended Berkeley Packet Filter) and XDP (eXpress Data Path). In standard Linux networking, when a packet arrives at a network interface card, the OS kernel allocates a `sk_buff` data structure, pushes the packet through netfilter hooks, checks conntrack tables, and processes iptables chains.

When operating at millions of packets per second, CPU cores burn up to 40% of their cycles simply managing Linux kernel memory allocations. Cilium attaches eBPF bytecode directly to the network driver’s XDP hook. When an incoming packet arrives from the Top-of-Rack switch, the eBPF program parses the packet header directly in the NIC ring buffer. It checks an in-memory BPF hash table, translates the destination Service VIP to the targeted Pod IP, updates checksums, and transmits the packet directly to the local container veth interface—bypassing the entire Linux TCP/IP stack in under 2 microseconds.

For more architectural frameworks on eBPF networking and low-overhead infrastructure design, review our deep-dive analysis on Cilium eBPF Service Mesh: Eliminating Sidecar Proxy Tax and cross-reference our kernel security teardown in eBPF Tetragon Security: Real-Time Kernel Observability.