You’re standing in the cold aisle of the data center, or perhaps just staring at a web console from your home office, and the sinking feeling hits. A critical production VM is down. You check the Proxmox logs, and there it is: the dreaded Out of memory: Killed process. You look at your dashboard and see 128GB of physical RAM. Your VMs are only assigned 80GB. The math doesn’t add up, and the frustration is visceral. You’ve been told ZFS is the gold standard for data integrity, but right now, it feels like a resource-hungry monster that just ate your uptime. I’ve been in those trenches. I’ve spent nights debugging why a kernel with “free” memory would suddenly decide to execute a perfectly healthy QEMU process. The culprit is almost always a misunderstanding of how the ZFS Adaptive Replacement Cache (ARC) interacts with the Linux kernel. On Proxmox, ZFS isn’t just a filesystem; it’s a sophisticated memory management subsystem running inside another memory management subsystem. When they clash, your VMs lose. We’re going to fix that today by mastering the math of ARC sizing. ## Why is my Proxmox server killing VMs when I still have “free” RAM? The disconnect between what you see in the Proxmox GUI and what the Linux kernel sees is where the danger lives. By default, ZFS on Linux (ZoL) is configured to take up to 50% of your host’s physical RAM for the ARC. In older versions of ZFS, this was as high as 80% or even the entire memory stack minus a small buffer. The theory behind ARC is beautiful: it uses “free” RAM to cache frequently and recently used data, making your spinning disks feel like SSDs and your SSDs feel like NVMe. ZFS is designed to be a “good citizen.” When the kernel needs memory for a process (like a VM starting up or a database inside a VM ballooning), ZFS is supposed to release its cache back to the system. However, there is a latency gap. Releasing ARC memory isn’t instantaneous. It involves evicting pages and updating internal pointers. When a Proxmox VM suddenly demands 4GB of RAM to handle a burst in traffic, the Linux kernel looks for immediate availability. If the ARC is holding that memory, the kernel may decide it doesn’t have time to wait for ZFS to clean up its cache. It panics, triggers the Out-Of-Memory (OOM) killer, and picks a “high-scoring” process to terminate. Since your VMs are usually the largest memory consumers, they are the first to be executed. You end up with a crashed VM and a ZFS cache that is still sitting there, smugly holding onto 64GB of data. ## The “Costly Trap” of Default ZFS Memory Management The biggest mistake I see practitioners make is trusting the “dynamic” nature of ZFS. They assume that because ZFS can shrink, it will shrink fast enough. In a high-density virtualization environment, this is a fallacy. Another trap is the “1GB of RAM per 1TB of Storage” rule. This rule of thumb was designed for enterprise storage servers (NAS/SAN) where the primary job of the machine is serving files. In a hypervisor environment, the primary job is compute. If you have a 100TB array on a Proxmox node with 128GB of RAM, following the storage rule would mean giving 100GB to ZFS, leaving only 28GB for your VMs and the overhead of the Proxmox OS itself. That is a recipe for a catastrophic system hang. We need to flip the script. We calculate the memory your VMs need first, add a 10% buffer for the hypervisor overhead, and then give ZFS the “leftovers,” within a specific range. ## How do I calculate the exact byte value for ZFS ARC? ZFS does not accept “16GB” as a configuration value. It requires bytes. This is where many administrators trip up, using decimal gigabytes (1,000,000,000) instead of binary gibibytes (1,073,741,824). To the kernel, this difference matters. The formula is: `GiB * 1024 * 1024 * 1024`. If you want to cap your ARC at 16GB, the math is: `16 * 1024 * 1024 * 1024 = 17,179,869,184 bytes`. When you set this value, you are telling the ZFS module: “You can grow, you can optimize, but you hit a hard ceiling here.” This ensures that even if your storage workload is massive, the kernel always sees a significant portion of RAM as truly available for the QEMU processes. ## The Actionable Playbook: Tuning Proxmox ARC Without a Reboot You don’t need to take your whole cluster down to fix this. We can apply these settings live and then persist them so they survive the next maintenance window. ### Step 1: Audit Your Current ARC Usage Before changing anything, see what ZFS is actually doing. Proxmox includes the `arcstat` and `arc_summary` tools. Run this in your shell:

arc_summary

Look for the “ARC Size” section. It will show you the current size, the target size, and the maximum size (c_max). If `c_max` is roughly half your RAM, you are running on defaults and are at risk. ### Step 2: Set the Live Limits We can push new limits directly into the kernel’s runtime parameters. Let’s say we’ve decided on an 8GB limit for a 64GB host. `8 * 1024^3 = 8,589,934,592`. Run these commands as root:

echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max
echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_min

*Note: I recommend setting min and max to the same value in virtualization environments to prevent constant allocation/deallocation thrashing, though some prefer leaving min at 50% of the max.* ### Step 3: Persist the Configuration The `echo` command only lasts until you reboot. To make this permanent, you must create or edit a configuration file for the ZFS module. 1. Create the file: `nano /etc/modprobe.d/zfs.conf` 2. Add the following line (replace the number with your calculated byte value):

options zfs zfs_arc_max=8589934592

3. Update your initramfs to ensure this setting is loaded early in the boot process:

update-initramfs -u -k all

### Step 4: Verify the Change Wait a few minutes and run `arc_summary` again. You should see the “ARC Size Max” (c_max) reflecting your new value. Your VMs now have a guaranteed “no-fly zone” where ZFS cannot encroach. ## Does limiting ARC hurt my storage performance? This is the empathetic mentor moment: Yes, technically, a smaller cache means more requests might hit the physical disks instead of RAM. But we have to talk about “perceived performance” versus “system stability.” In a Proxmox environment, a 5ms delay in disk I/O is a nuisance. A VM being killed by the OOM killer is a critical failure. Furthermore, if you are using modern enterprise SSDs or NVMe drives for your ZFS pools, the performance penalty of a smaller ARC is often negligible. The latency of an NVMe drive is so low that the “hit” of missing the RAM cache is measured in microseconds, not milliseconds. If you find that your disk I/O wait (iowait) spikes too high after limiting the ARC, the solution isn’t to give ZFS more RAM—it’s to add an L2ARC (a dedicated fast SSD for caching) or upgrade your metadata vdevs. Don’t sacrifice your system’s stability to mask slow underlying hardware. ### Frequently Asked Questions Can I use a percentage instead of bytes in zfs.conf? No. The ZFS module parameters for Linux require an integer value in bytes. Using shorthand like “16G” will result in the configuration being ignored or causing a module load failure. Should I also limit the ZFS Dirty Cache? The `zfs_dirty_data_max` parameter controls how much data can sit in RAM before being flushed to disk (the Transaction Group). While related to memory, it serves a different purpose than ARC. For most Proxmox users, the default (10% of RAM up to 4GB) is sufficient and doesn’t need to be touched unless you have extremely high-throughput write workloads. Does Proxmox’s “Memory Ballooning” help with ZFS ARC? Actually, they can conflict. Memory ballooning allows the host to reclaim RAM from a VM. If ZFS sees that “reclaimed” RAM as free, it might try to expand the ARC into it. Then, when the VM needs the RAM back, you’re back in the OOM killer race. Capping the ARC is the only way to ensure a predictable environment.