You’re standing in the cold aisle of the data center, or perhaps just leaning over a rack in your home lab, and you hear it: that rhythmic, metallic “click-chirp” of a mechanical drive entering its death throes. If you’re running 22TB or 24TB high-density Helium drives, that sound isn’t just a hardware failure; it’s the start of a high-stakes mathematical countdown. You check your dashboard, and there it is—a “DEGRADED” status on your primary ZFS pool. Now comes the moment that separates the architects from the hobbyists. Do you trust your RAIDZ2 array to survive a 72-hour reconstructive surgery where every single remaining drive is hammered with 100% read duty cycles? Or are you breathing easy because you opted for striped mirrors, knowing your resilver will be finished before your coffee gets cold? The anxiety you’re feeling right now is the “Resilver Death Spiral.” It’s the realization that as drive capacities have ballooned, our rebuild windows have stretched into days, not hours. In the era of 2TB drives, RAIDZ2 was the gold standard for safety. In the era of 24TB drives, that same math can become a suicide pact for your data. We’re going to strip away the marketing fluff and look at the cold, hard physics of disk I/O, the brutal reality of Unrecoverable Read Errors (UREs), and why your choice of pool topology is the most expensive decision you’ll make this year.

Why is my ZFS pool slow even with twelve drives?

One of the most common frustrations I hear from engineers moving to ZFS is: “Declan, I bought twelve enterprise SAS drives, but my database performance is worse than a single SATA SSD. What gives?” The answer lies in a fundamental law of ZFS architecture that many gloss over: A ZFS pool’s random IOPS performance is roughly equal to the sum of the IOPS of its VDEVs (Virtual Devices), not the individual drives. If you take twelve drives and put them into a single RAIDZ2 vdev, you have exactly one vdev. In terms of random write and read operations, your pool will perform like a single (albeit very large and very safe) hard drive. You’ve gained massive sequential throughput—great for moving 4K video files—but you’ve absolutely strangled your IOPS. For a database or a Proxmox VM cluster, this is a death sentence. Every time the OS wants to write a small 4K block, ZFS has to calculate parity across the entire 12-drive stripe. Now, imagine those same twelve drives configured as six mirrored pairs (striped mirrors). You now have six vdevs. Your random write performance scales linearly, giving you roughly six times the IOPS of the RAIDZ2 configuration. When your VM asks to write data, it only has to hit one pair of disks. This is the “High-Density IOPS” secret. If your workload involves small, random operations—which describes almost every modern operating system and database—striped mirrors are the only way to keep your latency in the single digits.

Is RAIDZ2 actually safe for 22TB high-density drives?

We need to talk about the “Resilver Death Spiral.” When a drive fails in a RAIDZ2 vdev, ZFS doesn’t just copy data to a new drive. It has to reconstruct the missing data by reading every single block from every remaining drive in that vdev, calculating the parity, and writing the result to the replacement. On a 22TB drive, even if you’re sustaining a healthy 200MB/s, you’re looking at a minimum of 30 hours for a rebuild. In reality, with metadata overhead and fragmented pools, this often stretches to 48 or 72 hours. During those three days, every drive in your array is working at maximum capacity. This is exactly when a second drive—likely from the same manufacturing batch and having lived the same number of hours in the same thermal environment—is most likely to fail. If you lose a second drive in RAIDZ2, you’re still “online,” but you are now running “degraded-degraded.” If a third drive hits an Unrecoverable Read Error (URE) during this window, the vdev collapses. Your entire pool is gone. Striped mirrors mitigate this risk through speed. Because a mirror resilver only requires reading from the one surviving companion drive, the process is significantly faster and doesn’t stress the rest of the pool. You can rebuild a mirror in a fraction of the time it takes to parity-calculate a RAIDZ2 vdev, drastically narrowing the “window of vulnerability.”

How do I calculate the real cost of ZFS storage efficiency?

The “Capacity Trap” is what lures most people into RAIDZ2. When you see that a 12-drive array of 20TB disks gives you 200TB of usable space in RAIDZ2 but only 120TB in mirrors, the 80TB difference feels like a massive financial loss. At $15/TB, that’s $1,200 of “wasted” hardware. But I want you to look at the “Operational Tax.” If you choose RAIDZ2 for a high-IOPS workload, you will eventually be forced to add L2ARC (SSD caching) or SLOG (Separate Intent Log) devices to mask the latency of the mechanical drives. You’ll spend hundreds on NVMe drives and PCIe adapters just to try and fix a problem that striped mirrors solve natively. Furthermore, consider the cost of expansion. With striped mirrors, you can grow your pool two drives at a time. Need more space? Buy two 22TB drives, create a new mirror vdev, and add it to the pool. Done. With RAIDZ2, you generally want to keep your vdev widths consistent. If you started with a 12-drive RAIDZ2, adding more space properly usually means adding another 12 drives. For many businesses, that’s a capital expenditure hurdle that’s much harder to clear than buying a pair of drives every quarter.

What is the best way to expand a ZFS pool without losing data?

Expansion is where the “Agility” of mirrors truly shines. In ZFS, once a vdev is created, you cannot (easily or traditionally) change its width. If you have a 6-drive RAIDZ2, you can’t just “add a 7th drive” to that vdev to get more space. You have to add a whole new vdev to the pool. When you add a new vdev to a pool, ZFS begins striping new writes across all available vdevs. If your pool consists of one giant RAIDZ2 vdev that is 90% full, and you add a second RAIDZ2 vdev, ZFS will heavily favor the new, empty vdev for all new writes. This is good for capacity, but it creates an “unbalanced” pool where the new vdev is doing all the heavy lifting while the old one sits idle. With striped mirrors, expansion is granular. You add a mirror pair, and ZFS immediately has a new “lane” for IOPS. Because the vdevs are small (only 2 drives), the rebalancing of data—while still not automatic in the way some other file systems handle it—is much less of a performance bottleneck. You can also upgrade a mirror pool by replacing drives one by one. Replace one 12TB drive with a 22TB drive, let it resilver (fast!), then replace its partner. The moment the second resilver finishes, the vdev expands to the new capacity.

Proxmox ZFS Cheat Sheet: How to build your pool for maximum IOPS

If you’re ready to move away from the RAIDZ2 trap and build a high-performance pool, here is the playbook I use for enterprise deployments. We’ll assume you’re using Proxmox or a standard Linux distribution with ZFS on Linux (ZoL). 1. Identify your drives: Use `ls -l /dev/disk/by-id/` to find the unique serial numbers of your drives. Never use `/dev/sda` style naming; it can change between reboots. 2. Set the correct ashift: For modern Advanced Format drives (4Kn or 512e), always use `ashift=12`. This ensures the ZFS block size aligns with the physical sectors of the disk, preventing massive write amplification. 3. Create the striped mirror pool: zpool create -o ashift=12 my_high_performance_pool \ mirror /dev/disk/by-id/drive1 /dev/disk/by-id/drive2 \ mirror /dev/disk/by-id/drive3 /dev/disk/by-id/drive4 \ mirror /dev/disk/by-id/drive5 /dev/disk/by-id/drive6 4. Tune the recordsize: For VM storage (Proxmox/KVM), a `recordsize` of 64k or even 16k can significantly reduce the overhead of small random writes. For general file storage, the default 128k is fine. zfs set recordsize=64k my_high_performance_pool 5. Disable atime: Unless you have a specific requirement to know exactly when a file was last read, disable access time updates to save on write IOPS. zfs set atime=off my_high_performance_pool 6. Set XATTR to SA: For Linux systems, setting extended attributes to “System Attribute” mode improves performance for small file metadata. zfs set xattr=sa my_high_performance_pool By following this layout, you’ve built a pool that can survive the failure of any single drive, and potentially up to half your drives (as long as you don’t lose both halves of a single mirror). More importantly, you’ve built a pool that can be rebuilt in hours, not days, and provides the IOPS necessary to keep your applications snappy.

Frequently Asked Questions

Can I mix drive sizes in a ZFS mirror pool? Yes, you can. However, each mirror pair will only be as large as the smallest drive in that pair. If you pair a 12TB drive with a 20TB drive, you will have 12TB of usable space. This is a great way to repurpose older drives while slowly migrating to higher densities. What happens if both drives in a mirror pair fail? If both drives in a single mirror vdev fail, the entire pool is lost. This is why high-quality enterprise drives and proactive monitoring (SMART tests) are non-negotiable. If you are extremely risk-averse, you can use “three-way mirrors” (three drives per vdev), which provides incredible safety and read performance at the cost of 33% capacity efficiency. Is RAIDZ3 a better alternative for 24TB drives? RAIDZ3 allows for three drive failures, which solves the “safety” problem of RAIDZ2. However, it makes the IOPS problem even worse and the resilver times even longer. RAIDZ3 is excellent for massive “Write Once, Read Never” archives, but it is almost never the right choice for active production data.