You’ve likely been there: it’s 2:00 AM, and your monitoring dashboard is screaming. Your Ceph cluster, which supposedly has 400TB of raw capacity, is throwing HEALTH_ERR because a handful of OSDs have hit the full_ratio threshold. You look at the aggregate usage and see you’ve only consumed 220TB. On paper, you have nearly 50% headroom, but in reality, your cluster has ground to a halt. This is the “Ceph Wall,” and it’s almost always the result of a misconfigured CRUSH map failing to account for OSD density. The frustration is real. You’ve invested in high-capacity 20TB drives to modernize your aging 8TB nodes, only to find that Ceph is stuffing the smaller drives to the breaking point while the new, expensive drives sit half-empty. It feels like the algorithm is working against you. But CRUSH (Controlled Replication Under Scalable Hashing) isn’t broken; it’s just doing exactly what you told it to do with the parameters it was given. Balancing a heterogeneous cluster requires moving beyond default settings and taking manual control of how weights and failure domains interact.

Why is my Ceph cluster reporting full when I have plenty of free space?

The core of the problem lies in the probabilistic nature of CRUSH. Unlike traditional RAID, which uses a centralized metadata table to track where every block lives, Ceph uses a mathematical function to calculate data placement. This is what allows Ceph to scale to thousands of nodes without a bottleneck. However, math doesn’t care about your hardware budget; it only cares about the weights assigned to the buckets in your CRUSH hierarchy. When you mix drive sizes—say, 8TB HDDs and 20TB HDDs—in the same root, the CRUSH algorithm attempts to distribute Placement Groups (PGs) based on the weight of each OSD. By default, an OSD’s weight is its capacity in terabytes. An 8TB drive has a weight of 8.0, and a 20TB drive has a weight of 20.0. In a perfect world, the 20TB drive would receive 2.5 times more data than the 8TB drive. The “trap” occurs because PGs are discrete units. If your PG count is too low, the statistical distribution becomes “clumpy.” One 8TB drive might end up with three extra-large PGs just by chance, pushing it over the 85% nearfull mark, while its 20TB neighbor is underutilized. Because Ceph prioritizes data integrity, once a single OSD in a placement group’s acting set hits the full threshold, the entire cluster may stop accepting writes for that pool to prevent data loss. You aren’t out of space; you’re out of balance.

How does the CRUSH hierarchy define my failure domains?

Before we can fix the balance, we have to understand the map. The CRUSH map is a tree structure. At the bottom are your OSDs (the leaves). These are grouped into host buckets, which might be grouped into rack buckets, row buckets, and finally the root. The power of CRUSH is its ability to enforce physical separation. If you set your failure domain to rack, Ceph ensures that replicas of the same object are never stored in the same rack. This is fantastic for uptime—you can lose an entire top-of-rack switch and your data remains available. However, this hierarchy also dictates how weight is aggregated. The weight of a host is the sum of the weights of its OSDs. The weight of a rack is the sum of its hosts. If Rack A has ten 20TB drives and Rack B has ten 8TB drives, Rack A is significantly “heavier.” CRUSH will send more traffic to Rack A. If your CRUSH rules aren’t carefully tuned to handle this disparity, you’ll see massive performance bottlenecks as the network links to Rack A become saturated while Rack B sits idle. This is why we strive for “homogeneity within the failure domain.” If you must mix drive sizes, try to distribute them evenly across your failure domains rather than clustering all large drives in one rack.

What is the difference between CRUSH weight and OSD weight?

This is where many seasoned sysadmins get tripped up. There are two different “weights” in Ceph, and they serve very different purposes. 1. CRUSH Weight: This is a long-term setting, usually based on the size of the disk (e.g., 12.75). It determines the probability that a PG will be assigned to that OSD. You change this when you want to permanently change the data distribution. 2. OSD Weight (In/Out Weight): This is a value between 0 and 1 (e.g., 0.85). It is a temporary override used by the ceph osd reweight command. It doesn’t change the CRUSH map; instead, it tells the cluster to artificially reduce the amount of data sent to an OSD that is filling up too fast. The “Costly Trap” is relying on osd reweight to fix a fundamental capacity imbalance. Using osd reweight is like putting a band-aid on a broken leg. It solves the immediate nearfull error, but it creates a “fragile” map where the mathematical distribution no longer matches the physical reality. If you have to reweight an OSD down to 0.7 to keep it from filling up, your CRUSH weights are wrong.

How do I fix OSD imbalance in a heterogeneous cluster?

If you’re staring at an imbalanced cluster right now, don’t panic. We can fix this without taking the cluster offline, but it will involve some data movement (backfilling). Here is the professional playbook for re-aligning your CRUSH map.

Step 1: Audit the Variance

First, we need to see how bad the skew is. Run ceph osd df tree. Look at the %USE column and the VAR (variance) column. A healthy cluster should have a variance close to 1.0. If you see OSDs at 1.5 and others at 0.6, your PG distribution is uneven.

Step 2: Normalize CRUSH Weights

If your OSD weights don’t match their actual sizes (common after manual interventions), normalize them. ceph osd crush reweight-all This command resets all CRUSH weights to match the actual size of the devices. Warning: This will trigger a significant amount of data movement. Do this during a maintenance window or when IO load is low.

Step 3: Enable the PG Autoscaler

In modern Ceph (Nautilus and later), the PG Autoscaler is your best friend. It monitors the amount of data in each pool and adjusts the PG count to ensure that the “clumpiness” of data is minimized. ceph mgr module enable pg_autoscaler Set your pools to on: ceph osd pool set <pool-name> pg_autoscale_mode on This ensures that as you add those 20TB drives, the cluster increases the number of PGs so the data can be distributed more granularly.

Step 4: Leverage Device Classes

If you have a mix of SSDs and HDDs, ensure they aren’t in the same CRUSH root. Ceph should automatically detect device classes, but you can verify this with ceph osd tree. You should see ssd or hdd next to the OSDs. If you want to ensure a specific pool only lives on your new, high-density 20TB drives (assuming they are a different class or you’ve tagged them), you can create a specific CRUSH rule: ceph osd crush rule create-replicated fast_storage default host ssd

Step 5: The “Upmap” Exception Table

If you have specific PGs that are just “stubborn” and won’t move despite correct weighting, Ceph Luminous introduced the balancer module. ceph mgr module enable balancer ceph balancer mode upmap ceph balancer on The upmap mode is incredibly powerful. It creates an exception table that moves individual PGs from overfull OSDs to underfull ones without changing the CRUSH math. It is the most surgical way to achieve perfect balance.

Can I mix different drive sizes in the same host?

Yes, you can, but you must be mindful of the “bottleneck of the slowest link.” If a host has one 20TB drive and seven 8TB drives, that 20TB drive will be handling significantly more IOPS than its neighbors. If the host’s backplane or network card is already saturated, that large drive will become a latency tail that slows down every write operation to that host. In a perfect world, you want your hosts to be “balanced units of consumption.” If you’re adding density, try to upgrade hosts as a whole, or at least ensure the high-density drives are paired with sufficient NVMe WAL/DB devices to handle the increased metadata and write pressure.

Frequently Asked Questions

Why is Ceph rebalancing taking so long? Rebalancing (backfilling) is throttled by default to prevent impacting client IO. You can speed it up by adjusting osd_max_backfills and osd_recovery_max_active, but do this cautiously as it will increase latency for your applications. What happens if I hit the ‘full’ ratio? When an OSD hits the full_ratio (default 90%), Ceph stops all writes to any PG that includes that OSD. This is a safety mechanism. To recover, you must either add more OSDs, delete data, or temporarily increase the full_ratio to allow the balancer to move data off the full drive. Should I use ‘reweight’ or ‘crush reweight’? Use crush reweight for permanent changes based on drive capacity. Use osd reweight (the 0.0 to 1.0 value) only as a temporary emergency measure to keep an OSD from hitting 100% while you wait for new hardware or for a proper rebalance to finish.