Why does Linux still swap when I have 768 GB of RAM, and at least 50% of it is still free?

[alpha754293]

@quilt

You are confusing write-amplification and wear leveling. The mechanism of a log/spool file destroying an ssd is the same sector being rewritten thousands of times until that specific sector fails. Wear-leveling done by the ssd controller should mitigate that but more primitive storage like (older?) usb sticks, sd cards and the like don’t do this.

Gotcha.

re: wear levelling
Again, that depends.

I don’t know how much wear levelling happens with random I/O on NAND flash devices given that the theory of operation for random writes is that (you can) end up writing small amounts of data, over and over again, as it traverses the entire physical map of the NAND flash cells/modules repeatedly.

For sequential writes or where the writes either for an entire file or a part of a file happening over and over again, it is quite entirely possible that even with 4 kiB random writes at the max I/Os/s, that you can exhaust the finite number of NAND flash cell/module’s program/erase cycle endurance limits before said NAND flash cell/modules start to fail, with going over the sequential transfer rate’s write endurance limits which is how most drives are rated, for warranty.

It’s bad when you kill drives annually. It’s great that it happens annually such that you can file a RMA and at least for the first one, MOST companies will eventually replace it, once you have provided sufficient data that you were well within the TBW and/or DWPD spec as advertised on their website/product spec sheets.

@quilt

CoW in itself shouldn’t lead to write amplification

Depends on how you end up “copying” on write. (i.e. are you ONLY copying the modified block or are you copying the entire file again, only to modify said modified block?

If it is the latter, then you WILL get write amplification because it is re-copying and re-writing unmodified blocks to free blocks, in addition to copying the modified block to the free block.

But if it is the former, then the total amount of data written for modifying a block, would only be the size of the modified block.

So it really depends on how CoW is implemented, at the block level.

@quilt

(unless you defragment continuously; which is a non-default option on btrfs, not sure about zfs)

ZFS fragementation is free space fragmentation.

(cf. the thread here)

(And yes, it is super annoying that ZFS decided to talk about fragmentation in terms of free space fragmentation rather than file and/or data fragmentation. I still hold the hypothesis/theory that you can actually have 5 blocks per vblock in ZFS, write to 4 of them, in groups of 5 blocks, such that you can still never exceed the 80% utilisation rule-of-thumb, and still achieve 100% fragmentation in ZFS.)

But again, if you are having to re-copy/re-write the entire file, just to modify a single block, then write amplification will occur, as a function of CoW (vs. if you were only copying just the block to be modified, whilst leaving the rest intact, but that will lead to file fragmentation, which for SSDs is less of an issue, but it would kill HDDs (in terms of performance), if you do that sufficiently (because you’ll end up where you’ll have lots of interspersed files, especially with/for smaller files).

@quilt

If the filesystem block size is 4k and the physical block size is 4k, a CoW filesystem would just update that block and write it elsewhere, whereas a ‘normal’ filesystem would update it in place. But on a proper ssd controller it would decide on the device level to write to a different block any way if it deems it necessary for write leveling. So in both cases 4k get written.

if logical and physical block size is the same, yes.

But even if you were to talk about 512b blocks (for both logical and physical blocks), you can end up with:

[unmod][unmod][unmod][unmod][mod][used][used][used][used][used][free][free][free][free][free] ==>

[unmod][unmod][unmod][unmod][mod][used][used][used][used][used][new][new][new][new][new] ==>

[free][free][free][free][free][used][used][used][used][used][new][new][new][new][new]

(this is the scenario where you copied and wrote the entire file for the ONE block that has changed.)

vs.

[unmod][unmod][unmod][unmod][mod][used][used][used][used][used][free][free][free][free][free] ==>

[unmod][unmod][unmod][unmod][free][used][used][used][used][used][new][free][free][free][free] ==>

(where you ONLY copied and wrote the block that changed, leaving the rest of the file intact, but increases file fragmentation such that the unmodified block occupy the first 4 blocks, but you would need to skip 6 blocks (in this example) to read the 5th block of the file, which was the block that was copied and modified on write.)

So it really depends on how CoW is implemented.

For btrfs, again, it really depends on how it is deployed.

I haven’t done much with btrfs since it was available as an option I think either in SLES 12 SP4 or like SLES15.

(There are other disadvantages with btrfs (vs. ZFS) that btrfs is still working hard to catch up to (e.g. support for RAID5/6/7) whereas ZFS already has those features in place, working, and most of the time, enabled.)

(cf. Btrfs vs Ext4, Btrfs vs. EXT4: A Comprehensive Comparison of File Systems in Linux (2025) | DiskInternals, Comparing Btrfs and ZFS: Features, Pros, and Cons - CorpIT)

I don’t really spend any time with btrfs on account of lack of stable RAID5/6 support.

(cf. When will there be btrfs raid6 support? - #4 by mbod - Kernel, boot, graphics & hardware - EndeavourOS, https://serverfault.com/a/1168921, BTRFS Raid Confusion - #20 by twin_savage)