New to BTRFS - Shrinking filesystems with Btrfs?

Operating Systems & Open Source Linux
, , ,
1

I’m looking to set up a filesystem that on some occasions might need extra space, but in other moments might need to shrink.

And this needs to be live at all times. The services using the data on the filesystem cannot go down and needs to provide its data at all times.

I’ve been told Btrfs is the ticket, but also that live resizing is potentially risky.

I have no experience with it, so my first reflex is to set up a test scenario. Can people tell me if what I describe down here is good, what would be better, and what I should be aware about, working with Btrfs?

My test:

Add 6 1GB disks to my VM. Add them all as Physical Disks. Create a single volume group with them, then create a single logical volume with that entire size.

Use mkfs.btrfs on it. Mount it, write it full of data and reboot.

Having rebooted, I now delete all the written data from the directory the volume is mounted as.

Now I want to take out disks, so I need to remove them from the volume group and then as physical disk

I’m reading that I should do something like this first:

sudo btrfs balance start -dusage=75 /mnt/data/

I’m reading this balances data blocks across all devices.

And check progress with this:

sudo btrfs balance status /mnt/data/

And then shrink the filesystem:

sudo btrfs filesystem resize 4G /mnt/data/

This seems to work. However, when shrinking it to 2GB, it refuses, even though there’s free room:

:~$ sudo btrfs filesystem resize 2G /mnt/data/
Resize device id 1 (/dev/mapper/vgdata-lvbtrfs) from 4.00GiB to 2.00GiB
ERROR: unable to resize '/mnt/data/': No space left on device

Apparently this is due to metadata taking up space.

At this point I’m wondering if putting Btrfs on LVM is even a good idea.

This is starting to get chaotic, so I’m going to stop and ask if people have advice for using Btrfs?

2

What does btrfs filesystem used say? 2GB is a very small partition nowadays, and btrfs reserves some space for safety (system and metadata space), since on btrfs running out of space can be dangerous due to it’s copy-on-write nature. I believe by default btrfs allocates chunks in 1GB size, so 2GB means one data chunk, one metadata chunk and no space for system. But maybe the defaults are different depending on the device size. Not sure…

It’s not. Most things you’d want LVM for are already supported by btrfs out of the box. If you keep things vanilla there’s probably no issues, but doing snapshots on both lvm and btrfs could cause bad fragmentation, etc.

EDIT:

https://btrfs.readthedocs.io/en/latest/Glossary.html#glossary

chunk
A part of a block group. Chunks are either 1 GiB in size (for data) or 256 MiB (for metadata), depending on the overall filesystem size.
1 Like
3

Seems too complicated. What are you actually trying to accomplish here?

I would suggest that you give BTRFS a thin provisioned disk and tell BTRFS it is an SSD. BTRFS will use discard (or you can explicitly enable it).

You can then keep the BTRFS on the bigger side and the underlying storage will only use as much space as what you put in.

4

@peteru @quilt Sorry for the late reply.

I’m really looking towards a 5TB fast storage array, with 1TB NVME SSDs, that may at moments have to be expanded if I’m expecting incoming traffic to be more than that in a 24 hour window , though this is expected to be exceptional.

And so, I need to be able to increase the filesystem size… but ideally also decrease it again later. Replace disks, even, if they go bad. Take em out, maybe. All while everything remains up. The shrinking part is what I’m not experienced with. I’m used to using LVM and ext4 and XFS, but they don’t shrink. So some googling brought me to BTRFS, which I have no full experience with yet.

Yes, I was using small partitions, my idea was to use simple numbers on a vm to just test things a little bit, get to know the commands and such.

I don’t have anything to test with yet, so I’m testing on VMs.

5

Do you mean shrink by removing a device or shrink the filesystem itself?

Why exactly do you need to resize so much, and perhaps using subvolumes can be a solution to that reason?

Other than that I think BTRFS is probably the best filesystem for this, I.e its flexibility. For me it worked great so far; I’ve replaced the root drive on my server by adding the new device, then removing the old one, for example (and dd‘ing the boot partition). And I’ve converted my servers storage hdds by adding a drive and converting raid0 to raid10 while online with all files staying available.

It’s a great idea to test in a VM, but going down to 1 or 2 GB devices is not representative since BTRFS reserves relatively large chunks. If you can you could repeat with maybe 10 GB per device instead of 2… though in any case you should have backups before doing these kinds of operations on a live system!

6

OK. Yes, BTRFS is going to work fine for this.

See: Btrfs features | Forza's Ramblings

You can simply create a BTRFS filesystem over all the drives and set your desired level of protection. Personally I run with data profile of raid1 (2 copies) and metadata profile of raid1c4 (4 copies of metadata)

See: Btrfs RAID profiles | Forza's Ramblings

When you want to remove a disk, just do btrfs device remove <dev> <fs>. Similarly you can replace a failed disk with btrfs device replace and that will be much faster.

See: Adding, removing and replacing devices in a Btrfs filesystem | Forza's Ramblings and Guide on replacing a disk in a Btrfs filesystem | Forza's Ramblings

It really is very simple. You don’t need to tell BTRFS to resize the filesystem. You place the filesystem on a group of devices and tell BTRFS what level of protection you need for your data. As you add and remove devices, the filesystem can grow and shrink automatically. The only caveat is that if you want to remove devices, you need to delete enough files to fit on the remaining drives.

1 Like
7

@quilt I mean removing a disk. Which should be the same as replacing a disk as it goes bad, I think.

I might not need expanded size at all times. So I’d rather be able to take disks out again.

From what I’m reading, having even the root drive as BTRFS is a good idea. But root drives are usually limited to physical slot in the motherboard. So there’s that: what if the root drive goes bad? What if there’s only 1 slot. If there’s more than 1, can I spread it over them? What about where the bootloader is located. What about GRUB?

Thanks for the advice so far.

@peteru Thanks for the links.

One of the resize articles mentions expanding root, but I’m still stuck with the same questions on “what if the root disk dies” and what about where physically boot and GRUB and such are located. Is that all just fine, btrfs handles it? Or are there other considerations?

8

This works well on btrfs. You do btrfs device remove ... or btrfs device replace ..., depending on if you have a device ready to take its place or not. Btrfs will move all blocks to the other devices on the filesystem and then remove it. FS stays online and useable during the operation. No need for shrink in that place.

Usually in 2025 you’ll have a /efi or /boot that is on a FAT32 EFI partition. What I personally do is to rsync /efi to a /efi_backup subvolume on my btrfs filesystem so I have it backed up. If your drive actually goes bad you’d need to recover from a backup anyway. if it’s a gracefully failing SSD that goes read-only, or you have salvageable bad sectors, you could boot from a live linux and btrfs replace from there, if you can find a way to have a new disk connected (e.g. usb enclosure). Personally I did swap boot drives like this, but in a system with 2 slots. Add the new drive, partition it, dd over the /efi partition, then btrfs replace the filesystem, reboot and voila. Only downtime was a poweroff to add the new drive, then a poweroff to remove the old one. Filesystem with all snapshots still available.

Yes, you can spread the FS over multiple drives (with any raid level, or ‘simple’ just spreading the blocks around). Just btrfs add the new device and the FS will expand onto the new one. If you balance after btrfs will spread out the blocks so both have similar use.

Well if the drive actually dies, it dies. Nothing can help you there except for backups. Where the bootloader is located is usually in the efi partition under /efi or /boot/efi (depending on the distribution). You can have /boot on a btrfs partition on any modern distribution but the efi partition of course always needs to be FAT32.

9

While thinking about this, I just realized I’m not sure what kind of hardware this project will be on.

Because if it’s something like a Dell poweredge, I could just make a number of physical disks into a virtual disk that acts as the root, and have the poweredge system take care of redundancy of the root disk.

10

Avoid hardware RAID. What happens when the Dell craps itself and you need to move the drives elsewhere to get at your data? With software RAID (aka leave it to BTRFS), you can take the drives anywhere and get going.