Arch Linux - 'No space left on device' with only half disk space used and tmpfs empty

Timoral · May 23, 2016, 8:47pm

For some unknown reason, my filesystem is showing as full. I am using a BTRFS filesystem on an ssd with Arch Linux 4.4.11-1 lts. I have already checked to see if my tmpfs is full and in fact it is empty. It shows as 0 available, 100% use but only 35G used when my size is 111G.

I can't do anything as my system thinks it if totally filled..! I've booted to a usb and ran btrfsck --repair /dev/sda3. No fix there.

What is going on...?

My outputs:

df -h

Filesystem Size Used Avail Use% Mounted on
dev 3.9G 0 3.9G 0% /dev
run 4.0G 9.0M 4.0G 1% /run
/dev/sda3 111G 35G 0 100% /
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup
tmpfs 4.0G 36K 4.0G 1% /tmp
/dev/sda1 511M 5.7M 506M 2% /boot/efi
/dev/sdb 1.2T 1.1T 33G 98% /mnt/btrfsarray
tmpfs 803M 0 803M 0% /run/user/1000

btrfs filesystem df /

Data, single: total=109.73GiB, used=33.21GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=1.01GiB, used=768.80MiB
GlobalReserve, single: total=272.00MiB, used=8.86MiB

Eden · May 23, 2016, 8:53pm

Whats the output of df -h and btrfs filesystem df /

Timoral · May 23, 2016, 8:55pm

df -h

Filesystem Size Used Avail Use% Mounted on
dev 3.9G 0 3.9G 0% /dev
run 4.0G 9.0M 4.0G 1% /run
/dev/sda3 111G 35G 0 100% /
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup
tmpfs 4.0G 36K 4.0G 1% /tmp
/dev/sda1 511M 5.7M 506M 2% /boot/efi
/dev/sdb 1.2T 1.1T 33G 98% /mnt/btrfsarray
tmpfs 803M 0 803M 0% /run/user/1000

btrfs filesystem df /

Data, single: total=109.73GiB, used=33.21GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=1.01GiB, used=768.80MiB
GlobalReserve, single: total=272.00MiB, used=8.86MiB

Eden · May 23, 2016, 9:28pm

what does btrfs subvolume list / return?

after that try btrfs balance start /

btrfs balance status / will show you the progress, might take alittle while.

Timoral · May 23, 2016, 9:31pm

btrfs subvolume list /

ID 260 gen 139223 top level 5 path var/lib/machines

btrfs balance start /

Returns:

ERROR: error during balancing '/': No space left on device

...

Timoral · May 23, 2016, 10:06pm

Okay... so I deleted /var/lib/gitlab/ which is what I was installing when everything buggered up.. For some reason I now have 77G back!?

Filesystem Size Used Avail Use% Mounted on
dev 3.9G 0 3.9G 0% /dev
run 4.0G 9.0M 4.0G 1% /run
/dev/sdc3 111G 35G 77G 31% /
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup
tmpfs 4.0G 36K 4.0G 1% /tmp
/dev/sdc1 511M 5.7M 506M 2% /boot/efi
/dev/sdd 1.2T 1.1T 33G 98% /mnt/btrfsarray
tmpfs 803M 0 803M 0% /run/user/1000

I think it happened when I was running these commands:

# export PATH=$PATH:/var/lib/gitlab/.gem/ruby/2.3.0/bin
# sudo -u gitlab -H gem install bundler --no-document
# cd /usr/share/webapps/gitlab
# sudo -u gitlab -H bundle install

Which I was getting errors with...

Very odd...

Timoral · May 23, 2016, 10:32pm

Aaand its full again...

df -h

Filesystem Size Used Avail Use% Mounted on
dev 3.9G 0 3.9G 0% /dev
run 4.0G 9.0M 4.0G 1% /run
/dev/sdc3 111G 35G 0 100% /
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup
tmpfs 4.0G 36K 4.0G 1% /tmp
/dev/sdc1 511M 5.7M 506M 2% /boot/efi
/dev/sdd 1.2T 1.1T 33G 98% /mnt/btrfsarray
tmpfs 803M 0 803M 0% /run/user/1000

What is going on?

Marten · May 24, 2016, 8:49am

What does btrfs filesystem show output ?

Timoral · May 24, 2016, 5:15pm

btrfs filesystem show

Label: none uuid: e03ca155-91fe-41a6-a5aa-28adf65a7e77
Total devices 1 FS bytes used 34.07GiB
devid 1 size 110.74GiB used 110.74GiB path /dev/sdc3

Label: none uuid: a0f4e991-ba5e-4c36-97ef-f694ff9a5fa4
Total devices 5 FS bytes used 1.08TiB
devid 1 size 232.89GiB used 232.01GiB path /dev/sdd
devid 2 size 232.89GiB used 231.88GiB path /dev/sde
devid 3 size 298.09GiB used 296.00GiB path /dev/sdf
devid 4 size 232.89GiB used 232.88GiB path /dev/sda
devid 5 size 149.05GiB used 149.01GiB path /dev/sdb

/dev/sdc3 is my root device. So the filesystem is showing as full even though it really isn't...

Timoral · May 24, 2016, 5:27pm

I just deleted Half life from my steam folder and I now have 77G back again?

df -h

Filesystem Size Used Avail Use% Mounted on
dev 3.9G 0 3.9G 0% /dev
run 4.0G 9.0M 4.0G 1% /run
/dev/sdc3 111G 34G 77G 31% /
tmpfs 4.0G 0 4.0G 0% /dev/shm
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup
tmpfs 4.0G 36K 4.0G 1% /tmp
/dev/sdc1 511M 5.7M 506M 2% /boot/efi
/dev/sdd 1.2T 1.1T 33G 98% /mnt/btrfsarray
tmpfs 803M 0 803M 0% /run/user/1000

Half life definitely isnt 77GB so something is wrong...

Marten · May 24, 2016, 5:48pm

Now you have some free space maybe scrub it.
sudo btrfs scrub start /
You can check for errors during and after with
sudo btrfs scrub status /

For 110G it should not take too long. Then balance it.

Timoral · May 24, 2016, 5:54pm

OK just scrubbed 33GiB. I wonder if the ruby script for installing gitlab was the issue

Marten · May 24, 2016, 5:57pm

Where there any errors detected ? Run a balance as well and thats about all I know how to clean up the filesystem.

I just did a google and found http://nathantypanski.com/blog/2014-07-14-no-space-left.html

From link
> Note Metadata, DUP - that’s 81% usage. The btrfs wiki says that 75% or greater metadata space usage can cause No space left on device errors, even when the filesystem isn’t full.

Could be that your meta data was 77% used.

Timoral · May 24, 2016, 9:51pm

I didnt see any no. Thanks for all the help I'll keep that link for later. I'll have to remember to or set up tasks periodically to run a scrub and rebalance