The small linux problem thread

The first file (Big Business) doesn’t:

 mpv "Big Business (1988).mkv"
Failed to recognize file format.
Exiting... (Errors when loading file)

The second (Alice) does, but it eventually starts vomiting errors and halfway through it just nopes out entirely.

I guess the first is indeed a sparse file, less just shows it as zeroes. The second one is filled but that’s expected given that it plays (partially anyway).

Shows the full size:

 rsync -hP "Alice in Wonderland (2010) BORKED/Alice in Wonderland (2010).mkv" "Big Business (1988) BORKED/Big Business (1988).mkv" ~/Videos
Alice in Wonderland (2010).mkv
          7.99G 100%  107.33MB/s    0:01:11 (xfr#1, to-chk=1/2)
Big Business (1988).mkv
          8.53G 100%  104.28MB/s    0:01:17 (xfr#2, to-chk=0/2)
 ls -gohl ~/Videos | grep mkv
-rw-r--r--. 1 7.5G Mar 29 01:56 Alice in Wonderland (2010).mkv
-rw-r--r--. 1 8.0G Mar 29 01:57 Big Business (1988).mkv

Interestingly du now shows both as the full size too:

 du -h ~/Videos/"Big Business (1988).mkv"
8.0G    /home/tarulia/Videos/Big Business (1988).mkv
 du -h ~/Videos/"Alice in Wonderland (2010).mkv"
7.5G    /home/tarulia/Videos/Alice in Wonderland (2010).mkv

But then again I didn’t use the --sparse option for rsync so it makes sense (same result with cp too).

I don’t really get the point of this then though… I can create a 1TB sparse file on a drive that has only 100 gigs left of space, I mean… that’s cool but why?
If I were to start writing actual data to that file I’d run into issues anyway because the space obviously isn’t being “reserved” for later use… so I don’t really get why this is a thing? Like why have a different apparent size when it is inaccurate either way?

One example where sparse files are useful:

  • You run Clonezilla or Partclone on a 10TB disk that only has 50GB of data on the file system. (Partclone skips over the empty-space portion of the drive and creates a sparse file.)
  • You want to get some files off the disk image…
  • Extracting that 10TB disk to an image file will only use about 50GB of space, and only take a couple minutes to write out as a sparse file. As a non-sparse file it would need 10TB of free disk space (if you even have it) and take maybe 6 hours to write…

Other examples include downloading torrent files or ddrescue to make a disc image… You don’t need to use up all the disk space and wait for it to all be allocated up-front, before you can even start writing anything.

1 Like

OK the use of cloning or creating a disc image I get, but what’s the advantage for a torrent (or any download for that matter) here? I don’t really get it.
Say I have 100 gigs left on my drive and I download a 200 gig torrent. Now normally the client would check available space, but let’s say it doesn’t. It creates the 200 gig sparse file and then what…? As soon as the download is crossing the 100 gig threshold we’re getting I/O errors so what’s the point then?
It could just not create any file and write the data as it comes in, which would have the same effect, no?

The zero’d sparse isn’t actually allocating/reserving the disk space so what’s the advantage here?

A) Your system doesn’t have to freeze up and wait (several seconds) for the whole 200GBytes of disk space to be allocated, before it can continue and even get started downloading.
B) If you’ve got 20 torrents running, but most of them only downloaded 10MB before the seeders each dropped off, why would you waste disk space on all of them?
C) You can check the difference between du and ls to see how much progress has been made.

Right but it doesn’t have to allocate it beforehand anyway is what I mean. When I download a file, whether it be via Torrent or in a Browser, the file can start at 0 and just grow as data comes in, it doesn’t need to allocate the whole 200 gigs right away, right?

I am not sure about this, and it may depend on the file system too. But when the header says “the next 200GB belong to me” and then the file ends after 10MB, that could cause problems.

It’s exactly how downloads (or even file copies in progress) work though.

Download a file in chrome and check the download directory, the filesize shown is climbing as the download progresses. Although I haven’t checked what du shows in that case.

Uhhhh…
That does not seem right.

This link https://repo.radeon.com/amdgpu/6.3.4/el//main/x86_64/repodata/repomd.xml is indeed 404

If you’re on Fedora that’s expected. The variable that goes after el/ doesn’t exist on Fedora (hence the //):

 bat /etc/yum.repos.d/amdgpu.repo 
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /etc/yum.repos.d/amdgpu.repo
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ [amdgpu]
   2   │ name=AMDGPU 6.2.4 repository
   3   │ baseurl=https://repo.radeon.com/amdgpu/6.2.4/el/$amdgpudistro/main/x86_64
   4   │ enabled=0
   5   │ gpgcheck=1
   6   │ gpgkey=file:///etc/amdgpu-install/rocm.gpg.key

I haven’t dug deeper into this but I believe the variable’s value is determined in a %post scriptlet when installing the repo RPM.

Mentioned this here too:

Depending what you need those repos might be irrelevant though.


Just tested BTW and it does indeed also grow in du.

1 Like

Well, the difference between downloading a file in the browser and downloading a torrent is that file downloads in the browser work in a linear fashion, while torrent downloads are chunked, and the chunks may come down the line in random order. Hence the need to preallocate complete file sizes, sparse or not, and fill them in using memory-mapped file access.

1 Like

That makes sense yeah… although as noted the sparse isn’t allocating anything, no? Since it doesn’t actually occupy the space. Unless that’s some filesystem internal wizardry for optimisation or something.

I’m a bit confused what’s happening with my Output redirection…

I wrote a short script to supply 2 files and calculate the VMAF, and I want the final score in the output file.

#!/bin/bash
if [ "$2" == "" ]; then
	echo "vmaf reference distorted"
	exit 1
fi

# https://github.com/Netflix/vmaf/blob/master/resource/doc/ffmpeg.md
ffmpeg 	-an -sn \
	-i "${1}" \
	-i "${2}" \
	-lavfi "[0:v:0]setpts=PTS-STARTPTS[reference]; \
			[1:v:0]setpts=PTS-STARTPTS[distorted]; \
			[distorted][reference]libvmaf=log_fmt=sub:log_path=${2%.*}.sub:n_threads=20" \
    -f null - \
	|& grep 'VMAF score:' | tail -n1 > "${2%.*}.score"

I’m using log_fmt=sub because it makes it easy to check low-scoring frames (because they get loaded as a subtitle in mpv).
Anyway, when I use this without the redirection I get what I want:

❯ ./vmaf in.mkv out.webm
[Parsed_libvmaf_2 @ 0x7f4bfc006600] VMAF score: 98.251637ate=N/A speed=3.89x

well… except for that garbage at the end. But when I use the redirection, my output file looks like this:

frame=   42 fps=0.0 q=-0.0 size=N/A time=00:00:01.71 bitrate=N/A speed=3.42x    
frame=   87 fps= 87 q=-0.0 size=N/A time=00:00:03.58 bitrate=N/A speed=3.59x    
frame=  135 fps= 90 q=-0.0 size=N/A time=00:00:05.58 bitrate=N/A speed=3.73x    
frame=  182 fps= 91 q=-0.0 size=N/A time=00:00:07.54 bitrate=N/A speed=3.77x    
frame=  227 fps= 91 q=-0.0 size=N/A time=00:00:09.42 bitrate=N/A speed=3.77x    
frame=  272 fps= 91 q=-0.0 size=N/A time=00:00:11.30 bitrate=N/A speed=3.77x    
frame=  317 fps= 91 q=-0.0 size=N/A time=00:00:13.18 bitrate=N/A speed=3.76x    
frame=  363 fps= 91 q=-0.0 size=N/A time=00:00:15.09 bitrate=N/A speed=3.77x    
frame=  410 fps= 91 q=-0.0 size=N/A time=00:00:17.05 bitrate=N/A speed=3.79x    
frame=  456 fps= 91 q=-0.0 size=N/A time=00:00:18.97 bitrate=N/A speed=3.79x    
frame=  503 fps= 91 q=-0.0 size=N/A time=00:00:20.93 bitrate=N/A speed=3.81x    
frame=  550 fps= 92 q=-0.0 size=N/A time=00:00:22.89 bitrate=N/A speed=3.82x    
frame=  596 fps= 92 q=-0.0 size=N/A time=00:00:24.81 bitrate=N/A speed=3.82x    
frame=  642 fps= 92 q=-0.0 size=N/A time=00:00:26.73 bitrate=N/A speed=3.82x    
frame=  686 fps= 91 q=-0.0 size=N/A time=00:00:28.57 bitrate=N/A speed=3.81x    
frame=  733 fps= 92 q=-0.0 size=N/A time=00:00:30.53 bitrate=N/A speed=3.82x    
frame=  779 fps= 92 q=-0.0 size=N/A time=00:00:32.44 bitrate=N/A speed=3.82x    
frame=  825 fps= 92 q=-0.0 size=N/A time=00:00:34.36 bitrate=N/A speed=3.82x    
frame=  870 fps= 92 q=-0.0 size=N/A time=00:00:36.24 bitrate=N/A speed=3.81x    
frame=  915 fps= 91 q=-0.0 size=N/A time=00:00:38.12 bitrate=N/A speed=3.81x    
frame=  961 fps= 91 q=-0.0 size=N/A time=00:00:40.04 bitrate=N/A speed=3.81x    
frame= 1006 fps= 91 q=-0.0 size=N/A time=00:00:41.91 bitrate=N/A speed=3.81x    
frame= 1052 fps= 91 q=-0.0 size=N/A time=00:00:43.83 bitrate=N/A speed=3.81x    
frame= 1098 fps= 91 q=-0.0 size=N/A time=00:00:45.75 bitrate=N/A speed=3.81x    
frame= 1143 fps= 91 q=-0.0 size=N/A time=00:00:47.63 bitrate=N/A speed=3.81x    
frame= 1188 fps= 91 q=-0.0 size=N/A time=00:00:49.50 bitrate=N/A speed=3.81x    
frame= 1234 fps= 91 q=-0.0 size=N/A time=00:00:51.42 bitrate=N/A speed=3.81x    
frame= 1280 fps= 91 q=-0.0 size=N/A time=00:00:53.34 bitrate=N/A speed=3.81x    
frame= 1325 fps= 91 q=-0.0 size=N/A time=00:00:55.22 bitrate=N/A speed=3.81x    
frame= 1370 fps= 91 q=-0.0 size=N/A time=00:00:57.09 bitrate=N/A speed=3.81x    
frame= 1414 fps= 91 q=-0.0 size=N/A time=00:00:58.93 bitrate=N/A speed= 3.8x    
frame= 1460 fps= 91 q=-0.0 size=N/A time=00:01:00.85 bitrate=N/A speed= 3.8x    
frame= 1506 fps= 91 q=-0.0 size=N/A time=00:01:02.77 bitrate=N/A speed= 3.8x    
frame= 1547 fps= 91 q=-0.0 size=N/A time=00:01:04.48 bitrate=N/A speed=3.79x    
frame= 1590 fps= 91 q=-0.0 size=N/A time=00:01:06.27 bitrate=N/A speed=3.79x    
frame= 1634 fps= 91 q=-0.0 size=N/A time=00:01:08.15 bitrate=N/A speed=3.78x    
frame= 1676 fps= 91 q=-0.0 size=N/A time=00:01:09.86 bitrate=N/A speed=3.77x    
frame= 1720 fps= 90 q=-0.0 size=N/A time=00:01:11.69 bitrate=N/A speed=3.77x    
frame= 1764 fps= 90 q=-0.0 size=N/A time=00:01:13.53 bitrate=N/A speed=3.77x    
frame= 1810 fps= 90 q=-0.0 size=N/A time=00:01:15.45 bitrate=N/A speed=3.77x    
frame= 1855 fps= 90 q=-0.0 size=N/A time=00:01:17.32 bitrate=N/A speed=3.77x    
frame= 1893 fps= 90 q=-0.0 size=N/A time=00:01:18.91 bitrate=N/A speed=3.76x    
frame= 1939 fps= 90 q=-0.0 size=N/A time=00:01:20.83 bitrate=N/A speed=3.76x    
frame= 1985 fps= 90 q=-0.0 size=N/A time=00:01:22.74 bitrate=N/A speed=3.76x    
frame= 2029 fps= 90 q=-0.0 size=N/A time=00:01:24.58 bitrate=N/A speed=3.76x    
frame= 2076 fps= 90 q=-0.0 size=N/A time=00:01:26.54 bitrate=N/A speed=3.76x    
[Parsed_libvmaf_2 @ 0x7f0a380067c0] VMAF score: 98.251637

It seems like this is capturing and redirecting some control characters too though because when I cat the file instead, I get this…?

❯ cat out.score
[Parsed_libvmaf_2 @ 0x7f0a380067c0] VMAF score: 98.251637ate=N/A speed=3.76x

How do I make it write only the final score line :confused:

From what I’m reading it seems the way the “updating lines” are achieved is by outputting a \r, so for tail everything is one line?

edit:
aha, got it:

…
	|& grep 'VMAF score:' | sed 's/.*\r//' | tee -a "${2%.*}.sub"

i.e. replace everything before (and including) \r with nothing.

I’ve given you several solutions to this problem before: The small linux problem thread - #6374 by rcxb

You can use col -b, vt100.py or one of the several programs named ansi2txt.

Something else that should work in this case: tr '\r' '\n'

Mh interesting, didn’t realise that was the same kind of thing.

But oddly, that somehow breaks the output entirely:

# no piping
❯ ./vmaf in.mkv out.webm
…
[Parsed_libvmaf_2 @ 0x7fd7180067c0] VMAF score: 98.251637ate=N/A speed=4.42x

# sed pipe
❯ ./vmaf in.mkv out.webm
[Parsed_libvmaf_2 @ 0x7f343c006600] VMAF score: 98.251637
❯ tail -n1 out.sub
[Parsed_libvmaf_2 @ 0x7f343c006600] VMAF score: 98.251637

# col -b pipe
❯ ./vmaf in.mkv out.webm
[Parsed_libvmaf_20@ 0x7f7620006980] VMAF=score:298.251637ate=N/A speed=44.4x
❯ tail -n1 out.sub      
[Parsed_libvmaf_20@ 0x7f7620006980] VMAF=score:298.251637ate=N/A speed=44.4x
                 ^                             ^

Not sure why it would be translating the first space to a 0 and the one on the score to a 2?

That output is… interesting

❯ ./vmaf in.mkv out.webm
…
frame=
2037
fps=104
q=-0.0
size=N/A
time=00:01:24.91
bitrate=N/A
speed=4.35x




frame=
2084
fps=104
q=-0.0
size=N/A
time=00:01:26.87
bitrate=N/A
speed=4.34x




[Parsed_libvmaf_2
@
0x7f88400067c0]
VMAF
score:
98.251637

Isn’t \r just a carriage return? Why is it translating spaces too :thinking: Or am I having a misconception of what a carriage return is or does? I thought it’s going back to the start of the line? Which clearly spaces don’t, so… confused

col -b isn’t very smart, it works in the simpler cases and it’s convenient as it happens to be included in the base of every unix system. One of the other tools (that you’ll have to find and install) are sure to do better.

Having a boot-related brain fart here…

Fedora 41

A while ago, before ZFS supported 6.13, I did this sudo grubby --set-default /boot/vmlinuz-6.12.15-200.fc41.x86_64 to have my ZFS-pool available, and now that 6.12 is the only kernel that boots, anything else that is listed (in Grub) results in kernel panic.

I fixed this before, but did not write down how and google-fu is currently AFK


Edit:
Well…
We do not speak of this

dracut --list-modules
dracut --kver 6.13.8-200.fc41.x86_64 --force
grubby --set-default /boot/vmlinuz-6.13.8-200.fc41.x86_64
reboot

Everything just works!

2 Likes

May I dare to ask why you are using root on ZFS and not root on BTRFS with Fedora?

On a scale of 1 to dear god, how bad is it that I do not understand your question?

2 Likes

Ohhh I think I got you wrong. I though you installed Fedora on ZFS but you only wanted to make your ZFS pool available on Fedora. I meant to ask you why you went through the trouble of installing Fedora on ZFS when it comes natively with BTRFS as a filesystem that has largely the same features but I understood you wrong.

1 Like

Yep.
I got a 1TB boot SSD (BTRFS)
4TB project-dump/general software
36TiB useable ZFS pool as a data grave

1 Like