FFMPEG's demuxer reduces file size

I have two video files (MKV container, AV1 codec) that I want to concatenate and I did it like so:

ffmpeg -f concat -i input.txt -c copy "out.mkv"  

Interestingly enough, the first file has 12GiB and the second has 15GiB, but the concatenated result has only 19GiB (70%). Do MKV containers or videos in general have such a high overhead or did something go wrong?

Demux into raw video files and do concat (not really recommended) but expect things to not work as expected.

1 Like

I don’t really understand your proposal. Given that I still have the raw files I don’t need to demux into raw files and I could use those instead. However, having separate files results in more manageable encoding times :wink:

Edit: I’m also not entirely sure if concat doesn’t just remove the overhead of both files or something like that. The ffmpeg manual states the following:

I also did a concatenation with MKVToolNix and the end result has 26GiB…which increases my suspicion regarding ffmpeg.

MKV remuxing is known to screw up timestamps in FFmpeg. I have never gotten a stream to output proper timestamps when reading from MKV.

You have to reconstruct the timestamps completely, then concatenate.

You also might not be copying all streams, and just copying the primary streams. For this, look up how to use the “-map” function to copy all streams.

2 Likes

Should that not be fixed with ffmpeg -fflags +genpts ...?

@Azulath As already mentioned, MKV may contain a lot of stuff, some of it could be multilingual subtitles and audio tracks. If all the extras are omitted in the concat operation, the resulting file size will be lower than expected. You could use ffprobe or mediainfo to examine the source files and get a list of streams.

1 Like

Hmm…I will look into this.

Do I have to add this as well or is it added implicitly?

That is probably the issue. I did an fprobe on the ffmpeg concatenated file with the following result:

ffprobe version n5.1.2 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.2.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp--enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[libdav1d @ 0x55f9e882ee00] libdav1d 1.0.0
Input #0, matroska,webm, from 'out.mkv':
  Metadata:
    ENCODER         : Lavf59.27.100
  Duration: 03:48:18.37, start: 0.000000, bitrate: 11545 kb/s
  Stream #0:0: Video: av1 (Main), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1608, SAR 1:1 DAR 160:67, 23.98 fps, 23.98 tbr, 1k tbn
    Metadata:
      DURATION        : 03:48:18.364000000
    Side data:
      Content Light Level Metadata, MaxCLL=602, MaxFALL=184
      Mastering Display Metadata, has_primaries:1 has_luminance:1 r(0.6800,0.3200) g(0.2650,0.6900) b(0.1500 0.0600) wp(0.3127, 0.3290) min_luminance=0.005000, max_luminance=4000.000000
  Stream #0:1(eng): Audio: truehd, 48000 Hz, 7.1, s32 (24 bit)
    Metadata:
      title           : Surround
      DURATION        : 03:48:18.366000000
[libdav1d @ 0x55f9e8832500] libdav1d 1.0.0

Wheras this is the concatenated MKVToolNix result:

ffprobe version n5.1.2 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.2.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp--enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[libdav1d @ 0x559cbd96d6c0] libdav1d 1.0.0
[matroska,webm @ 0x559cbd955e00] Could not find codec parameters for stream 5 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x559cbd955e00] Could not find codec parameters for stream 6 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x559cbd955e00] Could not find codec parameters for stream 7 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #0, matroska,webm, from out.mkv':
  Metadata:
    title           : vid
    creation_time   : 2022-11-28T17:31:19.000000Z
    ENCODER         : Lavf59.27.100
  Duration: 03:48:18.37, start: 0.000000, bitrate: 16168 kb/s
  Chapters:
    Chapter #0:0: start 0.000000, end 457.958000
      Metadata:
        title           : Chapter 1
    Chapter #0:1: start 457.958000, end 641.141000
      Metadata:
        title           : Chapter 2
    Chapter #0:2: start 641.141000, end 946.446000
      Metadata:
        title           : Chapter 3
    Chapter #0:3: start 946.446000, end 1200.866000
      Metadata:
        title           : Chapter 4
    Chapter #0:4: start 1200.866000, end 1536.702000
      Metadata:
        title           : Chapter 5
    Chapter #0:5: start 1536.702000, end 1801.300000
      Metadata:
        title           : Chapter 6
    Chapter #0:6: start 1801.300000, end 1933.598000
      Metadata:
        title           : Chapter 7
    Chapter #0:7: start 1933.598000, end 2096.428000
      Metadata:
        title           : Chapter 8
    Chapter #0:8: start 2096.428000, end 2188.019000
      Metadata:
        title           : Chapter 9
    Chapter #0:9: start 2188.019000, end 2707.038000
      Metadata:
        title           : Chapter 10
    Chapter #0:10: start 2707.038000, end 2808.806000
      Metadata:
        title           : Chapter 11
    Chapter #0:11: start 2808.806000, end 3083.581000
      Metadata:
        title           : Chapter 12
    Chapter #0:12: start 3083.581000, end 3317.648000
      Metadata:
        title           : Chapter 13
    Chapter #0:13: start 3317.648000, end 3449.947000
      Metadata:
        title           : Chapter 14
    Chapter #0:14: start 3449.947000, end 3816.313000
      Metadata:
        title           : Chapter 15
    Chapter #0:15: start 3816.313000, end 4040.203000
      Metadata:
        title           : Chapter 16
    Chapter #0:16: start 4040.203000, end 4141.971000
      Metadata:
        title           : Chapter 17
    Chapter #0:17: start 4141.971000, end 4243.740000
      Metadata:
        title           : Chapter 18
    Chapter #0:18: start 4243.740000, end 4518.514000
      Metadata:
        title           : Chapter 19
    Chapter #0:19: start 4518.514000, end 4671.167000
      Metadata:
        title           : Chapter 20
    Chapter #0:20: start 4671.167000, end 5057.886000
      Metadata:
        title           : Chapter 21
    Chapter #0:21: start 5057.886000, end 5180.008000
      Metadata:
        title           : Chapter 22
    Chapter #0:22: start 5180.008000, end 5383.545000
      Metadata:
        title           : Chapter 23
    Chapter #0:23: start 5383.545000, end 5597.258000
      Metadata:
        title           : Chapter 24
    Chapter #0:24: start 5597.258000, end 5760.088000
      Metadata:
        title           : Chapter 25
    Chapter #0:25: start 5760.088000, end 5861.856000
      Metadata:
        title           : Chapter 26
    Chapter #0:26: start 5861.856000, end 6343.379000
      Metadata:
        title           : Chapter 27
    Chapter #0:27: start 6343.392000, end 6424.807000
      Metadata:
        title           : Chapter 1
    Chapter #0:28: start 6424.807000, end 6526.575000
      Metadata:
        title           : Chapter 2
    Chapter #0:29: start 6526.575000, end 6607.990000
      Metadata:
        title           : Chapter 3
    Chapter #0:30: start 6607.990000, end 6791.173000
      Metadata:
        title           : Chapter 4
    Chapter #0:31: start 6791.173000, end 7076.124000
      Metadata:
        title           : Chapter 5
    Chapter #0:32: start 7076.124000, end 7462.844000
      Metadata:
        title           : Chapter 6
    Chapter #0:33: start 7462.844000, end 7849.563000
      Metadata:
        title           : Chapter 7
    Chapter #0:34: start 7849.563000, end 8419.466000
      Metadata:
        title           : Chapter 8
    Chapter #0:35: start 8419.466000, end 8999.546000
      Metadata:
        title           : Chapter 9
    Chapter #0:36: start 8999.546000, end 9192.905000
      Metadata:
        title           : Chapter 10
    Chapter #0:37: start 9192.905000, end 9630.509000
      Metadata:
        title           : Chapter 11
    Chapter #0:38: start 9630.509000, end 9966.345000
      Metadata:
        title           : Chapter 12
    Chapter #0:39: start 9966.345000, end 10068.113000
      Metadata:
        title           : Chapter 13
    Chapter #0:40: start 10068.113000, end 10393.772000
      Metadata:
        title           : Chapter 14
    Chapter #0:41: start 10393.772000, end 10688.900000
      Metadata:
        title           : Chapter 15
    Chapter #0:42: start 10688.900000, end 11075.620000
      Metadata:
        title           : Chapter 16
    Chapter #0:43: start 11075.620000, end 11513.223000
      Metadata:
        title           : Chapter 17
    Chapter #0:44: start 11513.223000, end 11665.876000
      Metadata:
        title           : Chapter 18
    Chapter #0:45: start 11665.876000, end 12052.596000
      Metadata:
        title           : Chapter 19
    Chapter #0:46: start 12052.596000, end 12510.553000
      Metadata:
        title           : Chapter 20
    Chapter #0:47: start 12510.553000, end 13698.365000
      Metadata:
        title           : Chapter 21
  Stream #0:0: Video: av1 (Main), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1608, SAR 1:1 DAR 160:67, 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      BPS             : 8482616
      DURATION        : 03:48:18.365000000
      NUMBER_OF_FRAMES: 328432
      NUMBER_OF_BYTES : 14524747499
      _STATISTICS_WRITING_APP: mkvmerge v71.1.0 ('Fortitude') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2022-11-28 17:31:19
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
    Side data:
      Content Light Level Metadata, MaxCLL=602, MaxFALL=184
      Mastering Display Metadata, has_primaries:1 has_luminance:1 r(0.6800,0.3200) g(0.2650,0.6900) b(0.1500 0.0600) wp(0.3127, 0.3290) min_luminance=0.005000, max_luminance=4000.000000
  Stream #0:1(eng): Audio: truehd, 48000 Hz, 7.1, s32 (24 bit) (default)
    Metadata:
      title           : Surround
      BPS             : 2994309
      DURATION        : 03:48:18.366000000
      NUMBER_OF_FRAMES: 16438023
      NUMBER_OF_BYTES : 5127142710
      _STATISTICS_WRITING_APP: mkvmerge v71.1.0 ('Fortitude') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2022-11-28 17:31:19
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s
    Metadata:
      title           : Surround 5.1
      BPS             : 448000
      DURATION        : 03:48:18.368000000
      NUMBER_OF_FRAMES: 428074
      NUMBER_OF_BYTES : 767108608
      _STATISTICS_WRITING_APP: mkvmerge v71.1.0 ('Fortitude') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2022-11-28 17:31:19
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:3(ger): Audio: dts (DTS-HD MA), 48000 Hz, 6.1, s16p
    Metadata:
      title           : Surround 6.1
      BPS             : 2644165
      DURATION        : 03:48:18.368000000
      NUMBER_OF_FRAMES: 1284123
      NUMBER_OF_BYTES : 4527593576
      _STATISTICS_WRITING_APP: mkvmerge v71.1.0 ('Fortitude') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2022-11-28 17:31:19
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:4(ger): Audio: dts (DTS-ES), 48000 Hz, 6.1, fltp, 1536 kb/s
    Metadata:
      title           : Surround 5.1
      BPS             : 1508883
      DURATION        : 03:48:18.368000000
      NUMBER_OF_FRAMES: 1284123
      NUMBER_OF_BYTES : 2583655476
      _STATISTICS_WRITING_APP: mkvmerge v71.1.0 ('Fortitude') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2022-11-28 17:31:19
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:5(eng): Subtitle: hdmv_pgs_subtitle
    Metadata:
      BPS             : 23231
      DURATION        : 03:19:44.235000000
      NUMBER_OF_FRAMES: 3634
      NUMBER_OF_BYTES : 34801087
      _STATISTICS_WRITING_APP: mkvmerge v71.1.0 ('Fortitude') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2022-11-28 17:31:19
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:6(ger): Subtitle: hdmv_pgs_subtitle
    Metadata:
      BPS             : 25025
      DURATION        : 03:27:21.150000000
      NUMBER_OF_FRAMES: 3672
      NUMBER_OF_BYTES : 38917529
      _STATISTICS_WRITING_APP: mkvmerge v71.1.0 ('Fortitude') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2022-11-28 17:31:19
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:7(ger): Subtitle: hdmv_pgs_subtitle
    Metadata:
      BPS             : 805
      DURATION        : 02:08:38.891000000
      NUMBER_OF_FRAMES: 68
      NUMBER_OF_BYTES : 777123
      _STATISTICS_WRITING_APP: mkvmerge v71.1.0 ('Fortitude') 64-bit
      _STATISTICS_WRITING_DATE_UTC: 2022-11-28 17:31:19
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
[libdav1d @ 0x559cbda42c80] libdav1d 1.0.0

This more or less corresponds to the real source file. (The chapters seem to be messed up though.)

You should use +genpts (GENerate Presentation TimeStamps) if there are problems with the timestamps in the output file (out of sync and so on), otherwise not.

As mentioned by @FurryJackman you can use the -map option to select the streams that ffmpeg should map. It’s been a while since I’ve used ffmpeg from the command line (I use FFQueue) but this could work:

ffmpeg -f concat -i input.txt -map 0:v:0 -map 0:a:0 -map 0:a:1 -map 0:s:0 -map 0:s:1 -c copy "out.mkv"

Above command should map video, the 2 first audio streams and the 2 first subtitle streams - but look it up!

EDIT: Added 0: to map args as file indexer…

2 Likes

Have you heard of av1an? I use it to split source files and then encode using av1an. It concats with lsmash or ffms2(ffmpeg) plugin for vapoursynth.

1 Like

@Danese and @Argone: Thanks for those suggestions. For the time being though, I have decided to just use MKVToolNix as it fulfills my requirements :slight_smile:

1 Like