This is a guide based on my experience with AV1 Software Encoding. As it is AV1 guides are spread out and there is not an all encompassing thread. I will start with libaom or aomenc It is the initial encoder that was released and generally is the slowest, but also best one for overall quality of encodes.
A little information before we start.
What is a video codec?
It is a language of sorts that stores video information and is sent to a decoder which a player uses to display an image on a computer, phone, tablet, or television. There are audio codecs and video, and text and so on. What I am focusing on is a more recently developed video one known as AV1.
The gist is for each evolution and subsequent codec that is released you are able to do one of two things or land inbetween. For instance h264 aka avc was developed in the mid to late 2000s. It is what we call a more efficient codec. It can handle higher bit rates (the rate at which an image is written to a display). Bit rate is also the determining factor of how big a video or audio file can be. In this case a movie generally has audio and video included in a single file.
What I am getting at is that going from mpeg2 to avc was a huge upgrade in what image quality could look like. Going from h264/avc to h265/hevc is a similar step forward where you can at similar bit rates have but the newer one having a higher quality image output.
Using hevc over avc you may see up to 50% better image quality (highly dependent source material codec). If both files are 8GB in size, the hevc in theory would look better than avc depending on what setting and parameters were used to convert to hevc and avc. Another way to look at it is via compression. Taking a avc source input and outputting it to hevc can lead to smaller sizes while retaining much of the overall fidelity. So you can have higher quality at the same bitrate or shrink the file size down and have the same next to near the same quality.
I would like to add that bitrate is seen as kbps (kilobits/second) or mbps (megabits/second) which is how much information is sent per second from the video file to a player. Depending on the codec it can be smaller and still retain most of quality to that of a less efficient codec. The thing is that it is a general idea as to what quality might appear. It still has to be decoded on the end of a video player. It comes down to to a software decoder or a hardware decoder which convert the codec to a visual signal the display can output an image.
All of this, compression is lossy. Which means the image will never be 100% of what it looks like from source. But it can be close enough. It can be 95% of the source in appearance all while being a smaller file size leading to less network load. Leading to a smaller bitrate.
It is highly dependent on source material and how much actual information you have to work with. For instance you can compress bluray video more easily and shrink the size than an already small size file. i.e. older youtube videos, dvds, etc. The more information the source material has the better.
Blurays are fantastic because they already have high bit rates and high fidelity. So using bluray rips are a good source to attempt at compressing while retaining most of original fidelity. It requires tinkering with parameters to find what you are willing to end up with. There are trade offs of course. Some loss, but in the best of circumstances it leads to no or very little fidelity loss.
As I said the blurays are a good starting point for compression because the more a source file has in terms of fidelity and bitrate the better compression can be. Smaller video sizes are harder for encoders to work with because the smaller the size the less information there is to compress. The more information the better it can compress.
The special sauce is basically the more a encoder has to work with the more it can do. When you use smaller already compressed or sources the less it can do.
You might ask why do this at all if there is any kind of loss?
The answer is storage space is expensive and you can cut down on the use by compressing video and audio. As well as less network strain and bandwidth consumption if you host your own media server (plex, jellyfin, etc).
The issue is most codecs are licensed out by royalties for software developers. They have to license hevc and avc and mpeg2 and mpeg4, etc.
This is where AV1 comes in.
AV1 was created by The Alliance For Open Media (aom) as an open source, royalty free codec. This simply means to include it in applications does not require licensing fees. Anyone can do whatever they want with the source code and not have to worry about violating licenses or having to pay for it.
As I said hevc is ~50% better at compressing video than avc is. So what does AV1 offer that hevc does not? Well, in theory with proper parameters you can see 30 to 50% better compression with av1 than hevc. Again we can either have better quality at the same bitrate or have a smaller bitrate and have it the same quality as the source/input file.
The issue is every time we take a step forward with a new codec the processing power needed increases. Compressing mpeg4 to h264 is easier and takes less time than h264 to h265. As computation increases and hardware accelerators begin to come out you will see a wider adoption. But generally speaking new codecs don’t receive wide spread adoption until years after the fact. When development and need come together you see actual widespread adoption.
AV1 is still an infant being released initially in 2019. libaom or aomenc was released at this point. Which the issue is it is extremely slow and not threaded well at all. So multi core machines don’t benefit natively from aom. It is painstakingly slow by itself. SVT-AV1 is a encoding application that is developed by intel and adopted by aom as a production ready encoder. As it is libaom is not production ready as it requires additional tools and hacks to speed it a long. Svt on the other hand is much faster and more wide spread. Tools such as ffmpeg and handbrake already have added svt-av1 to their arsenal of encoding functionality.
Though you may find people willing to use libaom for absolute quality:compression over speed. I am one of those people that have spent over a year encoding using libaom to get the best compression to quality ratio. In this guide I will get you started on the AV1 adventure and will focus on libaom as that is all I have really used since 2021.
Need is coming with the rise of 8k and beyond. AV2 is already in the works, h266 as well. Netflix and Google are big time investors in AV1 as it can reduce costs of 4k streams. When everyone has adopted hardware decoders, it is a win win. Lower bit rate than previously all while offering higher quality video.