I know LZMA is popular but is there any competition to one-man made compression? lol
and how does microsoft compress their entire os installtion to fit inside a dvd or ~5gb ish while uncompressed goes over 10gb?
LZMA2 is the next step from that and it's my own preferred format. It is desgined to work best with 4 threads so that means better performance and it makes splitting files into blocks easier(something I do often). It is also better at compressign already compressed data, similar to tarballs. One downside is it uses quite a lot of RAM, though for some that isn't a problem.
One thing to consider though is that there are many different compression formats and each is better for certain things such as text, binary format, media etc. So basically, just mess around and see what works best for the things you archive.
As for how Microsoft does what they do, it is very similar to how Valve compresses their games so well when doing Steam backups. Where as standard compression formats are designed to work best for a variety of data types, both Valve and Microsoft both know exactly how their files are constructed and therefore know the best compression patterns and algorithms for their files.
^This use 7zip