Hello all,
What are your thoughts on the best file format to use to archive (cold storage) your game collection (from multiple sources, GOG, Steam, ROMs, disc-based, previous gen consoles, et al)?
Old gamer here, that together with Steam Deck and Miniforums PCs, noticed that its a great time (and finally possible!) to have one’s gaming collection portable, local and with a cold storage backup - just in case internet is out or the portable collection has issues.
Am leaning towards ISO for the following reasons:
- Compression is not the goal, but readability and quick access to data when (rarely) needed.
- ISO has come a long way, don’t think the format will disappear anytime soon.
- Handles disc based games and multiple volumes easily (think GOG games with multiple 4Gb files ).
- Linked to the previous point, copy/moving one big file is more convenient/faster (handy, when installed, those EPIC DRM free freebies can sometimes can reach 30k or 200k files…with many being files with a few Mb)
Or is there a format that is better suited for the task that am not aware of?
Thank you kindly for your time,
I personally tarball then zstandard my local GOG installers, software, source code, etc. Tarball takes care of the multiple files, and zstd is extremely quick to compress and decompress.
If you’re a Windows user, this 7-zip frontend with zstd integration can take care of both steps. On Linux you can just build zstdandard then run it with something like:
./zstd -T112 -17 'input.tar' -o output.tar.zst
I use compression level 17, but I believe you can go up to 21 with diminishing returns. The uppercase T adjusts your thread count.
My best result so far has been zstandard taking the Unreal Engine 4 source from 70GB with dependencies to only 16GB after compression, but clearly pre-compressed game installers won’t match that compression ratio 
3 Likes
I had compression enabled on a ZFS pool, but the results for games, even with 1MB recordsize, were not good. Does manually compressing the entire folder produce better results?
That’s difficult to say. Anecdotally, my hugest GOG games like Kingdom Come Deliverance (80GB exactly) only compressed to 76.2GB after running it through zstandard; a 5% savings.
The files contained within GOG installers are already compressed, so running something like your Steam games folder through zstandard would likely yield better results.
1 Like
I got all my games on BTRFS with in-line compression and get around 1.35 compression ratio out of level 3.
Stuff with lots of files compress very well. Modern games with large archive folders (textures and stuff) are already compressed for the most part.
Always good to save space and have everything accessible all the time.
Why cold storage? You should be able to share the files over the network if not locally stored. All my old non-Steam stuff sit on a share on my NAS.
Problem with compressing entire libraries is that you have to uncompress everything for some files if you need them. Gladly, we got options today. No need for zip files ever again.
Compressing yields better results when transfering files (as opposed to moving thousand of small files) but reckon mostly its because of the file size rather than compression - compression gains would be small as one would expect.
I have the “silly” ultimate goal of having one file per game for achiving purposes as to have a “clean” archive that continues to grow to a respectable size - am gamer since the 90s to its pilling up, Doom 1993 is not retro, I remember like it was yesterday 
My concern with zip or tar is that a couple of my old zip files were paid by a visit by the CRC infamous error, as somewhen downline when upgrading from 3.5’’ to CDs to BD-R…something happened.
Cold storage because as these would be the “ultimate” backup to keep just in case, as the portable/playable/drm-free will live happily on SSD installed regardles that I can plug on a Steam Deck, or any PC with a USB-C port - and in case something happens or data is corrupted, worst case the cold storage would have my back.
Would agree that there is no need to zip files nowadays, hence the ISO approach that sounded appealing my brief experience with the format has been positive - usually with disc images.
depends how you want long you want to spend decompressing your games.
if you zip em you can control the amount of compression applied. with less being faster to decompress.
if you stick em in an iso. they will go in as is with no compression but the files will be immediately accessible
the ultimate, is rar them and then rename the file as an iso.
the rar file should then be mountable as an iso image.
giving you compression and mount ability.
Interesting, did not knew that about rar.
No observations were made regarding data integrity, am I too assume that you guys trust the formats you suggest as you did not experience any data loss or issues (be it on .tar .zip .iso or .rar) such as not being able to access the contents of the containers?
Perhaps my concern is no longer justified nowaways ad overblown by losing the original Lemmings as .zips became unreadable - which I still can’t find for sale.
So on the topic of compression I am currently benchmarking this to make a decision. I will benchmark XZ, ZSTD19 and ZSTD22. As soon as my machine is done compressing I will let you know how it looks for GOG games.
Yeah ZSTD 19 isn’t gonna cut it. Default level 3 will do most jobs. Decompresses at 2.4GB/s on my 7900x. Level 10+ will take some time, not what I would use for anything that I need at some point.
For me the goal is long term archival. I can store the installed games on an SSD no problem, but the installers I want for eternity* stored on disc in case the world goes to heck or one of those game providers out of business.
* or for the time being
Horizon Zero Dawn installation files from GOG
uncompressed 82.1GB
zstd19 81,8GB
zstd22 81,5GB
xz 81,8GB
Crysis installation files from GOG
uncompresses 7.4GB
zstd19 7.4GB
zstd22 7.4GB
xz 7.4GB
Tomb Raider 1 installation files from GOG
uncompressed 511.5MB
zstd19 498.2MB
zstd22 492.8MB
xz 494.7MB
Conclusion is, it is not really worth compressing the binary installers.
Thanks for the data @H-i-v-e. For compression am all in in agreeing not worth the effort/gains, so will skip on ZIP and maintain my inclination for ISO volumes as I still see perhaps two use cases:
a) many GOG games have multiple or dozens of 4Gb volumes (more a personal taste to see one file per game)
b) other sources like Epic freebies, ROMs et al where installers are not provided/available
Even for long term achival (I goal I share), having it in ISO or a container has a plus side for future self: when the time comes to backup or transfer the collection to a new media it would be faster I think - as copying a few thousand large files is much faster hundred of thousands of files of smaller size.
1 Like
Fair enough, one advice that I want to give you, both xz- as well as zst-Archives feature checksums. On top of that I place my files in a ZFS pool, which has checksumming on a per record basis, nothing is more troublesome than to think you have something saved, only for it to be corrupted when the time comes. Checksums will not save you from that, but they will let you know.
Thank you all for your feedback, got some food for thought! Not yet ready to fully switch to ZFS but will get there, will start with ISOing as a trial and see how this fares.
Many projects in parallel - but must say, first timer on L1 forums and such quick and detailed feedback is inspiring and will definitely start to post here often and share my projects (one of them is how-to to run older games in new systems - for which I will take the chance to switch/learn the Linuxverse ways!)