Frequently when starting a game in Steam on Linux I’m presented with a window that says “Processing Vulkan shaders…” This can take several minutes to complete, or actually just sit there ad infinitum. There is a skip button, but I’ve struggled to notice a difference between skipping and not skipping.
So dumb question time: what is a shader cache, and why should I wait for this thing to finish compiling? Heck, what is a shader anyway?
There are games which want to cache frequently used shaders, some games it doesn’t make much of a difference and others such as racing/open world games the caching helps–Dirt 4 for example waiting helps reduce FPS dips. For the most part your mileage varies from game to game.
I think it just makes the game run faster on its first run after some update has happened. I skipped it once (Dying Light) and when the game launched, it hanged for a bit and stayed inactive longer than usual. Granted my spec was i5-6600k+GTX970 so maybe more current hardware can tolerate skipping said precaching step.
Shaders are used for a variety of things. To be used under Linux they need to be compiled usually using LLVM, or on newer versions of Mesa using ACO as shader compilers.
This is normal and can’t be circumvented, the issue with that however is that it takes away CPU cycles for the compilation which - depending on the game and overall load - can result in performance dips or short hangs where nothing can be rendered (because no shader = nothing to render).
Compiled shaders are also incompatible between driver versions, so simply saving the compiled shaders somewhere doesn’t work.
The solution that DXVK (which is the underlying project used in Proton) came up with is saving the shaders that it encounters during the games runtime in a file, the shader cache.
When it encounters an updated driver stack and therefore invalidated compiled shaders, it can re-compile them ahead of time based on the shader cache.
An additional benefit is also that shader caches can be shared, because they are essentially just text files.
As said before how this actually affects gameplay depends on a variety of factors. Personally I just don’t want to deal with potential headaches so I just let the compilation run.
There are games that are very light on shaders, others are fairly heavy. My FF14 Shader cache for example is roughly 2.5MB, while the cache for Overwatch provided by Lutris is roughly 23MB.
If you run Overwatch without a shader cache you will notice a lot of hiccups for a long time until it builds up its cache, which is why Lutris provides one. After compilation it is fairly smooth even on the first match.
edit:
As for “why does it take so long to compile”, that’s difficult to answer as we don’t know your hardware. But it also depends on the amount of shaders (see above). Why it sometimes never stops compiling I can’t say, never had that happen personally.
Shader cache compilation isn’t unique to Linux or vulkan or proton either.
A few games using dx12 on windows do it (eg borderlands 3) and elite dangerous rebuilds it’s shader cache every driver update.
It’s just maybe more frequent on Linux using proton as described above.
OK, I’ll bite. The hardware I have the worst problem is my gaming laptop, an Asus ROG Zephyrus M15. It’s got an Intel i7-10750H and a GTX 2070 Max-Q (paired with Intel GT2 iGPU.) Not sure why that would take so long to compile on reasonably competent hardware.
OTOH the compilation time on my Ryzen 7 1700 + Vega 56 desktop is usually more tolerable.
Shader compilation is entirely on the CPU so the GPU is not really relevant in this case. I might be wrong here but I think the shader compilation is single-threaded and that CPU doesn’t have a super high power budget, so that might be part of the reason.
I can’t tell you why it takes so long tbh. But as mentioned before it’s not purely the hardware but also the size of the cache. So unless you’re comparing the same game, you can’t really compare.
I’m not sure how Steam handles this but I think when you cancel it, it will try again on next launch. So maybe when it’s hanging for long, just try again?
You can disable it in the steam settings, it was taking my 5900x roughly an hour to process Metro Exodus shaders prior to running every time there was an update it would peg my CPU at 100% till it wrapped up. I’ve left it off since then, and haven’t had any issues playing Metro or any other games with it off since, YMMV.
I find it very convenient to run the processing in the background. You can set this all up in steam settings menu or skip it all together. The pre-processing is a notable difference in my games and for me, it runs in the background so when steam updates things automatically and wants to redo all the pre-caching, it’s all background task so I don’t notice it at all except for new games. And with 39 nice values and multithreading, this process scales very nicely with additional cores and doesn’t block more important processes.
The background processing for some reason I don’t understand, really screws up Cyberpunk. So for now I have to leave it disabled. And oddly I also have to delete the CP2077 shader cache manually from time to time. See here:
I suspect that the Intel iGPU is confusing the shader compiler. But I have no way to verify that.
It doesn’t process in the background during play, just treat it as part of the installation process. If you are background processing during play, expect heavy load of the CPU potentially screwing up your framerates.
The background processing usage is that you dont have watch the progress bar on steam when launching, it’s just a convenience feature.
After deleting the shader cache, for whatever reason, it’s smooth as silk. No idea why. Until the next time Steam decides to re-compile the shaders anyway. Hopefully this will be fixed in a future Cyberpunk or Steam patch.
And perhaps the original problem was simply that the cache needed to be deleted, as opposed to the background processing option in Steam. As long as It works…
If you see negative impact, you can disable pre-caching all together (Steam → Settings → Shader Pre-Caching). At the moment you can’t check or uncheck on a per-game basis, so this might screw up your other games that benefit from it.
I know that some games do this during runtime as part of the loading process, but I don’t have any of them installed that I know of.
As a workaround you could probably put something like this in the launch commabd:
rm /path/to/dxvk-cache; %command%
But I’m not sure if that might screw with the launch itself.
Yeah sometimes the processing shader cache thing gets held up. It does eventually complete. I’ve struck the issue myself where I had to wait 20mins to play a game because of this processing box popup, couldn’t even skip it. And the game didn’t even need the shader files…
Something valve is going to need to work on; don’t let it block people from playing etc…
PS. My shader cache size atm is almost 10GB! lol
Make sure to file a bug report, I’m sure this is exactly the kind of thing they want to know.
How many games do you have installed O.o usually the shader cache for a single game is maybe 20-50 MB O.o
I think it aso helps if your drive is an SSD (be it SATA or NVME), instead of spinning rust.
I deleted all my shadercache data on all my drives and reconsolidated them into the user .steam folder via symlink. See if that helps.
Something was definitely causing steam to crash, likely bad cache file or permissions.
I don’t have too many games installed but because these cache folders have existed forever (years), they’ve just blown up in size!
Until a couple DXVK versions ago the caches were a lot bigger. I don’t know what they did, but when just opening the game with a newer DXVK version the caches shrink massively.
Lutris’ cache for Guild Wars 2 is like 130 MB, but that’s from a way old DXVK version. After opening it in a more recent one that shrunk to roughly 20 MB, so that could also be part of it.