Looks like there is more than one bug:
passthrough may be referring to this: Proxmox GPU Passthrough crashes host RTX PRO 6000 and RTX 5090 - Linux - NVIDIA Developer Forums
Others:
MIG support on RTX Pro 6000 Blackwell - Linux - NVIDIA Developer Forums (situational, less bad)
Xid 119 GSP Timeout on RTX 6000 Pro Blackwell (575.64.3) under Load – Reproducible Crash - Linux - NVIDIA Developer Forums (this one seems bad if ollama is going to crash under heavy load)