FP64 being neglected?

Hardware Hub GPU
, , ,
1

Hello everybody!

In the last few months I’ve been getting more and more interested to GPU acceleration for scientific workloads and Machine Learning, and I think I’ve noticed something I would like to ear your take on.
For what concerns AI and Machine Learning, depending on your workload and depending on if you are interested in training or inference, it looks like there’s been a shift towards lower precision types like FP16, FP8, FP4, INT8 and similar. This looks to be framed in the context of looking for a compromise between the increase of complexity and reliability of ML models, and the GPU industry seems like is pushing to serve this demand through better support for these formats on their higher tier silicon, mainly driven from the LLM AI bubble. This is ok, I think.
On the other hand, I’ve discovered that consumer GPU have artificially locked FP64 performance with ratios even worse than I thought (1/64 of FP32 FLOPs).
More “professional-grade” GPUs like RTX Pro Blackwell lineup, cost a lot more. This is an issue as it increases upfront cost for business purchases, makes FP64 graphical power inaccessible to individual scientists and engineers and inflates research institution costs for computing projects, expenses that are passed on to the consumer and citizen either through goods prices or through more taxes needing to be allocated for the same amount of GPU silicon area.
Moreover, it looks like even those pro-grade cards have heavily cut FP64 performance, with even more expensive GPUs finally offering some decent support. But even those very-high tier cards look like they have been neglecting FP64, somewhat, to better serve the lower precision needs from AI and datacenters.
Have I been getting all this wrong?
Do good options for FP64 scientific computing on GPU exist on the new market at lower prices or do institutions need to spend tax money on outrageous hardware prices? Do not-ancient FP64 accelerator options exist for hobbyists and small scientific teams with limited funding? Is it actually true that FP64 is kinda being neglected in favor of AI-tailored hardware? Of that’s the case, do you think that even if the AI LLM bubble bursts, the computing hardware flooding the used market won’t be as useful as one could hope in scientific computing?
I would like to get your opinion, as you probably know better than me. Especially if you have been professionally involved in scientific computing, Physics simulations and large scientific dataset analysis.
I will be thrilled if you have been actively involved in the deployment of compute power for astrophysics projects, because it’s the scientific field in which I’m currently more interested in. Thanks!

3 Likes
2

Haven’t done physics sims in a while but I do multi-terabyte datasets that are mostly a mix of bits, UInt8, UInt16, UInt32, FP32, and FP64. For the latter GPUs are pretty much irrelevant as analysis runs fine on the CPU. There’s a few things I could GPU accelerate but with like 9900X or 9950X the main bottleneck’s writing the code to effectively read data off NMVes and utilize CPU cores, after which the bottleneck is NGFF thermals.

If you solve those things with the workloads I’m running then it’s mostly like ~2.5 minutes per TB for analysis coming off a 4x4 drive, with some cores idling since it’s IO bound. Right now the slowest workloads go CPU bound on 9900X, dropping to ~5 GB/s data throughput. So if effective GPU offload is possible, something of which I’m somewhat skeptical, it’d save like one minute per run. Versus how many days of programming time to set it up plus the code maintenance costs? Machine time is cheap, programmers are expensive, so I’d expect probably GPU acceleration would just run up bigger losses over time rather than being able to amortize its setup cost. Different if you have BLAS kinds of hotspots, but we don’t.

Also the parametric models I run lack the parallelism to go effectively on GPU and FP64’s totally overkill. Accuracy’s like ±10% if you’re lucky and FP32 results match FP64 to like six significant digits. I’d look at FP16 via AVX-512VL but .NET 11 is the earliest C# might have support.

Where we do use dGPUs is torch, mainly panoptic, instance segmentation, and classification. Mostly natively UInt16 data by but UInt8’d probably be fine there. The rest of it’s FP32 by default but FP16 or UInt16 fixed point likely wouldn’t make any meaningful difference to accuracy (been too busy with other things to bench it out). There’s again no use case for FP64.

It’ll be irrelevant to what we do. To the extent there might be 12 V, rather than 48 V, GPUs released we can’t use them. Fire safety ban as 12V-2x6 isn’t suitable for unattended use at those power levels. ¯\_(ツ)_/¯

UDIMM and storage pricing reverting to summer 2025 levels would be helpful, though.

2 Likes
3

Then I can understand how FP64 alone is not that important for your use case.
In my case, I’ve seen GPUs providing a sizeable boost when doing signal analysis over long time series with parametric models that are complex to evaluate.
Also, if you are sampling probability distributions like in Bayesian analysis where you want to sample a multidimensional posterior probability distribution and estimate its integral over the parameter space to get the evidence, some algorithms have been developed to improve parallelization to the point that GPU acceleration makes sense.
In one specific case they claim to have seen time lowerd from ~20 hr on CPU to ~30 min on GPU (I think the test was done with an Nvidia A100, but don’t quote me on that): [2601.02336] The Sequential Monte Carlo goes NUTS: Boosting Gravitational-Wave Inference

Absolutely!

1 Like
4

FP64 alone is not that important in nearly everyone’s case.

Viewing performance from the perspective of how quickly data can be moved around rather than worrying about limitations in terms of raw compute for various datatypes will tend to paint a more realistic picture of how systems will perform in general but even more so with large scientific datasets.

Even with your special case of a compute heavy FP64 workload how close are you to hitting the theoretical compute limits of your current system? Odds are you’re still not constrained by raw compute but by bandwidth or latency limitations somewhere along the line.

1 Like
5

I was going to offer the Radeon VII until you mentioned “not ancient” and “new market”.

So… Arc Pro looks to be the best answer for you. They’re still markedly more expensive than consumer-level GPUs, especially with the B60 basically being a B580 with more VRAM, but at $800 for 3TFLOPs it’s vastly better on ratio than the consumer level stuff (beyond some argument for the regular B580 at a bit over half rate due to higher clocks despite doing it half rate).

2 Likes
6

We’ve got some stuff in that direction. The current state of the code people are using is it’s sufficiently inefficient FP64 Fortran people throw clusters of workstations at basic analyses. I got as far as validation testing an FP32 SIMD reimplementation of the compute kernel that should be a couple orders of magnitude faster before the project went on indefinite hiatus for other priorities.

It’ll come back around eventually but, without having a CPU based characterization of how 128-512 Markov chains in parallel behave (potentially 768 on Olympic Ridge), it’s difficult to assess whether widening to a dGPU would be worth it. My personal guess is it’ll probably turn out to be more effective to run like 64 or 128 at higher boost per core.

Certainly plausible if the compute fits a GPU but is is awkward on a CPU. I regularly see similar in torch with AMD desktop hardware.

7

I think most of your reasons are accurate. From a GPU pov fp64 is a niche case too. It’s not needed for graphics nor for AI and ML. Basically only science and engineering need it.

AMDs data Center offerings have better fp64 which is why the top500 supercomputers are dominated by amd instinct.

On the retail side Intel seems to have 1:4 fp64 to fp32 but it’s not clearly stated in the specs? Intel also offers a lot of vram per euro so that might be a good budget research option? Intel historically has also been very good on compilers and math libraries and that still seems to be the case lately with oneAPI, openvino, etc.

I’m not very optimistic about retired server stuff though. Today’s data Center stuff is highly integrated and very high power. It’s not a PCIe card you can simply plug in… 10y ago my uni had a cluster with gtx1080 ti‘s. Nowadays hpc gpus rarely come in PCIe any more and need specialised cooling. Power density is dramatically higher.

It’s ironic since historically the whole „gpgpu“ stuff was spearheaded by researchers building clusters with ps3‘s and programming openGL shaders for compute. Now the researchers who need fp64 are an afterthought.

3 Likes
8

To give some examples of scenarios where FP64 can be almost mandatory…
Well, fluid dynamics (Athena++…) and numerical relativity (Einstein Toolkit…) in certain situations, especially when sharp gradients, shock discontinuities or conserved quantities are involved, can be heavily reliant on having FP64 precision, and are quite heavy to run while being easily parallelized in specific computing approaches.
Hamiltonian Monte Carlo can rely big time on FP64 on very structured posteriors with sharp gradients while being parallelized, and Sequential Montecarlo, like other montecarlo methods, can rely much on FP64 precision to distinguish between “impossible” and “very unlikely” in log-probability space and when evaluating acceptance probabilities.

Metropolis-Hastings are poorly parallelized, I think. What kind of MCMC are you employing?

Maybe some profiling during code execution can give me a better idea. But I already encountered a scenario in Gaussian Processes where Cholesky decomposition numerical stability required adding excessive jitter on the diagonal of the matrix when using FP32, as small numerical rounding errors during decomposition can break numerical stability by making large covariance matrices ill-defined.

What kind of code are you using/building, if I may ask?

Yeah, that’s my worry too… that hardware has burned a lot of resources to be produced, and subtracted allocation to consumer stuff, and will be hard to redeployed in other context when retired from AI datacenters.

Yeah, I had that feeling. The Arc Pro options seem not too bad, given the market context. But I’m not ecstatic about it, either.

Just to be more clear, it’s nice to receive product recommendations, but I wasn’t looking for that. This thread was created to spark a discussion and understand more about the market and what do people more informed than me actually deploy and for which workloads.
Many thanks to people that helped with those suggestions, too. It may be useful to me in the near future and possibly to other user roaming around on this thread.

2 Likes
9

Can’t you just refine the mesh in these examples to accomplish the same result or similar convergence at FP32?
Also if you’re not constrained to FVM you can use higher order elements during discretization; this can boost accuracy of an FP32 simulation at the cost of more compute needed for solution; if your GPU’s FP32 compute is significantly faster than FP64, you may end up with a faster and more accurate solution on FP32 as long as your solver is setup appropriately.

1 Like
10

Myself, I haven’t been doing fluid dynamics simulations, but a friend of mine has been using FVM to study the dynamics within the structure of two celestial objects when they collide. I think he’s stuck with FVM.
In other contexts, I’ve heard that FVM (finite volume method, for anyone not in the field) handles better sharp discontinuities like shock fronts. Don’t quote me on that, though…

11

Yeah most pure CFD is FVM because it offers some pretty large benefits in conservation of mass flow; but there are some schemes for bringing higher order elements to FVM through sub-cell processing but I don’t actually know the benefits and drawbacks to them.

my 0.02 on the whole conservation benefit FVM offers is that if you’re relying on it for your solution to converge, you’ve already messed up at creating a suitable mesh.

EDIT:
I will admit that meshing is more of an art than a science and there aren’t very many programs that produce consistently good meshes.
One of the fundamental problems I see is that software that specializes in mesh creation will often be stand alone and not integrated into the rest of the simulation software and thus miss out on knowing what the boundary conditions at specific parts of the geometry are that should inform the fineness of the mesh in that area.

12

Sorry, I should have been more clear. I wasn’t suggesting that FP64 isn’t useful just that theoretical performance alone is rarely the limiting factor no matter what you’re doing. Even synthetic benchmarks often struggle to realize performance that approaches theoretical limits. For actually useful code lining up data fast enough to keep execution units fed is pretty universally what holds performance back rather than running into theoretical compute limitations.

13

Additionally, there are situations in astrophysics where a lot of physics at small scales influences what happens at very large scales, so you would need huge meshes with a very small spacing.

My bad for not understanding. I’m not a native English speaker, I’m Italian so… pizza!

Yeah, but like, the same is true for FP32, but with FP64 you have 1/64 the TFLOPS with respect to FP32, but you don’t need to move 64 times the amount of bits from memory, right?
If I’m not mistaken, then the TFLOPs cut may be more noticeable on most cards than the increase data throughput needs.

1 Like
15

Depends on use case. Primarily C# for CPU intensive things, as mentioned above, generally via PowerShell CLI. Python for torch, as usual, and some other applications. R, C++, and Fortran at various times. Probably not as much Julia or Rust as we ideally should be using from a technology perspective. Business need for near term productivity tends to trump longer term investments in bringing up additional languages, though.

Most of the work in the application space we’re in has used various flavors of differential evolution. There’s GPU accelerated variants but I wouldn’t say it’s clear DE is the most computationally efficient option at this point and the Fortran’s so inefficient population sizes have been kept quite small. So it’s more that it’s been found to work better than some of the other stuff people have tried.

FWIW this is basically what I do in our domains. It’s kind of standard people just throw FP64 at stuff but if you bother with small amounts of additional code and fairly minor bookkeeping many tasks collapse to an FP32 core with one O(N) use of FP64 at setup time and then again in results reporting. In some cases it’s actually been easier and more accurate to code the translation using Q64.

16

That’s true but just moving data to the GPU over PCI can negate any benefit. The B580 mentioned earlier for example supports PCI 4.0 x8 and FP64 at 1/32 of the FP32 rate. You need to be doing a fair number of floating point operations (around 30 according to Google Gemini) on every byte or compute won’t be the limiting factor because the PCI bus will be saturated. Storage and system memory can introduce similar limitations.

17

I will look into it for sure. I’m still not sure how you get the same numerical stability as FP64, but I’m now curious. Thanks for the fresh perspective!

Any suggestions on how to accurately monitor and keep track of that with minor performance degradation during debugging, testing and benchmarking?

18

Yep. Most of the compute kernels I write lack this density and, if they have it, often aren’t operations that are easy to parallelize.

If we could get GPU acceleration gains on large dataset analysis at least 4x8 wouldn’t, in principle, be a bottleneck since the data’s coming off a JBOD 4x4 or 5x4 NVMe. But lots of stuff I run exceeds ~15 GB/s read or write bandwidth to DDR. And often there’s no funding to rewrite the code to cast the precursors to a GPU so that becomes GDDR bandwidth, it’s involved enough nobody wants to do it, or somebody’s done GPU acceleration in a research paper but there’s no production version of the code.

Something else I hit fairly often is CPU implementations of a workflow have some components that could be GPU accelerated. But, to do that, you have take the workflow apart, do a bunch of fiddling to get the GPU packages set up, write translations between packages, and frequently reimplement internals of one package so it can talk to the GPU version of whatever it normally calls on the CPU.

Since machine time is cheap and programmer time is scarce it’s common our best business choice is to background stuff to 8-12 CPU cores while working on other things with the remaining four cores. So our GPUs spend a lot of time in fan stop.

Least invasive is just to measure CPU only versus GPU enabled runtimes and compare. Slowdowns with light compute kernels aren’t uncommon. It’s not just PCIe and DDR, it’s the overhead to push the data to DDR (if it’s not already there), tell the GPU driver to DMA it to GDDR, tell the GPU what to do once the data arrives, and then pull the results back for whatever the CPU’s deciding about next.

Even with the BLASy stuff torch runs for deep learning the threading and synchronization models used to talk to the GPU are one of the primary determinants of performance. From how smaller models run it looks to me like probably dispatch and retrieval remain the bottleneck even when GPU training and inference show substantial gains over their on CPU forms. But I haven’t had opportunity to go through the torch documentation and see if there’s performance counters or other profiling data which could confirm that (top level documentation just suggests benching different threading models).

If you’re writing the compute kernel typically you have an excellent sense of its compute versus IO density and the likelihood of being able to effectively parallelize across GPU cores. But if scientific computing here means research, though, a lot of research code doesn’t run often enough to break even on the labor of developing and maintaining a GPU version.

Also, in many fields graduate students mostly lack the requisite programming skills for decent CPU implementations, much less GPU ones, meaning consistent research assistant or associate funding and staffing is needed for sustainable code curation and positive ROI. Most commonly what seems to happen is one atypical grad student gets something working for their thesis and then nobody ever uses it again.

2 Likes
19

I´m not an expert in this field, but I read that emulation of FP64 is possible using Ozaki scheme, cuBLAS capabilities in CUDA Toolkit 13.0 Update 2 make this feasible. Maybe that’s achievable using an RTX5090.
Unlocking Tensor Core Performance with Floating Point Emulation in cuBLAS | NVIDIA Technical Blog
[2508.00441] DGEMM without FP64 Arithmetic - Using FP64 Emulation and FP8 Tensor Cores with Ozaki Scheme

3 Likes
20

Generally just watching stats in hwinfo can expose poor utilization.
Specifically use profiling tools if they’re available… if not then log time deltas for increasingly narrow sections of code.

21

Strange, at my university people propagate the notion that, depending on what you have to do, if you are interested in absolute computation performance the choice is either Fortran or C/C++.
Now Julia is starting to become more and more adopted due to the favorable trade-off between performance and easy syntax.

Got it. Actually, to answer another comment:

Yes, after people here pointed out that in FP64 calculations you may be more limited by moving data to and from the GPU, I asked Gemini specifically for emulation schemes to leverage FP32 acceleration and lower bandwidth requirements.
You were spot on about the memory limits. I looked into emulation strategies to leverage FP32 throughput, and found that “pseudo-FP64” is often the way to go on consumer cards.
Specifically, using Kahan Summation for integration, Iterative Refinement (or the Ozaki Scheme) for linear algebra, and Double-Single arithmetic for general math allows to trade cheap FP32 compute for better bandwidth efficiency.
I’m sharing a table that Gemini output:

Scenario Do you have “Pro” HW (A100/H100)? Do you have “Consumer” HW (RTX)? Optimal Choice
Grid Simulation (GRMHD) Yes Yes Mixed Precision (Store FP32, Compute FP64). Saves 50% VRAM.
Linear Solver (Cholesky) Yes Yes Iterative Refinement. Algorithmically superior speed for same accuracy.
Stochastic/Monte Carlo Yes - Native FP64. The math is complex, random access is heavy; optimizing is rarely worth the dev time vs just using the raw power.
Stochastic/Monte Carlo - Yes Double-Single (DS). Native FP64 is too slow (1/64th speed) on RTX cards to be viable.

Unless you have enterprise hardware with full 1:2 FP64 support, the better FP32 acceleration support on consumer GPUs is enough to compensate for the overhead to jiggle the conversion.
Sometimes these schemes can help in pushing calculation forward without needing to wait for the full FP64 block to be migrated to the GPU.

It’s true that you need to have control over the code to optimize different sections depending on the workload, and you will spend more developer time in doing so, so not sure if it’s the right strategy considering that

Now, regarding the following statement…

With the “publish or perish” in science, even among academics there’s significant inertia to change the status quo and adapt to new features.
And understandably so if considerable time and resources have already been spent in, for example, developing a code for supernova explosion simulation, which is one of the particularly compute-heavy workloads I can think of. Or stellar population synthesis and galactic chemical evolution codes…

Yes, that’s the point. Sometimes it’s hard to balance the need of coming out with new more accurate simulation results or new simulated models for research papers as soon as possible and being stuck having to wait compute completion, or needing not to spend time writing code that probably 10 people on earth will actually use regularly.

I wouldn’t confidently state “excellent sense”… if you are not a computer science major but you do something else, I have the feeling that you are totally put into water and told to learn to swim by yourself. Although the challenge can be fun, sometimes…

…Exactly!

More to think about, thanks. It’s stuff you can know but sometimes you need someone else pointing at it again every now and then

Thanks for the suggestions!

2 Likes