Looking for some advice and recommendations on used AI GPUs

Last year i convinced work to let me buy a couple of P100’s off ebay and build a basic inference box to try out some stuff running models locally, at the time it didn’t really go anywhere but i have been playing with agents for the last few months and kicked off another side project to create a POC where i’m fine tuning a small model and using open webui as the front end so our sales team can log in and ask questions about their sales.

I’ve been trying to talk them into letting me buy a DGX spark to make this go a lot faster but I’m in the chicken and egg situation where they are open to it once they see results but to get decent results i need better hardware that can fine tune a larger model.

I’m currently limited by my dual p100 box as it can only really fine tune a 3 or 4b model and a model this small is struggling to have that conversational consistency that you get with a larger model.

As i am in New Zealand getting my hands on used GPUs is not that easy and is expensive. The DGX Spark is going to cost about $10k and a used A100 80GB is like $12-15k

Anyone got any leads on where i might find better pricing or should i look at renting GPU time at a place like runpod for the fine tuning and proof of concept, anyone got any experience using these GPU rental services.

Any advice appreciated.

Rent a GPU online first. We toyed with the idea of buying and setting up the hardware but after calculating the initial cost, importantly also the cost of electricity and cooling, and crucially how quickly the GPU became “obsolete” by the time we’ve done our research and meetings, it just made more sense to rent a GPU especially during initial proof of concept. For some things, we will eventually get something inhouse but that’s likely going to wait until our other efforts are more matured and the pace of AI change mellows a bit more to justify the investments.

if you factor in the amount of time you rent the GPU for and how much that equates to real world cash though, it could work out cheaper if you just get a middle of the road GPU. For most situations, you probably won’t need a 5080 or 5090..you could probably even get away with a 4070ti (which is way cheaper)

Those GPUs are fine for inference, the Dual P100 rig i built is great for inference and can run 24B and 30B models depending on quant no problem. What they cannot do is fine tune those models. Creating the LoRA takes a lot more VRAM. I managed to get a config for fine tuning that runs at 15.8gb per card training a 4B model, right on the razor edge of what these P100 can manage but to train anything larger i need a lot more VRAM.

Doing a bit of research (and i may be wrong and am happy to be corrected) A single A100 80gb can train a 24B - 32B model efficently, but if you want to go further (and you could go bigger) you would need to start compromising. There is a reason these frontier models from Anthropic and Open AI take months and thousands of GPUs to train. Massive models and huge datasets need massive VRAM.

So yeah I’m looking at renting a A100 or H100 by the hour for training say a 24B model that should fit on the dual P100’s for inference.

Can you do the job with 64GB?

If so, a pair of V100s might be the way forward (if you can get hold of them), or - for a slightly different approach - a pair of R9700s would still be waaaaay cheaper than a Spark, even at retail price.

Of course, I can only speak about the R9700s in terms of inference (they’re around 3090 performance in Vulkan), I’ve never tried training.

Of course, this is assuming you need to train the models at all; it might be worth scaling back a bit and asking if you can achieve the same result with an MCP server or two?

The R7900 is an interesting option, didn’t think of that but i have no experience with using AMD for this. My fine tuning work flow is setup for nvidia. Will look into that. The only issue is those cards are unobtainium over here, as are the intel 32gb B70s. I have considered a couple of V100 as they are a drop in replacement for the P100 into the setup i built. I designed and 3d printed fan shrouds that mount to the front and back of the cards to mount 120mm noctua industrial fans and route the power cables to create a push-pull setup to ram air through the cards and it works really well, both cards pulling ~220w each during multi hour long tuning runs and cards peaks at about 80c.

The only thing I am not sure about is if 32gb is going to be enough either.

I have already built a bunch of tools that it can call to retrieve the data it needs to answer questions and manages who can access what and so on, that’s all handled seperatly and the point of the fine tuning is so the small model can understand the kind of questions that will be asked and allow it to use the right tool to get the information and format an appropriate output. These models are too small to figure this out on there own but the fine tuning has been working great for the most part. I am just hitting the limits of what a tiny 3b 4b model can do.

I have a pro subscription to chat gpt codex and when using that with openclaw connected to the same tools for testing and development the results are incredible but we do not want to use that for real with our live data. hence the experiments on locally running model. But its clear the system works. and what i want to find out is how big a model do i need to train and run to get ‘enough’ capability. The 4b model shows it can work sorta. Using a frontier model shows how insane it CAN be, no fine tuning no nothing, its just figures it out. what im trying to do is find where in the middle we need to land.

Sorry, kinda got off on a tangent there.

Maybe try bigger models? I mean, you can run Qwen 3.6 35B A3B in 32GB comfortably, even 24GB depending on context length, with great token rates. Maybe just give it a try without training in your P100 setup? Got to be worth a try. The 27B version would probably be even better out of the box, but I’m not sure whether the response times would be enough on P100s (even on an R9700, you’ll only get 30t/s decode).

I have to say…if I was in your position, I’d probably be focusing on the inference case since that’s the majority of the workload, and the fine-tuning only has to happen once per model (if at all, given the possibility of using Qwen 3.6 or similar). Just renting GPU instances can solve the one-off jobs if necessary.

So interestingly i just found some R9700 in stock at one retailer here and they are NZ$3800, to get 2 would basically be the same price as a 1tb DGX Spark NZ$7700 (4tb models are $9900)

64gb vs 128gb. would be able to run a pretty decent sized model on the spark.

Yes i have tried and it is better obviously but still not enough to get away without fine tuning. Hence the investigation.

True. But it’ll be much, much slower.

2 Likes

Really? a spark would be slower than R9700 ?

For inference, yes - when the model fits within VRAM. See the best-case scenario here:

…versus the R9700 with Vulkan here:

Spark FA on - 3661 / 56 (PP/TG)
R9700 FA on - 5620 / 139

The Spark isn’t designed for inference workloads, and the gap grows when you add a second R9700 - the TG performance stays roughly the same, but the PP performance grows by 30-40%.

1 Like

Interesting.

Tell me about it! :sob:

First of all, is fine-tuning really needed? Are you sure some extra prompt-engineering + maybe some RAG wouldn’t allow you to achieve what you want?

If fine-tuning is absolutely necessary, then how often are you going to be fine-tuning models? I don’t think you’ll be doing such on a regular basis, in which case a more expensive GPU might turn out to be a waste of money.

If your dual P100 setup is enough for your current inference needs, then it might be more cost-effective to get a cloud GPU at runpod or lambda cloud, use a handful of A6000s/PRO 6000s/L40s/A40s/A100s/H100s to fine-tune a larger model for way less than what you paid for a single P100s, and use the P100s to run the inference.

In case you want to upgrade anyway, even something like an used 3090 or 4090 would provide a noticeable perf uplift coming from those P100s.

As for the DGX Spark, it has lots of unified memory, but its compute power is pretty similar to a RTX 5070, and its memory bandwidth is way too slow at 273GB/s. That’s almost a third of what your current P100s deliver, and on par with a RTX 4060 (which has slower memory bandwidth that its predecessors, such as the 3060 and 2060).

You might be able to train and run larger models, but inference performance will be somewhat poor, especially if you plan to use dense models.