So I was doing a little bit of research on budget AI GPUs as I wanted to build a second rig with some leftover parts for bedroom gaming and didn’t want to necessarily use the 4090 for a local LLM so I googled and came across some hits like Teslas, P40s, P100s, etc, and I’m also aware of the 5060 Ti 16GB but I didn’t really want to spend too much money as the 5060 Ti is no longer ‘cheap’ (not sure it ever was, and especially if you want multiple for more VRAM). Let me describe my use case first.
My company has introduced a comically low usage limit for our AI subscriptions so rather than rely on a personal subscription with usage limits what I’d like to do is be able to ingest a bunch of Oracle Cloud Application books as the training material (for SaaS Financials, EPM, etc) and then query against it, but, rather than using something like an indexer+ElasticSearch to search for known phrases, I want to be able to pose business scenarios against it, and prompt it with something like “I have an invoice that isn’t hitting the correct asset book, here is the current configuration of two books, please reference the 26C Fixed Assets book” and then either I attach the PDF or it’s already trained specifically from the data.
Since I would like to utilize it for work it would be better if the model responses were accurate more often than not. I am not sure if this is necessarily correlated to GPU horsepower or the training configuration and what a good model to use would be, as I’m still very new to all of this.
Could I get by with some ebay pickups or would a single 4090 be really my best case for something as described? If budget options are good enough for this, I’d rather stick with nVidia for compatibility reasons.
I’m not sure what I can add to the conversation, but I am interested to see where this goes.