I have a nice opportunity to spec and deploy a solution to run a fully local LLM in an academic environment for around 200 users. This will be a test bed for our STEAM teachers, students, and will eventually be open to more users as well. The goal is to have a completely local environment to test out the different capabilities of open source LLMs using ollama.
My question would be regarding the hardware. Would a couple of the Nvidia DGX work for this purpose or would building out a dedicated box be a better solution? The budget is around 8k but I may be able to get up to 10k depending on which way the wind blows that day.
I run a much smaller setup in my homelab with only a few users and an 2000 ADA SFF 16gb card but don’t really have any experience with a larger deployment.
Thanks for the suggestions and thoughts.
How big are the models desired for inference? Will this system also handle fine-tuning or just inference?
I think at this budget, a CPU-based solution will allow for larger models and even some fine-tuning. Dual socket AMD Epyc 9004 or 9005 series with total 24 channels of DDR5? Given enough RAM, it is possible to run the DeepSeek V3/R1 base models with quantization.
Perhaps if budget permits, a 16 or 24 GB consumer GPU can be added to improve performance with smaller models.
Model size would be likely kept between 7 and 14 billion to start. My worry is that the STEAM directors would want more as time goes on but scaling is absolutely an option going forward. My guess would be no more than say 40 concurrent users at a time but would like to allow for more if it’s possible. I’d rather not have to have faculty and students schedule time on the hardware, but rather just be able to log in and go.
I’m no expert in this field (quite the opposite), but 40 concurrent users on ollama strikes me as a budget problem… Just based on this thread
Maybe a different story if you don’t mind rather aggressive queuing?
Curious if I’m way off base here, have you already solved this problem?
Not married to using Ollama, It’s just what I have experience with. It’s fine with 5 concurrent users in my homelab but I don’t know how well that will scale. Totally open to learning something else if there’s a better solution.
Hey, I used to work in higher ed up to last years as a sys admin. I did some support for the AI supercomputer we had, but it was more on the end user support side.
You can go either way. My previous employer had systems built with RTX 2080ti and RTX 6000 to run the LLM. Main reason they went that route was cost effectiveness and the ability to do in place upgrades. Though they bought a few DGX before I left and I was hearing good stuff about it, however, the cost to performance wasn’t where they wanted it to be.
Personally I would reach out to your vendors and start getting quotes while also factoring stuff like labor/resources that’ll be needed for each. Then go with what will meet your budget. Our primary was Dell, but we ended up working out a deal with NVIDIA directly.
If this is for testing purposes could be worth sitting down with major stakeholders and using a cloud based solution for initial testing to see if they are interested. If you are in the US I’m aware of a public university that might rent out at cost to other institutions, but I’m not sure what that process would entail off the top of my head.
We don’t really have any hardware vendors as such, aside from apple and google (chromebooks). They didn’t have an IT department for around 5 years until they hired a director 2 years ago. I was brought on at the beginning of this last school year and am a network engineer who now gets to play while wearing a bunch of different hats. It’s really fun but also can be a bit strange.
I do know that the director and all the higher ups want a fully local solution (which I agree with and think is great). Reaching out to NVIDIA is a great idea, I’ll do some digging next week and see if I can make some meaningful contact over there. I think the DGX stuff is cool but I’m betting that building something out will get more bang for the buck so to speak.
How is Linux support for these Nvidia “developer mini pc” devices like the DGX? I believe they require a custom Ubuntu distro called “Nvidia Jetson Linux” which drops support for older devices pretty fast. I would be wary to recommend one for long term institutional use unless the Linux support policy is clear.
As I understand it, the original Jetson Nano is already e-waste, with the last supported release based on Ubuntu 18.04:
https://old.reddit.com/r/JetsonNano/comments/wz034x/nvidia_abandones_jetson_nano/
So I would reach out to your procurement department, probably under finance/CFO, to see what options are available. If you’re public you probably have quite a few options available via the State. Google may not be bad start as they do have some LLM/AI focused products, though in the cloud mostly, which may very well reduce a lot of overhead and I don’t see why using it as an initial test bed wouldn’t hurt since they will very likely be willing to give a demo.
For on prem I would consider reaching out to all the major names(Dell, Lenovo, HP ,etc) as you will be able to get service contract or at the very least reliable warranty service. Smaller vendors like 45 Drives can be very good as well, but you can run into different risks with them.
Were they using an MSP?
Well here’s some more of that “interesting” stuff. We are private so most of the state stuff is not available. As for the procurement department … well that’s me and the director as well. That five years they had a “consulting company” which didn’t fit their needs.
I’ll reach out to some vendors and see what we can set up and what they can offer. It’s really is a fun place to work, just a bit unorthodox shall we say.
Not sure if you got this going, but this is better treated with vLLM. I am building a software stack to commercialize for 2026 on just this use case. I found that I can do about 15 concurrent, moderate use instances with a nemo_mistral 12B on a 48GB rtx 6000, so you will need some good hardware to do this, or really reduce token count. The application and use of the LLM will vary your hardware requirements (min is for audit, legal and finance so quite context heavy).
Nowhere near actually getting things going yet but thanks for the info.