First things first I already have a NAS that runs Unraid and hosts my plex server and multiple users for document storage.
I’m looking to use an Epyc Cpu to build a system that is mainly for VM’s that are rendering machines, but I would like it to take over as the main NAS, and the existing one I have to become a backup that only needs to be turned on every few weeks as its pretty rubbish. I would like advise on how to build a cluster that can also be access over the internet if i need to work offsite.
Hi, can you describe what you mean by the word “Cluster” in this context?
Typically it seems a cluster of computers are several real or virtual machines, networked together to share resources, or to split a task up into smaller chunks for parallel operations, or for fail over/ high availability.
Do you mean a cluster of VM’s to render in parallel? Or are you looking to have several epycs with a vm on each?
Would this be better than one larger VM doing the same work?
Do all the machines need to be accessible to the internet, or might a single host/management system be more useful remotely?
Because it looks less of a cluster, and more of just a remote machine that you want to feed work to from several remote places?
I work in construction so I work on CAD software mostly but I render models and designs too but it means my pc can’t be used when rendering.
I want the VM’s to use GPU passthrough but also be able to remote in from offsite. I’m looking at a VM to render, a VM for doing modeling and drawing on, and then the rest of the CPU to handle resource management on the NAS. I meant a cluster of NAS devices that can act as storage like when you have multiple racks of hard drives connected to a single CPU and an access point so I can access the storage from offsite. Such as sharing models that are stored on the network and are too big to send otherwise. And by having more than one NAS device the secondary can be a backup that isn’t always on.
Having a second NAS that acts as a backup is not a cluster. It is just a backup.
Even SAN systems with redundant components don’t call themselves clusters. At least I don’t think so.
At the point you’re spending a half million dollars on a SAN, redundant everything is simply expected. That is where technology like NVMe Fabrics come into play and multiple controllers can access the same drives. If one controller fails it gets booted off and the backup takes over flawlessly. In theory.
So it is like a cluster but in storage services I don’t think they call it that.
Can you think of a better way of describing OP’s request for help?
Looks like he want suggestions for half a rack of equipment, and I’m not so sure the technical term itself is such an issue.
Maybe “complete home lab” or something, in which case cluster certainly sounds more sophisticated
But I am way off topic.
OP wants:
•A render machine
•A main storage box
•A backup storage box
•Network access for remotely managing jobs
And is willing to have dedicated machines for each…
Cluster is just a term for an artificial layer between physical nodes and purpose-driven nodes. Exists for storage and for storage servers both. I guess the OP means cluster in a sense of a bunch of HW.
Based on my current understanding, just watch the latest video from Level 1 on Synology.
There are different ways of doing the same, but your post does not give me the confidence you have a clear idea yet. No offense meant, just that anything we say will sound patronizing as a result.
That said:
NAS handles the processing of data itself, so there is no need to have your local CPU manage the storage resources. There are nice solution that will provide main NAS and backup NAS. So that would be a cluster of 2 NAS servers…
Having compute nodes separated is a great idea. You could just shutdown a node on your main workstation and boot it up on your render machine to render. (The simplest way to split work between nodes enabled by iSCSI) This would a be compute node cluster and we would need way more information to recommend a solution from libvirt on a gaming PC all the way to vSphere running in IBM Blade server
Note down your workflow and we can help pinpointing places NAS would help you out a lot.
That would a cluster of compute nodes. These can still have local drives.
That would a cluster of storage nodes - you would still have both running most of the time as synchronization takes time between them and that time might be longer than your workflow.
There is also such a thing as a storage cluster where data is the thing being moved around, not the data service. similar, but not the same.
Sure, you can have central storage and node storage and even on the same NAS server. Note that this is not related to the previous 2 points. You would not want to run nodes off-site due to the slow connection between storage and compute. Full Duplex 100MBit is really the bare minimum for any such solution.
All of that said, it is 100% recommended to separate workflow into nodes even if you are still running it all from one computer. Buying a second one is almost always cheaper than making the first one faster.
