In my ever evolving bit of tinkering I will be rebuilding my Unraid server.
I have gone a bit crazy (for my modest little homelab) and plan to cram a bunch of SSDs in it (at least 8x 2tb Micron 1100s). I’ve had 10g networking for a while and it is well deployed throughout the house.
That said, because I have an addiction, I got the Ubiquiti Beast on day one. The 25g ports call to me. The beg me to pair them with those SSDs and make file transfers go burrrrrr!
For a hobbist/tinker-er, is there any reason to try to get those ports in use. Or am I not going to get any practical benefit from them.
I’m open to totally manufacturing a reason to justify the build out.
I use many 25g connections for VMs and Ceph. 50 gigabit when bonded, it’s nice. I will say, 25g is the fattest connection you should be aspiring to, unless you are doing something exotic. Single client/server without RDMA tops out around 15/20 gigabit.
Id be curious to see if you can even get 10gb on those 25gb ports for file transfers from one user an a VLAN to the server on another VLAN. Some people testing show that you cannot if you ever have to cross a VLAN, which means if that is the case you would need a completely flat network to make use of the Beast in such a scenario. But im curious to see another person do some testing and see.
My main rig, NAS, and my inward facing VMs are on the same Vlan. IoT, Guest, Isolated (internet only) are the other VLANs I have set up.
Sounds like I need to build this for Science!
I’ve got parts coming but will start working to get all this together. Maybe I can do a few posts along the way. Depending on what happens I might need some suggestions on getting things optimized.
From my own looking into this a while back it is “best” to buy some slightly older enterprise gear like a 2 port SFP28/ QSFP+ network card set & DAC cable or transmitters & optical cable to directly connect a workstation and a server of some kind. That is, assuming you have spare PCIE slots open with enough lanes to saturate the NIC. Ebay usually has some 25, 40, 100 gigabit “kits” You can pick from, just be sure it’s what you need. And plan on adding a cooling fan to the cards.
With three 2 port cards you can connect 3 devices in a ring, or can probably bond two 2 port cards for direct device connection, again assuming a new enough card AND motherboard with a PCIE 4.0 or similar high bandwidth slot available. It will only work at the speed of the slowest part (PCIE 3.0 card or motherboard for example, and if the card needs x16 or x8 lanes your motherboard better provide it)
You will probably be happier doing that and treating it like a direct connect high speed Thunderbolt 4 cable that goes a long distance, plus it is simpler and will use less power for the same actual result since your 10gigabit network will remain untouched.
So…. I already have two of those cards and was planning to put one in my main rig and the other in the NAS. Plugging them both into the UDM Beast’s two ports. I also have the DAC cables to plug in.
Maybe I do a direct connection between the two and drop the second port on the dual Nic cards to the UDM for everything else to have access? I have a fair bit of stuff on 10g that would be able to hit that 25g connection.
Never done a direct connection via SFP ports and not through the larger network.
I am still curious what I can crank through that UDM though.
IMHO, crossing VLANs for file transfers does not make a whole lot of sense.
Why should I create a firewall rule so that PC1 on VLAN10 can talk to TrueNAS on VLAN20, when I simply can add a VLAN10 to TrueNAS, so the PC1 can directly talk to TrueNAS, without going over the firewall?
You still can say that the TrueNAS SSH or webGUI is only listening to VLAN20, so I don’t even see a security advantage.
If you’ve exposed the entire host with no firewall in teh middle, you’re exposing it to any and all kernel bugs, including for example possible problems in ICMP/IDP/TCP packet handling, etc.
In a home lab its probably less relevant, but in a real business network you want to use least privilege.
Sure. I can say that the webGUI should only listen on
10.10.20.10 on VLAN20, which is not reachable by PC1 on VLAN10.
Same for SSH.
Similar for NFS. There you can also work with host allow, so that only PC1 with the IP 10.10.10.11 can access the share /pool/data over 10.10.10.10 from TrueNAS.
If that concerns you, you could simply use ufw or nftables.
There is still no need to take a detour over the firewall and bother the firewall with that load IMHO. Not even for security concerns.
The whole point of a dedicated firewall is that it is not vulnerable to the same kernel exploits as the hosts (and their firewalls) you are protecting.
You can have all the host firewall rules you like, if the TCP/IP stack has a bug in it you’re still fucked.