I had a nightmare that my storage box went down and all my backups, more importantly my media server, went down and I couldn’t get a board/cpu. The GPU craze is getting to me.
Anyhow, I’d been planning to build redundancy into my long term storage and upgrading to 10gb. Any advice on what kind of switches I should use or card setup in my personal pc would be much appreciated. I’m worried that I’ll much up the OOB stuff and be putzing around with a vga cable and keyboard to fix things all the time. I want to set it and forget it for the next 5 ish years. Somewhat newish switches are ok. I really only need a few ports at the 10gb + level. Things like my pi’s don’t need much. I am upgrading my ML box and do want to bring data over quick sometimes. Looking for advice, gotchas, direction.
VLAN capable switches have upgradeable software, and they can die too.
You’re connecting the NAS boxes directly to be able to sync/backup to each other without using any switch ports, that’s good.
Then, from each, you have another two 10Gbps uplinks (i don’t know much about these, do they have more than 2xSFP+ ports?). intending to go to two separate physical switches, for redundancy purposes, and your computer would then go to both, for redundancy purposes.
And then, you’d need to connect this pair of switches to your regular network — so you’d need 2x4port 10Gig + some other stuff.
You could get a pair of CRS305, that takes care of your 10Gbps redundancy (they’re 4xSFP+ ports + 1gig RJ45 for <150 each) and hook them up to any plain old gigabit switch.
For this wifi breakout-non redundant part of the network, you could use e.g.:
CSS610 which has 2x10G SFP+ ports and 8x1gig RJ45 for $100
or you could do a CSS326 with 2xSFP+
and 24x1gig ports instead of 8
or you could do a unifi switch pro 24 poe, which will happily take two 10Gig SFP+ inputs and provide a bunch of RJ45 for your low speed devices and do POE so you can power your access points once you decide to ditch orbi
You could also use a couple of pi-kvm or a pair of tiny pilot thingies to help you debug your nas boxes when you’re away.
the TrueNAS boxes I want are 10Gbase-T (RJ-45), not SFP+. The second switch is actually OOB management. It doesn’t need to be 10gb. The CR305 could work for the OOB management switch.
I’ve never hooked boxes up directly before. The TrueNAS mini X does have dual 10gbase ports. Will they just find each other? Since they aren’t being propagated to DHCP, would I need to manually assign an IP address to those ports in a matching subnet so they can see each other?
I tried a similar layout with some more basic hardware and the network stalled. I think I had a broadcast storm when I hooked everything to the ORBI. I may need a 1gb aggregation or something in between the lab switches and the orbi.
I was looking at upgrading my centos 6 fibre channel san OS to a newer version but thanks to en epic post by nx2l trying to help some poor dude and due to some “interesting” decisions made by the vendor with their path moving forward… It may be better to ditch the fibre channel and go 10gbe.
I digress, with your setup you’ll need the truenas mini X+ not the X for 10gbe. As coincidence would have it I have a a brocade icx-7250 holding down the floor. I see on ebay these are going for about $450AUD (aussie) and you can pick up some single (70AUD) or dual port(110AUD) 10gbe intel x520-DA? (? being 1 or 2 port respectively) and you’ll need some DAC twinax cables ($15AUD from fs dot com) and some 10gbeSPF+ (30m $90AUD) for your truenas (if that doesn’t support DAC.
Sos you could acheive what you wanted for about ~$1000.00AUD not including the truenas obvious.
Note: make sure you get the licensed version (8 port) of the brocade otherwise only 2 of the 10gbe ports will work. And vrrp should work fine for you. Seems overkill but that’s half the fine right
I’ve just costed it out for me so thanks for posting the question
You are right about the Mini X+. From what I understand, the SFP add-in card is optional. It doesn’t come packaged unless you get the 64gb octa core model, which is overkill for me. I am moving all my VM’s, including Plex, off of TrueNAS. That saves me ~$500US/Unit.
I think the add-in is just a x8 standard PCIe slot (maybe even full height…not sure) so I wonder if getting my own SFP+ card and installing it into the 32gb Mini X might not be the ‘cheap’ option.
The ICX-750 seems a little bigger than I need for the home lab. I’m not sure about the ‘upgradeable’ to 10gb. It looks like either way, moving to SFP is the way to go for 10gb and higher speeds, or I’m out ~800/switch.
Just be mindful that installing “external hardware” in these proprietary units may cause warranty issues. i.e. take it out before calling support
the icx are cheap and come in 24/48 ports. Seems a decent path for stability and their OS is similar to cisco IOS IIRC.
If you don’t need 10gbe (really at home who does?) you can just port-channel on a managed switch and most support lacp fast mode and trunking. That’s gonna cut costs a lot.
If you happen to have RJ45 10 gig, and are forced to use a twisted pair cable, you can get an RJ45 transceiver instead of a fiber transceiver. They’re more expensive than laser or dac (about $30-$50 vs. $20 for fiber vs. $15 for a 3m DAC), but maybe still acceptable depending on how many you need.
I looked at pricing for truenas boxes, … the hardware is nice, but looks overkill, as if you’re paying $300 markup for nicely picked components - you could build a more kick-ass regular am4 system (and even get working ecc if you pick the motherboard/cpu right), while saving money.
I have a custom storage box now. Did the whole DIY, bought really expensive noctua fans so it’d be quiet, got the HBA. It’s too big, too loud, and the idle power draw is too high.
If you run across a mini-itx case with 7 hot swap bays, let me know. That’s the thing that’s pushing me toward the Mini X. That and the support aaand the environment tab in the GUI (You can see which drive it’s talking about in the GUI. That integrations is kinda cool).
I could avoid 10g for now. I’m just worried about the transfer from my old system to the new system. It’ll take…a long time.
and due to some “interesting” decisions made by the vendor with their path moving forward… It may be better to ditch the fibre channel and go 10gbe.
aaand the environment tab in the GUI (You can see which drive it’s talking about in the GUI. That integrations is kinda cool).