Painful VMWare lab with 10Gb DACs and no switch

After playing with VSAN for a bit, I decided to try something different. VSAN can be a pain to set up with no switch and three 10Gb dual port SFP adapters… I have one host that way under-performs the rest. It’s an old thing, but still functions well, and happens to be in a Fractal Node 801 case with 4x4TB drives, 4x6TB drives and 3x250GB SSDs.

I started this adventure out downloading a copy of TrueNAS to play with, and ejecting the system from VCenter. There is a learning curve since I had never used TrueNAS and only played with FreeNAS a little. I’m a Synology guy. How hard can it be? (Don’t ever ask yourself that question. The gremlins are waiting to hear it!)

I got everything to come up, it worked flawless. That is until I tried connecting the two ESXI hosts to it to access an NFS share. I tried everything. I got one to see it, but not the other. Then I tried iSCSI. Same deal, opposite host no less. Checked my network settings many times, and it ever worked for both. Gave it overnight to perc. Still was unable to see the iSCSI stuff on one host.
Since I’m used to Synology, why not try (gasp) Xpenology… Only I couldn’t get it working at all on my server either as a bare-metal or virtual.
Now I’ve spent most of a second day with this thing and I finally loaded ESXi 6.7 back up on it clean, all drives wiped completely. Then I thought, let’s try one more thing. Let’s load up TrueNAS as a virtual, create the VSwitches, VMKernal port groups, etc. Add, drives, blah blah…
For whatever reason, this worked, once I created iSCSI shares. They all showed up on the hosts just fine even though the settings were identical on the bare-metal version. Tried NFS, and it still failed in the same spectacular way. I think it’s because of no real 10Gb switching gear. Might need to pick some up. That an a new RAID controller. Adaptec 5805 ain’t so fast. But it all works for now, and frankly it is a LAB, so not too concerned if it blows up.

What did I learn from this? Well, I can now say I have configured TrueNAS a couple of times, will only use XPenology on old HP Microservers where they don’t cause me headaches and direct links with 10Gb DACs can be painful to use without a switch.

Speaking of switches, has anyone tried the MikroTik CRS305-1G-4S+IN (4x 10Gb SFP + 1Gb) switch? It looks like it would work for my home lab pretty well, just curious if anyone here is playing with it and what your experience has been with it.

I have been using the 8 port version for a few years. No complaints. I’m not doing anything fancy with it, just running OM3 patch cables and cheap transceivers from eBay. Can confirm I get the full 10gig when copying between SSDs across the network.

Thanks Gordonthree. Now I’ll have to take a look at the 8 port! Update on the rest. I went back bare metal yesterday on the TrueNAS box and got it working perfectly using iSCSI. Not my ideal, but after a few benchmarks, I’m way happier than a virtual on the same box. My disk benchmarks went up by 400MBs on longer files to 1066MBs from 649Mbs on 4x hard drives no less. Once I saw that, I added more SSDs in the mix, and it’s pure delight watching my VMs that are using SSDs! They’re faster than the single SSDs I was giving them before!

Having a lot of fun with old hardware and a little bubblegum and duct tape! =)

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.