Considering getting another Threadripper or Intel 7900X X299 - Here's why

Hardware
,
1

Hi all - to give you some context I’ve got a basic 1950X setup on a Gigabyte Aorus Gaming 7 X399.

My initial plan was to virtualise with KVM/IOMMU; I may still go that route, but before I do that I want to try running XenServer on my box and see if it works.

Having done some research into XenServer, I like the ‘ease’ of use factor offered by it, given that I’ll be mounting all VMs to iSCSI shares from my primary FreeNAS/ZFS box. Being a level 1 hypervisor though, it means XenServer itself is the host.

Quick mention - I will be adding an Ubiquiti EdgeSwitch™ 16 XG (10G) switch here soon and tying my core systems over SFP for low-latency links. This will greatly improve iSCSI performance.

Having recently taken Fedora 26 for a whirl, I want to spend the next few months looking to migrate my development experience over to that distro (and potentially ween myself off Mac OS - more on that later).

In any case, by Q2/Q3 2018 I need to have a ZFS replication box up and running; I already have a LSI HBA card (that I need to flash to IT mode when I have some spare time) and I can breakout to 8-HDDs with a 2x SFF8087-to-4xSATA cables. P.S. I got this a while back when initially planning to build this stack in rack-mount chassis cases.

The change in direction that I may be looking to take now is as follows-

  • Run XenServer on the existing Threadripper box and test my PCIe passthrough mileage.
  • Add a second Threadripper system - OR -
  • X299 + Intel 7900X (as ideally I don’t want to spend more than $1000 on the CPU)

The reason I mentioned intel above is that it provides -

  • Option to install FreeNAS to the metal and virtualise VM instances (FreeBSD/FreeNAS has an Intel requirement)
  • ECC QVL is generally better supported on the Intel platform, on TR4 it isn’t always straightforward.
  • Thunderbolt support. I could potentially move this box to a dedicated server room and have a remote ‘terminal’ over Corning Thunderbolt optical cable - I’ll be pulling a Linus there - this helps keep heat and noise to a minimum in my work-area.

So what’s the purpose for the second box? primarily a Fedora 26 Workstation for dev/coding tasks. I also want to run FreeNAS with PCIe pass-through so that it can be the ZFS replication box as well.

That said, the bulk of my ‘tried and tested’ VMs will be moved into ‘operation’ on the Xen-stack.

Where I’m looking for advice & thoughts -

  • Deciding between Threadripper and 7900X for the second system; apart from being able to run FreeNAS (with virtio) support on the metal, would you throw caution to the wind and just get another TR4 box?
  • The impression I’m getting is IOMMU is more or less ‘ready to go’ out of the box on the Intel platforms. @wendell could you please comment on this? I recall you had a good experience with the Asus Strix X299-E board?
  • Going with the Asus X299 ROG’s Rampage VI Extreme - There’s value to be had in the 10G NIC
  • Should I just wait for WS boards to come out first before moving forwards?

Apologies for the longish brain-dump, and appreciate your thoughts as always.

2

ECC on x299? Did I miss something? :slight_smile:
I think I heard something on thunderbolt being supported via a PCIe card on some x399 boards? (Gigabyte?)
Out of the box iommu support is better on x299, yes.

2 Likes
3

Damn foppe would be correct. 7900X Spec

If you’re thinking of staying with threadripper (for ECC) then I can recommend using FiberChannel HBA to interconnect your data highway.
Qlogic FC HBAs can be found easily on ebay. There’s 16Gbit options that cost a bit, but considering the cost of a thunderbolt add-in adapter and the overhead of Intel pricing, maybe this balances things out? Also lowers reliance on thunderbolt for high-speed connectivity.
The cable for fibre is much cheaper too, lol. (The prices of optical thunderbolt, lol. ‘it is what it is’)

4

Wow, no ECC for the 7900X /mind blown. Guess they are really pushing everyone towards Xeon where the money’s at eh?

1 Like
5

That, or they want ppl to switch to AMD. Idk Intel’s mind. :slight_smile:

Anyway, wrt Thunderbolt – it may be that boards launched after jan 1 have built-in support, since that’s when the licensing changes happen? Otherwise you’re limited to PCIe add-in cards.

1 Like
6

Hi mate - Could I go this route with FC then,

I you take a scenario of 2 client PCs and 1 FreeNAS box, would the config be such that I’d need 3x QLogic 2700 cards and a single FC switch? (forgive my ignorance here, new to this. I’m looking at this from a ‘networking’ aspect, although I realise this is a FC SAN setup…)

Do please recommend any better FC switches, and I’m assume all of these are a ‘managed’ variant.

EDIT: It looks like I’d only need FC HBAs between the two FreeNAS boxes?

Workstation PCs ----> 10G Ethernet/SFP ----> Ubiquiti EdgeSwitch™ 16 XG (10G) switch —> Primary FreeNAS box over 10G Intel NIC

FreeNAS Primary FC HBA —> FC Switch ------> FreeNAS Replication server (FC HBA)?

^^ Is this how you’d configure it?

7

This depends on your FC needs - how many devices need this high io.
If there’s only two machines that need high bandwidth communication directly to each other then it’s as simple as A to B connection [FC-P2P] , skipping the ubiquiti.
The ubiquiti device can connect to each machine independently via ethernet.

If you wanted multiple computers to connect to the FreeNAS machine I myself would use a switch, but you could use loop-mode (aka token ring) if you felt that was a better option. I’m sure you’ve found that out by now (I am at the limit of my knowledge without googling constantly, lol).

8

As for googling, yup I share your sentiment.

I’d also need the FC link to the Threadripper box no? Rather than iSCSI over ethernet, over FC would provide the high IOPS. Sure, it ain’t vdevs on SSDs but I don’t want ethernet just bloating latencies further.

I’m just not entirely sure how the FreeNAS (FC HBA) <----> Threadripper XenServer (FC HBA) would ‘work’ hmm.

9

Setting up FC-LoopMode (‘token ring’) will allow communication of all devices without the need of a switch (incl threadripper box). https://en.wikipedia.org/wiki/Arbitrated_loop
I swallowed a gulp of fear reading this: http://www.mellanox.com/blog/2015/12/top-7-reasons-why-fibre-channel-is-doomed/

Anyway considering the current needs (and ignoring the scaremongering) I’ve seen that Qlogic works with FreeNAS on a few forums.


all the secrets: https://www.google.co.nz/search?q=freenas+qlogic+fc+arbitrated+loop

1 Like
10

Out of the box iommu support is better on x299, yes. (@foppe)

BTW, I was researching ECC options on X299, and this looks rather interesting-

Xeon W-2145 8/16
Base 3.7 GHz
Boost 4.5 GHz
TDP
140 W
Cost/1000 $1113
LGA2066
DDR 2666-ECC support
48 PCIe lanes.

Since I already have a Threadripper box, it just might make sense having an Intel alternative 8 Cores/16 threads is alright for Workstation use. As long as I can get IOMMU playing nicely, the ‘premium’ would be OK.

Update scratch that - CD8067303533601 is an OEM/tray microprocessor