[ADVICE] Build a new server or be funky with PCIe?

Hardware Hub Storage
, , ,
1

Hi! I have a TrueNAS… NAS, and it’s pretty beefy to me with a 16-core Epyc and 256GB of RAM.

All it does is store files and basically nothing else, but I do have a few apps on it like Plex, Tailscale (for entry), and Pi-Hole (currently not in use). I didn’t need to transcode often or at all with Plex, but I set it up for my parents and grandma, and they both have 1080p TVs.

I built another server with almost the exact same hardware, but I put a GPU in it. I’m thinking now, that might’ve not been the best choice.

EDIT: BETTER POST

The NAS as it is today

I want some advice:

  1. Should I use another server for Plex, Pi-Hole, and who knows what else.
  2. Should I keep my current server running Plex, but be funky with PCIe and add the GPU?

Here’s the situation (red lines PCIe lanes in use):

image
image1188×958 178 KB

  • 5 x PCIe slots have 24i SAS cards for internal connections → 8x PCIe lanes each.
  • 1 x PCIe slot has a 16e SAS card → 8x PCIe lanes.
  • 1 x PCIe slot has a 2x25Gb ConnectX-6 → 4x PCIe lanes.
  • 2 x M.2 ports using NVMe drives → 8x PCIe lanes.
  • 1 x SlimSAS port used for 8xSAS drives → 8x PCIe lanes.
  • 1 x SlimSAS port used for 8xNVMe drives → 8x PCIe lanes.

EVERY PCIe PORT IS FULL!

Possibilities

The board has 120 lanes available. Not sure where the remaining 8 lanes are going. Probably onboard Ethernet 'n USB.

But I have 44 available PCIe lanes just sitting around doing nothing.

The issue: Plex + GPU

My issue right now is the GPU situation. I want plex transcoding to be on the GPU rather than hogging up my limited CPU time.

Solutions

1. PCIe Bifurcation

This is the fun option, but the least practical and possibly the most-likely to create a slew of issues.

With clever use of PCIe bifurcation cards, I could better utilize my PCIe lanes. How to mount cards in the case is another problem though. In need someone with 3D printer tech. And the Storinator chassis has very limited space with all the wires.

Those SAS cards don’t have ports on the outside, but they do need to vent. If I could take a couple and share them across a single PCIe port, I could add a GPU without building a whole 'nother server.

Pros

It’s possible and lets me put a GPU in there.

I only have to maintain one server (YES!), and I only have to pay for the power of one server.

Cons

It sounds insane.

I don’t have a 3D printer, so finding a way to mount the cards without them touching the board is gonna be tough.

It creates more of a mess in the back of the Storinator chassis.

If I restart the NAS or put it down for maintenance, EVERYTHING goes down. But then again, everything needs the NAS anyway, so I don’t think that changes much.

Home Assistant is a different server for a reason, and I could always put Pi-Hole on a Raspberry Pi.

2. Build an App Server

I had some SuperMicro JBODs lying around, and I retrofitted one to put an almost-equivalent server inside. It’s in there right now running with TrueNAS and the Plex app container, but it’s not doing much.

Pros

This is the solution I chose already, and the server’s already built and running TrueNAS v24 with Plex installed just sitting there waiting for me to figure out how to transfer it over this weekend.

I like having two servers because I can restart my app server without messing up my NAS and visa versa; although, all my apps require the NAS anyway. If I ever wanted to do camera monitoring, that can sit on the app server.

The app server also lets me run VMs if I wanna finally get into Linux which I probably won’t mess around with; although, I’ve been curious about running a VSCode server.

Cons

I already have 2 servers in my server room making a ridiculous amount of noise. Adding a 3rd is more maintenance; something I’d like to avoid ideally.

I need to find a good OS for apps. I’ll be creating a separate thread for that discussion.

All my traffic has to go over Ethernet rather than going directly through the SAS connections. 10Gb should be enough for that, but there’s latency involved, and that’s gonna hit my NAS pretty hard to the point where my PC’s 25Gb connection won’t have the full bandwidth available when working on large multi-hundred-GB files.

Another con is I couldn’t, for the life of me, get it to write data from the App server to the NAS from one of the app containers. I don’t like spending all that time wasted struggling to figure out why something that should work isn’t working.

3. Build another server in the HDD chassis

I have two Storinator chassis. One holds 128 2.5" SSD slots and the other has 60 HDD bays.

The one with 60 HDD bays is connected via 3 x SAS Expanders to 3 x external SAS ports on the main NAS’s chassis.

image
image697×796 42 KB

If I remove that external SAS card, that actually frees up an entire PCIe slot for a GPU. I’d still need another server, but it can be the server for those HDDs rather than them having to be part of the main server.

If I wanted, I could make this an HA cluster, but TrueNAS doesn’t support it, and the hardware is slightly different.

Right now, I have a low-profile 2-slot NVIDIA RTX 4060 (waste of money since I’ve never used it), but I could instead buy a 1-slot full-sized Intel Arc GPU which has great support for transcoding.

Pros

I can keep everything on my main NAS. Talk about convenience!

I can put any future external SAS connections on the second NAS which frees up this NAS to be more versatile.

If my main NAS goes down, I technically have another NAS available now with the exact same data; although, Plex would go down in this case if the main NAS goes down :laughing:.

But… In the case of accessing that data and moving things around and recovery, it’s better to have two NASs with the same data than one that has 2 sets of the same data.

Cons

I’d have 2 servers to maintain and no TrueCommand subscription. “Annoying” and “tedious” are words that describe the experience of maintaining multiple servers that don’t auto-update.

4. Buy new TVs :troll:

Instead of spending any of this money on server gear and maintenance, I could just buy my family members new TVs :laughing:.

But really, that’s a today issue. It doesn’t help me in the long run and actually costs more.

5. Another suggestion

@mutation666 made a great suggestion.

I posted about it here:

2

I dont think you get SAS support off that port on that board

Do you really need so many sas cards? What drives are connected what the expected throughput? you can just get an expander or disk shelf to give you a ton more connections.

Also dear god the power used on those drives has to be crazy if you have that many.

3

I need all of those drives. I can’t take them out now because I’d have to rebuild the pool which is hundreds of terabytes, and I’m already halfway full.

My other Storinator chassis has 60 HDDs in it and is being used like a disk shelf as it is.

5. Reducing the number of cards in the main NAS

How to make it work

If I wanted to change this around, I could easily make that 128-drive SSD Storinator into a disk shelf.

I left 16 of the 128 bays open for hot spares that I’m not using. Right now, 4 need to be populated as 2 are the boot drives and 2 are the apps drives.

Only 112 drives are actually in my dRAID (of 7 vdevs). Surprisingly, if I made this a disk shelf, and didn’t put the boot drives in at 16-bay area, I could actually add an 8th vdev (something I could actually use!).

For those 112 drives, I need 5 x 24i SAS expanders, 6 if I wanna do all 128 bays.

In terms of external SAS cards, I have 1 already with 3 ports in use. Then I need 6 more ports which is 9 total → 2 more cards. And don’t forget 9 thick external SAS cables coming out the back.

The final setup

In doing all of this, I could use the server chassis I just built to house all of this hardware:

  • Both boot drives and the two apps drives would go in the front hotswap bays of the SuperMicro JBOD.
  • 1 x slot for the Mellanox 2x25Gb NIC.
  • 2 x slots for the NVIDIA RTX 4060 GPU.
  • 3 x slots for external SAS cards.

That leaves 1 slot for a Coral AI accelerator or anything else I might wanna add.

This is definitely a possibility!

Speed loss

I’d lose a bunch of speed from my SSD cluster though I’m not really stressing it that much anyway. As long as I can write at and read at a minimum of 25Gb, I’m fine for now.

U.2 NVMe drive issue

The last issue is what I do with my Intel Optane 905p drives. They’re U.2 NVMe. Not sure if this SuperMicro chassis optionally supports NVMe like some others I’ve bought. I’d probably need a newer chassis or backplane for that or I could somehow shove them in the motherboard part of case. Just don’t shake the NAS, and I’m good!

4

Could do an A4000 1 slot card to save a slot compered to the 4060 only slightly lower perf
would allow extra connectivity for the u.2 drives

Do you max out your current HBA connections bandwidth? Trying to figure out how many connections you would really need to max the SAS controllers.

1 Like
5

Last year I purchased 3 NetApp DS2246 disk shelves after seeing the LTT video on their purchase of a NetApp (SAS-HDD) rack. There’s 24x 2.5" drive bays for SAS/SATA per disk shelf (2U units, came with 2x IOM6 modules+dual PSU’s). Best thing is the ability to cascade the shelves courtesy of said IOM6 modules, meaning you’d only need 1 card connecting these chassis to the main server. Another card in your 2nd server (plus some extra cables) and the whole system suddenly becomes High Availability-ready.

The IOM6 module has 6Gb/s connectivity to the drives, there are IOM12 modules that cost 15x more (give or take: one EU seller has 8€ for an IOM6 and 112€ for an IOM12, both on clearance discounts) for just twice the bandwidth. netAPP uses standard SFF8087 cables connection the IOM to the network card. For your use case, using 6 DS2246 units (144x 2,5" disks) and one pair of 4xSFF8087 external cards you can divide the array in 2x3 disk shelves, each connected independently to the network cards and still have redundant access/high-availability for the entire array while upping the effective bandwidth to 1200Gb/s.

HTH!

1 Like
6

@mutation666 I can’t do an A4000 in the 2U JBOD case for my #5 suggestion.

If I wanted to do it in my current Storiator 4U chassis, that’d work provided I had another server for the 60 HDD zpool. That’s the #3 option I had: “Build another server in the HDD chassis”.

In that case, once I remove the 16e SAS card, I’ll have room for a 1-slot GPU, and the A4000 works for that purpose.

I thought the RTX 40 series had better video processing and native AV1 (which I don’t need today but might in the future). That’s one reason I chose that card over the 30 series, and the A4000 is based off the 30 series. Also, I’ve heard drivers in Linux suck for NVIDIA, and their enterprise gear requires some finagling.

Lastly, cost was a factor. I tried to keep costs low because I just need something to transcode. 90% of the time, I’m not using it; my parents are, but CPU decoding sucks, and not having a GPU limits other things I might wanna do like fully-local cameras (except, they do compression in the camera these days).

@Dutch_Master Thanks for the suggestion! Sounds like that’s a potential avenue I could’ve taken 3 years ago.

At this point, I’m not wanting to build something completely from scratch. I already have a bunch of hardware, I’m trying to utilize it as best I can without spending yet another bucketload of cash.

Also, I think you mean 1200Mb right? 1200Gb is 150GB/s. That’s not even in NVMe territory. It’s common for my SSD array to read and write data in the multiple gigabytes per second; so much so, that my desktop’s NVMe drives are the limiting factor.

7

Yeah, my bad. It’s 12Gb/s, just a simple typo :stuck_out_tongue:
Seagate says so anyway:

8

#5

I’m leaning toward #5. Uses less power overall and is less maintenance.

If I ever do video recording with cameras, I could always create another server and HDD array just for that purpose.

Here’s what #5 looks like:

image
image1814×1126 272 KB

I actually don’t know how those U.2 Intel Optane drives will fit unless the backplane has NVMe (it doesn’t). I’ll actually need to solve that issue too.

I don’t have enough external SAS cables, and I’d need 2 more 16e SAS cards. I’d need 3 more SAS expanders as well. That’s all $$$, and I’d have to burn the money I’d already spent on hardware for a second server and 5 x 24i SAS cards on this server.

I don’t mind doing this if it’s the right thing to do. Given my situation, that seems likely. This was in the cards, but I didn’t choose it because it wasn’t necessary. Well… Now it might be.

#3

I was thinking #3 might work because I don’t have to add another physical piece of hardware, and I get two NAS’s, but this won’t work for High Availability, so it’s basically paying for 2 servers but only using 1.

image
image1817×1178 243 KB

This is if I build another NAS into the existing HDD server case and remove the 16e SAS card, that leaves 1 slot open for a GPU, but I’d still be jam packed in here!

#2

This was my original plan because I wasn’t being creative enough apparently:

image
image1343×869 192 KB

It was adding a second server just for apps that could now fit a GPU. It costs 4 extra drives though, 2 for boot and 2 for apps.

Let’s just say I wasted a bunch of money here for not stepping back and thinking about this. I had a reason for the decision I made. I was planning on building a camera server as well, but it wouldn’t need a GPU since today’s IP cameras do that compression in the camera.

Helpful?

Does that help you guys see what I built and what I’m trying to accomplish?

9

Remember when I said this?

I must’ve traveled back in time and bought these for myself because there they were in my storage bins; exactly every part I need.

EVERY part:

  1. 6 x 24i SAS expanders with 8e inputs.
  2. 2 x 16e SAS controllers (I need 3, and it’s in the server I’m taking apart).
  3. A SlimSAS cable that has two MiniSAS connectors. Exactly what I need to connect boot and apps drives in this SuperMicro JBOD.
  4. Low-profile adapters for everything (except one of the 16e SAS controllers…).

The only thing missing was an NVMe-capable backplane. Not sure the model I need to find though. For now, I’m gonna shove them next to the motherboard and hope nothing gets shorted.

I must’ve already planned this very change-up early last year and forgot. Good documentation is key! Had I documented this for myself somewhere, I wouldn’t have wasted money trying to build another server when I already planned and paid for a solution :confused:.

1 Like
10

I ran into a snag that’s hard to fix. My SuperMicro JBOD has a SAS backplane. It’s not direct-attach; therefore, I need to give it SAS connections, or it won’t work with my SATA drives.

That means the SlimSAS → 2 x MiniSAS cable I used is worthless. Now I need to have yet another SAS card, so my PCIe slots are 100% full again :frowning:. Thankfully, that includes a Graphics Card this time.

11

Can order a different backplane they are pretty easy to source

12

Do you know where at? Any model numbers to look for? I’d need one with SATA and NVMe.

I also need to buy some 4-pin Molex splitters as while these PSUs are designed for lots of power draw, they didn’t give enough connectors.

13

What chassis? Ebay has a ton of backplanes on it.

14

I’m 99% sure it’s this chassis (SuperChassis 216BE2C-R609JBOD):

Side Note: For the life of me, I couldn’t get that SAS card to detect any drives through the SuperMicro SAS Expander Backplane even though that same card worked previously when I tested only 2 drives when it was the app server in this same case. I didn’t even change the motherboard or anything else!

But what did work was using their 4 x MiniSAS output card with a wire going into one of the 16e cards. It looks stupid, but it worked!

15

whoa, you are spinning 144 disks?
If you have less than 3 PETABYTES of storage, that is your problem.

22 TB CMR SAS / SATA drives are freely available
30 TB NVME U.2/U.3 drives are also freely available with 60 TB drives entering the market.

A storinator using the 16 empty slots with 22 TB drives can take 350 TB Raw
You would use 4 drives of redundancy and still have 260 TB usable

Migrate your data and begin decommissioning the other drives.
You now have an empty server to do computations and a storage appliance doing storage appliance things.

16

What has your CPU time all tied up? Why not just use software transcoding and forget about it, especially if it rarely actually needs to transcode?

Just throwing out a basic question… My little single-16tb drive jellyfin setup has very little in common with this.

17

It does not (mobo page)
Slimsas > 2x nvme or 8 sata drives per port

Which server/chassis models for everything and what model of ssd and hdd are you using? What sas cards and expanders, as well? Its pretty important in knowing those so that we know what you need to accommodate and what you have to work with.

Haven’t we all? lol

Agreed. There is no reason to get an rtx card or even a recent card if you can just do software transcoding. Upgrade that 16 core to a 32 core and throw 8 more threads at the problem, and you can do av1 with ease. No need to have the extra inactive power consumption of a gpu at all, and you won’t have to waste 1/2 slots and 16/32 lanes. If it isnt even doing av1 (it wont 99% of the time) the cpu should absolutely take care of it with ease. Itll be cheaper, too, if you were going to do an a4000. Theres an epyc 7542 32 core on ebay right now for 320, and the rest sit at just under 400. Far cry from the cost of an a4000.
If you want to do a gpu anyways to reduce need for cores, then go older. Unless you’re super into av1 and absolutely require it, h265 and h264 should be perfectly fine. You said yourself that people are going to be using 1080p older stuff with it, and with tvs theres no way they support av1 unless they’re super recent haha. A good one that can 2 transcodes and is single slot is a quadro p600. It takes very little power as well, and is 30 bucks on ebay right now. If you don’t need av1 its a perfect little card. BTW you can use transcode uncapping drivers to allow for unlimited transcodes. LMK if you decide to go this route, and I will send you the github links. You can also just search for em. If you don’t want to mess with that, the natively uncapped quadro p2000 is 120 on ebay and is still single slot. More power consumption, but can handle more.

Speaking of which, you’re handling like 200 disks and virtual machines on a 16 core cpu? It’s pretty cheap to upgrade to 32 core, like I said above. The epyc 7b12 (64 core) is 670 on ebay, and the 7b13 (64 core milan) is currently 1000. For your setup I would probably recommend that epyc 7542 though, it would be just right for this and you could have some overhead for software transcoding. :slight_smile:

1 Like
19

I’m using about 115TiB of data at the moment, but it increases over time. ~30-40TiB/yr, but that should be slowing down now.

It’s not all HDDs. Only 60 are HDDs in this system.

It’s not that simple.

First, I already own these drives, and the power cost is up to $240/yr. Negligible.

Second, these are HGST 10TB drives, and I only paid $2-4K for 100 of them. It’s not even comparable to the price of 20TB+ drives. I haven’t had one go bad yet either!

Plex is the only thing pegging my CPU and only when transcoding which almost never happens. It basically sits there doing nothing otherwise.

Still… When transcoding 4K HDR content off a disc with tone mapping, it simply can’t keep up. And that’s the situation that causes problems for 1080p SDR displays. Adding a GPU gives me headroom.

I already changed it but for documentation purposes:

  • 60 SAS HDDs in a Storinator XL60.

  • 128 SATA SSDs in a Storinator XL60 (only person in the world with this setup).

  • 6 x SATA SSDs for the boot pool, Apps, and HDD metadata (2 each).

  • 4 x NVMe Optane drives for metadata of the SSD pool.

  • It was 5 x 24i LSI 9305 SAS Controllers, 1 x 16e LSI 9305 SAS Controller, and 3 x 28i 8e SAS Expanders from Adaptec. But now it’s 9 x 28i 8e SAS Expanders from Adaptec and only 3 x LSI 9305 SAS Controllers.

I agree this could be a solution, but it’s unnecessarily expensive. The 4060 low-profile was $400 or so.

The 8-core version of this Epyc was already around $400, and I have a 16-core already. A 32-core would cost way more and also produce more heat, take more power, and run slower per core.

I don’t trust buying CPUs on eBay anymore. Been burned too many times in the past with consumer processors to do that.

Also, I’m on Milan with a 16-core 7313P. Rome would be a downgrade. I also have a two 8-core 7252Ps for whatever reason; I should only have 1.

I’m not running any VMs; in fact, I’m not sure what I’d use them for. It’s just Docker containers for a few apps.

The server is basically file storage with Plex. I might do more with it in the future, but I haven’t done anything crazy except Plex which had performance issues when transcoding; the one thing I currently wanted to fix.

2 Likes
20

I think I’d buy another machine to run plex on. Just link your current library to it. A gen 8 intel with igpu can do h264 hardware transcoding… I run an i7-9700 with no gpu and can transcode multiple streams.

An Epyc struggling with it seems kinda weird - you sure it’s not keeping up? It’ll take up your entire CPU to transcode as fast as it can, and then stop once the file is finished and just feed it to the client. I don’t know Plex, but in jellyfin it’s as simple as turning off AV1 and an older CPU can keep up fine with software transcoding.

21

20+ TB drives are $30 per TB, pretty much in the middle of what you paid.
But that’s still quite a bit to reinvest.

That’s 600 TB raw
probably 400 TB usable?

That’s a thing, though do you have enough in reserves for failures?
I am assuming so as you purchased 100 and only have 60 spinning.

makes sense

jeez

jeez

Don’t blame you
I just posted in the recently acquired thread a MoBo I received with the IO shield on top of the bare sockets

A stable NAS is seldom desired, but always underappreciated until it’s gone.

As stated:

2 Likes