Freenas road to Threadripper

My journey started about 3 years ago in 2016. I had a WD PR4100 running Plex and it was super slow to transfer files to and had issues streaming more than 2 movies at a time. I had heard about FreeNAS so I took a really old AMD phantom II and am3+ motherboard and some old hard drives I had laying around and started playing with it. 11.1 was still in beta so I started off with 9 which worked quite well. “Once I had things figured out.” But the read and write speeds were a little slow for my liking but faster than the My Cloud. I had also heard about Unraid so went to try to install that and no matter what I did I couldn’t get it up and running. I also knew I would have to pay for it after I reached a certain amount of drives. So I decided to stick with FreeNAS.

In 2017 right before I finished my testing 11.1 stable was released so I did some testing with that and decided to move forward with that build. Next came the decision to either stick with really old hardware or to get a rack server because I already had acquired a 42U rack (wife almost killed me when I came home with it). I came across a HP ProLiant DL360 G7 8-bay server with 2 6 core Xeon’s and 64gigs of ram for a really good price that I just couldn’t pass up. I pulled the trigger on that and decided that for the speeds I wanted to transfer things at I should put 2tb SSD’s in this unit. (Yes I know overkill). I also picked up an Intel x540t2 10gig NIC. I started off with 4 drives in a raidz setup. And I held onto the WD PR4100 for a bit and used it for a back up of the FreeNAS unit. Quickly I realized 4 Micron 1100 ssd’s wouldn’t last the year. I picked up 4 more thought about just adding them to the pool to basically make a raidZ2 but realized I needed the space more so I rebuilt the Pool. Thing made me realize how important it was to plan ahead.

In the fall of 2018 after running into issues with the WD My Cloud not reporting a bad drive I decided it was time to build a secondary FreeNAS box that would house only Spinning rust drives that I could do nightly backup’s to. I picked up a Rosewill 4U case that came with 12 hot swap bays. Should have looked around more and gone with a Norco RPC-4220 case it’s about the same price. I had recently upgraded my dad’s tower so I had an AMD fx8350 and ASUS TUF SABERTOOTH 990FX laying around so in went that and now 8 4tb wd red drives. I also picked up a LSI SAS9201-8i card. I ran these 2 servers until the spring of 2019.

That spring after looking at the usage data I realized I should just combine these into 1 server. Needed the cores of the xeon’s but I also needed at least 8 2.5” bays for the ssd’s in a hot swap. So I found a 12x2.5 hot swap 3x5.25 cage to put in the Rosewill case. I went with a Supermicro X8DTH-6F dual LGA1366, (2) Xeon x5690, 192 gigs of ddr3 ecc ram, LSI9201-16i, and 4 more ssd’s.

This ran great struggled a little when I was editing video and Plex was transcoding 1080p to SD/720 when I had 4 people hitting the server at the same time from outside the house but never really slowed down noticeably. Found a bad cable in one of the SAS to SATA breakout cables went to replace that cable and replace the battery on the board because anytime the power was disconnected it would try to boot from a hard drive and not the flash drives. And the board wouldn’t start up. I spent 3 hours trying to figure it out. All I had was a blinking BMC light. And it wouldn’t start up at all. (We’ll get back to my stupidity in a bit)

I decided that maybe it was time to make one finally upgrade because I was adding a LSI9201-16e to the server so I could just add disk shelf’s when I wanted to add more storage. (I’m a bad data hoarder). So being tired and frustrated with the server (up too early for kid’s football games). I drove the hour and ten to microcenter and picked up a 2920X, Gigabyte x399 Aorus pro atx and 128 gigs of GSKILL AEGIS DDR4 3000 ram. Unfortunately they didn’t have an air cooler so I did up having to order one from Amazon will be in Tuesday 10/22. I got home and started taking everything apart moving stand off’s to where they needed to be for the new motherboard and taking picks of all the parts I know was working perfectly to sell to try and recoup some money from the new parts I had bought. I looked at the motherboard carefully thinking maybe it was a capacitor or something that finally went. And I was going to try and just replace that to get it working again to sell. I then noticed the pins for the power switch, restart, and stuff had a label next to them. I remember looking the manual super carefully and the manual had it numbered 1 to 20 bottom to top. I thought that was weird but most motherboards are. Well the label on the board its self showed it numbered 1 to 20 top to bottom. So at 10pm Sunday night not being able to just let things go I plugged the board into a known good power supply put the CPU’s back in and put the coolers back on with a stick of ram in each. I then waited for the BMC light to turn on and low and behold after a second or two I was able to short the pins that were labeled power switch and it turned on and went to the bios no problem. After some back and forth I decided to continue forward with the Threadripper figuring that using a processor on a 14nm with a lot more cache with higher clock speeds is worth the upgrade.

Things learned:

  1. Always look at the motherboard carefully when building it may be different than the manual
  2. Never jump the gun to replace something just because you’re tired and frustrated. It could be something stupid
  3. Never bring a 5 year old to Microcenter you’ll want to rip your hair out. Lol.

If all goes well I’ll be able to update this on Wednesday letting you know how the new Threadripper is doing with FreeNAS, Plex, and the VM’s I have in FreeNAS.

3 Likes

Is it possible to change the tags from “hardware” to “blog”?

Cause there’s no question here.

And it was a good read btw. Please keep us updated!

2 Likes

Looks like someone did.

Fingers crossed the Noctua NH-U12S TR4 and ZOTAC GeForce GT 710 1GB DDR3 PCIE x 1 actually come in today. Tracking says they are out for delivery. Just happy they still make pcie X1 video cards i didn’t realize that one of the X16 slots was only a X1 slot until i started planning out the layout. just glad i found one on amazon prime.

1 Like

Day 1 of FreeNAS up and running on Threadripper 2920X.

Parts:

  • Rosewill 4U case
  • 1000 watt older modular EVGA power supply
  • Gigabyte x399 Aorus pro
  • Threadripper 2920X
  • Noctua NH-U12S TR4-SP3
  • GSKILL AEGIS DDR4 3000 MHz ram
  • Slot 1 - LSI 9201-16i
  • Slot 2 - LSI 9201-8i
  • Slot 3 - Zotac GeForce GT710 pcie x1
  • Slot 4 - Intel X540T2 10Gig NIC
  • Slot 5 - LSI 9201-16e
  • FreeNAS on 2 SanDisk Ultra 16GB in UEFI
  • (12) Micron 1100 2tb ssd RAIDZ
  • (8) WD Red 4tb RAIDZ
  • NetApp DS2246 24bau Disk Shelf
  • (1) old 128 ssd (jails drive) will be moving it a 3 ssd raidz shortly

Setup:

Putting everything on the motherboard wasn’t hard. The board came with a nice connector deal to put all the power and reset button stuff into. I had downloaded and put the updated bios on a flash drive so that was number one thing that needed to be done. Then i set the video card to slot 3 so it would always go there first for video. Also turned on virtualization (that took some searching in the bios) and turned on the ram over clock so i would get 3000 MHz on my ram. Then once those settings were saved and i set the usb’s to be the first to boot i let the machine boot up. It did a initial boot to set all the settings from my previous build. Then rebooted and started up no problem. The previous build was bios boot and i had issues with reboot when i added a drive so i put in an old hard drive into the disk shelf let FreeNAS recognize it and show up in the web GUI. Then I restarted the machine. When i did that it tried to boot from that hard drive so i shut down the machine and used a virtual machine to “update” the flashdrives to be a UEFI boot drives and put them back in the server. I checked the BIOS to make sure that those were the first things to boot. And because they were a UEFI drive they are automatically the first things on the list. Again it did the initial boot deal and then startup no problems. I’m happy to say it is running great. 24 hours no issues with running anything.

Quirks:

  • No temperature read out for the cpu
  • The system doesn’t actually shutdown i have to give the command wait a minute or 2 then hold down the power button to get it to shut off.
  • The Noctua NH-U12S is a hair tall for the case. I was able to get it closed. But wished i had gone with the shorter cooler.

As of right now I’m definitely happy i made this move the system actually saturated my 10gig network without a problem even when i have multiple computers on 10gig cards hitting it. Plex actually seams more responsive in house and my friends say it seams a little faster on their end too. My Ubuntu VM is faster and more responsive also. The disk shelf is just as fast as if i was placing the drives directly into the machine. I will be adding a 24 bay 3.5 inch disk self down the line to add even more storage eventually.

Weird, didn’t @wendell say FreeNAS doesn’t boot on threadripper? :thinking: Did I miss something?

1 Like

Not sure but if @wendell did or not but it boots and runs great.

1 Like

Please plan to get some little SSDs to mirror for the boot drive, FreeNAS on USB sticks stopped being a good idea when they started using a mutable ZFS pool for the operating system instead of the old ram disk setup they used before. The new wisdom hasn’t displaced all the old wisdom floating around, yet.

1 Like

@freqlabs thank you for letting me know about that. I will probably get some small m.2 drives to add to the board to set up as boot drives then.

Will i have to reinstall my plug-ins when i set those up?

1 Like

The plugins are installed on your jail pool, which is separate from the freenas-boot pool. Save a backup of your config and you should be able to restore all settings in a new install. It might even happen automatically when you import existing pools, I forget. But have backups anyway. Do be sure to unplug the old boot sticks before installing the new system, having multiple freenas-boot pools would trip things up a bit.

1 Like

Awesome thanks. That update will probably happen in a month or 2. Kinda hoping the wife will let me update my work station to Threadripper 3000 when those come out. My dual xeon v3 14 core workstation seams to be getting slower and slower when trying to edit video or transcoding video. After this upgrade she may make me wait till the spring.

What is your bios version out of curiosity?

The latest bios for that board is F2

Ohhhh interesting. I bet the hangup I have is the agesa version. That’s a pretty old agesa… hmm

1 Like

A couple posts from Reddit and elsewhere said that the BIOS needed to be updated for freenas to work. So that old bios may be the issue

I know it has been a while since I posted here and I just wanted to give an update on how things are going.

It has been an interesting 3 months. I started this box on FreeNAS 11.2 U6 since then it has been upgraded to U7. And now that FreeNAS 11.3 stable is out I am looking to upgrade to that. I managed to get my hands on a NetApp DS4246 and DS2246 disk shelfs. I found one of my original Mini SAS to SATA cables had a single bad cable. Which was causing errors to show up in FreeNAS. After trying 3 different drives that were all brand new I tied a regular SATA cable that fixed that issue so I had to get a new SAS to SATA cable.

FreeNAS recognizes the disk shelf dives with no issue at all. And every slot works as it should in each shelf. Great buys from EBay.

As for FreeNAS on the Threadripper has been great. I have had no issues transcoding Plex to 10 different streams at the same time. Most I’ve been able to test at once. With the speed of the ssd’s as my main drives I haven’t seen the ram get over 60mb even having 20mb designated for a virtual machines. 8 for a Linux machine and 12 for a windows 10 machine. I can honestly say upgrading the OS has been a breeze. Even though I’m still booting off of usb drives. The machine has been more than fast enough for everything that I do. I don’t think I’ll be in need to upgrade the hardware any time in the near future. Just need to add more drives and bigger drives. I just hope in the future that FreeNAS will read the temperature of the Threadripper CPU.

3 Likes

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.