Leon's Misadventures of Fumbling About In the Dark

[Leon_the_Loner]

For what must have been a few months now I've had parts lying around for a bitchin' home server/nas thing that'll run an ass ton of stuff for my home network. I haven't had the time with personal happenings, being a functioning human being, and work hours (my 60+ hour boys, have a drink along reading this post for me.)

But none of that shit is in the way anymore so it's time to my ass in gear. Grab your preferred nicotine delivery system, a mug of coffee, and a glass of your favorite vodka, your preferred OST going, and let's get to work.

The plan for the system is to run ESXi with freenas running virtualized with another couple of OSes. Probably debian and ubuntu for servers and media respectively. Because fuck paying for unraid, and fuck running plugins out the ass when I could just use a different OS to do any other shit. I have heard about plugins for freenas, and every single time I have seen it mentioned it is followed by threats of suicide.

Now before we get any further, I have never had any experience and I haven't read enough on ZFS, Freenas, or Vsphere as much as I probably should have at this point. So my unbearable stupidity will be coming out in full force during this experience. It will very much be a learning experience for me. But I've found the fastest and easiest way to retain knowledge for me is to just get my hands dirty and try it out.

I'll be writing this as I am doing the project, currently, the ONLY thing that I have done is put that bad boy together to see it POST, get in the bios, turn on VT-d, figure out the reseller for the board was not kind enough to leave the IPMI open, and have the sata controller fail to see 5 of the 6 drives. Yippee... Before we get to far, the part list.

CPU(s)
2x x5560 xeon
CPU coolers
Some dynatron coolers I found on ebay that sound like vacuum cleaners
Motherboard
SuperMicro X8DTL-3F rev2.01
RAM
4x SAMSUNG 8GB 2Rx4 LOW PROFILE PC3-8500R ECC Reg at 1033MHz. so, eh.
Storage
6x Hitachi HGST Ultrastar 7K3000 2TB
PSU
Some EVGA -850 80+ gold modular thing not really important
Case
Viper Z from rosewill It's actually a really nice case for the money with a very robust front I/O if that's your thing.

I got all of the hardware for around $550. If you really want to play with some server grade stuff but don't have the pockets the x55xx chips are an absolute steal. I'm not sure why, maybe I've knee capped myself with these

Now, The task for this first post is to get the IPMI working. So before I do all this I boot my boy up, look at the DHCP leases on my router, and plug in the IP of the only device I know could be it into the address bar. This brings me to a supermicro login page. The default login creds are "ADMIN" and "ADMIN." I try it, no dice. Time for plan B, google that shit.

Google search brings me to here Telling me I need to make a dos live disk with the firmware and ipmicfg on the disk.

So first I make a partion on the USB stick for fat32 and dd the freedos image to it. For those of you that are following along and are just getting your feet wet for the first time you can use dd to convert and copy files. You can use tools like rufus on windows and unetbootin on linux (unetbootin always makes an ubuntu installer for me so I never use it) but it's also a great tool to copy partitions over to other places and make backups and the ilk.

sudo dd if=/path/to/img of=/dev/sde1

It does it's thing and I can't see the usb stick in dolphin. Red flag one. It's telling me that the sectors are fucked and I did something wrong. So instead I do

sudo dd if=/path/to/img of=/dev/sde

And everything shows up fine. The one partion on my drive has been resized from 1Gb to 512MBs and we're all hunky dorry. I slap the firmware and ipmicg files on the drive plug it in and power on the black box.

Once I'm in I tell it I don't want to install and I'm into the command line. Once there I create a new user and give it operators permissions.

I give it the command

ipmicfg -user add 3 username password 3
Done

All set. I test it out and I'm in

Only problem is I don't have admin privileges and that means should I want things like DDNS if I want to access this box from the internet or run a mail server on it without occasionally not getting emails due to the IP being changed I'm I'm out of luck.

So I try to set it to factory defaults which goes like

ipmicfg -fd

And it fails

Well shit. Maybe I can make my new user an admin

impicfg -user level user-id# 4
Done

Ahhhh yes. It's good. It's in. I can login, and I can power the machine on and off from my web browser.

So what had been months of procrastination and legitimate busy schedules was about 4 hours of time after documenting everything for a forum post and laying the groundwork for future posts. Feels good man.

It's 3 o'clock in morning and I've still got some vodka to finish off. Hopefully this weekend I'll have another reply to this topic where I'm installing ESXi and freenas but more than likely it will be me finding hardware issues on the board and wonder whether or not to use the sas ports instead of the sata.

If you really just hated it leave a vote. If you would like to see my next adventure leave a vote. If you have sage advise please bestow it upon me.

• Post some more
• Never come back

Show results

[Leon_the_Loner]

Well well well friends it's been way too long since I started this, Damn near right after I posted this life got crazy again, crazy in a good/bad way but not in a "my mother is dead" way.

Grabbed that A+ finally, (protip, Just put your shitty resume out there with tags for IT and desktop technician. Save yourself a lot of time and the cost of the A+.) Moving, new job yada yada.

A small story about the move. The new job started and my lease was up very soon, so I went back to the parent's house for a month and a half. I thought, cool, I'll have plenty of time to work on this thing while I'm there. I get there with the stuff I need and throw the rest of the shit in a storage unit. Get there and realize there's no where to put a desk. So I take a stacking chair, load it with dumbells and mount my monitor arm on it I use a piece of left over MDF for my ghetto lapdog over the arm rests of an easyboy in the corner of my old bedroom.

I used that setup for a month and a half.

There was also no where to plug in any other system other than my PC. So it sat.

Finally back in my place, and room that needs some tech to make it feel like home.

Within that time I've moved from wanting to virtualize freenas with ESXi. I thought, Why not just use KVM instead? So I've decided on either running Proxmox or zen. I think I'll make another thread asking the question of which one, first, my misadventures.

So first thing, I forget my IPMI password that I wanted to use so badly. So I fix that again. After fucking up dd for the zillionth time refusing to use windows, netbootin never installing anything other than ubuntu no matter my settings, I just use rufus. On windows. Feeling defeated.

Whatever, IPMI works again. So I try setting up a samba (for those of you reading that don't know. samba is pretty much the open source implementation of SMB on windows) share to give to the IPMI virtual disk to install proxmox. Whatever the reason, java doesn't want to sink it's teeth into the console redirection tool, and the virtual disk wants backslashes and not slashes in the file path and I refuse to use more windows.

Whatever, make a proxmox usb and boot that son of a bitch up. Everything looks good! Let's start the install...wait, where are all of my drives? (This is around a month ago)

So I go through all the sas ports, and all the sata ports. One drive is still alive. Doesn't see my SSD (it's a 5-7 year old server board, wouldn't surprise me if it can't see the SSD.)

Luckily, a very friendly community member gave me an old raid card. Cool, At least now I don't need to rip apart my rig to test the drives. Let's just plug this in to the PCIe slot and plug the sata cables into... My sata cables were too short.

One monoprice order of an assortment of sata and usb cables and I'm back to trouble shooting. plug everything in and...Holy shit this raid card still works, Awesome. Let's have a look at the driv-

A google search brings me to a master password (guess the password was there just there to keep retard new hires from dicking with the settings) and I'm in. Show me the drives baby!

FFFFFFFFUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUCK

It was dead drives all along.

So now I'm here. Cool guy I work with that used to do security stuff says there's a place not from from where we work that sells used drives with something like a one month guarantee. So I'll hit them up.

In the meantime, What should I do guys? proxmox or zen? hardware or software. It's no longer a matter of money, and none of it needs to be the best thing ever. Just good enough to work for moving my music and video around the network and being able to run an ubuntu box with GPU passthrough to play some light games with friends. There's enough horsepower for some other VMs if I get frisky but I've got hardware for a lab environment laying around too.

[Leon_the_Loner]

Oh, oh buddy. It's been a long time of sitting on my hands and not being able to balance work life, social life, dark souls, and my bank account, but here we are.

Five dead drives later. The box is alive.

They're all still used drives, but S.M.A.R.T raises no alarms so off to the races I go.

Installing proxmox was a breeze, give it a drive and good to go. Drunk me thought he should keep the root partition at 300 GBs. Drunk me didn't think "maybe there are no home partitions on proxmox." He also thought 96Gb of swap was a good idea. HA. I do the funniest shit I tell you.

And there she is. Ignore the tabs. I know exactly what I'm doing.

I've made a VM for freenas. But the more I read the more it seems the only way to give freenas drives properly in a virutal environment is to use PCI passthrough. I kind sage gave me that raid card and I'm hoping I don't have to buy a real battery, just a small coin cell battery so the raid cards bios can remember to be set to Jbo- but that doesn't work, I ran an ubuntu live disk and got the smart info and it was not giving out the proper info...

Guess I've got some reading to do.

if anyone knows if passing the drives through virtio works fine (I imagine it's bad idea, extra layer in between zfs and drives is no me gusta) or if there's a different mode I should be setting the drives too, feel free to tell me.

[FaunCB]

Please for fucks sake stay I can't take it anymore. The most interesting stuff is coming from newbies breaking their shit.

[TheCaveman]

I made a similar box so I'll briefly go through what I did.

specs
Xeon E3 1230 v1
16GB ddr3 Ecc UDIMM
Intel c204 mobo
gt 210 (since I'm an idiot and idn't get a xeon with built-in graphics)
Lsi 9211-8i
3 x WD Red 1TB (I'm light on storage needs)
Intel Pro 1000PT Dual port
ESXI Freeeeeeeeeeee version
64GB Flash drive

I have 2 24/7 VMs, FreeNas and PfSense. I choose ESXI specifically because you really need to do PCI passthrough for the LSI HBA card to FreeNas (as you've learned by now lol). ESXI does PCI passthrough best, which is why I choose it. So FreeNas. I give my VM 8GB of ram, 4 Cores, the LSI HBA w/ 3xWD drives, and access to the vSwitch running on one of the mobo's ethernet ports. The LSI 9211-8i was choosen very specifically as well, since FreeNas HATES raid cards, but will play nice with a 9211-8i in HBA mode with the v20 firmware (MUST update or FreeNas will be mad). It all works though once you get through the configuration and set it all up. Reads and writes saturate GigE, and I'm happy with that (for now).

PfSense is my other 24/7 VM, and PCI passthrough was also a driving factor with this one as well. I wanted a dedicated port for the in-coming WAN connection, as well as a dedicated port for the LAN connection (although this one can be done without if you just put the LAN on the VM vSwitch). I give PfSense 2 cores, 512MB of ram, and 8GB of storage space and it runs gr8.

I'd really recommend going ESXI. The alternatives just don't do PCI passthough as well, and if you want to do FreeNas, it's as must-have. Also, REALLY consider grabbing a 9211-8i. The FreeNas forums are full of people rocking this HBA (like me ) and the support from FreeNas really makes it a seemless experience.

Also, I've done IMPI and Bios flashing on Supermicro stuff for my brothers' build (which I will post along as well as a long form explanation of my ESXI box eventually). He has a 12 ram slot version of your motherboard, and he has a pair of x5670's (which I also REALLY recommend, 12 threads each at 3.33Ghz turbo is NICE), with 96GB of DDR3 registered, a 1070, a 128GB samsung SM951, a Kingston 240GB Hyper X 3K, a 1000 watt platinum PSU, and a 1TB WD Blue. Flashing the bios on that board is a BITCH, and I think he lost the IMPI credentials that I setup for him (and will now also have to redo, although that one can wait).

[Leon_the_Loner]

Please for fucks sake stay I can't take it anymore. The most interesting stuff is coming from newbies breaking their shit.

I'm never too far away. I just don't get the chance to sit down and waste a night on tech stuff with the second Job.

To caveman. Even if ESXi is free to use, proxmox is open source, so even if it's more difficult I'd use it unless totally impossible.

[TheCaveman]

To caveman. Even if ESXi is free to use, proxmox is open source, so even if it's more difficult I'd use it unless totally impossible.

Good luck. Not sure if it was Citrix XenServer or Proxmox that I looked at, but I know it was one of them and it required a lot of command line tom foolery to get pci pass through working. I preferred to just use the GUI in Esxi, but best of luck figuring it out.

[Leon_the_Loner]

a lot of command line tom foolery

My favorite.

[Leon_the_Loner]

Hey yo, what it do, big money leon comin back at you with a little update.

Bought myself an m1015 raid card that works great in IT mode so I hear and got it all soop’d up with the nitrous and ready to be passed through to the freenas VM. Went and trolled myself over here

Also, what’s this?

ANOTHER SERVER? LEON, DUDE, YOU DON’T EVEN HAVE THE OTHER ONE RUNNING YET!

I know man I hate myself as much as we both hate regular liquorice. I’d like to see if I can throw one of my old 290s in there and mine some monero after I pick up a custom vega 64. maybe run a separate web server on this one, I dunno man. I don’t even remember what I wanted to run on the other one anymore, but I’ll document it as well.

[FaunCB]

Obligatory joke of playing with yourself in the dark.

[Leon_the_Loner]

Ohh, been awhile, promise I haven’t been sitting on my hands, I was busy until the end of the year and started playing DnD. Super fun, anyone want to play hit me up.

But on to the tech fumbling. Since then my first priorities were first: revive server from memtest destruction, 2 make the Vega 64 card work on my system on linux.

So the story with the server is I made an ubuntu VM and ran a memtest in it, like a retard, I know. Then whatever hybrid ISO setup proxmox has afaik it does not work with dd. Had to go back to windows and use rufus. It is up though, and has been running a 7 days to die server successfully for the past two weeks no problems, so yay?

Now comes getting the vega10 cards to work with linux in a capacity that doesn’t break everything, or run like ass.

And here’s a little tip sheet of things I tried before hand, and what worked.

For those that are looking for tips on setting it up on ubuntu go get the mainline kernel (latest is 4.15-rc7) from the ubuntu repos and add the padoka and oibaf ppas. Afaik that’s everything you might need to get it up and running. There are 3 major component groups.

Kernel group
You’ll want either an amd-staging version of some kind, or a release client of 4.15 (if 4.15 official mainline version is out when you read this that’s what you want) on ubuntu it’s stupid easy to install the kernel there.

I’ve tried arch as well, and most of the info I got came from trying to install it there and I was approaching the weekend and didn’t want to compile the kernel again. Just sitting at the screen having to use the laptop. If you are trying to setup on arch I would just compile the (at the time of this writing) the 4.15-rc7 (or clone the git for even more bleeding edge you emo punks) and had gotten a bootable kernel following the arch wiki but after trial and error and knowing what I needed to do I tried Btrfs and for whatever reason the kernel wasn’t having it, might need to go into the .config more and enable more btrfs modules.

hardware driver group
This group is a bit of a mix, since the amd driver stack is essentially all 3 of these groups, but whatever.

I started with a bunch of info cut and pieced together from the arch wiki and phoronix, they kept talking about radeonsi and I’m thinking, “southern islands? okay.” and install the xf86_video_ati driver the first time I try. This is wrong. You want the amdgpu driver. I’ll explain what radeonsi is in the next part

Mesa drivers

Mesa is the driver that handles things like OpenGL (and if this mesa-vulkan-drivers package isn’t lying to me) and vulkan. I grabbed both the padoka and Oibaf ppa’s and have the 17.4 version mesa drivers installed.

So with that there’s the quick guide, if you can’t figure out the rest yourself you should probably stay away from vega on linux for the time being. It honestly probably be easier and better if you did this on arch instead. You will likely have to manually configure xorg

Problems I ran into without the new kernel and drivers

• the performance even on the desktop and 2d image applications just sucked. it was definitely running around 30fps or lower. Things like krita weren’t responsive, but ran, 2d games could run, 3d was spotty.

• Multiple monitors did not work out of the box, I never tried to setup X to use the second monitor, but the performance was already garbage like I said

• not that it’s a huge deal for me, but software center would have a fit and never be useful for anything

Most of those issues were fixed after the upgrade. Software center still hates to install downloaded .debs (sucks cause it was so quick to do that) but you can just

dpkg -i /path/to/deb
apt-get install -f

and you’ll be set, although if you’re trying this you probably don’t need to know that (unless your brave and want to try anyway, in which case, god speed)

I have only tested insurgency (a source game) so more testing needs to be done. It runs very well, but I did have an issue with mouse input. Might have something to do with the tablet plugged in at the same time. Krita runs a-ok from the 2 minutes I used it.

For some reason undertale doesn’t want to start ¯_(ツ)_/¯.

[Leon_the_Loner]

Here’s an update on games I’ve tried. Most for only minutes at a time, but it’s super promising, and provided they continue to run as they have, my booting to windows will be exceedingly frequent.

My mouse issues were probably due to my ps3 controller. Dunno why but right now the left stick will actually act as my mouse.

Insurgency runs perfectly, as I expect any source game to do.

Hotline miami 1 & 2, obviously, run perfectly.

Civ 5, started a game with proper settings (mostly maxed out @ 1440p) and it ran perfectly

7 Days to die, ran okay, I wasn’t getting great fps on windows but it was better, I don’t think I set the settings as high in windows though. Plays alright, no error or issues.

Knights of the old republic 2 runs fullspeed, also as expected.

But here’s the big boi,

Witcher 2, cutscenes play at fullspeed, and in player controlled areas plays just fine, as with all the games so far. It’s not a very stable framerate but it’s not going below 30 if my eyes aren’t deceiving me.

This is a huge improvement from the drivers I used on an R9 290 that this game should sing on in windows. When I last tried Witcher 2 it played, but it suuuuucked. Audio was broken, framerate was in the single digits. Just unplayable.

I’ll be playing a lot of witcher 2 in the coming days, if I say nothing that means I tweaked the settings and it ran without issue.

Once kernel 4.15 is added to the official arch repos I’ll probably move my home folder and play from there. It was massively easier to get the latest bleeding edge drivers (amdgpu, mesa) on there than it was to hunt down the ppas for mesa on ubuntu.

Another tip. Use 16.04. afaik, 17.04-10 is broken as shit. Either way, 16.04 has better support afaik.

Here’s the settings I used in witcher 2. I wasn’t trying to give myself the greatest fps or skimp to make it run better, and for linux I was actually amazed.

As you can see, the only things I turned off was blur, and vsync. ubersampling sucks in windows as well (at least when I tried crossfire 290s.) I’m pretty stoked.

[Leon_the_Loner]

Update, Metro 2033 redux runs perfectly. no slowdowns, except at loading screens, but it doesn’t hurt anything, just freezes for a second and plays out the dialogue normally. I’ve played a few hours of it now, awesome game.

Persona 5 on RPCS3 runs better than it does on my windows machine. Don’t ask me why, it’s still running openGL. Vulkan support for linux in RPCS3 isn’t quite there yet.

Almost always I get a message talking about how openGL is taking too long and asking if I want to kill it or wait. By the time I get that message the game is running so…