Projects and general bedlam

redocbew · May 24, 2023, 5:35am

Long Rambling Introduction

I’ve threatened to do a blog or build thread or something a number of times on this forum. This one’s got a bit of a different spin on it.

As visitors of the lounge know, in the past year I’ve gone back to school to prepare for career #2 as a physical therapist and plan to leave technology behind as a profession.

I had plans to jump on the AI/ML bandwagon. Not doing that. I had plans to do a netbooting Raspberry Pi cluster running Kubernetes. Not doing that. I had plans to set up a wireguard tunnel from Linode so I could access the machines on my home network from school or anywhere else. Not doing that. I had plans to build an XCP-ng machine that would host NAS(I know, bad idea), plus a VFIO gaming VM, plus dev sites, and so on. Not doing that either.

These are all great projects that I wouldn’t discourage anyone interested in these areas from pursuing, but that’s not going to be me. To that end, the homelab is in need of some serious reorganization.

So, I have plans. I’m planning things. I’m not just building a nifty machine because I want one.

I’m not. Really. I mean it.

Wait… did I just say “reorganization” when I meant “ruthless cutting to the bone”? Ugh. Yuck. Gak. You see? This is why I need to leave the corporate life behind.

The Purge Begins

Over the past few weeks I’ve been selling many of the components I no longer need.

There were some switches, some fans, some spinning rust, a pair of speakers, a few HBAs, a few solid state drives, a pair of 2080 Ti’s, a 3970X, a 7232P, a TRX40 motherboard, a set of ECC DDR4 dimms, a few 40GbE nics, a bramble of Raspberry Pi’s, 128GB of DDR4, a Unfi AP, and way too many RJ-45 transceivers.

Yeah for a while there I had shit everywhere. Many leftover amazon boxes and an alarming amount of bubble wrap was employed here. At least this time I managed to start selling the old stuff before buying the new stuff.

I mean, I haven’t sold all the old stuff(yet), but I’m still calling that a win.

So, of course the first thing I buy is more hardware.

The New Machine

I don’t want to build an entire machine just for gaming that’ll sit idle most of the time. I like to compartmentalize things into VMs for easy backup and restore in case something dies, and I don’t see myself being able to get rid of windows altogether any time soon, so that all means I’m not done with virtualization or VFIO.

But…

download (1)

I may have waffled a bit on my choice of components.

waffle waffle waffle

Specs:
New stuff:
Ryzen 7 5700G
~~Noctua NH-L9x65 cooler~~
be quiet! Dark Rock Pro 4 cooler
~~ASRock B550M PRO4~~
~~ASRock X570S Riptide~~
Gigabyte X570S Aorus Pro AX
128GB DDR4 3600
~~EVGA SuperNOVA 850 GM~~
XFX Speedster SWFT 319 Radeon 6800 XT
Seasonic Focus SGX-750 750W SFX PSU
Lian Li O11 Dynamic Mini
2 x be quiet! Silent Wings 4 140mm PWM intake fans
1 x be quiet! Silent Wings 4 120mm PWM intake fan
1 x be quiet! Silent Wings 4 120mm PWM exhaust fan
Intel X550-T2 10g nic

Reused stuff:
4TB Rocket Q4 NVMe SSD
2 x Sabrent 1TB Rocket Q4 NVMe SSD
~~Kioxia 960gb CD6-R NVMe SSD~~
~~2 x 1TB WD Blue SATA SSD~~
2 x Intel 960GB D3-S4510 SSD.
~~EVGA GeForce RTX 3070 XC3~~
~~Intel X520-DA2 with 10g rj-45 transceiver~~(rip)
~~1 x Noctua NF-F12 iPPC 3000 PWM exhaust fan~~

I waffled on fans. I had already sold two of the three Noctua fans I had before deciding to do this build which required me to buy some new fans. The third one I later found had developed an annoying buzzing noise which I didn’t notice while it was in the closet.

I waffled on the motherboard once. At first I went with a uATX board so I could have a second PCIe slot for the 10g nic, then I went with the X570S Riptide from Asrock. There’s a minimum of shenanigans with this board in regards to which slots are affected depending on how you populate the others. The last two SATA ports are disabled if you’re using all the M.2 slots, but I only have four drive bays in this case anyway. The main reason for waffling was I wanted a third PCIe slot for the Kioxia SSD. I’m almost ensuring a bottleneck at the chipset with this much storage behind it, but if it’s worth doing then it’s worth overdoing. Thanks Mythbusters.

And chipset fans are annoying. That’s right. I said it.

jack-nicholson-deal-with-it

I was forced to waffle on the power supply. The O11 Mini takes only SFX power supplies, so I couldn’t reuse the one I bought for the home server, and the new one somehow got destroyed in shipping to the point where they refused to even deliver it.

I waffled on storage once. I’ll get better endurance from the Intel SSDs which I also had left over from the home server build, so I’d rather use those and add the WD Blue’s to the pile of stuff to sell.

I waffled on storage twice. I won’t sell all of the WD Blue’s. I’ll fill up the other two drive bays and give myself a little extra local storage.

Final assembly

After lots of waffling final(really) assembly began. It was mostly uneventful except for mounting the motherboard where I forgot that Der Bauer is German and like the rest of the world uses the metric system in the things they make.

Installing Fedora is up next.

tjw · May 24, 2023, 5:44am

same

install tailscale, it’s that but seamless and really elegantly done

redocbew · May 24, 2023, 6:38am

I’ve heard of tailscale. Seems unlikely I’ll get to it before school starts again in August, but maybe.

I also shrunk the server rack in the closet.

There’s the mini which is the primary NAS. The synology for backups, a router, a switch and a patch panel. The UPS seems a bit overspec’d now, but that’s ok.

Without a 4U box filled with a dozen hard drives, a GPU and a bunch of fans there’s not much in the rack which makes noise anymore. If need be I can move the rack underneath a desk or something.

regulareel · May 24, 2023, 6:45am

Hey man, Im also in healthcare auxillary services cosplaying here as sysadmin. You’ll do fine there too.

In retrospect, you should have been in the radiology unit. Its got cool tech stuff in there that I’m sure you’d appreciate.

redocbew · May 26, 2023, 2:10am

There was an addition.

How do people take non-photo bombed pictures of reflective things? There must be some trick.

Level1_Amber · May 31, 2023, 7:18pm

I love it!!

Edit: Lol I didn’t even realize it was an L1 Tux! Even cooler

Gnuuser · May 31, 2023, 7:39pm

I want one! ( L1 tux)

redocbew · May 31, 2023, 7:56pm

Store dot level1techs dot com

I got mine a while ago, but it’s been a while since I’ve had a desktop PC, so the dude hasn’t had much visibility until now.

Trooper_ish · May 31, 2023, 8:01pm

Have you tried borrowing a small mirror off the SO?

Otherwise, ask Den-Fi … (as you know, that forumite does smexy photos)

redocbew · June 5, 2023, 5:29am

Been working on this machine here and there when I can.

It’s running Fedora 38. I’ve been a long time Ubuntu user, but I wouldn’t really say I’m distro hopping, because that implies I could be switching to a different distro for this machine at some point in the future. That’s for sure not happening, because I’m more interested in using this machine than tinkering with it.

Got my ZFS pool up and running.

pool: tank
state: ONLINE
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sda     ONLINE       0     0     0
	    sdb     ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    sdc     ONLINE       0     0     0
	    sdd     ONLINE       0     0     0

They’re all “1TB” drives, but two are 931.51GB, and two are 894.25GB, so I did a striped mirror. I’m thinkin this is going to be zvols for the VMs. Currently got all but one of the VMs installed. A few are old and I’ll be migrating away from them eventually.

It seems kernel 6.3 and ZFS aren’t playing nice yet, so I’m staying with 6.2 for now. I tried using the Copr 6.1 kernel as well which seems fine, but virt-mananger had crashed a few times since I set it up. Not sure if that’s related to the kernel or not, but 6.2 does work with zfs 2.1.11 which is currently what’s in the repo, so I figure why not use that.

The two Sabrent nvme drives are also being used for VMs. The windows gaming VM will be installed to one, and my general purpose Linux VM will be installed in the other. This way the host machine is pretty much just a bare hypervisor, so backups will mostly just be the VM disks and configuration.

As it turned out, I had to waffle on networking also. The old Intel SFP+ card I had was dead, and the cheap replacement I found was indeed very cheap and wasn’t really worth using. I’ll be sticking with the onboard 2.5g nic for now as I don’t want to spend any more money on this build at the moment.

Wrapping up VM install and configuration is up next, and then on to the GPU pass-through for the gaming VM.

ThatGuyB · June 6, 2023, 1:00am

There’s a difference between distro hopping and hopping distros.

anon7678104 · June 6, 2023, 1:13am

gg

redocbew · June 8, 2023, 7:12am

Wrote a cron job for exporting VM images and configuration. I figure since I have the space I’ll keep things simple and replicate all of this to the local pool(/tank). Having everything in one spot should also make it easier to sync it to the NAS, and I’d kind of like to set that up as a replication job in TrueNAS instead of also copying to a network share or something in this script.

#!/usr/bin/bash

for vm in $(virsh list --name --all)
do
  vmstate=$(virsh domstate $vm);
  startvm=false;

  if [[ $vmstate == "running" ]]; then
      virsh shutdown $vm;
      startvm=true;
  fi

  vmdisk=$(virsh domblklist $vm --details | awk '/disk/{print $4}');
  israw=$(echo $vmdisk | grep -c ^/dev/);

  mkdir -p /tank/vmbackups/$vm;
  virsh dumpxml $vm > /tank/vmbackups/$vm/$vm.xml;

  if [ $israw = 1 ]; then
    qemu-img convert -f raw -O qcow2 $vmdisk /tank/vmbackups/$vm/$vm.qcow2;
  else
    cp $vmdisk /tank/vmbackups/$vm/$vm.qcow2;
  fi

  if [ "$startvm" = true ]; then
    virsh start $vm;
  fi
done

Edit: Because I like pain, I updated the script.

H-i-v-e · June 8, 2023, 7:01pm

These words be praised. Now I only need to stop messing up my systems by rash decisions. In the end I hope that libvirt/qemu and kubernetes will come to my salvation. ZFS’s rollback features has already saved me a couple times.

This has been happening for me for quite some time. My workstation is not unstable, everything is fine, except that virt-manager crashes every now and then when I press things it does not like. I have no idea why, it did not happen in the ancient past, but in the last year or two it has been crashing on a regular basis. I tried to use cockpit instead, but virt-manager is so simple to use.

redocbew · June 8, 2023, 7:15pm

Yeah it’s weird. It doesn’t affect the VMs that are running, so I just start virt-manager up again and everything is fine. It doesn’t seem to be related to the copr 6.1 kernel either as it’s still happening on 6.2.

redocbew · June 9, 2023, 3:00am

zfs 2.1.12 has support for kernel 6.3

redocbew · June 25, 2023, 5:04am

Home Gym PC

I built a NUC maybe a year or so ago now as an attempt to escape windows 11. Windows 11 became a thing here when my previous laptop died and I couldn’t wait to have a new one shipped, so I did the remarkable thing of going to a physical store to buy hardware.

After building the new machine I moved everything from the NUC into various virtual machines. That left the NUC free to be a ridiculously over spec’d video streaming device I can use during workouts.

This machine no longer needs a 4TB SSD, so I’ll sell that and replace it with a 1TB drive. It doesn’t need 32GB of memory either, but I’m not as interested in replacing that.

But wait! I should wipe that drive before swapping it out.

It’s fine.

In fact I’m not sure why I even bothered to(sort of) put the bottom back on, but you can see the mounting pegs screwed in which is how it’s going to attach to its VESA bracket.

I’ve used NUCs before, but never used this kind of VESA bracket with one. They’re pretty nifty.

Add image from JWST of the tarantula nebula as a background.

redocbew · July 3, 2023, 6:14pm

Every Monitor Must Get Stoned Armed

I’ve been looking for a monitor arm which could do all the things for a while now. I think I may now be an Ergotron fanboy.

There’s cables everywhere, but I need a new desk situation here, so leaving that the way it is for now.

Zszywany · July 3, 2023, 7:33pm

You’ve got some mileage out of the keyboard. Have you considered letting it die? /s
Nice arms though

redocbew · July 3, 2023, 7:40pm

But yeah… it passed into the realm of being entirely utilitarian quite some time ago. I don’t even know how long I’ve had it.