Sysadmin Mega Thread

I’ve just experienced an interesting issue. mdadm failed to start my /home array. (raid0 fite me) It gave me an error in dmesg:

[  666.293291] md/raid0:md127: cannot assemble multi-zone RAID0 with default_layout setting
[  666.293292] md/raid0: please set raid.default_layout to 1 or 2

I fixed it with:

echo 1 > /sys/module/raid0/parameters/default_layout

But I’m curious, does anyone know why this happened? I had a simple kernel upgrade and this suddenly starts happening. :confused:

3 Likes

Rework to the FS drivers. What was the file system on those disks? BTRFS, F@FS, and EXT4 have had some rework/cleanup done.

Well had a fun couple days trying to resurrect a server for a temporary AD backup…

Just heavily reinforced my mindset that you should almost always re-install Windows if you plan on repurposing hardware.

After doing everything humanly possible to restoring windows to normal because it refused to boot from a CD or USB I go to make domain controller and get errors. :sob::sob::sob: Basically windows still likes to believe its on a domain even if you change back to workgroup and tell it no, you are not.

So now after 2 days of work, updates and wiping here I am putting in a hard drive with the ISO on it just to re-install windows. Also had to scrape the key because all docs are nonexistent for the server.

Good stuff

7 Likes

:thinking: I should post some rack pics tomorrow

5 Likes






9 Likes

Damn son

What do you use it for? :grin:

Heating and white noise

9 Likes

And for the record I fixed the tilted rails on the bottom server immediately after posting the pictures.

3 Likes

Lmao.

Is it all FreeBSD and jails?
KVM?
Bhyve?
Got some fancy apps on there?
All networking/infrastructure?

TELL ME

:wink:

2 Likes

Yesterday I was offered a position as a C/C++ developer for a financial/credit card transaction company.

Looking over details and discussing with the family this weekend. But it looks like I made it, boyos.

7 Likes

Damn I was going to call you out on that.

I’ve done the exact same thing though.

2 Likes

In the rack:

It’s FreeBSD on the one with all the drives, it is the storage server and has a bunch of jails that have accumulated over time. I’ll be managing some bhyve VMs with libvirt on it as well; I think one is already running there as a proof of concept.

It’s also FreeBSD on the next one up with the SSDs. This one already runs some bhyve VMs managed with libvirt, and I stage/operate on git repos here.

The gigabit switch is boring but the 40GbE switch above it runs an OS based on NetBSD.

Oh and the Thinkcentre on the floor is running FreeBSD at the moment but I might put OpenBSD on it instead to supplant the AT&T provided gateway and serve as a bridge between the 1G and 40G networks.

Raspberry Pi in the mess of cables on the floor you probably can’t see but it runs NetBSD and I use it to connect to the serial consoles of the machines that don’t have IPMI.

Desktops:

On the left is FreeBSD running one VM I actively use right now (managed with vm-bhyve, haven’t migrated it to libvirt yet) and there are a bunch of git repos I work on with this machine as well. On the right it’s currently running Manjaro KDE but I haven’t been very happy with it and I think I’ll give Ubuntu 19.10 a try today.

The VMs are various Linux distros and FreeBSD versions for testing and development purposes.

3 Likes

I had originally racked it level but a half-U off so the hangers didn’t hang right, and when I realized this a week or so ago I fixed the front but got distracted and forgot about the back until now :smiley:

1 Like

A tutorial on this would be :ok_hand:

2 Likes

The Pi image comes from here http://www.invisible.ca/arm/
I have USB serial cables plugged in to the servers.
Setting the OS on the server up for serial console is pretty FreeBSD specific and hardware dependent. I guess that’s the “draw the rest of the owl” part. The BSD section could use some more content, I’ll consider writing it up.

1 Like

:thinking: I should post pics of the case mod I did to the supermicro chassis to install a hot swap ssd bay.

1 Like



Sadly I didn’t take pics of the process, it was pretty intense the whole back panel had to be removed to work on it without risking damaging things

4 Likes

Imo, next step should be a UPS, a proper vertical PDU and a vertical cable channel.

Forgot I wanted a UPS. I also have been looking for more servers to fill in gaps :smiley: but I probably need to do some power measurements under load before I go too crazy. The rack is drawing 3.59A idling right now, according to the Kill-A-Watt. And that’s not measuring the desktops.

A vertical power strip is a good idea. I got one at walmart that I use in the garage with built in USB charge ports. If they are powerful enough to supply a Pi I might grab another one of those for the rack. Though it does have kind of obnoxious cheap outlets on it, maybe it’s worth getting something a little more solid.

As for cable management, I have enough velcro straps to keep things under control for now. A channel would be nice but it’s not a high priority.

1 Like