So I’m starting this blog now, three months early, because there’s going to be a lot of stuff that needs figured out before I actually buy hardware/have the money in December. And hooo boy is this one gonna be an expensive rebuild. Maybe Threadripper Storm Peak will be out by then, or whatever Intel’s been up to.
The Goal:
I want to have one PC, with an insane amount of RAM for a desktop (probably 512GB), which can load a VHD/similar disk image over the network into a RAM disk and boot from that. I’d like to have multiple images stored on the NAS - one with Windows gaming stuff, one with Linux development stuff, etc - that I can choose from at boot and then flush changes back to that storage whenever something on the OS drive (I’d still have network storage connected) gets modified that I want to preserve.
No local storage except a device for the bootloader.
That means I’ve got several steps to figure out. Connecting to an iSCSI target as a storage device is easy… it’s everything afterwards that’s the hard part. Do I:
Have a local bootloader which connects to the iSCSI target, prompts to select an image, and then mounts that image as a RAM disk and boots.
Have the iSCSI initiator actually select and boot the image which would leave each OS image to do its own RAM disk setup/configuration.
Run some kind of hypervisor I’m unaware of that would handle most of this and accept the performance hit.
Dunno yet, these are all just shower thoughts so far.
Now this is all very similar to what Windows PE already does, minus the network boot… the RAM disk is assigned X: at boot. So, maybe I have something to investigate there? GRUB can initialize a RAM disk, that’s an option. I see something about adding EFI entries to the Windows boot manager, maybe that’s a thing?..
I have no idea what I’m doing, but it’s gonna be an adventure and I’ll document it here as it happens slowly over the next few months.
It sounds like you’re talking about TFTP and PXE boot. TrueNAS has a TFTP server built-in, and most NICs these days should be able to manage PXE booting with a little help from your local DHCP server.
Switching between OS images easily might be a thing… not sure offhand exactly how that would work.
That way you’d have no local storage at all, but it might be easier to run Proxmox or XCP-ng, and just have all your VM images pulled in over the network. XCP-ng calls them “storage repositories” and it’s pretty easy to define them as either local, or via NFS or iSCSI. Not sure what the analogue is for Proxmox, but there’s probably something.
Could be… PXE is something I’ve somehow avoided my entire life, so I have no idea what I’m getting into. Will do some reading on it.
A lot of the suggestions I’m seeing elsewhere right now are for mounting the boot drive over NFS which is specifically what I’m avoiding.
Yeah, I was looking through Hyper-V documentation too and I see it supports basically what I’m trying to do. If I can’t figure out a better way to do it on bare metal, I’ll probably just virtualize it and eat the performance cost. Not in a hurry though, so I’m gonna try a few things first.
The difficult part seems to be that I want the OS image mounted as a block storage device.
If you’re not going to be sharing the same data across multiple operating systems, then yeah I don’t see a reason why you’d be tied to NFS. iSCSI should work, it’s just like you said, a difference in how you boot from it.
I don’t think Storm Peak will be here by December. Probably OEM only in early 2023, and DIY summer of 2023 if AMD sticks with past practice. I was considering building a wrx80 workstation on 5000 series, but I think I’ll be waiting for Threadripper 7000. My workloads take advantage of higher clock memory and TR 5000 relies on bios updates on 3000 series boards. I think storm peak will hit at the ideal time where you won’t have to pay the early adopter tax on ddr5, and the boards will be built with adequite power delivery for cooking ram and beastly CPUs
Am learning things here at the moments, so I might not respond directly to everyone’s posts but I am reading them.
Also it’s occurred to me that one downside of what I’m trying to do is that by loading the entire OS drive into RAM instead of mounting it over the network, I’m gonna be waiting at boot for the thing to transfer at least a 100GB image over the network. Which isn’t great, but 40G InfiniBand should make it tolerable at least.
Do you understand the trouble, and possibly the hardware requirements of what you are planning?
Also the fact that if by chance you are able to pull it off you’ll be on your own with regards to troubleshooting/backups?
You’re probably better off buying the cheapest possible fiber channel San that can take snapshots, and use that as your storage system…
I understand that I’m about to buy $800 of registered DDR4, a few NVMe NAS drives, and that writing bare metal EFI block I/O drivers as I type this is beyond what anyone on the internet can/will take the time to help with.
This one’s a project blog and not another help desk thread asking for someone to put together a sensible PC build. Am doing it for the same reason my home network runs QDR InfiniBand - I enjoy learning about new things and figuring out stuff that sounds interesting.
The EFI RAM disk enumerates in GRUB, as HD1 here. Configuring GRUB to just run on its own, standalone/without Linux, was a miserable experience as those of you who saw me venting in the Lounge about it today have probably guessed lol
This thing is gonna boot through so many different shells at startup
Gentlemen it’s been an honor. I’ll be spending the week learning how to use the EFI specification to write part of this thing that doesn’t appear to exist in the wild exactly as I need it.
Good thing C is about the only proper programming language I’m actually familiar with.
Out of curiosity I tried to copy the Arch installation ISO to that RAM disk to see if GRUB could boot it from there. It failed copying about 2/3 of the way through. Now that I’m looking at the source code, it turns out there are several problems which I’m in the process of resolving:
The EFI driver RAMDISK.EFI creates the block device and handle in memory but also loads it with a FAT16 filesystem. The driver itself enforces a max 512MB size on the RAM disk (if > 512, set = 512 and continue), and the underlying GetDiskSize() & related function calls use UINT32 types which limit me to 4GB.
I’m removing the FAT filesystem entirely and rewriting the RAMDISK driver and MKRAMDISK command to use UINT64 integers so they can make an arbitrarily large block device. We’ll see how many problems that causes once I get the environment set back up to compile it.
