I have a couple of devices that are not practical (or possible) to install Linux on.
For most of them I’ve fixed this by using PXE boot and running from NFS and this works great.
The problem is with systems that only have a wireless connection, hence PXE boot isn’t possible.
So the question becomes, can I boot from USB and then switch to the NFS installation?
Running completely from USB is too slow and I have a perfectly fully working environment right there on the NAS, just need a way to boot into it.
I’m guessing adding wireless support to initramfs, then using the NFS installation as rootfs.
But at some point the connection will most likely be dropped/handed over?
Has anyone tried this before or any idea if I can find some documentation about getting something like this done?
This is mainly to get Ubuntu to run on my work laptop that I may not re-format and if this works I can run less wires to the my raspberry pi’s
Boot from NAS/SAN is fairly common in enterprise scenarios. At my workplace we strongly discourage the practice as it presents unique challenges over the life of the implementation that generally displaces any benefits. But I am sure other enterprises have different views. But boot from NFS? Not sure that’s the best option. I would think you’d want something more tolerant of communication failures/hiccups/anomalies- whatever you want to call it. I’m thinking iSCSI could be a good budget target for a home or SMB setting.
I anticipate this idea eventually proving not worth the effort but I am very curious what you find if you follow through. There might be better ideas though to accomplish the same function. Like boot from USB initial boot but then pass over to a faster local external disk. That could be a faster USB disk or, better yet, Thunderbolt if you have it. Then you could pretty much have the performance of an internal disk so you won’t be missing anything and it will pretty much behave like an internal disk in the configuration. Personally, I would make sure this option WON’T work before putting much effort into network boot especially when the network provided will be wireless. Best of luck to you whichever way you go.
Think this would not be an issue if I would for example create a ramdisk, copy over the OS and switch to that mount point, don’t think it’ll work with NFS as it would lose network connectivity doing the switch_root.
Although I do wonder what I need to get NFS up again, for example actually create the tmpfs mount points in memory, copy over what I need and once it’s up enough mount the ‘other’ stuff that takes a lot of space from NFS, like /home, /usr, /var
Think it’ll be as @mcclurken said and it would take to much time and research to get something working. Don’t think I’m experienced enough to be able to get this working in any reasonable timeframe.
It’s technically possible - it’s no different than e.g. decrypting root from initramfs, or setting up 802.1x for a network boot.
One tricky aspect might be switching over from one wpa_supplicant (the one in initramfs image, to the full fat one), you’d need to ensure that the new binary pages are fully loaded in ram before killing off the old network.
(it’s certainly technically possible as well, at least in C using madvise to mlock the binary and it’s libraries and config in ram prior to fork/exec-ing, there’s probably helper scripts floating around the internet to help with that)
I think what you want to do is entirely possible, I mean, Linux is awesome like that. But I still think in practice it wouldn’t be entirely stable. Here’s the thing, Linux REALLY hates it when you rip a disk out from under it. So you need something that kind of hides that eventuality from the kernel, which isn’t NFS for sure. Fiberchannel HBAs take care of it at the controller so the kernel doesn’t even know the disk is remote. We use that approach constantly at work, just not for booting. I personally manage a bunch of KVM hypervisors and setup a simple 2 disk local mirror for the OS while every bit of the data is remote on a SAN. This works well and doesn’t require any complicated configurations but it does require very expensive hardware out of reach to most of us without a corporate budget. Not to mention the noise and power to run that stuff at home.
That’s why I’m thinking iSCSI could work. You’re getting the benefit of TCP that way. You can set it up with FOSS on commodity hardware for next to nothing. But the bare minimum OS stuff would probably still need to be local somehow and loaded up in a ramdisk. But there might be some better ideas, at least more fleshed out ideas, if you look up PXE boot options. If the laptop can’t PXE boot natively I think you could rig something up of of USB. Maybe something like this. PXE boot will be only be as good as your network is fast and reliable though.
Still think this is probably more trouble that it is worth but at the same time the idea is intriguing. If you could boot your system just how you want it on any hardware you have at hand, surely that’s worth something? I imagine it’d be dog slow on WiFi though. But if you keep the base OS local and the data remote it should be workable as long as the WiFi is pretty good.
Sorry, long post, thinking out loud… For my home Linux systems I keep some data off on my local NAS. I was mounting these via NFS in fstab but I kept having kernel panics when the connection was interrupted. Tried it again with autofs but still had problems. So then I switched to SMB/CIFS, which is far more tolerant of interruptions-it is made for Windows after all, a big fan of reboots-and my problems vanished. So that could also be an option for this. It will ultimately be a cocktail of tech to make this work and if that cocktail is put together with tech that ‘buffers’, like CIFS or TCP or whatever, the end result should be better.
This is what switch_root does in initramfs. It is Linux specific, I believe, but it allows you to change what the kernel considers the / filesystem. Then you can erase or unmount whatever the original was. Usually the initramfs. It uses the mount --move support with a chroot and exec /init in there.
You can see the code at
I don’t know if that’s the original upstream code or someone’s fork but it is what came up first in search.
What I’m referring to is an unexpected loss of a ‘disk’, mount or path. Basically you have a mount and Linux loses access to that mount unexpectedly. It is unintentional so of course you’re not unmounting it. The OS can probably survive some amount of this activity as long as the files on it are not in use. But it will eventually catch up to you, at least on the systems and filesystems I frequent. I’m saying it not as some kind of expert on anything at all, just my experience. What you are describing with initramfs and chroot are different and, yes, work just fine so you’re absolutely correct. The vast majority of Linux systems I encounter use that technique in some way during boot. But once you’re up and running I think you want to be careful about those mounts or in some way build in resilience for when it does happen.
Well yes if you lose connection to your NFS server then your system hangs.
Which is actually a feature. I’ve had nothing but data loss and pain trying to use the NFS “soft” option that allows interrupting disk requests. It never works right.
But that’s just a feature of network filesystems. iSCSI is no better. If you lose connection to the iSCSI server everything hangs. Exactly like NFS. Exactly as if you just unplugged a USB hard drive.
USB is ok. But finding a good flash drive is hard - most people use nvme in a usb enclosure to get reliability and performance rather than some dram-less/slc-less USB stick.
@nvanlaerebeke how much ram does your target system have? What kind of system is it exactly?
@rcxb ath9k is there, sweet! I didn’t know you could setup ipxe project “did wifi”.
This spurred more of a discussion that I thought it would.
This is for a home setting and as hobby/interest so even if it would have a hiccup now and then it wouldn’t be as much of an issue.
I’ve been using my PI’s this way, just not over wireless, that’s where it gets tricky.
In almost all cases you can kick everything off using PXE, been using iPXE for a couple of years now. It’s just the damn wireless
@rcxb I did not know iPXE has wireless support, so will be the first thing I’ll try.
The reason this approach would be cool/come in handy are a few reasons:
No need for wires
Run around with the laptop or use it somewhere a cable cannot be used.
Would also make it so there is no need to run any wires to ‘my own’ IOT devices, the PI’s have wireless and a dongle for the older 1st and 2nd gen PI’s.
Keep the same environment across multiple systems
This way it’s possible to boot w/e device I have with the same installed programs and don’t need to maintain multiple installations that I ‘sometimes’ use.
This wouldn’t be for my main system of course, that runs from nvme
No storage requirements
All those IOT devices, or devices you just have to keep installing the same stuff on like vscode, spotify (i need my music) or my work laptop that needs to stay windows
Fast
It is network depended but have been using ISCSI and NFS for years now and it’s plenty fast for most things.
It would be over wireless this time, but an IOT device doesn’t really need a lot of bandwidth, and for the laptop it would be light use.
Example, in case of the Raspberry PI’s the SD card is just way slower then the network.
I’ve got a pretty good home network, storage is handled by a Synology RS818+ and the switches I have are all “business” grade, a Dell-PowerConnect-6248-48 , D-Link DGS-1210-10P for POE and TP-Link Auranet EAP245 for wireless.
I would not recommend anyone trying to run anything using their ISP router for example…, did not end well
So with that in mind I asked this question.
Unfortunately I’m not extremely experienced and knowledgeable about these subjects, my background is being a full stack developer who just likes to dip his toes in all of the pots and learn a bit about everything.
I’m filing away all the idea’s and adding them to my docs for when I have time to pick this up and give it a go as it’s seems it would be taking up a fair amount of time, although if the iPXE wireless pans out it might be doable.
There are some other things I need to get in order first, like fixing my backups after I moved like a year ago :(, the girlfriend has been keeping me busy.
Thanks a ton for all of the idea’s to get me to the next step(s)!
