My FreeNAS box completely locked up during a file transfer. Initially I thought it was a Windows issue and started troubleshooting that, but apparently my Linux laptop doesn’t see the NAS either.
I verified that the ethernet cable works by connecting another machine and scanning my network for that one.
The NAS beeps correctly when I try to boot it, but I get no access via my browser and the NAS doesn’t respond when I press the power button to shut it off. I have to long-press it instead.
This makes me think it simply doesn’t boot into the OS, so I’m suspecting that the USB stick which houses the FreeNAS install has failed.
Can I boot the NAS from a new stick and import my (encrypted) pools to that one? If so, is there anything special I need to keep in mind?
I have several very recent backups, so data loss is not a concern. I prefer to solve it without resorting to backups though, copying 12TB over a Gigabit network is a PITA.
the relevant specs :
MoBo : Intel Server S1200V3RPS
CPU : i3-4170
RAM : 32GB (4x 8GB) Kingston ValueRAM DDR3-1333 ECC
HBA : LSI 9211-8i in IT mode
OS : FreeNAS on USB stick (plugged directly into MoBo)
Pool 1 : 2x Crucial MX200 250GB (mirrored)
Pool 2 : 4x Samsung 850PRO 256GB (RAIDZ)
Pool 3 : 7x WD Red 4TB (RAIDZ2, connected via HBA)
If you didn’t export your pool with zfs export then when you zfs import your pool give it the -f flag and it should work. Since you encrypted the data make sure you know the key.
Sorry for not responding sooner, we had a bit of a storm here and everything tech-related got pushed back because of that.
I did a fresh install of FreeNAS on an old SSD and plugged that into a spare SATA port. Still nothing. The NAS does the usual “bip … bip bip bip” that I get with a normal boot … but afterwards nothing happens. No login screen on the Web UI, it doesn’t show up on the router either.
So the USB stick might be fine after all. I’m not sure if that’s good news. Now I have to keep looking and things may get expensive.
I never got it to output any signal on the MoBo’s VGA port even when new (CPU is an i3-4170 which has Intel HD4400 graphics), so I’m not sure how to troubleshoot this if it’s a hardware or BIOS issue.
I still have an old GTX 550Ti, not sure if that will make a difference. Guess I’ll find out tonight.
EDIT :
Nope, I only have the passphrase. Losing the data is no big deal though, the backups of my backups have their own backups.
A true pleasure to have a discussion with a NAS user who also backs up properly
I’m assuming you are leaving it for “bloody ages” before trying to bring up the webUI. Booting FreeNAS takes “bloody ages”. If so and you aren’t seeing a DHCP connection On your router or a ping on the static IP (if set) it may be the NIC.
Plug in the Gpu to test the hardware (try a Linux usb pen for quicker testing) that should tell you what is not working.
If it is the hardware you can just move the original usb pen and pools to a new Mobo and boot. I’ve done this 4 times now and it was fine every time. Thanks BSD!
Normally it takes around 6 minutes between pushing the power button and getting the logon screen in the web UI. If it’s not up by then, I typically wait until the 15 minute mark before attempting to reboot the NAS or the router, or unplugging and replacing ethernet cables.
I already tried both ethernet ports on the back of the NAS, no difference.
What I did notice just now is that with the SSD it immediately turns off if I push the power button. That is slightly different behavior from the USB stick, where it just doesn’t respond to the button at all.
The SSD is connected to one of the ports of the HBA, so perhaps the MoBo doesn’t recognize it as a bootable device. I’ll try plugging it directly into the MoBo and see if that makes a difference.
All I’m aiming for right now is to see a login screen on the PC. Once I have that, I have no issues wiping the entire thing and starting over from scratch. It’s more than a day of copying data, but that’s no big deal.
I still feel like I should be able to do better though. I still do my backups manually (last one was Jan 21st), and the NAS crapped out on me literally at the start of a fresh backup. Luckily I only really did some cleaning up in the last weeks and didn’t add any data.
Instant power off is a sign of a failed boot so it is just sitting at a bios prompt. Plugging a gpu in will tell you what is Broken. If not try resetting cmos or a battery pull. It may just be that the change of boot device has confused it (I had a haswell system that hated any config changes, crappy gigabyte board)
I managed to get display out from the motherboard’s VGA port. No idea why it didn’t work when I built the damn thing.
For some reason on my first attempt with a monitor it booted just fine, but threw an error because of an unreadable sector on da6. I then went to my PC to access the web UI, which was slow as hell. Like 56K slow.
I tried unlocking my “data” pool, which is on a pair of SSDs. No dice, after clicking the unlock icon I get greeted by a small loading screen instead of a password prompt.
Eventually I decided to reboot, but the NAS wouldn’t shut down. So I forced it and now apparently da7 went into write protect mode and the whole thing fails to boot.
In between the first and 2nd picture I did get the FreeNAS splash screen, so it does recognize the stick as a bootable device and attempts to boot from it.
At this point I’m considering just booting the NAS in Linux, formatting all the good drives, ripping out the da6 one and reconfiguring the NAS from scratch using the other FreeNAS stick I prepared earlier.
That’s the USB boot drive right (da7:umass-sim - umass-sim means USB mass storage)? If you have an SSD do a fresh install of FreeNAS to that. All your settings should be backed up on the big pool. You should be able to just import the pool and go about your business.
Apparently this error can happen if you use the wrong SSD. FreeNAS seems to be picky when it comes to flash controllers. One of the mods in the ixsystems forum suggests using an USB stick instead, but I’m doing that already.
Other people say it could be the result of plugging the ethernet cable into the wrong port, but I already verified that this isn’t the problem either.
Either way, it looks like there’s an issue with the original stick after all.
Without the encryption keys I’m not going to be able to import my old pools into the new install, so ly data is no longer any concern. I’ll just recover it from backups.
that would be weird, seeing as I had a HDD which was labeled da7. Perhaps FreeNAS switched the drives around? It would explain why unplugging 2 drives caused the error to move to da5.
I’d look it up in the UI’s disks screen, but that’s not an option if it completely refuses to boot.
If the stick did indeed go into read-only mode, I should be able to sort that out with Gparted. Time to fire up the laptop.
EDIT: nope, apparently not.
I already have a fresh install on a 16GB PNY stick which I had used for FreeNAs before, as well as on a 120GB OCZ Vertex 4 SSD. I could try it on a spare 850EVO, those are verified to work as boot drives.
Personally I don’t have an issue with using SSDs. I always ran FreeNAS from a USB stick because
the motherboard is set up for running from a USB stick (it has a USB port directly on the motherboard itself, for this very purpose)
when I was researching all the options prior to building the NAS, everyone suggested using USB instead of SSD because after the boot there’s little to no difference in terms of performance or longevity
and mostly because I simply had no free SATA ports anymore. Right now I do have one free port, but I do intend to replace that dead drive eventually. (or replace all HDDs with QLC SSDs if the prices drop a bit more or one more HDD dies on me)
There is some old wisdom that hasn’t died fast enough about the longevity of USB devices for FreeNAS boot media. FreeNAS used to be effectively read-only on the boot media and used ramdisks to operate from. This is no longer the case, FreeNAS now uses ZFS on the boot device and as such it really benefits from more durable storage with plenty of space to allow for updates to create new boot environments, snapshots, etc. Beyond that there are occasionally a few quirks about interactions between BIOS/EFI and USB chipsets and USB media that are not always handled well by the new boot loader since FreeNAS 11.2.
In your system you might benefit from another HBA (4i maybe?) for the pools. It is true that some onboard SATA controllers have bugs with things like NCQ TRIM that can cause problems, but if you have that specific issue I would gladly work to find either a driver fix or I know the quirks to set for working around it.
A 120GB SSD is well within the range of affordability these days, and that will be more than enough over-provisioned to be plenty durable.
Ah, okay. Yeah, that makes sense. So the USB stick wore out after all. That would explain things.
I built the NAS back in 2015, that “old wisdom” might still have been valid back then.
In the meantime I found out why the new installs wouldn’t boot. Apparently they simply had no network connections.
Resetting the network connections in the console forced FreeNAS to do a DHCP request, after which the NAS received the usual IP that I had assigned to it in my pfSense router. Gonna do a restart real quick to see if it sticks, wouldn’t want to have to do this every time I boot the NAS.
EDIT : That was a bit of a struggle, had to dig a bit deeper in order to make the settings stick.
So I now have access to the NAS again. Time for a fresh start, this is a good opportunity to rethink how I had the NAS set up and perhaps make some changes.
Perhaps an extra HBA indeed. Or I could replace some of the old small SSDs with new larger ones and rearrange the pools to suit.
Question though, you mentioned using mirrored SSDs for the FreeNAS install itself. How do you set up that kind of install? I only saw options to install the OS onto one drive, not two.
I ended up buying a pair of new 4TB WD Reds to replace the one that died last year and the one that’s now throwing up the unreadable sector error. I’ll try to sort that error out later and keep it as a spare drive.
As for the OS drive problem, I decided to go with mirrored 256GB 850PROs. I had 4 of those in RAIDZ for my music pool. Two of them will be used for the OS, the other two will probably become external drives or spares.
The old music pool will move to a pair of (newly bought) mirrored 1TB Crucial MX500s. That way I keep using the same amount of SATA ports, so no need for an extra HBA.
Now all I need is good enough weather to take the NAS outside and blow the dust out, then I can swap the drives out (and probably overhaul my cable management).
I got the NAS up and running again and all data has been restored from backups, so it’s time for a little update …
Those are the new 1TB Crucial SSDs. SMART tests are fine, scrubs don’t help either.
After some digging I found what seems to be the problem. Apparently this error is very common with this generation of MX500 on FreeNAS and FreeBSD.
Currently contemplating how I’ll handle this. I could buy a pair of 1TB 860EVOs and install those instead, that would be the best way to get rid of the error.
Not sure what I’d do with these MX500s then though.
