Is it possible to restore zfs-pools that are on other drives if a re-install of the proxmox?
I was reconfiguring several systems on one of the proxmox hosts after the IP change no longer connect…but I can see the MAC Address on the network, just not connecting. Right now working through the IPMI interface, but thought rebulding the OS drive and recovering the zfs-pools would be an option…but is it?
Absolutely.
I did this about a year ago when I wanted to replace my mirrored boot drives with a smaller pair of drives. (ZFS allows growing pools, but not shrinking pools).
I made a copy of the entirety of /etc, /var/lib/pve-cluster/config.db, exported the ZFS pools, reinstalled, re-imported the ZFS pools and then overwrote the default config with the one from the old install, and the system was right back where it previously was.
From the command line just type “zpool export poolname”
This will unmount/remove the pool from the system and prepare it to be imported on another system.
You can then connect the drives to the other system (or as it were in this case, the same system) and do “zpool import poolname”
If a host becomes unbootable or otherwise unusable, zfs will complain that the pool is in use on another system when you try to import it. You can usually still import pools that have not been properly exported, by using the “-f” option to force it at import time.
All of that said, Proxmox can be a little tricky when it comes to exporting ZFS storage. It has a bunch of services running to provide system statistics, and these will often make the ZFS pool appear busy, and unable to be exported.
If you run into this problem, you will need to find the offending service and manually shut it down using systemctl, before you can export the pool.
Unfortunately I can’t remember which service it was that caused the problem. It was an annoyance for me last time I needed to do this, but eventually after shutting them down I was able to export the pools.
After importing the pool(s) on the new proxmox install, if you are not importing the proxmox configuration, you may have to manually re-add the pools as storages in the web interface again, or the management interface won’t see them.
I can’t remember this part exactly.
A slight aside, but you may want to consider the below.
If the system is part of a Proxmox cluster, if you don’t duplicate the config, it will no longer be a part of that cluster upon reinstall.
You can duplicate the complete working configuration of that proxmox node by copying /var/lib/pve-cluster/config.db (and probably /etc too) to the new install, but then you might just be duplicating whatever networking issue you are trying to get rid of.
Unfortunately I did this quite a while ago, and while I am intimately familiar with managing ZFS pools from the command line, I can no longer remember the Proxmox particulars perfectly, as I do that so infrequently.
Hopefully this will at least be enough to point you in the right direction.
Too bad you didn’t back up your config before messing with it.
Actually, I just had a thought.
Do you boot this proxmox system off of zfs? If you do, did you ever snapshot the root pool (rpool by default)?
If you did, you can probably retrieve the old network configuration from that snapshot.
You can browse the content (read only) of ZFS snapshots without restoring them by navigating to mountpoint/.zfs/snapshot/snapshot name
From there you can look at the old content of files, and even copy the files out of the old snapshot without restoring the whole thing and overwriting the things you don’t want to change.
Maybe looking at “/.zfs/snapshot/snapshot name/etc/network/interfaces” will give you some hints as to how you used to have it working?
Heck, with a little luck you could just overwrite your current “/etc/network/interfaces” file with the one found in “/.zfs/snapshot/snapshot name/etc/network/interfaces” and - network wise - you will - after a reboot - be right back where you started?
(except for /etc/hosts, /etc/hostname, and /etc/resolv.conf Depending on what you have changed, you may have to restore those too)
Best of luck!
@mattlach I do use the default rpool for the OS drive, but I use a separate zfs pool on separate drives for my VMs. At this point the only connectivity I have to this particular host is through the IPMI. I was reworking the networking/subnets and as part of that was setting up a Proxmox backup server. Th eother hosts did not have an issue with the change in IP, but this particular host has…and I did not get the backups server in play yet…
Well, if you did snapshot the rpool at some point, you should at least be able to get at the files in /etc that brings the network back to where it was by restoring /etc/network/interfaces, /etc/hosts, /etc/hostname and /etc/resolve.conf, and then maybe you can try over again from there?
Otherwise, re-installing isn’t that difficult, but preserving config.db and all of /etc is definitely recommended. You don’t have network access to the box, but maybe you can just rsync them to a temp folder on the ZFS pool before exporting it?
I wish you the best of luck!
As previously mentioned none of this stuff is really fresh in my memory, but I may be able to give you enough info to point you in the right direction to find the rest of it on your own.
How did you reimport the VMs once the restoration of the ZFS pool was complete? I also wanted to say thank you, you have been helpful…
All of the VM config info is stored in the config.db file, so as long as you copy that over from the old install, everything will be there just as you left it (provided it can find the drive image/devices in the ZFS pool, which it should if you import that first. Automatically the ZFS pool should have the same mount location as before, so this should just work.
It’s possible some things are stored in various locations in /etc as well, so I copied that over as well, but in your case you may want to avoid the networking related config files as that’s what you are trying to avoid replicating.
Some of the VM and Container config stuff is stored under /etc/pve, but my recollection is that this folder is actually mounted in etc using fuse from inside config.db, so you shouldn’t have to worry about those. Still, it’s a good idea to back up all of /etc just in case something breaks.
You may want to disable autostart of VM’s just in case, and reenable it later. But this is unlikely going to be a problem, unless you use PCIe passthrough. But this is mostly a concern when moving from one server to another that doesn’t have the same PCIe layout. For a reinstall on the same hardware, it is unlikely to be a problem.
The concern here is that with different hardware, what used to to be the PCIe device you wanted to pass through, is now actually something important for the host, and with autostart it then yanks thank PCIe device away from the host right on boot, causing everything to crash.
Seems unlikely with what you are doing on the same hardware though.
Also, I’m sure there is a way to import VM’s manually should you not copy the config.db file, but I have never done it, so I am not entirely sure how.
Worse comes to worse, if you can’t figure it out, you could just create them again, and either select the correct ZFS block devices from the previous VM’s as your boot drives, or dd the contents over from the old ones to the new ones, though this is likely a pain in the ass if you have many VM’s.
So I copied the directories to a folder I created in the zfs pool in question. SO my question is how can I list or navigate the zfs pool so I can restore the vm configs and other data? I have been trying to find documentation on it but have not found it. I was able to create the directory and ls/cp to it but now it is not showing.
It seems like the new install is not properly mounting or seeing the files I have:
root@pve2:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
vm-pool 5.00T 2.13T 96K /vm-pool
vm-pool/temp 96K 2.13T 96K /mnt/temp
vm-pool/vm-1201-disk-0 109G 2.22T 9.80G -
vm-pool/vm-1900-disk-0 81.3G 2.21T 56K -
vm-pool/vm-1900-disk-1 81.3G 2.21T 56K -
vm-pool/vm-2001-disk-0 3.33M 2.13T 104K -
vm-pool/vm-2001-disk-1 109G 2.22T 9.49G -
vm-pool/vm-3001-disk-0 102G 2.20T 30.0G -
vm-pool/vm-3002-disk-0 148G 2.26T 14.7G -
vm-pool/vm-3003-disk-0 145G 2.26T 8.50G -
vm-pool/vm-3004-disk-0 60.9G 2.18T 3.69G -
vm-pool/vm-3005-disk-0 81.3G 2.20T 4.93G -
vm-pool/vm-3006-disk-0 81.3G 2.19T 16.2G -
vm-pool/vm-3301-disk-0 156G 2.22T 41.4G -
vm-pool/vm-3302-disk-0 148G 2.22T 39.5G -
vm-pool/vm-3304-disk-0 203G 2.31T 14.3G -
vm-pool/vm-3305-disk-0 111G 2.23T 9.10G -
vm-pool/vm-3306-disk-0 81.3G 2.20T 5.83G -
vm-pool/vm-3308-disk-0 81.3G 2.20T 6.41G -
vm-pool/vm-3309-disk-0 81.3G 2.19T 11.8G -
vm-pool/vm-4101-disk-0 60.9G 2.18T 5.63G -
vm-pool/vm-4102-disk-0 102G 2.21T 17.3G -
vm-pool/vm-4201-disk-0 52.8G 2.16T 14.1G -
vm-pool/vm-4202-disk-0 81.9G 2.20T 7.10G -
vm-pool/vm-4205-disk-0 102G 2.22T 6.96G -
vm-pool/vm-4208-disk-0 81.3G 2.20T 1.92G -
vm-pool/vm-4211-disk-0 81.3G 2.20T 4.56G -
vm-pool/vm-4212-disk-0 81.3G 2.20T 3.14G -
vm-pool/vm-4213-disk-0 60.9G 2.18T 1.68G -
vm-pool/vm-4213-disk-1 60.9G 2.18T 1.87G -
vm-pool/vm-4214-disk-0 60.9G 2.18T 1.70G -
vm-pool/vm-4215-disk-0 60.9G 2.18T 1.25G -
vm-pool/vm-4215-disk-1 60.9G 2.18T 1.25G -
vm-pool/vm-4215-disk-2 60.9G 2.18T 1.25G -
vm-pool/vm-4216-disk-0 60.9G 2.18T 1.23G -
vm-pool/vm-4217-disk-0 81.3G 2.20T 7.52G -
vm-pool/vm-4218-disk-0 81.3G 2.20T 3.95G -
vm-pool/vm-4219-disk-0 81.3G 2.20T 1.77G -
vm-pool/vm-4220-disk-0 60.9G 2.18T 1.93G -
vm-pool/vm-4221-disk-0 81.3G 2.20T 3.47G -
Even though the files show in the list, and I can see the files after mounting the ZFS pool, I cannot move the data across. If i look at the mount points, the data is not showing up…
I am kinda stuck because I am not sure what to do next and every document I have found does not show a direction to resolve this issue.
Sorry, didn’t get a notification of these posts for some reason.
It looks like you have properly set the mountpoint.
There are two other properties related to mounting that can/need to be set:
canmount and mounted
Well, I think “mounted” is not settable, it just displays the current status.
You can set the mount settings using the zfs set command.
-zfs set canmount=yes ‘name’
-zfs set mountpoint=path ‘name’
To see what they are currently set at, either individually do a:
or just
That might be it. Sometimes you have to tell zfs to mount them manually the first time (then it is done automatically after that, (unless canmount is set to noauto.)
Also try issuing the zfs mount command:
to tell it to mount it
No worries I will try it…but what is mounted does not contain any of the files, almost like it zfs has a parallel file structure that I can not access…
This is what I get…
root@pve2:/vm-pool/images# zfs get all |grep -i mount
vm-pool mounted yes -
vm-pool mountpoint /vm-pool default
vm-pool canmount on default
vm-pool/temp mounted yes -
vm-pool/temp mountpoint /mnt/temp local
vm-pool/temp canmount on default
root@pve2:/vm-pool/images# ls -lah
total 9.0K
drwxr-xr-x 2 root root 2 Apr 17 07:33 .
drwxr-xr-x 7 root root 7 Apr 17 07:33 ..
root@pve2:/vm-pool/images#
If I do a zfs list I get the following and can see in the Web UI:
root@pve2:/vm-pool/images# zfs list
NAME USED AVAIL REFER MOUNTPOINT
vm-pool 5.00T 2.13T 104K /vm-pool
vm-pool/temp 96K 2.13T 96K /mnt/temp
vm-pool/vm-1201-disk-0 109G 2.22T 9.80G -
vm-pool/vm-1900-disk-0 81.3G 2.21T 56K -
vm-pool/vm-1900-disk-1 81.3G 2.21T 56K -
vm-pool/vm-2001-disk-0 3.33M 2.13T 104K -
vm-pool/vm-2001-disk-1 109G 2.22T 9.49G -
vm-pool/vm-3001-disk-0 102G 2.20T 30.0G -
vm-pool/vm-3002-disk-0 148G 2.26T 14.7G -
vm-pool/vm-3003-disk-0 145G 2.26T 8.50G -
vm-pool/vm-3004-disk-0 60.9G 2.18T 3.69G -
vm-pool/vm-3005-disk-0 81.3G 2.20T 4.93G -
vm-pool/vm-3006-disk-0 81.3G 2.19T 16.2G -
vm-pool/vm-3301-disk-0 156G 2.22T 41.4G -
vm-pool/vm-3302-disk-0 148G 2.22T 39.5G -
vm-pool/vm-3304-disk-0 203G 2.31T 14.3G -
vm-pool/vm-3305-disk-0 111G 2.23T 9.10G -
vm-pool/vm-3306-disk-0 81.3G 2.20T 5.83G -
vm-pool/vm-3308-disk-0 81.3G 2.20T 6.41G -
vm-pool/vm-3309-disk-0 81.3G 2.19T 11.8G -
vm-pool/vm-4101-disk-0 60.9G 2.18T 5.63G -
vm-pool/vm-4102-disk-0 102G 2.21T 17.3G -
vm-pool/vm-4201-disk-0 52.8G 2.16T 14.1G -
vm-pool/vm-4202-disk-0 81.9G 2.20T 7.10G -
vm-pool/vm-4205-disk-0 102G 2.22T 6.96G -
vm-pool/vm-4208-disk-0 81.3G 2.20T 1.92G -
vm-pool/vm-4211-disk-0 81.3G 2.20T 4.56G -
vm-pool/vm-4212-disk-0 81.3G 2.20T 3.14G -
vm-pool/vm-4213-disk-0 60.9G 2.18T 1.68G -
vm-pool/vm-4213-disk-1 60.9G 2.18T 1.87G -
vm-pool/vm-4214-disk-0 60.9G 2.18T 1.70G -
vm-pool/vm-4215-disk-0 60.9G 2.18T 1.25G -
vm-pool/vm-4215-disk-1 60.9G 2.18T 1.25G -
vm-pool/vm-4215-disk-2 60.9G 2.18T 1.25G -
vm-pool/vm-4216-disk-0 60.9G 2.18T 1.23G -
vm-pool/vm-4217-disk-0 81.3G 2.20T 7.52G -
vm-pool/vm-4218-disk-0 81.3G 2.20T 3.95G -
vm-pool/vm-4219-disk-0 81.3G 2.20T 1.77G -
vm-pool/vm-4220-disk-0 60.9G 2.18T 1.93G -
vm-pool/vm-4221-disk-0 81.3G 2.20T 3.47G -
vm-pool/vm-4222-disk-0 60.9G 2.18T 2.78G -
vm-pool/vm-6001-disk-0 43.7G 2.15T 20.4G -
vm-pool/vm-6002-disk-0 39.2G 2.14T 15.7G -
vm-pool/vm-6003-disk-0 60.9G 2.17T 18.4G -
vm-pool/vm-6005-disk-0 32.5G 2.14T 21.6G -
vm-pool/vm-9000-disk-0 3.05G 2.13T 1.50G -
vm-pool/vm-9001-disk-0 25.4G 2.14T 7.90G -
vm-pool/vm-9002-disk-0 81.3G 2.20T 1007M -
vm-pool/vm-9003-disk-0 90.8G 2.21T 9.61G -
vm-pool/vm-9004-disk-0 246G 2.27T 85.2G -
vm-pool/vm-9004-disk-1 124G 2.21T 42.9G -
vm-pool/vm-9004-disk-2 17.5G 2.14T 7.35G -
vm-pool/vm-9004-disk-3 10.2G 2.14T 440K -
vm-pool/vm-9004-disk-4 30.3G 2.15T 9.98G -
vm-pool/vm-9005-disk-0 3.05G 2.13T 1.50G -
vm-pool/vm-9006-disk-0 50.8G 2.17T 7.63G -
vm-pool/vm-9006-disk-1 4.06G 2.13T 56K -
vm-pool/vm-9006-disk-2 102G 2.20T 30.4G -
vm-pool/vm-9007-disk-0 102G 2.20T 29.9G -
vm-pool/vm-9008-disk-0 35.4G 2.15T 13.1G -
vm-pool/vm-9009-disk-0 20.3G 2.13T 14.2G -
vm-pool/vm-901-disk-0 3.85M 2.13T 108K -
vm-pool/vm-901-disk-1 246G 2.26T 41.8G -
vm-pool/vm-9010-disk-0 20.3G 2.13T 11.5G -
vm-pool/vm-9011-disk-0 8.13G 2.13T 1.21G -
vm-pool/vm-9012-disk-0 12.2G 2.14T 2.10G -
vm-pool/vm-9013-disk-0 23.2G 2.15T 2.88G -
vm-pool/vm-9014-disk-0 89.4G 2.20T 8.04G -
vm-pool/vm-9015-disk-0 13.2G 2.14T 2.09G -
vm-pool/vm-9016-disk-0 106G 2.21T 24.3G -
vm-pool/vm-902-disk-0 3M 2.13T 96K -
vm-pool/vm-902-disk-1 102G 2.20T 25.7G -
root@pve2:/vm-pool/images#
That is very strange. Based on this you should be seeing your files in /mnt/temp.
It looks like it is properly mounted.
Maybe the backup failed for some reason? Or you inadvertently copied them to a different location?
Did you verify the pool was actually mounted before you copied the files to the temp folder?
I don’t know what to suggest at this point.
Just in case the permissions and ownership are wacky, can you try becoming root and see if you see anything then?
