Hi all,
I have proxmox installed on my homelab on 2x M.2 drives and I want to replace them with 2x SATA SSDs I have lying around. I don’t want to reinstall everything from scratch on the other drives so I thought I could replace them one at a time as if they were failing using the ZFS tool but I never had to do this process and since nothing is actually failing not sure what’s the optimal way of doing it. Can somebody give me some pointers or refer me to a guide?
Thanks in advance,
Leo
if they are in a mirror, then zpool attach → detach
zpool attach poolname m2_device new_sata, one sata drive to one m.2
should run a resilver; better to wait for it to complete,
and zpool detach poolname m2_device the m.2
You can attach both sata’s at the same time, but I do one at a time…
zpool replace poolname olddisk newdisk should do the job. It doesn’t care what the condition of the old drive is.
But zpool attach works just as well. Just takes more steps, but is tried and trusted among mirror-enthusiasts 
with SSDs, should be damn fast and painless (sequential resilver)
Choices
edit: just make sure you unplug the M.2 only after the stuff is done. If you use zpool attach, you end up with a 4-way-mirror with 2xM.2 and 2xSATA SSDs. Only after you ran zpool detach are the M.2 safely evacuated (yes, that’s the term) from the pool.
Thank both of you, and yes, they are in a mirror, I thought I put that, my bad 
So for what I understand in both scenarios I have to have all 4 disks connected and after running the commands, I physically remove the 2xM.2., right?
The way to reference the disks for the commands is with /dev/nvmXnY (basically how they show up on the Disks section within Proxmox GUI)? What’s the default poolname if I’m booting proxmox from these drives?
Sorry for the trivial questions, don’t want to mess it up.
Thanks!
zpool status will show you all pools on the machine with topology and, well…status.
No, it does not!
I was wrongfully under the impression that this is enough, but it isn’t.
The documentation is not great IMHO, when it comes to this.
You not only need to use the ZFS replace command (which basically gets your mirror up and running again) you also need to change the bootloader. Otherwise you can’t boot from both.
Scroll down to “Changing a failed bootable device”
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs
You most likely need this:
sgdisk <healthy bootable device> -R <new device>
sgdisk -G <new device>
zpool replace -f <pool> <old zfs partition> <new zfs partition>
proxmox-boot-tool format <new disk's ESP>
proxmox-boot-tool init <new disk's ESP> [grub]
oh right. Proxmox has to make a separate (non-ZFS) boot partition because the Kernel hates ZFS. Forgot about that.
but dd should just work then, right?
You want to offline the pool somehow before using dd, which is problematic on a running boot drive.
Yeah, stick to that Proxmox documentation.
Thank you for pointing that out. I have single SSD with ZFS on my Proxmox and dd cloned replacement disk ready (kinda “clunky manual backup”, haven’t come up with anything better yet)
Hmmm… could be. But DD would take longer, because it also copies zeros, right?
That is IMHO a missing feature of Proxmox.
OPNsense or TrueNAS offer a one click, one file backup.
Proxmox could offer the same. Then you could just reimport the config backup and after that the pool.
I am not sure if you not simply could copy /etc/pve/.
But yeah, it’s a shame there is no official way.
Based on the documentation, I shouldn’t need to add “grub” to this line because I have systemd-boot. Am I reading it correctly?
I seem to remember to have read somehwere that copying /etc/pve does the trick, but a quick search right now didn’t confirm it
this worked like a charm! thanks!
Only thing missing now is to remove from /etc/kernel/proxmox-boot-uuids the M.2 ids. I can do it manually but anybody knows if there’s a another way?
That is right, just use:
proxmox-boot-tool init <new disk’s ESP>
Really isn’t written well 
Strange, it did it for me. Maybe after a reboot?
really? not for me, I restart after replacing the first one, the id was still there and even after the second one as well 