Proxmox backup issue[solved]

I have been having issue backing a up Proxmox VM to local and network storage on my node. I can not crate a clone, template, backup or move this instance . The VM is working fine it’s my main traefik reverse proxy and docker host for my homelab. I have been wanting to replace the drive it is on with larger compacity one and am trying to crate a backup so I don’t have to start from scratch with everything I have setup. Error from proxmox is below . Thanks for the help.

[quote/] ()

Task viewer: VM/CT 125 - Backup

OutputStatus

Stop

INFO: starting new backup job: vzdump 125 --compress zstd --mode snapshot --node Homer --storage local --remove 0
INFO: Starting Backup of VM 125 (qemu)
INFO: Backup started at 2022-01-19 18:56:41
INFO: status = running
INFO: VM Name: traefik
INFO: include disk ‘scsi0’ ‘Local_VM-Storage:vm-125-disk-0’ 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating archive ‘/var/lib/vz/dump/vzdump-qemu-125-2022_01_19-18_56_41.vma.zst’
INFO: started backup task ‘a6048ee6-2ea2-407f-89eb-66ee16f5e8f1’
INFO: resuming VM again
INFO: status: 0% (516947968/536870912000), sparse 0% (100446208), duration 3, read/write 172/138 MB/s
INFO: status: 1% (5420351488/536870912000), sparse 0% (982360064), duration 36, read/write 148/121 MB/s
INFO: status: 2% (10783686656/536870912000), sparse 0% (1511763968), duration 74, read/write 141/127 MB/s
INFO: status: 3% (16158425088/536870912000), sparse 0% (1984389120), duration 116, read/write 127/116 MB/s
INFO: status: 4% (21536964608/536870912000), sparse 0% (2210033664), duration 211, read/write 56/54 MB/s
INFO: status: 5% (26873692160/536870912000), sparse 0% (2677395456), duration 561, read/write 15/13 MB/s
INFO: status: 6% (32617267200/536870912000), sparse 1% (6516051968), duration 790, read/write 25/8 MB/s
INFO: status: 7% (37694668800/536870912000), sparse 2% (11591938048), duration 799, read/write 564/0 MB/s
INFO: status: 8% (43330961408/536870912000), sparse 3% (17227628544), duration 809, read/write 563/0 MB/s
INFO: status: 9% (48432480256/536870912000), sparse 4% (22328156160), duration 818, read/write 566/0 MB/s
INFO: status: 10% (54103572480/536870912000), sparse 5% (27998711808), duration 828, read/write 567/0 MB/s
INFO: status: 11% (59299659776/536870912000), sparse 6% (33194397696), duration 837, read/write 577/0 MB/s
INFO: status: 12% (64938639360/536870912000), sparse 7% (38832119808), duration 848, read/write 512/0 MB/s
INFO: status: 13% (70044352512/536870912000), sparse 8% (43937366016), duration 857, read/write 567/0 MB/s
INFO: status: 14% (75214618624/536870912000), sparse 9% (49107095552), duration 866, read/write 574/0 MB/s
INFO: status: 15% (80901570560/536870912000), sparse 10% (54789582848), duration 876, read/write 568/0 MB/s
INFO: status: 16% (86126559232/536870912000), sparse 11% (60013445120), duration 885, read/write 580/0 MB/s
INFO: status: 17% (91384315904/536870912000), sparse 12% (65270734848), duration 894, read/write 584/0 MB/s
INFO: status: 18% (97082408960/536870912000), sparse 13% (70968160256), duration 904, read/write 569/0 MB/s
INFO: status: 19% (102272598016/536870912000), sparse 14% (76157808640), duration 913, read/write 576/0 MB/s
INFO: status: 20% (107512266752/536870912000), sparse 15% (81397006336), duration 922, read/write 582/0 MB/s
INFO: status: 21% (112781885440/536870912000), sparse 16% (86666223616), duration 931, read/write 585/0 MB/s
INFO: status: 22% (118686482432/536870912000), sparse 17% (92570218496), duration 941, read/write 590/0 MB/s
INFO: status: 23% (123893121024/536870912000), sparse 18% (97776324608), duration 950, read/write 578/0 MB/s
INFO: status: 24% (129239810048/536870912000), sparse 19% (103122870272), duration 959, read/write 594/0 MB/s
INFO: status: 25% (134542458880/536870912000), sparse 20% (108424724480), duration 968, read/write 589/0 MB/s
INFO: status: 26% (139881086976/536870912000), sparse 21% (113762619392), duration 977, read/write 593/0 MB/s
INFO: status: 27% (144962486272/536870912000), sparse 21% (117460004864), duration 1000, read/write 220/60 MB/s
INFO: status: 28% (150583902208/536870912000), sparse 22% (119477137408), duration 1140, read/write 40/25 MB/s
INFO: status: 29% (155700559872/536870912000), sparse 23% (123893915648), duration 1154, read/write 365/49 MB/s
INFO: status: 30% (161116717056/536870912000), sparse 23% (127939301376), duration 1175, read/write 257/65 MB/s
INFO: status: 30% (163876175872/536870912000), sparse 24% (129982189568), duration 1187, read/write 229/59 MB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
ERROR: Backup of VM 125 failed - job failed with err -5 - Input/output error
INFO: Failed at 2022-01-19 19:16:28
INFO: Backup job finished with errors
TASK ERROR: job errors [/quote]

What version of Proxmox are you using? What is the storage medium you’re backing up to? Local or network?

I am on Proxmox 6.2-4 , I’ve tried backing up to a local SSD “zfs”,HHD “zfs” and to a Network share “cifs”. All other VM’s backup to all of these storage mediums perfectly fine. Also all disks on the system pass S.M.A.R.T tests, Was thinking it was a mechanical failure with the drive the VM is stored on but it passes S.M.A.R.T. At a loss here with the “err -5 - Input/output error” Backup gets to 30% every time and fails.

Have you got enough storage in ‘/var/lib/vz/ for the backup to take place? or have a look a monitoring /var/lib/vz/ space as the backup hits 20%?

1 Like

Try doing a “move disk” from proxmox VM → hardware → hard disk → move disk, without deleting the source. You will move the location where your VM is running from, but still keep the old disk in place if needed.

Tried moving it and i get this error

i have around 4TB of space left on the share

Never seen that error before.

I usually remove snapshots as soon as I confirm the software I installed works and it does its job, so about 1 day later (or 3 if it’s weekend). I wonder if the snapshots are influencing that.

Is it possible for you to delete all your snapshots, shutdown the VM, then do a manual copy of the disk image with the VM off?

Is the share in the path /var/lib/vz? Because according to the above output, it looks like proxmox is trying to do an archive in /var/lib/vz, so it could be running out of space there. You could do a temporary mount into /var/lib/vz/dump to another location, like your share, using either local bind mounts or using NFS or sshfs if it’s a remote share.

Can you show us the output of df -h ?

no the share isn’t in the path of var/lib/vz . in “var/lib/vz/dump” there are just a few logs from me trying to backup

Yep, it looks like proxmox is creating an archive in /var/lib/vz, which is under / partition, which only has 115G, with 112G left.

If there is nothing important in /var/lib/vz/dump, try renaming that folder and doing a symlink instead.

mv /var/lib/vz/dump /var/lib/vz/dump_old
ln -s /mnt/pve/Network-Proxmox /var/lib/vz/dump

Then try doing the backup again, as you normally do. See if that fails. After all that is over, remove the symlink and rename the dump folder back with

rm /var/lib/vz/dump
#notice there is no trailing / after dump, do not put a trailing slash, you want to remove the symlink, not the folder
mv /varlib/vz/dump_old /var/lib/vz/dump

nope still get this " ERROR: Backup of VM 125 failed - job failed with err -5 - Input/output error" @ 30%

Is the VM turned off when you do the backup?

Last thing that I would try is to, instead of the symlink, do a temporary bind mount.

I’ve tried with it running and shutdown with all backup types ZSTD, LZO and GZIP. I have a strong suspicion this is a drive issue but my home assistant VM running on this drive backs up fine. I’ll back up all my compose files and docker volumes and start over on the new drive :confused:

1 Like

Usually i/o errors are coming from the drive. Have you ever run zfs scrubs on it?

1 Like

Found the problem :frowning: @ThatGuyB Thank you for your time and help :slight_smile:

3 Likes