Best performance deduplicated and central storage for proxmox cluster?

Operating Systems & Open Source Virtualization
,
1

Do you guys recommend anything outside of TrueNAS for deduplicated shared VM data stores for Proxmox nodes?

I am currently using Synology DS1522+ btrfs RAID5 + 10Gb on 3x 2TB SSDs and even with jumbo frames; it kind of stutters and the performance isn’t as fast as I had hoped (using ‘native’ VM hdd settings to get a real sense of the raw bandwidth)… there’s dips and non constant bursts when I was expecting to get at least 300Mbps 24/7

Screenshot 2025-05-12 at 2.38.58 AM

I’m considering Starwind vSAN Free may be a good choice, the other option is Linstor but their reputation on proxmox forums isn’t the best due to some licensing beef from many years ago apparently.

2

I recommend not using dedup, using mirrors instead of RAID, not using shared storage but local storage on Proxmox instead.

If you still really think you need HA, I would go with ceph.

3

I’m not sure I know what that means.

same

Throughput depends entirely on the kind of data you are transferring. random IO and sequential reads are totally different beasts. 300MB/s on random writes on a RAID5, which is probably memory starved because dedup,…delusional. 28MB/s might be 3-7k IOPS for all we know, which isn’t bad for that kind of network storage.
For sequential reads, 28MB/s is utter trash.

I wouldn’t put out the nuclear option just yet…Ceph is a totally different beast and some random consumer drives and general hopes for higher performance will only end in tears.

4

Well “real HA” aka “I can pull that drive, NIC, switch or whatever” and not “some parts are redundant, but please don’t touch X” is already the nuclear option.
I would highly discourage HA for any homelab and even most small businesses.

1 Like
5

Redundant switches, redundant ISP connections, UPS…it is getting expensive really fast. But HA just by having 3+ nodes allows for shared storage, load balancing “hot-swap”/offlining servers and VM/storage migration and also aggregated bandwidth benefits and bypassing chassis/architecture limitations.

Why I called this the nuclear option is because while it can solve problems, it also adds a lot more complexity and serious problems/fallout that usually isn’t needed and I always advocate for simplicity and ease of administration. Unless you really want to give yourself the business for the sake of it, not because it is the more efficient/optimal/fastest thing to do (like myself)!

2 Likes
6

Let me clarify.

I have 3 worker nodes for Proxmox. I have several “clones” and “similar” VMs across the 3 PVEs nodes - I want to save storage and migrate to NVME raid storage in a central NAS.

If I enable data deduplication, I think I may be able to store 10 copies of the same 1 “base” virtual machine on an NVME. This will make things faster because right now I am using a single old ssd 500mbps on each PVE.

My goal is NOT to have bulletproof storage, because I think I could run “poor mans HA” by having proxmox sync the zfs storage of important VMs to its local storage if the central server becomes unresponsive.

7

sounds great! So we’re talking about shared storage to fuel all VMs. I was thinking about that too last year.

Yes and no. In practice, you trade space savings from dedup for memory. “similar” != “same” and those dedup hashtables get very large very quickly. From my own experience (“similar” steam libraries + deduped dataset on all my zvols used for VMs) and other people sharing their experience…the cost in addition DRAM needed doesn’t justify the space savings on storage. Less available RAM also is less RAM for read cache.
These days, with low $/TB, getting more storage is just cheaper and more efficient 99% of the time.

Memory demand won’t go through the roof suddenly, oom daemon and swap won’t cause panic, killing threads at random, stalling the server,etc.

As the saying goes…friends don’t let friends use dedup.

Just make sure to increase swap if you wanna go for it and watch memory usage very closely :slight_smile: (and please post your dedup ratio and total space saved)

edit: I was assuming ZFS. With BTRFS, you will just cripple your read cache because BTRFS just straps everything as metadata into the page cache which is FIFO eviction. So probably has to get dedup metadata first from the drives before reading actual data. I think it is worse than ZFS at it. And ZFS dedup isn’t efficient in the first place. BTRFS just makes things slower rather than direct the burden to RAM to keep up performance.

I’d rather go harder on the compression. VM disks with OS files are very good for high compression ratios in general.

8

no way!
1.5 is probably a better guess.

But you can achieve 10x by snapshotting the base image. Also comes without any of those pesky dedupe side effects.

9

If you move all the VMs to a central server. How would your “poor mans HA” work when the central server becomes unresponsive, ProxMox would not be able to read from the central storage to sync to local.

For saving space, putting the VMs on the same storage and using Linked Clones might be better than dedup.

1 Like
10

I vaguely remember what I read a few weeks ago but its basically something like this: The VM itself is configured to use a datastore with a common name between local storage and the shared storage.

When Central storage fails, the vm would be restarted with the local zfs copy that could be as fresh as 1 minute snapshot from when it was working on the central node.

1 Like
11

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.