Erying MODT Proxmox Cluster

My New Proxmox Homelab Cluster Build with Erying MODT Boards and Intel 11980HK CPUs

I wanted to share my new Proxmox homelab cluster build featuring Erying MODT boards with Intel 11980HK CPUs. I’m currently in the testing phase with one system but plan to expand with two more systems to form a redundant cluster. Inspired by Craft Computing’s vide on this topic. Here’s the breakdown:


Hardware Overview

  • Motherboard: Erying MODT with Intel 11980HK CPUs (Engineering Samples)
  • PCIe Slot: 1 x16 PCIe 4.0 (currently unused, reserved for future expansion).
    This slot was planned for an Intel Arc card for transcoding, but onboard graphics has been more than enough.
  • M.2 Slots:
    • 4x4 PCIe 4.0 for a 256GB boot drive
    • 4x4 PCIe 3.0 with an M.2 to SFP+ Intel NIC
      (Note: I discovered during testing that none of my Mellanox ConnectX cards are compatible with Proxmox v8)
  • RAM: 32GB per node
  • Cooling: Intel stock coolers I had laying around from past projects.
    One node running all my services hasn’t driven the system past 90°C.
  • Chassis: 2U Rosewill cases for each system

Storage and Networking

  • Container Storage: NFS-based, hosted on a TrueNAS Scale server
  • Backups: Stored on a separate SMB dataset that is replicated off-site for redundancy

Services

I run all my services in Docker containers on LXC containers, with GPU passthrough for transcoding. Here’s what I’m running:

  • Sonarr
  • Radarr
  • SABnzbd
  • Emby
  • Code-Server
  • Tdarr (one node per Proxmox system)
  • Homepage

All services (except the Tdarr nodes) are set up for high availability.


Transition from Old to New

Old System:

  • CPU: 14-core Xeon (X99-based) in a 4U chassis
  • Networking: SFP+ NIC
  • GPUs: 1660 Ti and 1080 for transcoding (each could handle ~300 fps but drew 150W each!)
  • OS: Running Unraid

New System:

  • Performance: Each node achieves ~320 fps in transcoding with Intel GPU passthrough while drawing only 25W total per system (CPU + GPU, not including NIC, SSD, etc.).
  • Efficiency: With three nodes, I expect to exceed the performance of my old system, with the added benefits of redundancy, high availability, and significantly lower power consumption.

Why the Upgrade?

  • Reduced Power Consumption: From two GPUs drawing 300W total to three nodes drawing a combined 75W.
  • Smaller Footprint: 2U cases vs. a 4U chassis.
  • Improved Redundancy and Scalability: The cluster setup ensures no single point of failure, and services remain highly available.

Next Steps

  • Finalize testing and bring in two more nodes to complete the cluster.
  • When I build the rest of the systems, swap out the fans in the test system to Noctuas.
    The Molex fans are louder than they should be.
  • Consider future expansion using the empty PCIe slots (open to suggestions here!).

I’m really excited about how this build is coming together, especially given the lower power consumption and improved performance. I’d love to hear your thoughts, feedback, and ideas for further optimizations!

2 Likes

Why?

I write that the one of the two main reasons is Efficiency, why would you run three instead of one note? Is HA really that important? And do you really go for real HA? Network and switches also redundant?

I get why anyone would go the cluster route because of the learnings. But it is expensive and inefficient and slowish.

Great questions and points!


The Fun and Learning Factor

You mentioned one of the main reasons, and I’ll admit it’s not fully based on logic: it’s fun and a great learning experience. Running a cluster is an excellent way to gain hands-on knowledge, and I get to explore new configurations and setups. Despite adding multiple nodes, the cluster still consumes less power than my old single system, and I was looking for an upgrade anyway.


The Logical Reasons

  1. High Availability (HA) for My Needs
    I travel often for work, and during my homelab history, my dedicated storage and switches have rarely gone down while I’m away. However, I’ve had several instances where my container or VM host went down, locked up, or I misconfigured something while playing remotely with my homelab.

    These failures lead to text messages and calls about Emby, home automation, photos, and other services being down. Fixing the issues requires walking my non-technical partner through troubleshooting over FaceTime—a frustrating process for both of us.

    With this new setup:

    • I can take down 1–2 hosts for maintenance or upgrades without service interruption.
    • A host can fail, and I won’t need to rush to fix it.

    While this isn’t “true HA,” it’s exactly the level of availability I need based on the challenges I’ve faced.

  2. Cost Efficiency
    While three machines are more expensive than one, each node costs about $500 (including parts I already Had). This makes them the cheapest container/VM hosts I’ve built in years, especially since I don’t need to add NVIDIA GPUs or a platform with enough PCIe lanes for GPUs, network cards, etc.

    Overall, the cost for three nodes is close to what I spent on my last single host—but now I have three hosts instead of one.

  3. Scaling Out Instead of Up
    Instead of scaling up to a more powerful single host, I chose to scale out with multiple nodes. While it might seem inefficient, I believe it’s worth the trade-off. Each host runs fewer services, which should allow them to stay in a lower power state. Combined, the new setup is significantly more efficient than my old single host.


Performance and Speed

  • Efficiency: These new nodes are collectively far more efficient than my old host.
  • Slowish? Not Likely:
    • Spreading the load across three servers theoretically improves overall performance by providing each service access to a larger pool of CPU and RAM, as long as these were the limiting factors.
    • My previous experience with cheap thin clients taught me that slow or sluggish behavior often stems from:
      • Network Latency and Bandwidth: Why I’ve ensured this cluster uses 10G or higher networking.
      • Storage Latency: This time, I avoided clustered Ceph storage and upgraded my TrueNAS system to handle the load:
        • Optane metadata drives
        • NVMe L2ARC pool
        • 256GB of RAM
        • Sufficient disks to saturate 10G networking

I hope this explains my reasoning! Let me know if you have further questions or suggestions.

1 Like

That one I can understand.

That is what IPMI, VPN, SSH is for.

I guarantee you, the added complexity will add more downtime than it can shave off.
At my old workplace, we had two hospitals, connected with each other by two fiber lines. These lines went two different routes in case some bulldozer or something cuts a line. We had a Cisco HA Network. Sometimes they would loose the heartbeat for whatever reason and both think that the other one failed.
My boss said this funny sentence: “Uptime would probably be better, if I just drive to our office and change the plugs by hand in the rare case someone cuts our cable”

And like you said, your setup is not really HA. I bet if you do that, your Switch will act up, or its PSU goes bad or something.

I kinda to the same by using two hosts. Big tank TrueNAS with special vdevs and lots of ARC. Proxmox with fast NVMEs in mirrors. Best of both worlds. Your 10G iSCSI will still perform extremely poor compared to local NVMEs.

Wait, how are you achieving HA when you don’t use Ceph?
Centralized storage from TrueNAS?

I know this sound overly negative, but I would still go for it :grin:
Had it for some time and then decided it was not worth it.
Anyway, if you already have loads of “production” stuff running like Emby, you would probably be better off by just using three 50$ Dell Office SSF PCs and have fun with them.

I have used IPMI before with used HP or Dell servers, and while it was awesome, it wasn’t worth the loudness and heat of datacenter rack servers for me. That’s part of the reason I transitioned to custom systems with mobile-based processors. As I get older, I’m less willing to deal with the heat and noise.

I could invest in some of the newer, cheap IP KVMs and might still do that with my new system. However, I still rely heavily on my VPN and SSH for most issues I can fix remotely. Early in my career, I learned the hard way about being careful with commands like rm -r * in the root directory. After running that command, I had to wait seven days before I could get home to fix the server. SSH and VPN didn’t help much in that case, which is why my next server included IPMI.

With the cluster, if I accidentally mess up a node, it becomes a problem I can deal with when I get home. This approach also helps in the case of a hardware failure, allowing the rest of the cluster to keep running.


High Availability and Complexity

I agree that HA can add complexity in many cases. For example, my project with Ceph on Ubuntu using Cephadm was fun but turned into more of a headache than it was worth for my setup. However, my experience with Proxmox has been much smoother.

Turning a few nodes into a cluster has taken me only 20–30 minutes of configuration. As long as I have a dedicated 1G connection or a converged 10G connection for Corosync traffic, it has been rock solid. With three nodes, there is a quorum, which avoids the “who’s right” scenarios you mentioned.

In your case, you had the option to drive to the other site to replace hardware. For me, being days away from home makes that impossible. Asking someone at home to plug an HDMI cable into the correct device can seem daunting. I need the system to keep running until I can get back.


Switch Redundancy

You’re absolutely correct that I’m not protected against a switch failure. However, I recently upgraded my switches to decent-quality, name-brand models, so I’m optimistic they’ll have a good lifespan. In my experience, switches have been the most reliable devices in my rack.

Thanks to your suggestion, I’ll plug each node and a connection from my TrueNAS server into different switches. This way, either the router or both switches would have to fail simultaneously while I’m gone for it to cause an issue.


Storage Performance

10G-based NFS is not as fast as local NVMe storage, but it works well for my use case. Most of my bandwidth usage involves media files rather than container boot disks. Since LXC containers leverage the host OS’s base (hosted on a local NVMe), this has improved performance compared to VMs.

With a 10G interface and most containers’ boot disks stored in TrueNAS ARC (256GB) or L2ARC, the latency difference between local and NFS storage is negligible for me. While IO latency increased slightly (from 0–1% to 5–7%), I haven’t noticed any real impact on user experience.

The setup also provides significant benefits:

  • High Availability: Containers can move between nodes without data loss, even if a node fails between replication tasks.
  • Backup and Replication: It allows me to easily replicate ZFS storage and backups off-site.

Additionally, I already had a well-sized and well-specced TrueNAS server with replication configured from a previous storage upgrade, so it made sense to reuse it.


Hardware Choices

I appreciate your comments—they encourage me to double-check my decisions and consider new perspectives. I’ve gone through several iterations of homelab setups and tried a lot of different things based on comments.

I previously attempted a similar setup using $50–$100 Dell office machines, but I ran into networking issues. The onboard NICs were too slow, even with link aggregation, and upgrading to 10G would have required modding projects I wasn’t willing to take on.

That’s why I landed on mobile CPUs for low power consumption (similar to office SFF computers) and a platform with enough expandability to add a few key components like:

  • A 10G NIC
  • Multiple NVMe drives
  • Other upgrades as needed

This combination gives me the flexibility and reliability I need while keeping power consumption and noise levels low.

1 Like

I would like to point out that redundant network between two separate campuses are very different things than within a single rack.

but ignoring that.

considering these are being put in cases measured in rack units. high availability network links are easy, I’ve found that if you are picking up SPF+ network cards used, the cost difference between single and dual port cards is basically nothing, and the quad port cards are only somewhat more expensive. knowing that, each machine can be directly attached to each other. and second link between them on the network.

even if the rest of the network isn’t redundant yet – that’s not a reason to not reduce the number of single points of failure over time.

but I’m also currently on a crusade to slowly eliminate cloud services from as many peoples lives as possible. which means I’m currently very pro on prem HA projects.

1 Like

Dell and HP servers annoy me immensely because of their fan curves, but if you are patient you can get some great deals on supermicro boards and build a server that isn’t loud.

every server i build now needs IPMI or equivalent, but that aside, I really think that moving to a HA set up is interesting, and am looking forward to further updates.

Great idea about the redundant network! Since I’m only using the M.2 slot for the first 10G network, I might use the PCIe slot for a 10G or even 1G ring network dedicated to Corosync traffic. This would allow me to keep the primary 10G network for other traffic while adding some redundancy. Ideally, I’d just dual-link 10G for each server, but I’m limited on SFP+ ports on the switches.

I was always taught not to judge anything until you’ve either lived it or tried building it yourself. As a result, I’ve never been much of a cloud service person, except when it’s absolutely necessary or makes sense.(I would never personally try to host email. Every time I looked into it, I go NOPE)

Over the past 10+ years, I’ve essentially become my family’s cloud provider. I’ve had to balance homelab budgets and reliability while meeting cloud-level expectations. This experience has given me a huge respect for cloud services, though I still prefer keeping control in my own hands. I’m right there with you on that!

I started my homelab journey with a used Supermicro board. When I “upgraded” to Dell and HP servers, I was shocked by how difficult and complex it was to get similar control over their fans. HP’s approach of running fans at 100% just because you install a PCIe card without HP firmware was especially frustrating.

I still have one Dell R720xd that serves as my TrueNAS host, paired with a NetApp disk shelf. It works well for my needs, offering the amount of PCIe lanes, cost-effective RAM, and 2U size I require. I’ve managed to get the fan noise to a tolerable level, but if there were a reasonably priced replacement, I’d be all ears.

More updates will come as soon as AliExpress delivers the rest of the motherboards. Prices fluctuate, but I managed to get mine for about $160 each. Since one node can handle everything, adding two more should be painless.

2 Likes

this really depends on how much storage you need. you can get a lot of density in 2-4U. i myself have recently chosen a silverstone case with 6 5.25" bays. because icydock and silverstone both have 5.25" to 2.5" drive cages. where you can fit 4-6 ssds per dock. or they also have a m.2 5.25" enclosure which i believe will do 8 drives per bay.

of course if you need the density of spinning rust, they have the 3x5.25" bay to 5x3.5" drive cage. which would let you store something like 100TB of raw storage.

none of those are cheap mind you. it would probably be cheaper to look at a case that has drive sleds already. for example, the Athena Power RM-2U2123HE has 12 drive bays in a 2U size. i know nothing about its other features.

but if i may ask. why are you constrained to 2U?

1 Like

I suppose it’s a good problem to have, but I’m running out of space in my rack. A bigger rack won’t fit in the room, and I don’t have space for a second rack.

I’m currently using NetApp disk shelves and am saving the last 4U of space for another disk shelf. That leaves me with only 2U available for the actual host. If I replaced my current setup with a 4U system that can hold 24 drives, I could add a 12-drive disk shelf and achieve the same total drive capacity.

While this is an option, I’d prefer to stick with two of the same NetApp disk shelves for parts interchangeability. I already have a spare controller and power supply from the last shelf I purchased.

Currently, my Dell R720xd has all six external PCIe slots fully occupied:

  • 2 M.2 L2ARC drives (could combine these into one slot with bifurcation)
  • External SAS HBA
  • 2 Optane metadata drives
  • Dual-port 10G NIC

The internal PCIe slot is occupied with the HBA for the front bays, and all 12 front bays are full with a mix of SATA-based L2ARC drives and a six-drive vdev.

Hi, @Justin. Are these the videos you were talking about in your post?

Yes I am using those or similar ones from the same seller on Aliexpress. Here is the link if you are curios. Erying MODT

1 Like

Update: The rest of the parts will arrive on Monday, and I will complete the build and deployment. Let me know if you have any questions so I can address them during the final deployment and testing, and provide answers afterward.

In the process of purchasing the testing motherboard and CPU combo (link in the previous reply), I found some from a cheaper seller. It was strange because they were listed at half the price, around ~$62 USD. The listing is no longer available, and neither is the seller’s store (“Source Factory No.1 Store”). These are the last two boards, and they are set to arrive on Monday. I will also follow up on how they arrive, if they perform similarly, or if I got scammed and they turn out to be bricks, etc.