3-node 65w Proxmox Thunderbolt Network Cluser

Here is another video on a similar setup. Maybe this helps

On proxmox forum it has been worked out already:

Working config:

3 Likes

This is interesting. I recently set up this exact cluster (using actual Intel NUC 13th gen) and I am not getting those high (nor inconsistent) pings. Additionally, I am getting 20Gbps out of the box, not the 1Gbps that Wendell mentioned in the YT video. The only things that I did was in /etc/systemd/network/00-thunderbolt0.link I added

[Match]
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05

Then

update-initramfs -k all -u

and then added the following to /etc/modules

thunderbolt
thunderbolt-net

So, I am curious was it different with my setup. Also, I am having an issue with the Thunderbolt 4 interfaces not coming up after a reboot. I have to bring the interfaces up and down repeatedly on each host in order to get them up. Once they are up, they are stable.

3 Likes

Oh, frr looks like it might be nice if it doesn’t end up sending the traffic via non direct routes

The proxmox forums link has the udev rule you need. After that I’ll come back after reboot

@wendell

What hoops did you have to jump through for for 10-20gbps?

Yeah that is the one. I am just saying a guide around these instead of firewire makes more sense. At least for those of us in the states where the used market is not too expensive.
Much better bandwidth and more mature driver support.

@wendell makes great guides and gets to play with all the neat toys.

1 Like

Alex posted the gist above. On amd thunderbolt net was not working. On intel Initially I had the low pings and bad perf till disabling security and setting up the udev rules.

2 Likes

Interesting. Now out of curiosity since Ive not set thunderbolt udev rules. Is it possible to set a rule for the specific machines and otherwise leave security enabled for anything else?

I entirely realize that would complicate things and make for needing to label your port connections and not mix and match im guessing.

Just a curiosity is all. Or is it much more complicated to whitelist certain devices for no tb sec?

Id love to know the nitty gritty reason why. Is it that security limits or turns off direct memory access? Reducing perf?

1 Like

I tried to do this with 3 UM790 Pro but only one of the two USB4 ports works with network over Thunderbolt.

The left port sometimes works, like 1 of 100 times you try. But the right one always works. The support at minisforum can’t help as it seems too advanced for them.

I have the same problem with the left port if I run Windows on it too so I think it is a hardware problem or firmware.

1 Like

I’ve ran into the same issue with UM790’s this week - I thought I was going mad! TL;DR: It’s possible to make them work, but I’m only getting 11.9Gb/s…

I’ll preface this by noting that I’m using stock Proxmox 8.0.4 with no link security changes, udev rules (besides the one I mention below) or the like; I literally just installed all updates then added “auto thunderbolt0
iface thunderbolt0 inet manual” to /etc/network/interfaces and rebooted.

First: make sure to use certified USB4 cables.

Next: I believe UM790’s are extremely picky with port choice - I could only get links to work when the left port on one host goes to right of next. Left-Left or Right-Right just refused to work with heaps of “config error” messages and even an occasional “Maybe the USB cable is bad” in dmesg.

Lastly, even though the NIC’s were found they’d always be “noop state DOWN” in ‘ip link’ until I manually ran ifup against the interfaces on both sides. Someone in another forum suggested this may be a bug of the TB driver.

I eventually created a udev rule to do this automatically:

ACTION=="add", SUBSYSTEM=="net", ENV{INTERFACE}=="thunderbolt*" RUN+="/usr/sbin/ifup '%E{INTERFACE}'"

Sometimes this works perfectly, other times it’d cause Proxmox to boot without configuring any NIC’s.

If anyone’s got any suggestions on improving bandwidth or resolving the udev reliability, I’m all ears! For the later, my latest hunch is to subprocess ifup with a slight delay in order to not disturb other functions…

2 Likes

The ports are only 10Gb
Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M

Intressting, as for cables, i can use all USB-C to USB-C cables i have and get 10Gb/s on the right ports on all 3 UM790 Pro that i have. But i do i good cables to and that do not help to get the left port to work for me.

1 Like

I think about getting a UM790 Pro Cluster. Did you get it working reliable. I would like to deploy this solution in a small company.

Should i pull the trigger on the black friday deal?

Hi AdrianS

I have still not got it working. I would say don’t if you plan on using USB4. My plan now is to get good network adapters to put in the USB4 connectors.
The support at Minisforum have stopped responding to me.

/R

I decided to buy 3 Minisforum NPB7. I hope with a Intel cpu i run into fewer problems. Also the NPB7 has 2 2.5 Gbit ports. Each one gets 96GB of ram and a 2TB ssd.

Have you got the NPB7 yet? Just wondering how Proxmox works with the small and large cpu cores.

I have the 3 NPB7 setup and in a cluster but the thunderbolt connections dont work. When i plug them together i dont see any events in udevadm monitor. I added the modules thunderbolt-net and thunderbolt. I am new to linux and need some guidance to troubleshoot this issue.

i used this guide Thunderbolt Networking Setup ¡ GitHub

Ok i am a little bit further. When i plug the minisforum into my laptop with usb4 it shows the events (thunderbolt) in udevadm monitor but noch when i plug it in the other minisforum pc. Then it shows nothing.

1 Like

PS: I resolved this by changing the UDEV RUN statement to “/usr/sbin/ip link set ‘'’%E{INTERFACE}‘'’ up” instead

I’ve since found a new issue - if link is down I only see 3~4MB/s between the disconnected hosts when routing via the “middle” host. Not too sure where to look to troubleshoot this :confused: