Here is another video on a similar setup. Maybe this helps
On proxmox forum it has been worked out already:
Working config:
3 Likes
This is interesting. I recently set up this exact cluster (using actual Intel NUC 13th gen) and I am not getting those high (nor inconsistent) pings. Additionally, I am getting 20Gbps out of the box, not the 1Gbps that Wendell mentioned in the YT video. The only things that I did was in /etc/systemd/network/00-thunderbolt0.link I added
[Match]
Driver=thunderbolt-net
[Link]
MACAddressPolicy=none
Name=en05
Then
update-initramfs -k all -u
and then added the following to /etc/modules
thunderbolt
thunderbolt-net
So, I am curious was it different with my setup. Also, I am having an issue with the Thunderbolt 4 interfaces not coming up after a reboot. I have to bring the interfaces up and down repeatedly on each host in order to get them up. Once they are up, they are stable.
3 Likes
Oh, frr looks like it might be nice if it doesnât end up sending the traffic via non direct routes
The proxmox forums link has the udev rule you need. After that Iâll come back after reboot
Yeah that is the one. I am just saying a guide around these instead of firewire makes more sense. At least for those of us in the states where the used market is not too expensive.
Much better bandwidth and more mature driver support.
@wendell makes great guides and gets to play with all the neat toys.
1 Like
Alex posted the gist above. On amd thunderbolt net was not working. On intel Initially I had the low pings and bad perf till disabling security and setting up the udev rules.
2 Likes
Interesting. Now out of curiosity since Ive not set thunderbolt udev rules. Is it possible to set a rule for the specific machines and otherwise leave security enabled for anything else?
I entirely realize that would complicate things and make for needing to label your port connections and not mix and match im guessing.
Just a curiosity is all. Or is it much more complicated to whitelist certain devices for no tb sec?
Id love to know the nitty gritty reason why. Is it that security limits or turns off direct memory access? Reducing perf?
1 Like
I tried to do this with 3 UM790 Pro but only one of the two USB4 ports works with network over Thunderbolt.
The left port sometimes works, like 1 of 100 times you try. But the right one always works. The support at minisforum canât help as it seems too advanced for them.
I have the same problem with the left port if I run Windows on it too so I think it is a hardware problem or firmware.
1 Like
Iâve ran into the same issue with UM790âs this week - I thought I was going mad! TL;DR: Itâs possible to make them work, but Iâm only getting 11.9Gb/sâŚ
Iâll preface this by noting that Iâm using stock Proxmox 8.0.4 with no link security changes, udev rules (besides the one I mention below) or the like; I literally just installed all updates then added âauto thunderbolt0
iface thunderbolt0 inet manualâ to /etc/network/interfaces and rebooted.
First: make sure to use certified USB4 cables.
Next: I believe UM790âs are extremely picky with port choice - I could only get links to work when the left port on one host goes to right of next. Left-Left or Right-Right just refused to work with heaps of âconfig errorâ messages and even an occasional âMaybe the USB cable is badâ in dmesg.
Lastly, even though the NICâs were found theyâd always be ânoop state DOWNâ in âip linkâ until I manually ran ifup against the interfaces on both sides. Someone in another forum suggested this may be a bug of the TB driver.
I eventually created a udev rule to do this automatically:
ACTION=="add", SUBSYSTEM=="net", ENV{INTERFACE}=="thunderbolt*" RUN+="/usr/sbin/ifup '%E{INTERFACE}'"
Sometimes this works perfectly, other times itâd cause Proxmox to boot without configuring any NICâs.
If anyoneâs got any suggestions on improving bandwidth or resolving the udev reliability, Iâm all ears! For the later, my latest hunch is to subprocess ifup with a slight delay in order to not disturb other functionsâŚ
2 Likes
The ports are only 10Gb
Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M
Intressting, as for cables, i can use all USB-C to USB-C cables i have and get 10Gb/s on the right ports on all 3 UM790 Pro that i have. But i do i good cables to and that do not help to get the left port to work for me.
1 Like
I think about getting a UM790 Pro Cluster. Did you get it working reliable. I would like to deploy this solution in a small company.
Should i pull the trigger on the black friday deal?
Hi AdrianS
I have still not got it working. I would say donât if you plan on using USB4. My plan now is to get good network adapters to put in the USB4 connectors.
The support at Minisforum have stopped responding to me.
/R
I decided to buy 3 Minisforum NPB7. I hope with a Intel cpu i run into fewer problems. Also the NPB7 has 2 2.5 Gbit ports. Each one gets 96GB of ram and a 2TB ssd.
Have you got the NPB7 yet? Just wondering how Proxmox works with the small and large cpu cores.
I have the 3 NPB7 setup and in a cluster but the thunderbolt connections dont work. When i plug them together i dont see any events in udevadm monitor. I added the modules thunderbolt-net and thunderbolt. I am new to linux and need some guidance to troubleshoot this issue.
i used this guide Thunderbolt Networking Setup ¡ GitHub
Ok i am a little bit further. When i plug the minisforum into my laptop with usb4 it shows the events (thunderbolt) in udevadm monitor but noch when i plug it in the other minisforum pc. Then it shows nothing.
1 Like
PS: I resolved this by changing the UDEV RUN statement to â/usr/sbin/ip link set â'â%E{INTERFACE}â'â upâ instead
Iâve since found a new issue - if link is down I only see 3~4MB/s between the disconnected hosts when routing via the âmiddleâ host. Not too sure where to look to troubleshoot this 