So, here’s what I’ve been doing for the past few hours. Thought maybe someone will find it useful.
I’ve got a few VMs set up on Proxmox, and gave each of them 3 network cards. All 3 from the same hardware network card. That was the mistake, I eventually found out.
So, each VM has eth0, eth1 and eth2.
With respective network ranges of 192.168.0.0/24, 10.17.16.0/24 and 10.17.92.0/24
These VMs also had some firewall rules applied to them. For instance, INPUT rules that say: only accept traffic for this list of ports if it comes via 192.168.0.0/24 over eth0.
And some software that forms a cluster by communicating over this range was reporting errors.
Took me way too long to try something like this:
sudo iptables -A INPUT -j LOG --log-prefix "Dropped Packet: " --log-level 4
Checking /var/log/syslog, I found that the packets were coming in… but on eth2. They were properly marked as having come from 192.168.0.0/24, with source and destination port… but were being picked up by eth2 instead of eth0.
So, lucky me, the machine actually did have a second physical network card. I stopped the VMs, removed the last 2 network cards and added them back on this 2nd card, leaving 192.168.0.0/24 on the first card.
Restarting after this, everything connected as expected.
From what I can tell with my limited knowledge, this behavior is down to the 3 ifaces actually being the same physical card and there not being proper separation being applied.
Or maybe I’m wrong? I’m too tired to keep looking at this point. Any insights welcome, of course.