Waiting on my second vaccine so finally have a chance to post this.
I was just curious if anyone else had any experience with the networking aspect of of proxmox, ovs, a ruckus icx switch amd vlans as I’m at a bit of an impase.
Currently I have 3 vlans set up between 2 proxmox servers, their IPMIs, their virtual machines and 2 workstations. The short of it is from one workstation that is the only device not on the icx 7250 switch (connected directly to pfsense firewall and on a vlan not on the switch or used by any other device mentioned here) I can access all IPMIs, VMs amd proxmox hosts. This is as expected as I have the switch in routing modqe. However, from the other host that is on the same vlan and subnet as the management (ipmi and proxmox mansgement) interfaces, I can’t access anything other than the VMs. It can ping and receive responses from all of the VEs (svi in cisco garble) on the switch, ping the pfsense firewall and access the interwebs at large. I have tried every conceivable combination of tagged and untagged on the switch as well as making a vlan (and not) from netplan on the the host itself, and nothing works to give it access to even those interfaces on the same subnet and vlans. It can however ssh just fine into the VMs which are not on its subnet or vlan. I am at a loss as to what to do at this point as obviously I am failing to understand how something in the mix functions.
The Openvswitch implementation and documentation for proxmox is annoyingly sparse and non-standard. I will post on servethehomes ruckus forum thread but wanted to see if anyone here might have any suggestions first. No plans to post on proxmox as every thread that involves OVS is met with a, “why are you using OVS?!” Yes. Why am I using something touted as a feature of proxmox…
The host that can connect to everything is a windows machine with a chelsio nic with the vlan tag implemented from the nic configuration section inside of windows if that helps (its my last machine with windows, plan to change to linix, don’t judge me!)
Linux supports bridge vlan filtering these days within its native bridge driver, and that works fine with vhost-net acceleration, it turns out this is what most people were interested in having in the first place, making OVS popularity drop. I suspect proxmox was just late to the party with ovs support. Also, sr-iov is more of a thing these days compared to ten years ago when ovs was being advertised for high performance VM stuff - maybe they shouldn’t have bothered… but in any case.
Back to first principles…
good thing about all of this is you can use tcpdump
and/or tshark/wireshark
to see what’s actually going over which interface with which tags.
This includes incoming as well as outgoing frames.
In general, you’d use ovs-vsctl
to connect to do the switch configuring stuff. It which talks to the database daemon (saves config between reboots) and to the switchd which talks to the kernel and tells it what config to load.
You can use the tool to inspect its configuration and current state of its various ports.
Simply decide what VLAN tags you want where in your network, and check using ping/tcpdump whether the tags or packets are present, and where tags/packets are getting lost.
Avoid using VLAN 1 and 4093/4094/4095 . Various hardware vendors are “weird” and hijack those vlan ids for their own configuration handling…
Once you’ve determined where packets are being lost, if it’s within Linux layer3 you can insert firewall logging or additional routing tables and policies and so on. If it’s on L2, you can clear the switch and watch its fdb for the right mac address appearing on the right port.
Let us know how it goes.
I did not use the normal ovs commands as I assumed none of those would be reflected in the gui and wanted to be able too see all I could from the gui.
On the tcdump/wireshark side, no idea why I did not run that first, lol. I will try and run that and/or set up a mirrored port on the switch to see whats going on. Guess thats what happens after a marathon network setup and then find an oddball issue at the end that is being a pita!
The more I look into this, the more I am starting to suspect the ubuntu (nuc) machine. It is the oddball out. It does not seem to be doing vlans one way or the other. It does not even get a spot on the arp list of the switch. I guess I can run another distro from a usb and see what happens to its traffic. Pita.