Need Help: Proxmox Bridge Setup

Hardware Link

NOTE: This diagram is apparently misleading, it is obsolete and remains here simply so as not to confuse the existing posts. It is replaced with a second (better?) diagram later in the thread below.

Most of the info should be on the diagram. Proxmox and OpenWRT currently operate correctly except for not separating out the hardware ports. So I think I just need a pointer as to how to setup the Linux/Proxmox bridge/port definitions and I’m good to go. Thanks in advance for any assistance on this. The purple text denotes the port-definitions.

Proxmox is installed on this 6 port mini pc and openwrt is a VM inside of that?

Are you doing pcie passthrough for the openwrt vm?

Do you want proxmox to be VLAN aware?

last time I did it this way, but I had a switch with matching configuration between Proxmox and in my case PFsense.
With this config you tag the VNIC of your VM via Proxmox GUI.

#VLAN
auto enp14s0
iface enp14s0 inet manual

auto enp14s0d1
iface enp14s0d1 inet manual

iface bond0 inet manual
        bond-slaves enp14s0 enp14s0d1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4

auto vmbr2
iface vmbr2 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 10 20 30 50 60 70
1 Like

Correct.

My ignorance is such that I don’t know if that is possible/preferred/necessary, I would assume that it is at least possible.

Not necessary and not useful. The vlan labels simply indicated (poorly) how OpenWRT tags things internally to firewall the packets. In that regard, the vlan labels are misleading here. A better diagram will follow, I apologize for the confusion factor.

P.S. Thank you for taking the time to help.

Thank you for helping me, and in such detail. I think I’ve made this far more complex than I really needed, and beyond my ability to understand what is needed in much detail. Apparently, I didn’t know enough to ask the right questions. Another (better?) diagram follows.

Also, PFsense is not preferred because the OS has no drivers for the i226 hardware and I am intimately familiar with OpenWRT which is currently in place on another existing router.

A (hopefully better) diagram follows. This is data-flow based and grossly simplified. This configuration is all I need for 6 months or a year during which time I can learn about this subject a bit more and ask better questions if I need help in the future. Thanks to everyone for their help on this.

Proxmox-Diag-2.drawio

Wan is a hardware port. The upper Lan is a hardware port. The switch is external hardware. The lower Lan can be a hardware port or virtually connected to the Upper Lan hardware port, as indicated by the dotted box lines. A hardware port is preferred as will be shown in a third diagram in a moment. I could live with the above model for now, but the third diagram further describes what is the end goal of mine.

For this model, just assume that OpenWRT is the only VM.

This is what the end goal looks like. Things are much complicated by the various data rates of the system. The data rates have these features:

  • Wan is limited by router hardware to 2.5 Gbe (cost compromise).
  • Workstation to Nas is 10 Gbe.
  • 1 Gbe Lan ports all flow at full speed to either 10 Gbe port.
  • 1 Gbe Lan ports have 2.5 Gbe aggregate flow to Wan port.

If I figured this part out correctly, Router ports H0 thru H4 are passed-thru, so OpenWRT sees them as separate hardware ports and can bridge them as desired on the OpenWrt side. They are otherwise invisible to proxmox and the switch. The data flows at 2.5 Gbe to each router port.

Router port H5 is hardware passed-thru and sent to the switch (via external cat6 cable) as well as bridged so that proxmox can talk to the Lan. All proxmox ports bypass the proxmox firewall.

If it helps, I found this IOMMU info.

root@pve:~# dmesg | grep -e DMAR -e IOMMU
[    0.011923] ACPI: DMAR 0x00000000787C7000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.011956] ACPI: Reserving DMAR table memory at [mem 0x787c7000-0x787c7087]
[    0.135044] DMAR: Host address width 39
[    0.135046] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.135051] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 49e2ff0505e
[    0.135054] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.135059] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.135061] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff
[    0.135064] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.135066] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.135067] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.136841] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.357271] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics

In proxmox you can do this in the GUI. You create bridge interfaces for each of the physical interfaces (or a bridge of multiple interfaces) you want to use and then in the VM configuration you give it whatever bridge interface you want. You can use the same bridge interface with multiple VMs.

1 Like

yeah you just need a basic bridge. you can have however many physical ports in it that you want, i would recommend just starting with 1 physical port to make sure everything is working. at least one bridge network is required as that is how you configure normal VMs to get on the network. there are other ways, but this is the easiest. also the IP of the NIC in the bridge can be the management interface for ProxMox. can you get to your ProxMox gui? the setting is here:

if you can not get to the gui we can build you a config to add manually.

1 Like

Sorry for the late reply, real life and all that.
Thanks for all the help to everyone involved. I have everything operating well enough to be dangerous now.

Special thanks to @Dexter_Kane whom has helped me out on several occasions, on a wide variety of hardware/software issues.

The community here is greatly appreciated, if under-mentioned.

Just to put a ribbon on this topic, the config that was most applicable to my situation was to pass each hardware port (sans one) in it’s own Proxmox bridge individually to OpenWRT, then bridge the ‘LAN’ ports together inside the OpenWRT VM config. The OpenWRT bridge left the last port (last Proxmox bridge) as the ‘WAN’ port which is physically connected to the fiber internet (wan).

All this could have been MUCH simplified by not trying to max the hardware bit-rate. With the convoluted way that things are wired (physically) I have NAS file service at 10gbe , NAS and workstation Internet at 2.5gbe and wi-fi, phone, printer etc. at 1gbe. The best of all worlds, and a router with plenty of ponies to move the traffic.

To put a happy ending on this thread, the franken-cobbled budget setup I have, is configured as follows:

  • 2.5gbe wan (Internet) connected to a 1Gbe (average) symmetric fiber. Fiber hardware can go as high as 10gbe, but the data usually runs about 1-1.8 gbe.

  • 10Gbe connection from my workstation to my nas.

  • Previous tp-link router repurposed as a glorified wi-fi AP for the new router. (1gbe)

  • Dedicated static port for proxmox in case the router VM goes bonkers.

  • Smart Queue Management QoS (cake qdisc) for traffic shaping.

Thanks to everyone that contributed.

GRC Port Authority Report created on UTC: 2023-04-05 at 01:12:32

Results from scan of ports: 0-1055

    0 Ports Open
    0 Ports Closed
 1056 Ports Stealth
---------------------
 1056 Ports Tested

ALL PORTS tested were found to be: STEALTH.

TruStealth: PASSED - ALL tested ports were STEALTH,
                   - NO unsolicited packets were received,
                   - NO Ping reply (ICMP Echo) was received.