Host lost network connection after setting up a bridge

crashpb · August 31, 2021, 7:31am

hey,
so I have a fedora 34 SERVER box and I was planning to run opnsense as a VM and use my onboard NIC as my LAN .
setting VMnetwork as my LAN NIC on opnsense I couldn’t access its web gui.
so I created a new bridge and put my motherboard’s nic on it.
right now my lan is working fine but for some reason the host itself doesnt have network access.
some logs:

ip a:

   1: lo: <LOOPBACK, UP, LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever inet6 ::1/128 scope host
valid_lft forever preferred_1ft forever 2: enp117s0f8: <BROADCAST, MULTICAST, UP, LOWER_UP> mtu 1500 qdisc mq master B_PCI_NIC1_8
group default glen 1000
state U
link/ether f4:ce:46:a6:55:10 brd ff:ff:ff:ff:ff:ff 3: enp117s0f1: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc mq state DOWN group default q
link/ether f4:ce:46:a6:55:11 brd ff:ff:ff:ff:ff:ff
4: enp117s0f2: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc mq state DOWN group default q link/ether f4:ce:46:a6:55:12 br ff:ff:ff:ff:ff:ff 5: enp117s0f3: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc mq state DOWN group default q_
link/ether f4:ce:46:a6:55:13 brd ff:ff:ff:ff:ff:ff 6: enp@s31f6: <BROADCAST, MULTICAST, UP, LOWER_UP> mtu 1500 qdisc fq_codel master B_OB_NICO state L link/ether 60:45:cb:a0:c8:3e brd ff:ff:ff:ff:ff:ff
7:
wlp113s8: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 16:4f:eb:91:05:ab brd ff:ff:ff:ff:ff:ff permaddr 08:6a:0a:b8:62:73
8: B_OB_NICO: <BROADCAST, MULTICAST, UP, LOWER_UP> mtu 1500 qdisc noqueue state UP group default qle link/ether 66:a8:ac:70:b5:df brd ff:ff:ff:ff:ff:ff inet6 fe80::64a8:acff:fe70:b5df/64 scope link
valid_lft forever preferred_1ft forever
9: B_PCI_NIC1_1: <BROADCAST, MULTICAST, UP, LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether ba:95:c2:46:64:e8 brd ff:ff:ff:ff:ff:ff inet6 fe80::b895:c2ff:fe46:64e8/64 scope link
valid_lft forever preferred_1ft forever 10: B PCI_NIC1_0: <BROADCAST, MULTICAST, UP, LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 92:5b:e3:51:43:9f brd ff:ff:ff:ff:ff:ff
inet6 fe80::905b:e3ff:fe51:439f/64 scope link valid_lft forever preferred_lft forever 11: vnet28: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc fq_codel master B_PCI_NIC1_0 state I
link/ether 4e:5c:1a:13:5d:52 brd ff:ff:ff:ff:ff:ff 12: B PCI_NIC1_2: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc noqueue state DOWN group defau link/ether 36:ac:88:8a:b0:63 brd ff:ff:ff:ff:ff:ff
13: B PCI_NIC1_3: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc noqueue state DOWN group defau
link/ether 76:b7:bc:2b:cc:65 brd ff:ff:ff:ff:ff:ff
14: vnet23: <NO-CARRIER, BROADCAST, MULTICAST, UP> mtu 1500 qdisc fq_codel master B_OB_NICO state DOWN link/ether 4a:27:6f:82:01:dc brd ff:ff:ff:ff:ff:ff
15: virbr8: <BROADCAST, MULTICAST, UP, LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen link/ether 52:54:00:87:2f:10 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope
100
global virbro valid_lft forever preferred_1ft forever 16: vnet8: <BROADCAST,MULTICAST, UP, LOWER_UP> mtu 1500 qdisc noqueue master B_PCI_NIC1_8 state UNKNOW
link/ether fe:54:00:5a:52:77 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe5a:5277/64 scope link valid_lft forever preferred_1ft forever
17: vnet1: <BROADCAST, MULTICAST, UP, LOWER_UP> mtu 1588 qdisc noqueue master B PCI_NIC1_1 state UNKNOWN link/ether fe:54:08:5f:58:39 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe5f:5839/64 scope link valid_lft forever preferred_1ft forever
18: vnet2: <BROADCAST, MULTICAST, UP, LOWER UP> mtu 1588 qdisc noqueue master virbro state UNKNOWN group link/ether fe:54:88:89:20:3a brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe89:283a/64 scope link
valid_Ift forever preferred_1ft forever
19: vnet3: <BROADCAST, MULTICAST, UP, LOWER UP> mtu 1500 qdisc noqueue master B_OB_NICO state UNKNOWN grou link/ether fe:54:00:fe:4e:a4 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fefe: 4ea4/64 scope link
valid_Ift forever preferred_lft forever

ip r:

192.168.122.0/24 dev virbra proto kernel scope link src 192.168.122.1
nmcli con show:

note: I created the bridge-slave-enp0s31f6 myself later to solve the issue but its still not solved.

Mastic_Warrior · August 31, 2021, 12:13pm

I don’t have the ability to edit your post. could you put your iconfig output into a code block to help with formatting. Put you output in between these, please [code][/code]

risk · August 31, 2021, 7:22pm

For future reference, please surround preformatted text with triple back ticks, this is horribly difficult to read.

Also, how many physical interfaces and bridges do you have / and why? It looks like too many judging from your description.

Apparently enp117s0f# is a 4 port nic and you’re using the first port, and don’t have an address assigned - which is why it doesn’t show up in ip a, and it’s a slave to… which one?

I’m guessing enp0s31f6 is your onboard?

Essentially, if i understand correctly you need the following:

bridge … it needs enp0s31f6 added to it
a virtual nic/tap (other side within VM) that’ll be added to that bridge
another virtual nic (other side VM) that you’ll configure and let your host talk to the VM

You can check with tcpdump what goes over which interface and if things are getting lost somewhere, and maybe can then proceed to check firewalling.

crashpb · August 31, 2021, 8:04pm

rn I have a 4 port intel nic >enp117s0f* and my onboard nic > enp0s31f6.
I want my enp0s31f6 to be my LAN uplink (I connect this nic to my switch ) and I also want this nic to have a static IP and and serve my host’s web gui (cockpit) .

enp0s31f6 was part of “VM Network” originally however after giving this interface to my router’s VM I couldn’t access my router (opnsense).
basically the LAN part wasn’t working.

so I created a new network bridge called B_OB_NIC0 and added enp0s31f6 to this bridge; doing this now I can ping my router over LAN (basically LAN now works, for example my clients get dhcp addresses from my router) but I have LOST access to my web gui.
after searching the web and reading some guide I set enp0s31f6 as a slave for B_OB_NIC0 manually but that didn’t solve my issue.

basically right now host itself doesn’t see a network and I don’t know why.

risk · September 1, 2021, 9:19pm

BOBBY_NIC0 (illustrating that naming scheme doesn’t matter) should have enp0s31f6 as a slave.
Your LAN IP (for cockpit) should then be configured on BOBBY_NIC0.
You’d then add a virtual nic (other end inside VM) to BOBBY_NIC0.

That way, BOBBY_NIC0 is basically a switch, your host is plugged into it with a BOBBY_NIC0 interface, and it leads to a physical switch through the physical interface, and to your VM.

I see you have vnet3 on B_OB_NICO is that your opnsense lan interface?

crashpb · September 2, 2021, 7:35pm

hey,
thanks to you I have things working now as unintuitive as it seems (to me), by giving an ip address to the bridge (the B_OB_NIC0 bridge, I also have set my onboard nic as an slave for this bridge).

it seems as though I have connected some machines to a switch and to access one of them I have to give the switch an IP and use this IP to access that machine. (I know a bridge is not exactly a switch but still…, also I’m coming from ESXi so be kind).

… yes vtnet3 is my lan interface on opnsense.

oh and B_OB_NIC0 stands for Bridge_OnBoard_NIC 0.

searching the web I came across Open vSwitch, as I do lot of network fkery on my lab I might switch to that later.