Help needed! - Hetzner Bridged Networking for a KVM Linux Server

programster · June 6, 2020, 4:57pm

I have a load of VMs on various cloud service providers such as AWS and Digital Ocean. It’s gotten to the point now where I figured I could save money, have a fixed cost, gain some experience, and get much better performance if I was to just rent a dedicated Debian 10 server from Hetzner, pay for an additional subnet, and set up the VMs as guests under KVM (like I’ve done at home).

Unfortunately, I just can’t seem to figure out how to configure the /etc/network/interfaces file for acting as a bridge for my guests.

At home, this was really simple, and I just had the following example configuration:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto enp35s0
iface enp35s0 inet manual

auto kvmbr0
iface kvmbr0 inet static
    address 192.168.1.19
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.254
    bridge_ports enp35s0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

Unfortunately, it is not so simple with hetzner. For a start, I have a single static IP and a separate subnet that I bought for my VMs. There was no option to just have a single subnet with lots of IPs when I ordered the server, which would have been simpler/nicer.

The primary IP details for my server are:

address 95.217.122.234
netmask 255.255.255.192
gateway 95.217.122.193

The additional subnet I received is: 95.216.106.56 / 29

Network 95.216.106.56
Gateway: 95.217.122.234
Netmask: 255.255.255.248
broadcast: 95.216.106.63

I originally tried the following network configuration, which failed. (Failed meaning that I could no longer SSH into the server. I would apply a network configuration by performing sudo service networking restart, or just rebooting the server when I am connected via the KVM spider.)

### Hetzner Online GmbH installimage

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp35s0
iface enp35s0 inet static
  address 95.217.122.234
  netmask 255.255.255.192
  gateway 95.217.122.193
  # route 95.217.122.192/26 via 95.217.122.193
  up route add -net 95.217.122.192 netmask 255.255.255.192 gw 95.217.122.193 dev enp35s0

iface enp35s0 inet6 static
  address 2a01:4f9:4a:37d5::2
  netmask 64
  gateway fe80::1


# Add secondary subnet for VMs.
auto enp35s0:0
iface enp35s0:0 inet manual


# Add kvm bridge
auto kvmbr0
iface kvmbr0 inet static
    address 95.216.106.56
    netmask 255.255.255.248
    network 95.216.106.56
    broadcast 95.216.106.63
    gateway 95.216.106.56
    bridge_ports enp35s0:0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

Hetzner Documentation

Since then, I found this Hetzner documentation, for what looks like it is specifically talking about this problem.

I believe I should be able to use direct routing and would prefer that over a “brouter”, but I think if I did this, I need to recieve MAC addresses from hetzner, and I believe this is only possible when you buy individual IPs and not subnets, so I may need to swap my subnet out with a bunch of individual IPs?

I tried both methods with the following configurations, all of which failed for me. Perhaps someone can point out where I’m messing up?

Brouter FAILED

auto enp35s0
iface enp35s0 inet static
    address 95.217.122.234
    netmask 255.255.255.255
    pointopoint 95.217.122.193
    gateway 95.217.122.193

iface enp35s0 inet6 static
    address 2a01:4f9:4a:37d5::2
    netmask 64
    gateway fe80::1

auto kvmbr0
iface kvmbr0 inet static
    address 95.217.122.234
    netmask 255.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    pre-up brctl addbr virbr1
    up ip route add 95.216.106.56/29 dev virbr1
    down ip route del 95.216.106.56/29 dev virbr1

iface kvmbr0 inet6 static
    address 2a01:4f9:4a:37d5::2
    netmask 64

Direct Routing Bridge 1 (FAILED)

### Hetzner Online GmbH installimage

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

# deliberately no configuration for enp35s0 here

auto  kvmbr0
iface kvmbr0 inet static
 address 95.217.122.234
 netmask 255.255.255.192
 gateway 95.217.122.193
 bridge_ports enp35s0
 bridge_stp off
 bridge_fd 1
 bridge_hello 2
 bridge_maxage 12

Direct Routing Bridge 2 (Failed)

### Hetzner Online GmbH installimage

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

allow-hotplug enp35s0
iface enp35s0 inet manual

auto  kvmbr0
iface kvmbr0 inet static
    address 95.217.122.234
    netmask 255.255.255.192
    gateway 95.217.122.193
    bridge_ports enp35s0
    bridge_stp off
    bridge_fd 1
    bridge_hello 2
    bridge_maxage 12
    # route 95.217.122.192/26 via 95.217.122.193
    up route add -net 95.217.122.192 netmask 255.255.255.192 gw 95.217.122.193 dev enp35s0

Original (Working) Configuration

For references, below is the working configuration that came with my server and which I restore through a KVM console, every time I lose access to the server.

### Hetzner Online GmbH installimage

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp35s0
iface enp35s0 inet static
  address 95.217.122.234
  netmask 255.255.255.192
  gateway 95.217.122.193
  # route 95.217.122.192/26 via 95.217.122.193
  up route add -net 95.217.122.192 netmask 255.255.255.192 gw 95.217.122.193 dev enp35s0

iface enp35s0 inet6 static
  address 2a01:4f9:4a:37d5::2
  netmask 64
  gateway fe80::1

Side Problem - KVM Keyboard Input Pain

As a side note, using the “Lantronix SLSLP KVM Console” is a real pain in the butt due to various issues with the keyboard input that I can’t seem to resolve. If you have any advice on this it is much appreciated.

Tab completion doesn’t work, instead it seems to move focus so that typing anything further doesn’t work and you need to click on the window again.
y is mapped to z and vice-versa.
- is mapped to a funky B shpaed character, can get around this by using - on the number pad instead.
/ is mapped to -, can get around this by useing the number pad.

I have tried using sudo dpkg-reconfigure keyboard-configuration to change the keyboard input, trying both a Generic 105-key PC (intl.) and leaving it on the Generic 104-key PC because that is what the console appears to be using.

After setting the number of keys, I had set the Keyboard layout is set to English (UK), yet the mappings are still wrong.

Any and all help is much appreciated!

Hossam · June 8, 2020, 4:07pm

I haven’t reviewed the configuration carefully, but this caught my eye, why are you using eth0? Your interface name seems to be enp35s0.

Hossam · June 8, 2020, 4:31pm

I tried to read the question more carefully, but I don’t understand what you are supposed to do with the additional IP range? Is it on the same network interface? Does it use a vlan? Do you plan on configuring the new network on the host or you want it only for vms, if you do, how are you planning to do the routing with multiple gateways.
And by the way, since this is a dedicated server, you don’t have to restrict yourself to Hetzner documentation (which could be outdated), it’s better to consult official debian docs and forums.

programster · June 14, 2020, 6:35pm

Good spot. I’ll have to doubel check that I’ve search/replaced all eth0’s with enp35s0 and retry the configs.

programster · June 14, 2020, 6:37pm

The additional IP range is because I wish to set up virtual servers on the dedicated server, that are each individually addressable over the internet. E.g. websites with their own A record that go straight to the VPS. For all intents and purposes they are their own servers.

I am pretty sure I’ve found where I am going wrong and its to do with hetzners security. It appears that I have to use individual IPs rather than subnets, because only for these can you request a MAC address to use which will get through the firewall. I guess their subnets are only for internal networking or something.

programster · June 14, 2020, 9:00pm

The eth0 was one of several issues. I have created a blog post detailing all the steps one has to do in future with the example interfaces file.

The other thing that caught me out was that I needed to reboot the server rather than just restarting the networking service.

One also needs to request static IPs so that one can get MAC addresses and assign them to the guests.

Hopefully nobody else will have to make the same mistakes.

KCarterSr · April 21, 2021, 2:36am

@programster I joined level one just so that I could contact you since I could not find an email address for you anywhere (e.g Github) anyway…

I’ve been going over your post and your blog post about setting up a server for VM’s. I have a server on Hetzner and purchased an additional ip too. I understand (I think) how to implement what you’ve been talking about, but I need clarity on one issue (Access via internet).

You said that you have several websites in VM,s, but since you only have one public facing interface, how do you allow those sites to be reached from the internet?

E.g.
If you have a public Ip is 144.xxx.xxx.50 and your additional purchased Ip is 144.xxx.xxx.100, are you saying that the 144.xxx.xxx.100 address can be reached from the internet? (A site at 144.xxx.xxx.100:80)

If I implement the instructions in your blog example will that be possible?

Thanks
K. Carter

programster · April 21, 2021, 6:59am

Hey @KCarterSr ,
If you implement the instructions here you should be fine (I made sure to put exactly what I did so that I could redo it if I needed to set up another server).

…but since you only have one public facing interface, how do you allow those sites to be reached from the internet?

This is where the bridge comes in as shown in the tutorial:

You will have requested individual IPs for each server, and edit each of the vms in your KVM to use that IP and that mac address and they will be fine. You have just one physical interface, but each IP is public facing.

If you have a public Ip is 144.xxx.xxx.50 and your additional purchased Ip is 144.xxx.xxx.100, are you saying that the 144.xxx.xxx.100 address can be reached from the internet? (A site at 144.xxx.xxx.100:80)

Yes, each IP can be reached from the internet, but you need to make sure to set up the MAC addresses to match, otherwise traffic will be filtered out by the security handling.

Best of luck. It should all be in that tutorial. Please be sure to let me know if you think there is something wrong though and I’ll try to clarify or fix accordingly.

KCarterSr · April 21, 2021, 12:41pm

Thank you soooooo much.

I’ll let you know how it goes.

K. Carter Sr.

KCarterSr · April 23, 2021, 1:02am

Hello, me again.

I can’t get past this part.

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

# deliberately no configuration for enp35s0 here

auto  kvmbr0
iface kvmbr0 inet static
    address <primary IP here>
    netmask <netmask here>
    gateway <gateway IP here>
    bridge_ports enp35s0
    bridge_stp off
    bridge_fd 1
    bridge_hello 2
    bridge_maxage 12

Once I do this, I loose the server (Yes, I changed enp35s0 to the correct name and rebooted) and I have to install a fresh OS on the server to get it back. When I loose the server I can't even ping it.

However, if I ignore this part:
   # deliberately no configuration for enp35s0 here

and leave the original interface info too. I don't loose access.

Is it going to be a problem if I leave the original interface info.

Thanks
(Debian 10)
K. Carter Sr.

programster · April 23, 2021, 7:54am

Sorry you are experiencing problems. I can’t help you now, but I’ll have a look at this in my lunch hour and get back to you (UK timezone).

In the meantime, I have posted my server’s /etc/network/interfaces file to see if that helps in case I missed anything:

screenshot-99(1490)

… and my /etc/network/interfaces.d folder is empty.

I would check and re-check all the ip addresses and netmask in the config are correct

KCarterSr · April 23, 2021, 1:11pm

Firstly, I just want to thank you for being so responsive and helpful. Most people would not want to help some random person from the internet.

I have some ideas that I’m going to try.

Oh, by the way, I was wondering if you were on a KVM spider when you did all this?

If so, you would not have noticed that you no longer had access to your server when you removed the original interface lines and rebooted.

First thing I’m going to do is go through the steps and leave the original interface initially. After I’ve completed running your KVM command generator tool and everything else, then I will remove the original interface and see if that works.

If that doesn’t work, instead of reinstalling a fresh OS, I’m going to request a spider from Hetzner and take a look using ifconfig and see if anything looks like a problem. I’m sure I can get it working using all the information you have given on your blog. If I figure it out, I’ll let you know so you can add to your blog.

Also, if I get it working, next I’m going to try doing the same thing using the Linux version of VMWare instead of KVM. If that is successful too, then you can add that information to your blog.

One last odd thing.

In a fresh install of Debian from Hetzner, the interfaces file has something your original file didn’t. The last two lines have this:

\ # route 144.xx.xx.xx/27 via 144.xx.xx.xx (ignore the backslash)
up route add -net 144.xx.xx.xx netmask 255.255.255.224 gw 144.xx.xx.xx dev enp3s0 (I put in the x’s)

I don’t why they needed that. But I’ll remove it to see if it was actually needed. Any thoughts on that?

Thanks
K. Carter Sr.

programster · April 23, 2021, 2:01pm

When your server is working again without any changes, can you double check your interface before you set the bridge. I’m just wondering if the issue is that in my tutorial I have enp35s0 on the bridge and you may be copying that and only changing the IP addresses? It is quite likely that your interface is enpXXX and you will need to adjust that accordingly. Sorry if that wasn’t clear enough in the tutorial.
(things were so much easier when it was just eth0, eth1, etc).

KCarterSr · April 23, 2021, 3:45pm

No I had already changed that.
Thanks

modzilla · April 23, 2021, 7:51pm

These lines are actually there for the computer to get access to the internet. If a PC needs to contact another computer outside it’s subnet ( a range where all computers can talk freely to each other - eg 192.168.1.1, 192.168.1.10 and so on - netmask), it will talk to a gw (gateway/router). The router then either knows where to find the address and acts as a bridge or if it doesn’t it will ask another router and so on. So if the routes aren’t set right, you’ll loose access to the server as it doesn’t know how to send the packets back.

It appears as if you’re in a different subnet than programster, so you’ll have to change netmask accordingly. Your interfaces config should look sth like this:

iface …
address 144.xxx.xxx.xxx
netmask 255.255.255.224
gateway 144.xxx.xxx.xxx
bridge_port enp3s0

KCarterSr · April 23, 2021, 8:56pm

@modzilla

Thanks for the info about that line. I’ll keep that line in.

About your subnet info. Thank you. I had already done that.

programster · December 13, 2023, 5:02pm

Just an FYI for anyone who comes across this post, things have changed since Debian 11, and one now has to add bridge_hw <device> now. I just tried setting up again from my previous tutorials and it wouldn’t work until I added that. I have since updated my tutorials, but people may wish to see the info at the hetzner docs.