Questions regarding internet, 10 Gbit/s Ethernet, network booting

Networking Networking Hardware
1

okay, I hope this isn’t going to be too much of a mess.
I am planning to remodel my home network, firewall and NAS. Also l’m thinking about switching to 10 Gbit Ethernet for a couple of devices. At least for the momend I’m planning to build most of the hardware myself from stuff I have lying around and stuff I can get for free/cheap from a local company that’s upgrading their equipment. (mostly I just enjoy tinkering and learning new stuff)
Right along the lines “My web search isn’t your networking degree” I understand I know nothing and need to double check some stuff.

  • Is it possible to just switch out my ISPs router for my own pfsense box. Worst case I expect I need to clone the MAC adress of my current Router or is there more to it? I have fiber directly into the building to a small box that is connected to the router via a optical cable with SC connectors.
  • I’m not entirely sure if I should use the existing Cat 6A cabling or switch to optical cables with SFP+. As the network looks now I’d need two 10 Gbit switches anyway or I need additional cabling which is kind of a pain. However, 10 Gbit RJ45 switches are expensive and the SFP+ types seem to be a lot more affordable. I don’t know which way to go.
  • Does anyone know more affordable 10 Gbit RJ45 switches? The Netgear XS708T costs 493 Euros!
  • I am still not entirely sure about when to use a managed switch and when not. (this one probably shouldn’t be that hard for me to grasp but it is for me)
  • Currently I’m using a lot of VMs with pci passthrough on my main machine and I switch between them with my raspberry pi. If I’m going to have a fast NAS anyway would it make sense to use network booting for the different operating systems/images instead of having them in VMs. I have never used network booting but it does not seem to be used a lot anymore. Is there a reason for that?

I know I’m leaving out a lot of details but I’m not sure what is important and what is not. My current setup is kind of convoluted because it developed over time.

Thanks for your help! :slightly_smiling_face:

2

Probably yes. It might depend on your ISP but Comcast lets me use my own equipment (and it saves me $10 a month)

I’m unfamiliar with RJ45 as I use SFP+ instead. In my experience with SFP+ some switches complain if you use offbrand tranceivers, but they should still work. I use a pair of 10gbit Quantas and they complain about their transceivers but they still run and perform well.

It’s good for layer 3 networking. Great for use with VLANs. (Have one port on your router with all of the VLANs on it, and have it trunk to the switch and pick the VLAN ports on the switch)

If you’re talking about network booting your KVM virtual machines: Network booting isn’t necessary if you’re using network storage in the first place. Having something like a Ceph storage cluster will let you tie libvirt directly into the storage pool and it will read the image over the network. The VM won’t even know that it’s disk image is coming over the wire.

2 Likes
3

If you’re going to the hassle of recabling the house, might as well put in fiber. The cable itself is dirt cheap now, finding someone to fusion on the ends for you will be the only challenge. If you’re careful about it, you could use pre-made patch cables.

10g over copper is really a mixed bag. Even with cat 6a cable there’s pretty severe distance limitations, and the copper gear uses a lot of power compared to the optic gear.

Both Ubiquiti and Microtik have very inexpensive SFP+ switch gear now. I picked up an 8 port 10g switch from Microtik for under $250. So far it’s worked with any optic I’ve thrown at it, including some real cheapies from ebay. The last card I bought from ebay, a dual port HPE sfp+ card came with real HPE optics, I was surprised, best $60 yet for a combo that would have been over ten times that retail.

2 Likes
4

Good to know, thanks! From my limited understanding I thought it should work but you never know.

I read about that. Good to hear it works anyway (at least in your case). It seems latency with SFP+ is also a lot better.

hmm… so I can have the image on the storage server but run it as a VM on my local machine. That seems so obvious I didn’t think about it yet, thanks! :grin:

1 Like
5

Yeah, that was my thought. The alternative is buying 1000 Euros worth of switches and I’m not doing that.

At one of the tubes I fear there is not enough space left to pull the connector through. I need to think about that. I’ll start with devices that have the most benefit from 10 Gbit/s anyway.

The Microtic switch is the one that started this whole thought process. This is when I realized 10 Gbit is in my price range and I have enough usecases by now that would benefit from the bandwidth.

Thanks for the input @SesameStreetThug and @gordonthree !

2 Likes
6

The VM config lives on the server? How does that work exactly?

7

I’m not talking about the VM definition. I’m talking about disk images. VM definitions are just XML files, they’re rather small so there’s no reason to want to move them off of the hypervisor.

1 Like
8

I got it now, I was mainly asking because I thought you could run the VM from any client without having to import the VM definition, which seemed strange.

For OP: I am unfamiliar with Ceph and if it offers any further advantages but this can be done with any Samba/NFS/iSCSI network share.

Going this way means running the VM on the client and having to import the VM definition for every client you want the VM to run on. Running the VMs on the PC either with local or networked storage (as the client mentioned previously) and accessing them through VNC or ssh offers much more portability.

Network booting does not really apply here and is quite complicated if you want persistence. Network booting is not generally used with VMs, SesameStreetThug’s solution is much better and generally you cannot network boot a VM image (e.g. a qcow2 image) without a hypervisor.

1 Like
9

You can manage VMs remotely via SSH, from a computer that isn’t running the VM. This is useful if you want to run a single VM and have it accessible from multiple workstations.

1 Like
10

@anotherriddle before you go out and buy any equipment or connect a Pfsence box to your network, please do yourself a favor and talk to a Level 2 tech from your ISP. You may not be able to replace your ISP provided router with Pfsense. I know for a fact that if your ISP is Cincinnati Bell you won’t be able to. I tried it didn’t work so well.

2 Likes
11

Thanks for the advice :slightly_smiling_face: I’ll definitely talk to the guys at my local ISP. I just didn’t want to miss anything obvious before bothering them. I’m allready known for being the guy that asks all the strange questions :grin:.

Thanks! In hindsight it’s pretty obvious that this should work with everything that I can mount in linux… :thinking: just didn’t think that far ahead.

Now reading up on the different types of network shares is on my todo list.

1 Like