Need some direction on multi building network planning | Project Log

Hardware Hub Networking
1

There are 3 buildings on the property that would have their own networks. One building would also have the WAN drop and server closet. Each building needs to access the servers and WAN.

Know enough networking to be dangerous and not sure what the right questions to ask to get started. Have managed switches for each building, Router/Firewall appliance, NAS and servers all ready to go.

Managed switches going to do the work or does it fall on to the router? Router software is not set in stone at the moment but have basic experience with pfSense.

I like to learn as I do things so not asking for a step by step, just some leads on where to start. Set up a mini lab for proof of concept and then scale up.

Thanks.

2

Are the 3 buildings a house a garage and a shed, or 3 office buildings? … If closer to former and there aren’t a ton of users, it’s all as if you’d just tried to network 3 rooms.

pfSense is ok if you need a firewall and dislike router appliances (e.g. don’t want a Mikrotik or Ubiquiti and don’t plan on spending huge money on Cisco/Juniper/Arista/Extreme/…) Something like a plain old Debian, or OpenWRT if resources are tight is better IMHO as it’s more flexible and usually faster due to better drivers.

3

I know enough to know one thing. Planning.

Theres so many questions to find answers for.

Budget?

Are the buildings connected? if not how will you connect them for internet access never mind building to building LAN.

Whats the requirements for each buildings use? What are the current number of users and use cases for those users, whats the expansion expected to be over the course of the life of the network.

Distances? Where are the buildings, how will they connect?

will your WAN drop building be sufficient for a server closet for all three buildings or will you need something in each of those buildings? (you’ll at least need some networking to handle each building in each building i imagine.)

Server infrastructure all internal or any public facing DMZ equipment to consider?

Just computer network traffic or is there phone/VoIP to consider?

Security? (See wendells recent video about a couple of considerations for security of your equipment and protecting your server closet.)

Equipment manufacturers? Who’s the preferred manufacturer? Dell? Cisco, Juniper, HP, a mix? Something to carefully consider here, if you use pfSense, you’ll be the one supporting it, probably until your dead or its replaced. Is it good enough for your needs, is it worth the potential extra maintenance on your part for the potential cost savings? It gets tricky, 3 small buildings maybe you only need a few pfsense appliances if you know them well enough but they may not be the best choice if you end up having to support them because your the only one who knows how.

We have a small network that’s just one external network router and then just firewalls and switching, the routing is simple enough that dedicated routers just aren’t needed and the basic routing we can put on the firewalls is sufficient. Firewalling tech was more important.

Speaking of firewalls. you’ll need to consider where more advanced stuff is required, ips, ids, etc. filtering, access control, etc.

That’s some stuff that’s probably worth considering at least. Once you start digging down into requirements, budget, use cases etc for buildings and future expansion i think you’ll get a much better idea of what you’ll need and what you’ll need to plan to build into a network design.

@DeusQain and maybe @wendell might have some more refined insights to specific questions.

4

This is all within residential context. 5 acres with 3 houses, 1 shop (planned WAN drop and closet), and various wireless APs / security camera poles scattered about the property. Shop has a back room that is being retro fitted for a suitable server closest. Just caught @wendell video on server closest do’s and don’t, so great timing on that release!

Burial conduit will be ran to each building and cat7 ran to start with fiber to follow (ambitious plans for 10gig). Houses are within 100m of the server closet.

Initial thought is for each building to have their own private network; one house pc can’t just up and connect to another house’s pc. However, each building can reach out and see the NAS, Plex server, Cache server, etc. Also, get internet obviously. Traffic will be basic entertainment and home office use.

Budget is flexible. While no hard limit is in place, don’t have massively deep pockets. Dropping ~$600 on an appliance is not out of the question. However, equipment is in place to get setup initially. I’ll have to post up some model numbers.

So a router for each building overkill? One router/firewall handling the entire network suffice?

5

Yes, and no. If your point of egress is that of the “server room”, it makes little sense to give equipment to the specific sites. However, if there will be access points that will be handled by each little house by itself, with self-fixing in mind for the occupants, go ahead - it sounds however like there will be more than “a wire” coming out of the wall for each house… But that you can always fix later on. One/one cluster should be sufficient any day of the week, and a simple physical and virtual separation will do much.

Remember what was mentioned before - planning. And add separation of duties. Plan ahead as in preparing, not always taking the cost in money directly. Better to have a step by step build-plan, than having to revert separation due to heavy costs.

Also, 100m is the theoretical limit. Even with cat6+, it will be up to your equipment to be able to handle it - meaning you will have to have something powered in each house anyway… 10gbit ethernet equipment might sound sweet, but could cost you more than setting up fiber and converters that you can future-proof. Just an idea.

6

So would starting out with a topology diagram be good to start? Setting up a small scale lab so to speak in the shop would prove if our plan works, right?

Have a running list of equipment on hand. Of the managed switches, there is Dell 2708, 2724, and 2824, and a TP-LINK TL-SG1016DE. Routers include a pfSense SG-2440 and TP-LINK TL-R600VPN. Wireless APs are various Ubiquiti models.

I am pretty sure that inventory of gear will support a configuration that meets our needs.

1 Like
7

Yes, that indeed is a very smart choice.

8

Lets start with naming.

Demarc == WAN Drop building
B1 == House 1
B2 == House 2

One PFSense machine is enough to run the whole network.

Direct Burial Single Mode Fiber Optic cable will be your friend.
Look into 10GB Line Cards for connectivity between buildings, so that you will maximize connectivity between B1/2 and Demarc where the big servers live. If 10G is out of your budget, you can install 1G Fiber uplinks and upgrade later if you desire.

Your Topology can have multiple networks with ease. The connections to each B1 & B2 can be on their own network, and everything connected inside the Demarc can have their own Network. The PFSense machine can handle your routing, or you can connect L3 Switches together and create Access lists.

The issue you will run into with Copper cables is distance between buildings. Also, I wouldn’t use Cat7 for anything. In order to get 10G speeds you need their special connectors, and you need to properly ground out the connections. It’s annoying and not worth it. Cat7 is mostly deprecated these days.

This is just some initial thoughts, and I would consider these the tip of the iceberg.

1 Like
9

Do you have any links/articles on Cat7? My cohort settled on running the Cat7 but nothing purchased yet.

Is that caution only applicable to long runs? Is Cat7 suitable for interconnecting within a rack?

10

Cat7 is bs. It’s not any kind of standard but the cable is usually beefed up in some way meaning more expensive and harder to work with.

As @DeusQain said above “single mode direct burial is your friend” for connecting between buildings … that’s because it’s easy to upgrade speeds and reuse the same cable. You can also bury 1" or 0.5" ID HDPE tube and pull your own slightly armored indoor rated fiber (it’s certainly less <50c per meter)

For within buildings: cat6 or cat6a for cable length above 55m is enough to get you 10G ethernet. Even cat5e will do 10G within a rack or up to 35m if I remember correctly in case you already have random cables lying around. What you need to make sure you get is solid core copper UTP cat6/cat6a and none of that copper clad aluminum stuff. Cat6 is easier to work with than cat6a.

In terms of topology, I’d do one switch per building, one VLAN across all buildings for managing network gear. One VLAN across all buildings for guest network (only internet access).
In addition to that, one VLAN per house for “resident PCs”.

Your Dell 2724 and 2824 are ok-ish, … you can put $5 gigabit transcievers from fs.com into each to connect them to gigabit fiber. I’d save the 1016DE to put under someone’s TV, not a bad basic switch but spending 20-30 on a media converter to give it a fiber port is not worth it. TL-R600VPN is trash IMHO but well known and someone might be looking for it on eBay to implement dual wan failover using it, if it’s in good shape it might be sellable.
The 2708 is 8 port pure copper 1Gig, …

You’re missing 1 core switch and WiFi APs.

As a “core switch” (I guess it is even if it’s a small network) I’d get something like Ubiquiti ES-48-LITE or a Mikrotik CRS328. They have some 10G fiber ports to connect your router and storage/media box (pfSense for now). And another couple of SFP for the other buildings/houses and enough copper to use as a local switch.

For APs Ubiquiti uap ac lite or uap ac pro is not a bad choice, you could get Mikrotik wAP ac or hAP ac^2 to serve the role. (The latter have better specs / more features for the money but are less fast when it comes to pure WiFi).

11

Technically Cat7 isn’t recognized by TIA/EIA. It has shielding around every pair, and is designed to work with GG-45 or TERA connectors.

Within a rack, unless you are doing 10Gb over copper, Cat5e or Cat6 will work fine, If you are doing limited runs in the rack, Cat6 will be fine, but the more cables you cram together the more likely Alien Cross Talk will become a problem.

Smaller installations it usually isn’t an issue, but It can be.

I agree with @risk in regard to your switch situation.

12

Ubiquiti APs we have already. AC-LITE, AC-PRO, and AC-LR. Going to add on or two of the Mesh APs for a detached garage here and there.

A core switch is on the list to get. Eyeballing Ubiquiti switches since I enjoy their products so far. For expandability, would probably want to roll with one of the 16 XG switches. Though not sure what the comparisons are between the UniFi and Edge series are besides looks.

Think we are are settled on single mode fiber between Demarc / B1 / B2 / B3 / B+n, Cat6 in the Demarc, and finally Cat6 for each building’s network.

Working on a topology diagram little by little when I have the time. Still settling in and all have day jobs…

What SFP transceiver is recommended? Not much knowledge on fiber and see quite a few offerings in that $5 range suggested. Am I needing to do the Dell compatible model to match my switch? Then whatever other brand for the core switch?

1 Like
13

So really nice Customer Service at FS.com. Got linked to what modules I need.

Though multimode fiber was suggested. Asking what is the reasoning behind the two, multimode is suggested for runs less than 550m and singlemode going for 2km and longer. Leaves a lot in the middle though lol.

Got links for both types of fiber modes so if single mode is still recommend here (runs are max ~70m) then will roll with that.

14

I run SM and MM in the vault. I have experienced very little difference between the two and these runs are <10m.

At the end of the day it’s all about cost vs limitations.

They run MM at the GWCC, but for DreamHack because of where our NOC was, compared to where the drops were. We would have exceeded 600m for the longest of the runs. There is no harm in running SM if it isn’t cost prohibitive.

15

Hey, this sounds like my kind of project!

You’ve already been getting some good advice, so I’ll lay out how I would do it and you can take whatever parts are useful to you.

Gateway/Firewall

Pfsense or OPNsense on a capable, but not overkill 1U server (with AES-NI of course).

This will handle your LAN to WAN traffic, so IDS/IPS (Suricata), basic internet firewall stuff (block private IPs coming in from the internet, blacklists, blah blah), QOS, VPN tunnels, etc.

Router

I think that Unifi makes a lot of sense for you. With the Gateway handling the IPS/IDS and other heavy lifting (which Unifi routers aren’t particularly good at), you can give yourself the gift of easy Unifi deployment/management.

This will just route LAN traffic, so maybe some light vlan/firewall rules to separate admin, management, surveillance, basic users, etc. It’s mainly just there to push packets, unencumbered by WAN concerns.

One caveat here is that if you only have 1 public IP address, you’ll need to manually disable NAT on the Unifi router (presently can’t be done through the management interface) and have the gateway handle NAT. If you have a /29 block of IP’s (or more), you just need to make a point-to-point connection between the gateway and Unifi router, letting Unifi handle NAT (this will consume 3 public IPs).

As far as which model, if price is no object, you might as well get the XG. The Pro gateway is looking a little long in the tooth.

Switches

You should go Unifi all the way here to take full advantage of the centralized management. They should have everything you need in their catalog except for a 10GBASE-T access switch. I don’t know why they don’t have one yet. I believe the Netgear options are decent. You could also just go all in on SFP+ and use the XG-16’s (yes I know they have 4 10GBASE-T ports, but that’s pretty meagre if you’re looking for 10GbE everywhere).

APs

Ubiquiti has an AP for everyone. I’m sure you can figure out what suits each placement best.

Cabling

I would run redundant fiber lines between each building, creating a loop if possible for maximum redundancy.

The CAT6 vs CAT6A thing boils down to an inconsistency in standards. CAT6 lists 10Gb/s, but 10GBASE-T explicitly required CAT6A (I believe it makes no mention of CAT6). For me, I always go with CAT6A for 10GbE. It just eliminates a variable if you have a sketchy connection.

Rolls of CAT6A aren’t that expensive anymore, and the patch cables aren’t expensive at all.

Cameras

You might as well go with the Unifi cameras unless you have something else in mind. IMO, it beats configuring Zoneminder.

It’ll give you an excuse to get one of these babies so you can keep everything rack-mount form factor (no Cloud Key or little NVR).

Anyway, hope some of that helps. Obviously, there’s a few ways to skin a cat here, so you could take none of this advice and still get what you need.

16

At 40G/56G/100G , transceivers for multimode are cheaper, but require weird fiber terminations in some cases; it’s best if you have a lot of stuff to connect and at the same time have in house fiber splicing skills (in your case you just want to get preterminated single mode fiber, and at 70m between buildings get 100-120m of fiber just in case you want to move the equipment around and you don’t have to worry about splicing).

Single mode is the “install once use heck of a long time” solution, you pay extra for transievers but just keep reusing your lc-lc upc-upc 2 core cable for 100G, and if you want you can get temperature stabilized color lasers at different frequencies and mux multiple 100G or multiple 10G into a single fiber. Ideal if your fiber is burried and you don’t plan on changing it in the next 20-50 years.

If you get a 1" ID HDPE poly tube between to use as a conduit, you can pull fiber as needed (e.g. you really want multimode 100G in 5 years time, but have single mode). Professionally there’s fancy low friction special conduits and lubrication that goes on the cable and machines that push fiber into the conduit; at home and at short distances like yours you can get away with a fishing line connected to a plastic bag and a vacuum cleaner to pull the fishing line and dry pulling the fiber using the fishing line.

Can I see your shopping list from fs.com?

17

We have a running Google Sheet can be viewed here https://docs.google.com/spreadsheets/d/1dga4p9olpmCi12dJp0vtcZzhzZN6XnM2D7iCrqq0tf0/edit?usp=sharing

I dropped in both fiber modes and planned equipment from Ubiquiti.

@oO.o Speaking of cameras, we have been trying out these models on Amazon and like them so far - https://www.amazon.com/dp/B0777PNBY4. A mesh AP will most likely get installed next to a camera location. For a gateway/firewall, the Gateway Pro is what we settled on since it will have the UniFi controller backed in and cheaper in our case. I have a little pfSense box but is old, new ones cost more than the Gateway Pro. Same if we sourced and built a 1U server. In the end this is just a multi-household network with some shared resources but we do have a sort of “go big or go home” ideology haha.

@risk and @DeusQain We’ll roll with single mode fiber in pre-terminated lengths. What are the stress limits pulling the patch cables through conduit. Pulled thick bundles of ethernet and coax cable before and it was a massive PIA. Used a tractor and poured soap down the pipe to eventually pull it the whole way. With a 1" tube (any sources you recommend?) I imagine pulling a couple pairs of fiber through will be quite easy, right?

18

I’m definitely not an expert on surveillance. My experience is that I tried setting up Zoneminder, Xeoma and Kerberos (not that Kerberos) and concluded that they were all too clunky for the client to use without constantly asking me questions. Ended up going with Unifi. I haven’t heard anything from them since.

That camera is definitely a good value if you’re happy with it. I’d say, just make sure you have a management system in place that you like.

That should be fine. Just be aware of the throughput limitations for IDS, QOS, etc.

You can always start with the Unifi and then drop in the pfsense/opnsense box later.

How fast is your WAN (sorry if I overlooked this)?

1 Like
19

We are on 300Mbit Business Class from Comcast. Aspirations for Gigabit if we can work a deal with a contract. My brother works from home and my goal is to as well within the year.

20

Symmetrical?