Two offices, two servers(?), same data?

Hello, I need some help with my current venture.

A friend of mine has a small construction company that is about to open a second office (Site B). He has asked for my help in setting up their network and server infrastructure. We are setting everything up from scratch at both locations.

My plan is to place one physical server at each office that will handle data storage, Active Directory duties, phones, video surveillance and anything else that might come up.

I have ended up at this architecture for each server but I would very much like your input:

  1. TrueNAS Scale as the base OS
  2. FreePBX VM for phones
  3. Windows Server 2019 for AD. Each computer will be enrolled to the AD server and all user directories will be hosted on Samba shares on TrueNAS. This way employees will be able to log into any computer at any site and have access to their allowed data, as well as their user environment.

It is the first time that I will handle a multiple-site architecture and I don’t want to make rookie mistakes.

The need for two servers comes from the bad internet infrastructure. Site A has 20Mbit/2Mbit Down/Up ADSL, site B will probably have 100Mbit/10Mbit Down/Up. I considered hosting everything in the cloud but multiple people working on remote large CAD files with a 20Mbit connection will not work. Also, Site A’s internet connection is not reliable (and there is no cheap way around it), so we don’t want people not being able to work because the internet is down.

If I go down the two servers route, I need a way to “mirror” the two AD controllers (users, folder locations, etc.), as well as mirror the data and find a way to eliminate situations like concurrent editing of the same file on both locations.

Am I approaching this the right way or is another type of solution more preferable? Any input on this and any suggestions or recommendations will be greatly appreciated!

1 Like

I would ask him to outsource it unless you want to be “free” tech support for unlimited amount of hours, avoid AD and Windows Server etc unless you have a very good reason for it and be able to charge for it. It’s likely less hassle to use the user management in Samba and have a simple Powershell script that’s run upon login for each workstation. From what you’re describing they’re not going to be able to maintain it by themselves anyway which is why I would recommend doing it the least complicated way possible if not completely outsource it.

If you want to be the nice friend I would highly recommend to keep it as simple as possible, it may not be the best solution techwise but it’ll likely not matter in the end.

  1. Since you’re going for a storage server go with Core, it’s more robust and just works.
  2. Ditch the FreePBX idea, it’ll work poorly at best on DSL especially if you’re going to have users utilizing the connection for other things. I’m well aware of traffic shaping (HFSC is probably you best bet in that regard) but you’ll have so much issues with DSL and/or limited upload as it is it’s not worth considering. Get a PBX solution that works using mobile(cell) networks, it’ll work a lot better, there’s nothing to host locally. It’ll likely be a bit more expensive but it’ll pay off in the end because it will “always” work.

Have you considered that users may use the same files? That’s going to be an issue if you’re going for a dual server/setup solution. You might also want to look at 4G/LTE options to either replace and/or offload the ADSL connection.

Unless you don’t trust the users at all, running something like blocky (GitHub - 0xERR0R/blocky: Fast and lightweight DNS proxy as ad-blocker for local network with many features), Adguard Home etc and resticted firewall will go a long way and of course having some kind of antivirus-ish program installed. You can go IDS too but then again, if you’re not going to around to monitor it there’s limited usefulness of setting it up.

8 Likes

As @diizzy says, Windows AD, FreePBX, all that stuff needs maintenance… unless you’re bored of your skull, don’t do it.

…

…

This is a PITA to make so it works well… how many users, how many desktops/laptops?

…

So, I’m thinking… a bunch of laptops, a bunch of Syncthing folders over Tailscale as a virtual LAN, maybe one or two machines as extra Syncthing nodes, serving a dual purpose as long term “local” (non-cloud) backup storage.

I assume you’re getting office 365 / google workspace for each user too?

1 Like

Of course I am going to charge for it (friend rates), as system/network administration is my side job. I want to go with Scale because I can run Kube pods for any other services that might be needed. I am already running Scale for another client for a similar application, albeit being a single site setup. I am also much happier running Linux and KVM as opposed to FreeBSD and Bhyve.

That was the “no cheap way around it” referring to. At 1EUR/1GB, 4G is crazy expensive here. It might work only for PBX use but I would like to leverage the free outgoing call plan the ISP provides on both locations.

I am planning on running pfSense either bare metal or virtualized at each site, so setting up some bandwidth with the Traffic Shaper or Limiters for VoIP is not that hard.

6 local users to start, 4 at Site A and 2 at Site B, with 14 max for future expansion (6 + 8). I already have a working proof-of-concept environment for that with nested VMs on my homelab. I am also planning on adding the pfSense routers on the AD to have the ability to use the same AD user credentials for VPN connections for remote work (for on-site workers).

Not sure how we are going to handle Office yet.

You seem to have already made up your mind, I wish you the best of luck as you’ll likely need it.

as a total noob, can i ask a stupid question? why not go with a shared cloud storage (dropbox, google drive or microsoft drive)?
For example w dropbox, each user can restrict who shares their files. The files they work on are stored locally. Each person is responsible for their own security on their own pc. mobile accesss is available. The only restriction is two people working on the same file concurrently which gives sync errors.

1 Like

I don’t know man, ADs, pfSense TrueNAS stuff… replicated storage, roaming profiles, this can all work, … not sure it’s worth it.

… part I’m worried about is that setting all this up in the first place isn’t that simple, you’ll end up dealing with backups and hardware software upgrades, managing users, provisioning client machines.

It’s time you’ll spend.

On top, your friend, after paying for hardware they have to pay for (laptops, desktops, extra monitors), they’d pay for TrueNAS machine(s), and/or would you run pfSense in VMs (less hardware more hassle), Microsoft licenses for server related stuff.


On top of that, dust gets into a motherboard slot, TrueNAS dies, data is gone … for at least 2-3 days, what do people do?


Here’s an alternative plan (let me know location):

  1. Network wise:
  • buy/procure a pair of (used/refurbished) SFF PC, not latest gen 7-series/8-series, a bit of ram, stick a single large 3.5" HDD. (150-200 a pop per SFF server + 200 for the HDD for each site…).

  • for software I’d do TrueNAS Scale with Tailscale*, or just basic Ubuntu server with Tailscale and Samba for centralized network shares .

  • No VMs, no AD, you can provision storage accounts to share credentials across machines from a flat file, or a small script, easy peasy.

  • for network just get dumb switches and a horrible ISP router, but get a decent wifi. If ISP router sucks, stick a USB nic (or an LTE card or whatever) into your Ubuntu and enable routing.

Whatever you don’t setup, can’t go wrong, doesn’t need maintenance.

  1. Client wise:
  • get laptops, dedicated per person (1500-2000 per laptop, I’m guessing maybe + some external monitors not sure what software needs)
  • local accounts only, sync them across with Ansible
  • try Ansible roles to have repeatable/reusable/maintainable client configuration
  • keep most if not all data local-ish
  • get Tailscale onto each machine.
  • use Syncthing to have shared folders, include large HDDs into Syncthing, so that folks have data locally
  • office 365 is probably cheaper than your other options
  • beyond Syncthing shared folders, and centralized network shares, make sure you have full machine backups on each machine

  • Tailscale: it’s $6 a month/client, but you can get around it using headscale that you can host for your friend - it’s one container that needs basic http and takes almost nothing resource wise to host… no web UI on headscale though, but it works.

I know this doesn’t sound like sexy admin work, but … laptops die, servers die, 2-4 (or 6-10) people being barely able to do work being very grumpy for 2-3-4 days not being able to access data while you scrounge up a replacement, and then deliver a bill, sounds even less sexy.

You can do samba share on these small boxes, if you really want centralized / mounted shares, you can do Proxmox HA + Ceph, if you don’t want people to be stuck without central shares when upgrades go wrong or PSU dies.


So maybe 1d of work to set up, maybe 1d for provisioning all the clients, and maybe 15min per week to actively check and maintain and monitor everything is being auto updated, no alarms are tripping up + maybe 1d per year on upgrades, maybe round up to half-a-day with half-a-brain for each new user to provision both their account and hardware.

How much do you think your alternative would cost hardware+licenses+time?

4 Likes

I’d also be very hesitant going the inital suggested path, however I would elaborate a bit on @risk’s suggestion.

I’d a bit a hesitant to get refurbs, I’d grab the cheapest entry servers (Towers) or Workstations (with ECC support) you can find from Lenovo/Dell etc that can hold at least 1 NVME drive and 2 SATA drives and use those for TrueNAS.

Networking,
Grab something from Ubnt, or find something “embedded-ish” to run pf/opnsense on or as firewall/* whatever you feel comfortable with. For Wifi, grab something that OpenWrt supports and that’s based on Mediatek ARM64, setup as dumb AP and you’re done without needing to worry about cloud etc and if there’s a need to update software it’s not going to be a pain (I’d not recommended this route if you’re going to use it as a router/firewall). I would highly recommend that you get a managed switch, simply because they’re so much easy to work with and can cope with people doing stupid things like looping cables. Zyxel GS19**-series or such will be fine.

You’ll do just fine with Wireguard alone so I dont see the need to use Tailscale/Headscale but oh well.

I’d also recommend at least gettings Exchange Online, it works fine and people are familiar with it. As far as Office goes it’s a bit of a pain if you want to use non subscription variants as Microsoft really don’t want you to. You more or less need to split each license into a separate account otherwise it’s going to be a pain.

1 Like

To be honest - sounds a bit wasteful for basic gigabit file serving workload. There might be business aspects to it, refurbed stuff not looking professional enough and looking like the customer isn’t getting their moneys worth, but if it’s a “friend” going HA with old machines sounds like a better use of money IMO.


Re wireguard/Tailscale/headscale – Tailscale clients are more “convenient” to setup (e.g. small stuff like IP addressing taken care of), and you get to avoid worrying about physical network ACLs - which means you can use cheap off the shelf commodity/home or SOHO network hardware (easy to get, follow pictures on the box to get working).

I was looking to go for refurbished dell rack servers like R730s (we have the space), Brocade managed switches with 10Gb server-to-switch connections and Ruckus APs for Office and Guest SSIDs. A couple of PCIe NVME SSDs for VM and data storage, a couple of Sata SSDs for boot/root drive duty and some 3.5" HDDs for backup and NVR duties for each server. Replication from the SSD pools to the HDD pools and cross-replication between the HDD pools on each site.

Tailscale is the easy solution for mobile clients but I would stick to OpenVPN tunnels for site-to-site comms as I have done for years.

I would value warranty, ECC and piece of mind that hardware is new and not abused to hell. They’re pretty cheap anyway so it’s not like I’m suggesting to go for 1000$+ boxes. I don’t see what’s “hard” about deploying Wireguard clients, it’s very stright forward and it’s very easy to write a configuration file for import. I have no issues using it as-is for both mobile and “static” links. :slight_smile:

@sotiris_bos
Given your low upload OpenVPN is going to be fine even on dinky hardware but context switching quickly becomes an issue once speed starts to ramp up. The DCO kernel module is supposed to fix this but I’m not sure how well adopted that is yet.

R730s are like 7y old which is pretty crusty by now but performance-wise they’ll do fine, probably not the best choice looking at energy efficiency either.

This honestly seems like a job for a hosted file solution like Sharepoint / Google Workplace’s Team Drives etc. Installing a standalone domain just for this is gross overkill (especially as I’m afraid you don’t seem to grasp how AD works - “I need a way to “mirror” the two AD controllers”).

I can’t imagine you’re hosting gigabytes of data if you’re planning on syncing it on the fly over those incredibly low upload speeds (if you are, again I’m afraid this is unrealistic, even if you do block level replication), so something that allows the user to download a file, work on it locally, then sync back up is the way to go.

Ok, here’s a third opinion, initially trying to help you answer your own questions, but (like always) derailing later on.

  • Is cloud expensive for your client?
  • If not, would cloud not be a better deal, since they are not likely to grow fast very soon (at most 14 people)?

With Google Workspace, you can edit the same excel or word file at the same time with others. I think ms 365 might have something similar, but don’t quote me on that, never used it. With both, you also get emails and chat + VOIP (hangouts, or whatever google uses nowadays, or teams).

If you are dead set on fully local infrastructure, then:

  • Would it be possible to go lower with commodity hardware, but instead upgrade the internet plan at least in one location?
  • If you go with new hardware, are you (or rather your client) prepared to pay the premium for it?
  • Do you have a budget you are working with, or are you first trying to do a proof of concept to give a quote?
  • If you go with used hardware, do you have enough of it to ensure some amount of redundancy?

Not all companies need HA, but at least some replication between sites might be useful and gives you the option to launch a service or VM on the other side in case something goes wrong in one location. I’d still make one of the sites the main one and have it serve the other, the one with the higher upload bandwidth.

With Asterisk PBX you have the option to go with smaller codecs. G.711 a-law and u-law use quite a lot of bandwidth and are generally the default used in most PBX software AFAIK. We used to have a 5 Mbps dedicated line between Europe and Asia and we switched to g.729, with decent results. But this was a dedicated line, but it was also crossing the pond(s) through India to reach the SEA datacenter.

I think it wouldn’t be much of a problem if your sites are in the same country, but don’t get too far, like from New York to San Francisco. Marseille to Paris, or Napoli to Milano should do.

Speaking of sites and internet, if you are using the same ISP in both locations, ask them for a deal to get better bandwidth in their own network. You don’t really need more than what you have to connect to the internet (unless you have some heavy, many and frequent youtube users), but it would be nice if you’d get at least 50 Mbps (or better yet 100 Mbps) from one site to the other. Some ISPs have things like this by default, but you need to ask first.

  • Are you going to manage everything part time?
  • If you die tomorrow, or just want to end your IT career and move in a cabin in the woods, how easy would it be for your client to replace you?

These are questions anyone should be asking themselves. You should definitely make good documentation on the infrastructure, no matter what you use, but how easily (and cheaply) can you be replaced? The technologies you will deploy need to be easy to find knowledgeable people to hire (this is why you typically don’t deploy Gentoo or nixOS in production, even if they are better, most people only know Ubuntu, SUSE and RHEL).

I would argue AD is easier to manage, but finding someone who knows how to do it is hard and these guys ask for quite some money. And the documentation last time I tried learning AD was poor, no guides or anything. But maybe I’m misremembering.

I had a tech-newbie learn Samba via some shell scripts I made in about 4 months or so. He had no previous linux experience. And it wasn’t even something fancy, like TrueNAS GUI, it was plain CentOS. If you don’t get into configuring Samba as an Active Directory Domain Controller, then using Samba for CIFS should be fine (and cheaper). This gets rid of the need for Windows (both licensing and managing it).

Did you really have to go with some of the more expensive stuff? I have run HP gigabit Switches with balance-alb connections and ubiquiti unifi 5 APs, with real junk routers (seriously junk, 10 years+ uptime 1U servers and even 10 years ago they were deprecated core 2 duos with 8 GB of DDR2 RAM) and everything was fine for 300+ VMs and a few physical servers here and there (which later I retired and made them into VMs). And everything was gigabit, not even 10G. And we had 70+ employees. You are overspecing for 14 people, big time.

For the network infrastructure, since these are probably small sites, I’d go with cheap gigabit capable stuff, unless you find a real killer deal on 10G (which I doubt, compared to how many people are basically giving away gigabit stuff by now). Something like a ProtectLi router, but maybe cheaper should do. I wouldn’t go virtualized on it and I wouldn’t go second hand on the routers, but everything else could be. Up to you.

Making a site2site VPN with these would probably be hard, since you seem to be using some pretty awful internet plans, but it should be doable with hackery like DynDNS.


It just kinda hit me, but I think we’re going a bit too far with this, really. Both sites have terrible internet and I doubt they are equipped to handle the equipment.

  • Do you have a secure rack to put the servers in?
  • Is your rack going to be air conditioned?
  • Do you have a UPS to power all the high-powered electronics you were planning to get? For how long?
  • Is the electricity costs going to be worth it basically duplicating the infrastructure?

In some parts of Europe, electricity costs grew up 3x in price. Even back when I used to manage 5 racks, it was expensive. I was talking with my ex-colleagues a few weeks ago, when I left the company we only left 2 racks (which we collocated) and now they are trying to buy the best server hardware they can to lower electricity costs.

In all honesty, I think you should talk to your ISP and see if they are able to collocate a single server. If they can’t, look for someone who will, but typically with ISPs, you can get better bandwidth if you stay in their infrastructure and don’t go out to the “greater” internet. Worst-case scenario, you can go with dedicated VPSes (but these tend to be expensive for what they offer). But for 14 people, probably even a single 4 core 8-16GB of RAM VPS will suffice, as long as you have enough storage.

For collocating, get something newer and power efficient, maybe a 16 core epyc. Then virtualize the router on it (which I generally don’t suggest, but should be fine for up to 30 people - after that, the business has grown enough to make it worth going physical). Make the router a VPN and have everyone connect to the VPN with laptops.

Treat both sites like coffee shops, everyone just connects to a rando wifi, but use a VPN to connect to the infrastructure. Make a split-tunnel VPN (i.e. don’t redirect all traffic through the VPN, only what is necessary). Wireguard has that option to only give the routes to the internal network. You should be able to do the same with OpenVPN I think (just use wireguard).

Then get a cloud plan, or a VPS with tons of storage somewhere and do backups to it (I haven’t used restic before, but I keep hearing only good opinions on it, but considering finding people to know stuff, maybe you want to use something more easily understandable, like backupPC, which I recommend).

This way, you get to buy a single server, everything runs on it and the business has the potential to grow and even relocate without anything changing. And you also enable work from home, which I’m sure your client will appreciate when someone in the company gets sick and can’t come to the office, but is still capable of working (not to mention lockdowns).

Just make sure that if you get a collocated server, to get something like a Pi-KVM and a dedicated network for it, so you can potentially troubleshoot it from anywhere in the world, without having to ask the collocation support for assistance, which typically incurs costs. Just don’t open iDRAC or iLO on the internet. Even get a cheap Pi and make it a VPN to connect to, to then connect to the IPMI, if your server already has one and you don’t want to spend money on the KVM part of Pi-KVM.

But do ask for collocation perks, the ISP where I used to collocate servers were offering free internet KVMs during lockdowns (although they were so ancient, you were forced to use I.E. :face_vomiting: some old activeX stuff). Before that, they would offer free KVMs on-site, but that was only so you wouldn’t have to dress up in cloth hazmat and go inside the datacenter and potentially introduce dust and stuff in. Still, better than having to go downstairs and plug a monitor and keyboard in the servers (which we did when we first participated in racking the servers, because we had some strict connection requirements, although we were mostly just watching and guiding).

A single internet site will absolutely be a better deal than dealing with the split-brain problem with low resource interconnects. If you had a 200 Mbps synchronous plan on each site, I would have recommended some decent options with site replication and stuff, but as it currently stands, there are too many sacrifices to make, which would complicate stuff. It really could be doable to have 1 site serve the other and not have to bother with setting up the other site, but why bother when there are better options in this scenario that you don’t have to make any sacrifices, really.

1 Like

Everybody wants to just hand-wave away that “20Mbit/2Mbit” internet connection and “large CAD files”.

A local file server absolutely will offer much better performance for employees trying to work than hosting directly on “the cloud” would. There are plenty of options that will more efficiently sync changes between local and remote servers… even rsync will make a decent attempt, and putting the files in something like a git repo (or some other version control system which does proper file locking) might be pretty efficient. But even without that, just having all night to sync-up changes rather than people waiting on the slow internet all day will really improve productivity.

If folks want to keep pushing “the cloud” you should at least have strong recommendations on a native local client for a given cloud service that does all kinds of great local caching and extremely efficient replication, and maybe even a setting to only do so after work hours (though everyone needing to keep their PCs powered on every night would eventually cost more than a server and paid IT staff to design and maintain the system).

So I have set up several small businesses with similar requirements. Synology has been my go-to for this stuff.

They have built in remote replication tools and failover as well as vpn connectivity.

Also a warranty :wink:

I am all about TruNAS but when I do it for businesses unless they have the budget and the need for an iX factory system it is too much to maintain for me.

1 Like

I haven’t seen any mention of CAD files. Sure, it is a small “construction company,” but that doesn’t necessarily mean they would be working with CAD. There are places that still draw things by hand. OP would have to confirm what type of files he was planning to use Samba for.

The reason I suggested Google Workspace is because almost guaranteed people will be working with excel, so having access to edit files at the same time is a huge thing. I don’t know of other options that does this, but my knowledge is limited on this. Even Sharepoint only makes it so that a file gets locked when someone edits it (don’t get me started how awful it is). As for git, it would be good too, but you have to teach people to use git. Not ideal.

And you are overestimating what 20 Mbps can do. I get an average of 6.6 Mbps down on fast.com (with a max of 10 Mbps boost) and 19 Mbps up (yes, I get faster upload for whatever reason). I can even watch youtube as a single user. Without involving video streaming, I doubt they’d have any problem with 14 people working on a centralized collocated server.

I wouldn’t treat self-hosting in a collocated environment as “the cloud” because it’s your server, your hard drives, your OS, your everything except the internet (and even in your own datacenter, the internet connection is not yours either). Sure, I would treat a VPS as a cloud, because you don’t know what’s running behind your VM. Not to mention cloud services and SaaS in general. But don’t just lump everything in together as “the cloud” because you aren’t breathing on your server.

Anyway, when I collocated my servers, I wasn’t even worried that 100 Mbps down / 30 Mbps up wouldn’t be enough (which is what we had at the new office). It was enough to even allow everyone to watch youtube and still not feel sluggish when working on the samba server (VM) in the collocated hardware (through different ISPs). There were 20 people at the office, with the top reaching 25 (some just went in the open space, that is available for everyone in the building).

We had a 4G router that we used for when the internet was out, but before covid, it was used for outdoor days (once every few months, except during winter time, we used to work from outside bars, coffee shops or restaurants for the whole day). And even this was enough to satisfy at least 15 people. I don’t even remember the speeds, probably something like 30 down / 5 up, which wouldn’t be too far off from what OP has with ADSL in site A.

And my users were using VPNs. And I didn’t have a split VPN at the time and whoever got inside our network was not allowed to go out without going through our squid proxy. So literally all their junk traffic was going through the 100 Mbps pipe through our collocated infrastructure (or 20 Mbps on the 4G router) and out to the internet and the UX was still fine. Again, the office infrastructure was treated as any coffee shop, except it had a NAT and firewall (opnsense) and a printer on-site.

So you’re telling me that 3.3 Mbps per user will not be enough to work on a remote Samba server? Sure, the 333 Kbps per user would be a little atrocious - if all your 8 users would be trying to upload large files, all at the same time! I would bet money that will not happen at least 90% of the time.

Worst case scenario, you need 1 hour and 40 minutes to upload a 250 MB file, time in which everyone involved who has to view that file (and who uploads it) can do something else. I don’t know how large CAD files, if they are indeed working with CAD. And keep in mind that the same limitation would apply if they work between two sites, if someone from the other site has to see the file, so this is not inherent to the collocated infrastructure.

A case can be made if and only if the people in site A and site B never need to view large files from one another. That means that 2 local Samba servers would make way more sense, so people in the same site can share files with one another easier and the server can take care of backing up the files, if they work directly off of the samba servers.

But Samba doesn’t need a lot of power. I inherited a core 2 duo with 8 GB of RAM samba server, with 2x 1TB md RAID-1s that were serving 70 people. And the server wasn’t breaking a sweat. It was among the first things I virtualized (and funnily enough, still made it a 2 vcpu and 8 GB of RAM VM, but I moved from CentOS 6 to CentOS 8 - RIP, then I used centos2oracle.sh script to convert it to Oracle Linux). The reason I virtualized is because I wasn’t trusting a core 2 duo with a lot of dust in it and 5 years of uptime to not crash disastrously when it mattered most. It was running on borrowed time.

You can literally run Samba on a Raspberry Pi 4 (or an equivalent SBC) and users wouldn’t notice a difference, except for the network throughput if you go with 10G equipment (which again, not necessary for only 14 people). Not even sure you can make a case for a 2.5Gbps infrastructure (pretend that it’s cheaper than 10G for the argument, even though more often than not nowadays, used 10G is about the same price as new 2.5G), since people will be likely using laptops and be on wifi.

Balance-alb can be used to load balance traffic on 1 Gbps and get more out of it (and get some redundancy in the process). LACP is preferable if your switch supports it (you can get cisco switches that support it really cheaply, not to mention other brands like emc, hpe, fs.com, zyxel, huawei and others).

I don’t know something more efficient than ZFS send through a compressed connection (i.e. piping through bzip through ssh).

That’s assuming that people don’t need to view the files in the same day, meaning that someone would have to upload it sooner rather than later. But again, I don’t know the workflow, sotiris needs to confirm.

Might I suggest using the search function.
I didn’t make it up, that was a direct quote from the thread-starter:

Perhaps everyone arguing for cloud services just didn’t bother reading through his post…?

Any advanced client-side deduplicating backup software will do better. BorgBackup is a good place to start. Intra- and Inter- file identical blocks can be eliminated, even if non-aligned. I know deduplication can be enabled on ZFS to give it some of that, but really only at your own peril.

I’m interested how this is going to play out, I have my own Architectural/building firm.

That internet speed is a bummer, is the construction company doing any drawing in-house, or are they just opening drawings to view or print them? When you say large, are they over 100MB and do they contain XREF’s?

I’ve been an almost daily AutoCAD user for over 25 years (Oh god, I’ve become THAT guy), and something that’s always irritated me is how local cloud folders can sometimes annoy the app, almost as if it’s a semi-locked file and the OS/Cloud app can’t quite understand it. It also creates a few temp files, which can fortunately be stored locally if needed (*.BAK and * .*sv$).

I used Dropbox for years and most of the time it was OK, but it occasionally caused me problem - note this related to open and actively revised files. I gave Google Drive a go as well, but the same issue occurs. I’ve used SyncThing for the last 6 months and I’ve had no problems. On Windows you do need to be wary about file/folder creation/moving (ghost folder copies), and to avoid any problems I tend to create project structures on my desktop and once ready, only then move them to a SyncThing folder.

I have a Daily TrueNAS Core that is the master SyncThing, 2 workstations that are connected, then when a a good amount of work has been done (perhaps by mid day), I fire up another TrueNAS Core machine that auto-sync’s. Weekly I turn on 2 other machines that receive snapshot/replications of the SyncThing dataset. As a precaution, I have another TrueNAS machine that receives snapshots on a monthly basis.

For me I only go through this palaver because my files are worth money, each cad file cost someone between $1000-5000 to create.

CAD aside, I guess they’ll have construction documentation needs as well, photos of progress and phases, etc.

I’m not sure if any of this helps, I did drivel on a bit.

1 Like

Ok, my mind filtered that out somehow. Still doesn’t invalidate most of what I said, just the google workspace part. I’ve read the entire thread, before starting my reply, since nobody mentioned what I’ve added.

And I gave a re-read of the first post.

Unless the ISP doesn’t offer better intra-network speeds, then collocating a server in their DC is not ideal. But I’d be surprised if that’s not the case.

Businesses with the need for an active internet connection should look into active-standby methods (they are the cheapest, even if you don’t make use of load balancing your traffic), like adding a 4G router as a second WAN that becomes the default gateway when WAN 1 is unreachable (or on manual trigger). Unless they are completely in the middle of the boonies, with no 4G towers (which I doubt, considering he mentioned Euros) then I see no reason for site A to go offline.

And we still don’t know what this implies.

Will they have to share projects between site A and site B on the same day? If so, no amount of servers in both sites will mitigate having to wait for large projects to get uploaded to the other location.

If they just need to save their projects locally, then that’s fine and going with a Samba server in each location will do them wonders. And if the users generate 10 GB of data per day, transferring that through an unreliable connection overnight will probably work just as badly as trying to send it during the day. If the connection is unreliable during the day, what would make it not disconnect at night? For 10GB, it would take 11 hours on 2Mbps upload speed, if it reaches the max upload at all times.

2 sites gives you the ideal way to set backups and cross replication, but with a slow and unreliable pipe, it will do you no good. And it’s fine if it’s slow and reliable, rather than fast and unreliable, but both?.. We aren’t wizards and we can’t make work magic to make something bad be usable.

Back to two servers, I would set up samba on both sites, replicate it with zfs snapshots at night, then copy between pools locally what items have been added. I would do it with rsync for each user’s folders (so they also delete what things were deleted from the other side) and another one for the shared projects. Still, a conflict is likely to arise if two people edit the same project in the same day, the next day, the last one who saved will override all the other person’s work in both sites. You need a lot of coordination.

I wouldn’t suggest ZFS deduplication, at best, just back up everything using BordBackup like rcxb recommended, or restic, which looks inside file blocks and deduplicates data that goes to a backup repo, then use something like Czkawka, fclone or rdfind to find duplicate files and remove them, maybe symlink them if needed.

Without the unreliable connection, I’d suggest to set up a samba at site B and have site A connect to it as a test, to see if the bandwidth is really that much of a problem. But with this being the case, I still think it’s far cheaper to have a collocated server that you manage, and doing redundant internet connections at site A if that is really that big of a problem. If you can make it load balance, that’d be even better, but I don’t know how to do that without BGP (which is expensive and overkill).

You can be dang sure the collocated server will not go offline, or go offline for long because of the ISP or your collocation provider (things like internet going down in the DC are not unheard of, but are rare and get fixed fast). And then, you just need your people to connect to the server.

Colocation is going to be extremely expensive, I really hate my ISP (and they are the best of the worst).

5G/4G does not cut it. They limit upload to 20Mbits/s.

I am still waiting to get a quote on their “IPVPN” and “Ethernet from location A to location B” plans, but I am pessimistic since they offer these plans for multinational enterprises working in the tourism sector and I suppose their prices will be very steep, as are their leased line options.

Sort of an upside is that site B is getting a 1G Down / 100 Mbit Up connection, but that is not available at the location of site A.

I am currently looking at setting up a Samba domain at site B for starters, instead of going the Windows Server AD DC route. I would very much appreciate an option that allows for an active-active sync strategy, even (or especially, to avoid conflicts) if that means that if one user opens a file, that file remains locked for everyone else until said user saves their work. Is there any such thing as Samba remote caching or Samba daisy chaining?

Edit: Can I handle this essential replication with something like GlusterFS? I have never used GlusterFS before so I don’t know.