New home = new network. Need help with checking my network ideas. Thank you!

Networking
,
1
To help with reading, I created a Table of Content

Intro

Hello all!

First things first: this is my first post, I just joined the forum to post this. I’ve been watching LevelOne Tech videos for years, and the team always speaks highly of the forum community. I need a little help for networking setups and I feel I’m not proficient enough to not make mistakes. Therefore, I tought I’d ask for help here, to people who will know much more than I.

Also, I think this will be quite a long post. I feel I have to explain and describe a lot of things to be sure to be thorough enough because networking stuff needs to be. So… Sorry and thank you for reading, I guess?

Finally, since it’s my first post, a little bit about myself: I’m a 30-something french professional AV systems engineer (think meeting rooms, videoconferencing, auditoriums and such). Long time gaming and computer enthusiast, opensource advocate, privacy and online security tight-a**, I also dabble in networking (mostly surface stuff, both for personnal home networking and for professionnal use).

And please “forgive my french”. English isn’t my native language, but I tried to proofread as much as I could, sorry if some mistakes slipped through.

Now that that’s out of the way, on to the show!

What I have today:

I’m buying my first appartment and will be moving in in a few months. I’ve always wanted to create my own home network infrastructure, and I feel that this will be the perfect moment to start from scratch. To understand what I started to think about, let’s talk about what I already use.

The devices and use-cases I have:

  • Main Gaming/Workstation computer
  • Home entertainment/media consumption PC that doubles as a networked storage solution
  • IoT/connected robot vacuum cleaner
  • an android tablet, used as a remote control over wifi for several different pieces of software

Today I use my internet service provider’s all-in-one box (modem, router, DHCP server, WiFi access point) as the node to my home network. I use my home entertainement PC as a do-it-all box. It’s old hardware I had laying around that I repurposed: Intel i5-2400, 8GB DDR3, no-name 120GB system SSD, 11TB of HDD in one box. I installed a Ubuntu OS and it stores all my media files (movies, TV Shows, music) as well as personnal files (pictures, backups, etc.). It is not setup with any redundancy (RAID), but it is a redundant storage since all the personal/critical files are also copied on 2 other seperate drives in my main PC, as well as off location on a relative’s NAS. Except the media files, but those are really not critical and don’t need redundancy.

For media consumption, I mainly use Kodi on all my devices and I set up a seperate SQL server to share my database between all my networked devices, instead on a local one on each. This way, a movie will be marked as “seen” on all devices simultaneously and I can pause and resume playing a movie between 2 devices automagically. Works great. Since all media files are stored on the Media PC, I use simple smb file sharing on the network to access those from all devices.
I have both a Logitech Harmony universal remote (with a USB IR receiver) and the Yatse Android App on my tablet to control the thing. Perfect use for me.

For music, I use 2 seperate pieces of software that I would like to consolidate. On my main PC, I use iTunes ('cause I’m still rocking that iPod Classic, baby!), with music files stored locally. On the media PC, I use Rythmbox (also with files stored locally), that I find good enough, although I haven’t found a decent remote Android app to work with it yet.

Now, one of the goals I have for the new setup is to have everything properly stored on a NAS and have everyone target this NAS for reading files.

The new home, the new network and my goals:

  • New place has Fiber to the Home, and IPS offers a cheap $50/month 2000Mbps down/600Mbps up plan (yeah France…), so I’d like to take full advantage of that.
  • That means, at least, 2,5Gbps networking for devices that supports it
  • I want to run my own pfSense router to be able to set everything as I want and use the ISP’s box as a “dumb” modem
  • I want to have at least 3 seperate VLANs, with 3 separate SSIDs to separate devices on the network, acces, and improve security
  • I want to install a server box to host different VMs, hosting different services, including (but not limited to): a NAS, a home-automation server, my media file SQL server

Here’s what I thought of:

The modem:

ISP has 3 models of boxes available. The most high-end one supports FTTH and 2,5Gbps LAN (1 port). I would like to disable every “smart” function of the thing (DHCP, Access Point, DNS resolution, etc.) and only use it as a modem, connected to a pfSense router that would do everything. As far as I read online, it would mean set it as a DMZ server. Am I wrong? Is there another (safer) solution I should look into? What are your thoughts?

pfSense router

Up until very recently, I was thinking of having a pfSense router run as a VM on the “does-it-all” server. But then, yesterday, Wendell called it “the forbidden router” in his most recent video so I guess I won’t do that? All jokes aside, he made very good points. So I was thinking of buying a NETGATE 4100 BASE PFSENSE+ SECURITY GATEWAY. It would host my pfSense router and supports 2,5Gbps networking. Any feedback on those devices?

Having my own router means several things:

  • I can use a different DNS resolution server than the one used by my ISP, thus optimizing privacy. I was thinking about Cloudfare’s 1.1.1.1 service. Any thoughts on that? Do you recommend other solutions?
  • I have a paid subscription to PIA VPN services. Would it be possible to have it run at a router’s level and have every internet packet go through VPN tunneling and encryption?
  • By using both a different DNS resolution server and a VPN service, I would like to create a network wide add blocking system, like pi-hole. Is this something I can setup directly within pfSense or do I need to have it running on a seperate VM on my network?
  • I would like to set up at least 3 seperate VLANs with: 1 main VLAN with everything that need to communicate on the network (Main PC, Media PC, Android tablet, NAS, etc.), 1 seperate VLAN with dedicated WiFi SSID for guest use (with only internet connection, and no access to other devices on the network and a 3rd seperate VLAN for IoT things (so far, only my connected vacuum cleaner, but the list might expand) for only internet access.

Am I missing something? Is it something blatantly wrong with my train of thought?

The switch:

To provide wired networking to all devices that need it, I obviously need a network switch. It must support 2,5Gbps networking and VLANs. I found this QNAP QSW-M2108-2C (sorry I can’t post URLs). Any thoughts on QNAP devices? would it support what I’m trying to do here?

The access point:

As stated before, I would like to assign 3 different SSIDs to the 3 different VLANs. I could even go as far as having only one SSID broadcasted (the guest wifi) and unfiltered, while the IoT and main one would be hidden and with MAC address filtering. I found this TP-Link EAP610 (Sorry I can’t post URLs) that seems to support both multiple SSIDs and SSID to VLAN mapping. Is that all I’m looking for?

Note: the appartment is fairly small, a single Access Point will be enough, no need for mesh.

The almighty all-in-one server:

I found a website to buy refurbished servers on the cheap, and I found this HPE ProLiant ML350p Gen8 Server (sorry I can’t post URLs) for ~$500. Specs are 2x Xeon Eight Core E5-2665, 64GB DDR3. It comes already equipped (on-board) with 4x 1Gbps ethernet, but I can cheaply add a 2,5Gbps PCIe card. It also has a 6x 3,5’’ HDD bay associated with a HP Smart Array P420i/512M FBWC Controller RAID controller. I would stuff it with NAS rated HDDs and use it to run my VMs.

  • What virtualization server system should I use? Is ProxMox a good bet?
  • For the NAS VM, I was thinking about TrueNAS. Any thoughts?
  • I want to play around with home automation, so I will probably install a home-assistant VM as well,
  • To host my SQL media library, I’ll probably just install a Ubuntu or Debian distro that I would be able to remote in.

The death of Dropbox:

I try to use as little as cloud storage as possible (“the cloud doesn’t exist, it’s just someone else’s computer”), although I do still use Dropbox for 1 feature I like. My personnal Dropbox is always synced with my actual storage server, so whatever I put on Dropbox immediatly drops onto my personnal storage. With that, I use the Dropbox Adroid app feature that, whenever I take a picture, it is sent to Dropbox. Therefore, whenever I take a picture with my smartphone, it is send to Dropbox that is immediatly synced to my storage at home. I love it.
Now, the goal is to try to do the same, without going through Dropbox. Back in December 2020, Ryan made a video about NextCloud. Although he hosted it through Linode, it seems to be available for TrueNAS and I believe that the NextCloud app for Android could do the same “automatic sync” of pictures.
The issue here is: so far, I was talking about having storage available on my home network. Having a “NextCloud” feature on my NAS opens it to the internet. I worry about security here. How safe would it be? I guess I could have “NextCloud” set up within a small dedicated part of my whole storage, to safely seperate my home storage from my cloud storage?

What is this Spotify you keep talking about?:

I hate streaming media. I like the idea of owning my files and access them as I please, even if I don’t have an internet connection (which is rare, I confess, but still). I’m also fairly picky about music rendering and quality, that is why I am still using my good old iPod Classic I bought back in '07. Since then, I modded it with a 256GB SD card and custom firmware, to be able to read lossless audio (like FLAC). And I’m not keen to replace it any time soon.
That being said, there might be a few instances when having access to my home music library might be interesting. Once again, part of the “Ultimate Home Server!” series, Wendell demoed the Navidrome service, which runs on TrueNAS as well.
Navidrome works with Android clients and desktop clients (both Windows and Linux), so I’m thinking of ditching both iTunes for Windows and Rythmbox for Ubuntu and install SoniXD on both. But there are 2 features I still enjoy about iTunes: adding content and remote control.
When I wish to add content to my music library, through iTunes, I simply point to the folder of content I wish to add, and then iTunes scans it, reads the ID3Tag info of the files, creates a folder tree for artist and album in my “Music” repertory and stores it there.
I know Navidrome would read ID3Tag info as well and keep a clean library, but would it be also able to change my “Music” folder structure like iTunes does?
Also, when I listen to music with iTunes on my desktop, I can use an Android app on my tablet to remote in on iTunes, access all my library and control the playback. Any way to keep this with SoniXD?

Closing thoughts

This is where I am so far. I’ve been thinking on and off about such a solution for a few years now. Even if most (if not all) software listed above is OpenSource and free, it does involve some fairly expensive hardware and (I assume) a lot of time to set up correctly, and I foresee the endeavour being sweat inducing. But hey, I don’t have to do it, I chose to do it. But before investing any money and time in this, I thought of asking around for some help and have someone second check my findings (not too dissimilar to an OpenSource project).
I would like to thank the whole LevelOne Tech team to inspire me in this project and I would also like to preemptively thank any and everyone of you who will take the time to read my here novel and give me feedback about it.
I can’t wait to read from you.

Sincerely,

Poggo.

2

About that server: where are you placing that in your house? Given it’s former business-grade stuff, it’ll be loud, VERY LOUD! So, rethink your requirements if the server needs to live anywhere near inhabited space.

Tip: an old workstation is usually quieter, has better expandability when it comes to connectivity and maybe storage, can be disguised for the better half (i.e. partner) and is usually cheaper to purchase.

3

Proxmox, ESXi, xcp-ng. These are the main hypervisors used for home-use. Take the one you’re familiar with. I’m using Proxmox myself and can recommend the product. Debian based with ZFS and easy install and slick web management. Really easy point&click setup for VMs, passthrough, backup and storage. Many good videos are available to learn Proxmox step by step.
If you want a NIC for passthrough to your pfSense VM, check if that NIC is compatible with pfSense or FreeBSD in general. Nothing worse than no driver support for your new hardware.

I’m personally not very fond of “for NAS” harddrives as the only special thing I can see on them is a 100-150$ premium compared to 24/7 enterprise drives.

TrueNAS is a safe bet as a storage server and pairs well with Proxmox to make storage available via NFS or iSCSI to fuel all your VMs. But UnRaid and OMV are are also popular choices, but TrueNAS is known to be the more enterprisey professional storage appliance.
I recommend getting a cheap SSD as read cache for TrueNAS. If you spend a lot on HDDs, you may as well optimize your pool performance with some cache. ZFS loves memory and cache.

Old and good server hardware brings a lot of horsepower and many options, but 24/7 power-draw and noise are a concern for most of us home-lab owners.

About Nextcloud…I was running my one via turnkey-linux ISO via Proxmox and later moved it to being a TrueNAS plug-in. All my devices are synced with my homeserver now. With 600Mbps up, I see no reason not to do this yourself and also offer family some free family cloud :slight_smile:

Switches…I’ve always been a Netgear guy. But I’m planning to buy Mikrotik switches soon. They have a good reputation and are very cheap in comparison.

4

The all powerful server you’re describing is just a file / webserver basically?

I’m thinking:

Unifi switch+wifi (because it’s hassle free and wifi is speedy, there’s not much to tell… the switch and u6-lr or uy-mesh for your wifi. I have not lr and mesh, lr is a bit better but mesh is smaller - easier to place perhaps.

Option 1: enthusiast with slight appreciation to data hoarding

This is my preference, because torrents can run happily

  • node 304 in white, because apartment and not basement and you can put it in a corner and put a flower pot and plant on it, and because lots of space for a bunch of 18T HDD and/or flash
  • ASRock Rack > X570D4U-2L2T … Yes it’s 500+ but it’s got decent x550t2 nic onboard and ipmi and decent ECC support.
  • some sprinkling of components

Keep Ubuntu server on host it can do your routing, put apps in docker containers to allow for independent updating.

You can run pihole in a container for your DHCP and DNS and what not,… or any gazillion other options

This is super speedy and you won’t need to upgrade when your ISP decides to offer 5G or 10G home broadband

Option 2: I’m 30+something and life’s too short

This might require some significant compromising

Plug in 16G ram, a boot nvme, into an n5105/i225 mini PC get an 18T external USB HDD for your files. Run proxmox.

Not much to say about this option except it’s tiny and packs a punch.

Option 3: I don’t care for fast internet beyond speedtest numbers

I don’t see much of a point, you’re giving free money to your ISP.

Pay netgate, bring your old server but upgrade nic on it… not sure where you’ll keep the old server.

5

edited for having 1 reply

The one I had my eye on is a server-grade workstation. Although rackable, it is meant to be put on feet, as a desktop. See below:

hp-652065-b21-proliant-ml350p-gen8-server-8xsff-10112664
hp-652065-b21-proliant-ml350p-gen8-server-8xsff-101126642400×1800 404 KB

I had the hope it would be quieter then.

I agree that I thought about a workstation I could gather parts for, and handle quieter cooling myself. But I liked the idea of having native multiple integrated NICs and a RAID controller. And the 6 drives bay is a nice to have. If I build it myself, I’ll have to add PCIe network adapters and a RAID controller.

Thank you for your input. Prowmox seems like a good option.

I like the added value of 5 year waranty and early RMA that I have with Pro drives as well. And if I mount them in a bay, the vibration dampeners with NAS drives is nice. As long as I am down to invest for drives, might as well invest well (that’s how I see it).

I was thinking of having a SSD to run/host my Ubuntu VM. Should I have a seperated dedicated SSD just for TrueNAS caching?

Yup! That’s exactly what I was thinking about, and that’s the plan.

I often use NetGear switches at work and haven’t complained yet. Although they are quite expensive, and the affordable 2,5Gpbs ones I found seem to be “managed in the cloud” which I’m not looking for.

I’ll look into Unify but it seems to be a bit on the expensive side and I’m not a fan of the “app only” management. I’d rather have a basic webpage.

So just a “basic” Ubuntu distro that hosts everything including routing and NAS? Would I be able to better manage routing, IP adresses (with NIC passthrough), RAID and redundancy if I use seperate VMs with a hypervisor instead?

I don’t think that’s the route I want to take. If I’m investing in hardware, might as well do it right with RAID and redundancy.

Once again, part of the project is to have a better solution for data hoarding, mainly RAID and maybe personnal selfhosted cloud services.

Many thanks to all for your input!

7

It’s open source. Get the ISO and start a VM, nothing better than checking out web GUI and test out stuff with virtual disks before fully committing on a bare-metal install.
TrueNAS Scale can also cover most virtualization needs and Docker containers, so also test this ISO as well. If you can avoid complexity by not having both Proxmox+TrueNAS at the same time, management will be much easier.

My TrueNas ZFS pool provides all the virtual disks for VMs via iSCSI. But you can also keep an SSD for VMs in Proxmox, but I prefer having everything in TrueNAS for simplicity reasons and the cache applies to everything in the pool, including VM disks. Boot drive for Proxmox (+TrueNAS VM virtual disk) is the only drive Proxmox has direct access too. But that’s more a matter of taste really. I always recommend a cheap SSD/NVMe to boost pool performance, but it isn’t required and can always be added later. I just get allergic reactions when I see random read performance on my HDDs that could be avoided with a larger ARC :slight_smile:

And I agree with @risk when it comes to sizing of the server. It’s tempting to go all-in and overbuy. Storage and small VMs don’t need much horsepower to work. SATA ports, an old quad-core and maybe some additional NIC usually gets the job done.

8

I had two ISPs in France…

Orange

  • Very reliable service (not many outages)
  • Low latency (Latency annoys me more than bandwidth)
  • Their routers and WiFi suck, you should replace them
  • Some of their routers come with a fibre-RJ45 converter box… you can use this to do the media conversion for your home-made router (pfsense, opnsense)
  • Their 2Gb router was a pain in the butt as it had no multi-gig LAN ports. I tried doing aggregation in pfsense (putting pfsense behind 2x1gb LAN ports from Oranges router) but I didn’t notice any speed benefit from this
  • There is documentation on how to get PFSense to work but I had better luck with Opnsense

Free

  • Cheap 10Gb connection (The contract said 10, I got 8)
  • The fibre is shared… your bandwidth will change depending on your neighbours use
  • Higher latency than Orange
  • Multiple multi-day outages per year (This might have just been my bad luck, the fibre was cut by others working in my street)
  • Speed changes with the time of day, direction of the wind
  • ~2 connection resets per day
  • Their 8/10Gb router has 2xHDD bays, user upgradable RAM, SFP+ LAN and Bridge Mode (if you use bridge you lose all the other things the router was doing… this passes through to your new router)
  • The performance of their router (wifi and routing) was actually pretty good. I would not feel I had to replace it immediately.
  • The SFP+ LAN port made it nice to put Storage, Router and Virtualisation on the same cheap, fanless 8 port SFP+ from Mikrotik (~220€?). I used SFP/±>RJ45 adapters (~50€ for multi-gig, ~20€ for 1Gb) to attach wired devices (Ubiquity U6Pro access point, TV, Printer)

Both

  • I tried looking for details of their fibre so I could connect PFSense directly to the fibre via SFP+ connector. It’s not supported/documented. Different parts of the country have different topologies/connectors… I gave up.
  • Whole-home VPN worked through pfsense/opnsense… but I didn’t keep it (it was better to just use a VPN from the clients/servers that needed it)
  • Whole-home ad-blocking through pfsense/opnsense à la pi-Hole
9

Amen

and given current prices of second to last generation server boards even a server solution that supports ECC memory is pretty affordable nowadays

This:

comes with a 4 core/8 thread 3.6Ghz Xeon E5, has three pci slots to support 10gbit cards/nvme, has IPMI and supports bifurcation … 200USD … add 50USD for 32GB of ecc ram and you have a very low cost atx form factor server platform …

10

I don’t know if it’ll be better; it’d certainly force you to learn e.g. how routing and firewalling work, as opposed to “learning” where to clock to install something that magically does those things on your behalf.

It’s certainly possible to setup ZFS and a cron job to email you once a week with the status of the pool, and to configure two network interfaxes and enable routing between them, and maybe add a DHCP server, and some firewall rules, and install Samba and share some paths over SMB.

You can also use that Ubuntu as the hypervisor for VMs for whatever purpose you’d need/want a VM. (e.g. home assistant, because of it’s built-in backups and upgrades, and docker based add-ons, would be more interesting to run in a VM than e.g. a NAS)

11

Edit: typos.

That actually seems like the better option. I didn’t know that TrueNAS would be able to host VMs as well. That means I could run TrueNAS as a Hypervisor, run my NAS setup with it and have dockers or VMs to host everything else I want? I’ll look into that for sure.

Upon reading about SSD cache for ZFS and with the new info that I wouldn’t need a seperate system/hypervisor running if I use TrueNAS Scale, I was thinking of having 4x HDDs for raw ZFS storage, 1x 256ish SSD for ZFS caching and another 256ish SSD, with seperate partitions to install/run TrueNAS Scale and different VMs I’d like to run. Although, since TrueNAS Scale handles dockers, I would be able to run Pi-Hole as a docker within TrueNAS Scale. Your thoughts?

That’s the whole plan, yep.

That’s what I found as well. Their “Entry level” Livebox 5 doesn’t support LAG, and only their “high end” Livebox 6 has 2,5Gbps LAN. I’m gonna pay premium just to have this feature and not using anything else that comes with their high end product… The price of wanting nice things I guess. Although, I will still go and ask if there is any other solution. In my setup, all I need is a modem to connect to the internet, with FTTH and 2,5 Gbps. I really don’t need any other bells and whistle. So I’m gonna go and ask if they don’t maybe offer any other solution, although I’m not hopeful…

I’d love to hear your feedback on this. So far, I am using a client on all my devices. And since they all are connected at the same time, I was thinking I might as well have the internet traffic go through VPN as default. What were the drawbacks that made you get back to seperate clients?

I’m looking into a used/old server board+CPU+ECC RAM combos available. I found some cheapish things, but with some compromises (comming from old tech):

  • Lots of systems I find come with dual CPUs. Does it play nice with TrueNAS Scale and VMs?
  • Many of them only support SATA2 (3Gbps) on some or all SATA ports. Would that be a bottleneck for ZFS HDDs?
  • As far as I understand, ZFS doesn’t use a dedicated RAID controller and is instead handled by the CPU. How many cores should I allocate to ZFS file storage and caching?
  • Also, although it doesn’t use a RAID controller, if my board has SATA ports linked to a RAID controller, can I still use ZFS no problem? That wouldn’t cause any issues, it would still be used as a “dumb” SATA port? Also, could I (if I really have to) use SATA ports connected to seperate controllers on the board for the ZFS stripe?

I think it’s part of the process and what I’m looking for here. I want everything to work great, while learning (mostly with failure, I’m aware of that) and having the satisfaction of something I setup myself.

My conclusions so far are:

  • use a seperate box for pfsense (NETGATE),
  • find compatible old hardware to build my own does-it-all box (silence is key),
  • Setup TrueNAS Scale as the base system for this server,
  • Run the ZFS/NAS setup within TrueNAS Scale and a Docker for Pi Hole,
  • Sprinkle some switching and access point hardware on top and we should be golden.

Once again, thank you all for your input!

1 Like
12

I absolutely agree used Supermicro is great.

I’d like to add though that if you hunt around for deals a bit, you might be able to find cheap X10-based servers getting decommissioned because they’re running out of their support periods.

Performance/Watt is better and you get DDR4 + more PCIe.

1 Like
13

Yes it does, but your watt consumption will skyrocket, two sockets, double RAM slots, additional circuitry for the dual socket, plus the idea that if you need dual socket you really don’t care about power draw, and fan noise

HDDs? No, SSDs, yes

If you are not using deduplication (and you should not) then cpu requirements for parity calculation and compression aren’t really that high, the bottleneck will be the throughput of your spinning disks

Yes, just disable the software raid (that usually linux ignores anyway) and you will be good, as long as linux supports the specific chipset

14

Agreed, the challenge here in Europe is that the X10 platforms are priced much more that the x9s … you can get an x9-8core combo for 200EUR, the same on X10 is more than 600 for atx form factor
Rack servers are a different thing, but rack servers need a lot of space and make a lot of noise …

1 Like
15

TrueNAS always had virtualization as a feature. FreeBSD-based TrueNAS Core has a bad rep because of BHYVE which has quite some limitations compared to other virtualization like QEMU/KVM under Linux. With TrueNAS Scale being Linux-based, this caveat doesn’t apply anymore. I’m probably migrating to Scale in a couple of months just for simplicity, but I want to give this new product some more time to mature and because of me being lazy, also : never change a running system.

That’s some old-fashioned approach. With ZFS you have one big pool and things like RAM, cache,log or special work all the time to boost data that sits on some HDD to main memory or SSD speed. ZFS automatically picks the most important data and keeps it in cache. Things like VM images and frequently used files are on high priority. So you usually don’t need a second SSD pool. This would also mean double administration, maintenance, backup,etc. and is prone to human failure. Best practice is to keep a single large pool, but there are always exceptions.
And always take the cheapest reliable drive for the boot pool (boot drive). We’re talking about megabytes of logs per day of activity and occasionally reading OS files after a reboot.

No problem. There is some good support out of the box and by adding ‘TrueCharts’ library, there’s just hundreds of point&click installs for VMs and Containers. And with KVM under the hood, you can run basically everything.

No HDD known to mankind can exceed 3Gbps bandwidth. But SSDs will be bottlenecked by SATA2 and capped at ~60% speed for sequential reads/writes.

An old quad core can handle everything just fine. If you use deduplication or higher compression than the standard lz4 (which is basically free real estate), cpu load is higher. I recommend getting more memory/cache over a beefier CPU. My 6 core Ryzen is still oversized for 10Gbit/s streams and zstd compression or decompression, just to give you a reference.

16

The only experience I have with VMs is software based VMs with Oracle VM VirtualBox. When I setup a VM with it, I have to/can assign a number of processor cores to be “dedicated” to run the VM. I need to set RAM space and a virtual HDD as well. Is that not how it would work with TrueNAS Scale as well? Or is it dynamic allocating? Like, by default it runs all cores for the NAS system and when I create and run a VM that I have allocated cores for, those cores are no longer available for the NAS system? Or even more dynamic: all cores work for everything and TrueNAS handles the load balancing?

OK. So I just need a system drive, to boot on, with the TrueNAS “OS”. Then, I just create a big pool with all my HDDs, and a SSD cache, and then all data (other than TrueNAS system) would be stored on/in the pool? Either raw storage as NAS or dedicated storage for VMs. The dockers run within TrueNAS though, so these would run on the main system drive.

OK. So if I find a motherboard with 6 SATA ports, 2x 6Gbps and 4x 3Gbps, I can plug the HDDs in 3Gbps and SSDs in 6Gbps. Once again, as long as both controllers are supported by Linux, no issues with having the HDDs and the SSD cache running through separate chipsets?

So, what I thinking about is:

  • Old server board with Xeon CPU, 4 cores/8 threads, 8GB+ RAM, dual Gigabit, 6 SATA ports (with 2x SATA 6Gpbs minimum), add a 2,5Gbps PCIe card,
  • 4x 5TB HDDs with 256/512GB SSD for cache, in ZFS equivalent of RAID 5 for 15TB of total storage (both NAS and VMs),
  • 1x 128/256GB SSD boot drive for TrueNAS and dockers.
17
  • For the ISP router, there should be an option to switch it to Bridge mode. This turns it into essentially a media converter and allows you to use your own firewall/router.
  • Unifi APs are decent and fairly inexpensive for how well they work. You can either manage it via a phone app or using the Unifi Management Controller (there’s a docker container for it). The container is also not needed for it to run, just to do firmware updates and push config changes (VLANs/SSIDs), so you can turn off the UMC container if you feel so inclined.
  • TrueNAS Scale for its VM and Docker support should be able to take care of most of your other requirements. Unless you’re wanting to get into clustering immediately, it should be sufficient for the HTPC docker containers, pi-hole, steamcache, NextCloud/Syncthing, etc.
  • Recommend pfSense for the firewall/router. Relatively simple to setup multiple VLANs and subnets for your different networks fairly easily, you can setup your PIA VPN to transparently tunnel traffic from one/multiple/all networks through it if you feel so inclined, can setup FQ_CODEL for proper packet queuing, and could setup an OpenVPN server so you can remote back into your home network for file access without exposing anything it to the internet
18

I’m seeing some of the stuff on mixing storage, vms, routing. I’ve really benefited from having three separate boxes for:

  1. Routing (Opnsense/Pfsense): I play with this box too much but I can reinstall and restore a config in minutes
  2. Storage (TrueNAS): I try to touch this box’s configuration almost never. I have multiple accounts so (for example) my work and personal devices can’t see files they shouldn’t; my scanner can only add scanned files, not delete anything
  3. Application (Virtualisation, Containerisation… proxmox, docker on ubuntu, hyper-v on windows sever): This box is a play area. I mess with this a lot but it doesn’t take out my network (mostly) or risk my files

I had a quick look. It doesn’t have “bridge mode” which means your pfsense router would still have to go through the Orange router… this negates a lot of the benefit. Free was the one that gave me a proper, bridged multigig connection.

Putting everything through the one VPN was annoying. Some apps/sites don’t like it so you’ll have things like video sites saying “it looks like you’re on VPN… no content for you”. I’m sure I could get around this with routing rules, multiple VPN connections… but this sounded like a pain in the butt. I resolved to keep the network fast, light, scalable and generic. P2P virtual machines were VPN’d from within the VM, mobile devices could initiate their own VPNs through apps.

19

Correct. Everything is stored on HDDs. And ZFS, with higher storage tiers like cache and ARC (memory), checks your usage and keeps most frequently (MFU) and most recently used (MRU) data in memory or SSD. When I hit CrystalDiskmark on my Windows VM, I get 20-30GB/s reads because the VM is cached by main memory, but ultimately stored on some HDD. That’s why people always talk about “ZFS loves RAM”, because every spare byte is used to cache the really important stuff. L2ARC (cache) functions as a natural extension to memory, but at slower SSD speeds.

In TrueNAS, create pool, assign all HDDs as data vdev with redundancy of your choice (RAIDZ1 aka RAID5 is probably the option of choice). Then just drag the SSD(s) and make them cache. Done.

I have no experience with boards that have both SATA2 and SATA3 at the same time, so I can’t really

comment on that. But I recommend having some SATA port(s) and space for additional drives, for expansion in the future.

Sounds like a good plan. But if you can get some cheap used memory, I’d upgrade to 16/32GB.