I just joined here based on the latest YouTube video, so I’m not sure the depth of what everybody gets into here.
Would it make sense to have a section for Management / Monitoring? Or maybe that’s getting too “enterprisey”.
I run a bunch of VMs, so I use things like:
After looking at my last electric bill, something that might be worth considering in the “what”/“nice to have” category is whether 24/7 access(low power) is required or not which might lead to 2 servers for some folks. It will be different for everyone but for instance I don’t need financial records or CAD drawings 24/7 but I might want audio-books and obviously home automation 24/7. Maybe a “reliable” wake-on-lan server with an equally reliable sleep function might fit the bill.
One software package I personally find quite useful is Mediawiki, which is the wiki engine that runs Wikipedia. I find it quite a natural fit for accomodating lots of otherwise sparse and disjointed information in one place, and hyperlinking/categorising the hell out of everything.
Some examples of open source projects using this to track collective know-how is the sunxi project (Allwinner chipset support for Linux), and the elinux wiki, which is a collection of useful info regarding running Linux in an embedded context.
I’ll generally have a many-core Debian Linux server of whatever’s dirt cheap to run. Currently an old Bulldozer that spins up local NFS and CIFS shares, HTTPS, plus some SSH secured remote services. Probably more I’m not remembering.
It’d be nice if there was something easier than that but I haven’t run across it yet.
I do like running Part Keepr as a webserver VM to organize my electronic components. It’s a PHP and webserver based software. I pushed up a pre-built VM if anyone wants to use it (or reference what software versions an install needs).
Edit: and KODI! Since my media’s just a well organized network-transparent share to begin with. I like it.
automagic file sorting and such would be handy.
doc/xls/ppt/odt/etc could be considered “editable” documents, where as pdf’s would be documents to keep/file. The suck thing about pdf’s is they can have embedded text or be basically just jpegs of the original docuument, so they’d have to be ocr’d for easier sorting. something like gpdfreader works but its finnicky for getting the settings dialed so it isn’t trying to ocr a company logo or some shit. password protected archives could be considered backups. images/music/videos all have metadata that make them easy to sort/categorize. Musicbrainz can be used to organize/clean up tags on audio files fairly easily and accurately.
should the sorting be automatic or a cron job or something the user should have a button/right click option (in folder, right click, “sort this shit out”, automagically sorts files into appropriate folder structure.
Desktop search utilities have been done a million times, and i’ve never really found anything that works exactly how i want it. indexing/sorting/etc, documents with searchable snippets as metadata are sort of useful, but not all filetypes have those capabilities.
Also naming conventions can be another spaghetti bowl to untangle.
I remember several years ago google had made some sort of magical tool that can clean up massive data sets and correct minor differences (initial case, camel case, abbreviations (road vs rd) typo/trasposition) to have things in neater piles.
also in mixed platform setups (windows/linux) even stuff like backups can be a bitch because of file permissions and fs differences.
I don’t think there are enough fingers in a full football stadium to count all the times i’ve used linux to copy stuff to and from NTFS volumes to avoid permissions and hours of setting/changing permissions just to move shit from one drive to another, root said move, files fucking move.
There needs to be a standardized server base, and modular plugins bolted onto it for each functionality. NAS module, Backup module, Audio/video server, mobile sync module that handles ping things out to pants computers, book server, home automation system, firewall/vpn, maybe even something cool like a chat/federated file exchange thing.
The problem with having every module “containerized” is it sidesteps the common base and allows for breakage. The whole reason the google/microsoft/apple ecosystems are so damned useful and hard to avoid is the integrations from being on a common base. Gmail can see/do things involving your calendar, it can see the stuff you’ve bought on google pay, if you want to attach a file in an email it can just grab it from google drive instead of having to find out which drive it is saved on and upload it, instead of having to sneakernet a file over to someone or fuck with bluetooth/adhoc wifi transfer (and finding a program both people have on their phones) you just send them the link from google drive.
Convenience through integration.
the wheel doesn’t have to be reinvented. Not long ago i basically had a FTP server at home, because pretty much everything from my phone to my laptop, and probably even a toaster if connected to the internet can interact with FTP… Massively insecure, old as fuck, but the reason you can still get computers with serial ports/serial port cards/usb to serial adapters is because it works.
fugly isn’t always a dealbreaker, but broken/inconvenient always is.
idk, brain dumping ideas here.
also, on notes… i feel your pain.
I almost want something like terry davis had in temple OS. he had some sort of weird thing where you could have like images and 3d models directly in text files. i know terry was insane, but there were some nuggets in temple OS that were genuinely cool ideas.
I think in pictures and objects… Having a document format where you could have text/images/3d models/etc in one document would be handy for fleshing out ideas.
an interface somewhat like Mspaint/gimp drawing & measuring, simple 3d modeling, handwriting recognition, a graphing calculator, and an embedded python console or some sort of basic language to step through/automate things within your notes, excel-esque functionality baked in as well, some flowcharting capability would be nice, able to import images to trace or annotate. the ability to file away links/snippets of text and images from anywhere for later use. also being able to embed or link to files and manipulate them from within that interface (sifting through a csv file full of data thats analyzed and can be used as an object in the document) would be super handy.
google keep is barebones, but is easy enough to dump things into and do minor edits, but still super lacking. onenote is really nice and fairly close, but its still somewhat off and being tied to office kind of makes it an afterthought.
a proper notebook/brain dumping tool utility should be its own thing and not just a freebie tacked onto another suite.
onenotes on mega steroids…
I don’t have any server solutions, but have apps I can suggest.
I used to recommend notability before they turned into a f2p SaaS app. They had the not only the ability to sync audio with handwritting, but also the ability to apply OCR on handwriting (making it very easy to search through documents).
Recently I’ve heard of Xournal++, which has similar features. I might give it an install since my work doesn’t allow us to use personal devices
Currently I use marktext for all my typing notes. It is markdown, open source, and wysiwyg. If you want something more powerful than notepad, but lighter than word, I highly recommend it.
https://marktext.app/
On mobile, I use quillnote. It is my replacement for google keep. Since I use markdown for everything, it’s a nice feature. Furthermore, it also has an experimental nextcloud sync feature
This has more of a database/wiki format
https://wiki.gnome.org/action/show/Apps/Gnote?action=show&redirect=Gnote
Both of which I have heard of from this video
That is a great point of conversation, the real cost of running a home server is actually the power.
I am half way though posting my home setup details, but I have done the math several times and it is something anyone getting a setup started should consider.
I have 2 servers, a HP 24 port PoE managed switch and a ubiquiti USG Pro. Attached to the switch is 1x wifi AP and 3x CCTV cameras.
This setup draws about 700 watt 24/7. I live in Australia and power is quite pricy at about 0.34c per KW/h. This comes to around $2000 aussie per year just to keep the lights on on quite a simple and small home lab.
Can’t wait to see others annual cost in power, just to make my self feel better!
I have one main box (truenas) and run everything else in docker on Pi’s or the app has its own dedicated Pi
Storage
• TrueNas
• But with openzfs on linux now, I have been considering just running a linux box.
Media
• Plex is still my go to - because of the wife approval
• I do use Kodi as a backup to Plex
• Emby does look good and will be checking it out
• Ebook I’m just using Calibre the app (can make notes etc) and Calibre-web on the go (running on a PI)
File hosting
• Nextcloud on a Pi.
VPN’s
• Running OpenVPN on a Pi.
The single pain of glass to access it all
• Homer - I like how simple it is to add new links etc.
Knowledge repo
• I’ve been using Joplin for about 2 years now, and I use it in my job. I just use a private repo to pull and push the changes.
Backups
• I only backup the data on my truenas, everything else I use ansible for with a private git repo with all the dot files etc and the apps are in docker with compose files.
For blow away vms etc I’ve been playing with Kasm and it been a good experience so far
This is a very interesting question, definitely got me thinking this morning. I think it will depend on how much you want to tinker and how much control you want.
I think my ultimate set up would be a linux distro for the base and storage with zfs, with containers for my apps and something if not cockpit for a user friend way to manage it all - which just describes truenas scale, BUT gives me full control.
So NextCloud has .md support builtin (it will render a .md file in a directory on the web client) which is pretty handy for notes. Then there’s Obsidian which is a really nice Markdown editor.
That’s what I use to organize my lab notebooks, because I can gussy them up when I want to (insert tables, images, charts, etc) or keep them nice and simple. it’s as extensible as I’ll ever need it to be.
I also have a reMarkable 2 and that’s nice for writing by hand, marking up PDFs and the like, but I’ve yet to get it working well with NextCloud.
Thought I’d share my setups and what I think could be improved.
For backup I mostly use Duplicati. It isn’t perfect but it’s pretty good and reliable. You need to manually increase the block size if you have large backups though, otherwise it gets extremely slow and the local databases get massive (mine total about 12 GB for around 4TB backed up). It supports a huge number of targets and I use Jottacloud because it was the cheapest option when I got started. I’ll have to review it when renewal comes up.
I generally don’t bother imaging PCs for backups, if it ever gets that bad I’ll just do a fresh install of the OS and restore apps/data. Sadly Windows is a real pain when it comes to backing up application settings, so I have to keep notes about how I have things set up.
For email I use Gmail, and have Thunderbird automatically open for 30 minutes at 4 AM every morning. That allows Thunderbird to sync mail, after which Duplicati backs up the mail store files. I also have rsync run periodically to mirror Google Cloud and Google Photos. Unfortunately with Photos the ones it mirrors are not the original quality, but it’s the best automatic option I have found.
Speaking of Google Photos, I keep my original Pixel phone around because it gets unlimited original quality photo storage. If you just copy photos take on another device to it and allow them to be backed up, they don’t count towards your quota. I have a 50GB quota with Google because back in the day I got loads of it for free on various offers they were doing.
I should probably have some local offline backup, but at the moment I’m cloud only. I do convert old machines to VMs. They don’t work very well but are okay just for grabbing some stuff you forgot to migrate.
For music hosting, Google Music was good so they discontinued it. YouTube Music is rubbish but at least it’s free. It does store the original files you upload too, you can get them back via Google Takeout. Handy for backing up a large music library as there is no quota!
For “adulting stuff” I have a Fujitsu ScanSnap document scanner. All the letters I get go through that and then get filed away by year, and eventually shredded. It creates PDFs but doesn’t seem to OCR them, even though it says it does. I have been meaning to sort out manual OCRing using Tesseract. It’s a bit of a faff doing PDF to PDF with it, but it’s possible.
Passwords are stored in Keepass. It’s open source, syncs with Google Drive, has clients for all major operating systems including Android. I keep quite a lot of useful into in there, beyond just passwords.
VPN wise I use one for privacy going out. I am going to move to a pfSense router soon and will probably have a way to VPN in at that point. Also VPN out for all WiFi clients.
Knowledge Repository - this is the Holy Grail for me. I started a long time ago with OneNote (don’t @ me) and it is free now, but I don’t like it being tied to Microsoft OneDrive. I switched to Joplin a few years ago, with sync to a free cloud service (Good Cloud) that was supposed to be temporary but I never got around to it.
Joplin is a bit of a mixed bag. The text formatting is via Markdown which personally I don’t like very much. Often I just look at the plain text editor part rather than the rich text output. I can just format the monospaced text by hand faster than I can remember all the Markdown stuff.
Maybe it’s just me but newlines seem to be completely broken in Markdown.
Joplin lets you have multiple notebooks and tag each article. I currently have one notebook per year and a few for particular types of data like a complete list of all my computers, specs and manuals. I was thinking of changing to some other scheme, but I think tagging is mostly the best way to organize stuff. Yearly notebooks at least lets me see what year the note was created.
One thing about most note taking apps/sites is that they don’t recompress images. When you copy/paste and image on Windows it is converted to a PNG, i.e. lossless, even if the original was a JPEG. As such the amount of data you end up storing can be huge.
At the moment my server is an old Intel DQ77KB, which is a Micro ITX form factor board that uses 19V power. The idea was that running it from a laptop power supply should keep power consumption down, but I’m not sure it really does. Idles at about 35W with a Xeon E3 1265L v2 and 16GB RAM. It runs Windows 8.1, I know I know.
I want to upgrade it at some point. I need something with more than 1 PCIe slot. Stuff is just so expensive at the moment. What is cheap on eBay tends to be high power server parts, and I want something low power and efficient if it is going to be on 24/7.
One other thing I forgot: books.
I’ve been looking at options to digitize books lately. I did one by cutting the spine off and running it through my scanner. Then use ScanTailor, some image manipulation tools and finally Tesseract to OCR and compile a PDF. Oh and JPDFBookmarks to add chapter bookmarks.
I’ve been thinking about building a rig to do non-destructive scanning. There are several designs, the simplest I’ve found is just a sheet of plexiglass and a handle to lift it and hold the book open while you scan one page at a time. I say scan, you use a camera. Some people use phones but a Canon G10 is extremely cheap on eBay and can be controlled by a computer.
For me, my biggest “home server” requirement (now) is that all of my servers/systems MUST support NFSoRDMA because I am rocking a 100 Gbps network with little to minimal effort on my part.
I already have enough on my plate in terms of stuff that I need to do, so the less time that I spend on sysadmin and related tasks, the better it is for me.
It’s not uncommon for me to be processing a single archive 7-zip archive file that’s multiple terabytes in size, so being able to quickly and efficiently handle that much data is paramount to me.
I don’t like SSDs because SSDs have a finite number of erase/program cycles which means when you are processing large volumes of data as I tend to do, I will wear out a SSD very quickly and I don’t want to spend a lot of money on enteprise grade U.2 NVMe SSDs either.
This is why I still use a LOT of mechanicaly rotating hard drives. Not as fast of course, but I also don’t have as much of a finite life on them as I do when compared to said SSDs.
(As an example, I am currently in the process of using 7-zip/LZMA2/max compression to compress 7.3 TiB of data into a single 7-zip file, and then I am going to calculate double parity on using par2, and then I am going to be writing the results of all of that onto one of my LTO-8 tapes in a father-son setup/topology.)
And it’s unfortunate that non-SSD storage devices haven’t really gotten that much faster despite that everything else (CPU, RAM, networking) has gotten a LOT faster over the years. But not HDDs. They are still as slow as ever. And it’s WAYYY to expensive to have around 200 TB (roughly) of solid state (SATA or U.2 NVMe) storage.
(I’ve actually lost count what my total, approximate, raw capacity is across all servers and workstations.)
My use case for all that I have here is both for my personal and for my business. I am very much an advocate of owning all of your data and using services that I can maintain control of without reliance on a cloud service or external services but there are some exceptions in this list for now because I have not found a suitable replacement for those services yet. I have covered setting up the majority of these things on my YouTube channel.
Home Automation
– Home Assistant is hard to beat (or I am too biased to learn something different)
Home Surveillance
-Synology Surveillance Station
Home, Business & Cloud PC backups
– Syncthing for real time backups/revisions running in TrueNAS
– Maintain “retired” PCs are bootable VMs “just in case” is done via XCP-NG (not sure why other than “because I can” but there are some really old systems in there, I may have a data hording problem )
– TrueNAS to TrueNAS replication for on site backups & revisions
– TrueNAS to Backblaze for off site backups
– Weekly rotation of critical VM’s to USB NVME drives kept in off site safes
– Synology Active Backup for bare metal Windows systems backups
Music & Media hosting
– Recently moved to Emby seems much better than Plex, runs smoother and I like the UI better
–Synology Photos for backing up my phone photos
File Hosting
– Syncthing for all the systems me and staff use and for my personal data
VPN
– pfsense at work with OpenVPN & pfsense with Wireguard at home
Knowledge Repository
– Self hosted MediaWiki for write ups and documentation
– Self hosted Bitwarden for passwords
– Still using Google Keep until I can find something that works as smooth and syncs as well
Secure VMs / Lab Testing
– I keep VM’s running with auto updates in XCP-NG and as needed I stop them, clone them, use them as needed, then destroy when done. Great for labs or opening questionable files. Also have the ability to put them on their own VLAN with policy routing to keep them from going out my public IP.
Secure VMs
– for things like secure online banking or one time use
Easily create disposable VMs that are fully patched up
If you already run Xen as a hypervisor with nested virtualization or can tolerate having an appliance to use for such things, this sounds like a use-case for good ol Qubes Introduction | Qubes OS
I’ve tried many applications for everything. I run my stuff on Unraid with docker.
I prefer my apps to have a decent UI, I don’t like messing about in a terminal etc. just to get things done quick.
But from the dozens of apps I have tried, the two that I still use/use the most aside from all my media apps (emby/navidrome etc):
Documentation:
-Bookstack. It’s simple and has a great UI and has quite a lot of features. The only downside is that is more focused on documentation than quick note-taking and no native android/iOS app.
Password manager:
-Vaultwarden (Bitwarden). Not much too say here it’s just easy and works great. Browsers extensions, iOS/Android app.
Have used both of these for 2+ years now.
On the VPN topic, something I’ve been using that has been incredibly useful is Tailscale. It is built on Wireguard, and is insanely easy to set up and get going on any platform.
It doesn’t function as a standard VPN server + client model. Instead, each client is a separate node, is given a static IP, and is directly connected to each other in a Zero-Trust point-to-point network. There are really powerful features like specifying a node as your DNS server, resharing a node’s local subnet over the Tailscale network, and allowing a node to function as an “Exit Node” which other nodes can use as a full tunnel. It also has SSO.
Since my hosted servers span multiple private networks, Tailscale is perfect for me to connect everything together. I use the subnet sharing feature to have my local Docker network presented over Tailscale for management, and use a Pi-Hole node to function as my DNS (and block ads on all other attached nodes). No more firewall configurations, or me having to put WebGUIs for tools like Sonarr and Radarr accessible on the public internet. Now, any of my clients on any network can point to something like radarr.plex.hc (or whatever I want), and I can access that Docker’s WebGUI.
Tailscale is fantastic. It’s like Hamachi from back in the day without all the issues. Just wish they used the in-kernel Wireguard implementation when running on Linux for better performance.
Overlay networks such as Tailscare are great. Another one that works much the same way is https://www.zerotier.com/
just when you posted the video i started looking into self hosted books/comics/manga
I always searched for years then give up finding a solution for my collection. and the exact day your video goes live on youtube is the day I decided to go back and search for a solution for that XD
and guess what? i couldn’t find anything good or close to being user friendly , am decent I have my own nas and plex + ubuntu labox. but still found all the self hosted solutions for books too hard to use or lacking essential features.
i tried:
komga
calibre
mango
mango there is almost no documentation for it.
komga too hard for me didnt figure out how to install it yet
calibre cant read almost all formats these others do support like rar+jpg etc…
it can read jpg but doesn’t put them in a collection so its useless for that.