The Ultimate Home Server - Component: Knowledge Repository

Only cause I was in this world for a bit, for digital stuff they call it ICR (intelligent Character Recognition) :wink: But on the native app, couldn’t that be solved with the web interface itself? I suppose advanced stuff like drag and drop, or serious file editing would be challenging. It’s sad that licensing is the bigger issue than the actual technology :frowning:

1 Like

I’m a vim user and have always been a little bit jealous of org mode because it really does tick a lot of basic note taking and task management boxes. I think it’s somewhat a victim of it’s own awesomeness. It seems like it works so well in emacs that nobody really felt the need to create great integrations for it elsewhere.

3 Likes

yeah, i agree that the web interface should be able to do all of it, but due to some limitations on how browsers deal with touch interfaces a native app will feel smoother and faster on mobile devices than a web app. So if i were developing the project I would probably go the path of web first, integration into os second (so using the system calendar/document viewer/whatever), native apps third.

1 Like

Well we could always write it in Java and use this fancy logger I heard about :wink:

3 Likes

Yo cuold always try org mode in the evil mode of emacs.

1 Like

i’ve tried evil mode in emacs but found that while it meets the basic vim workflow it’s not great when doing more advanced things. I’ve even gone as far as using spacemacs but found I would fight the editor more than I like

1 Like

I am just starting to use a intel NUC with VNC and getting obsidian sorted on it a security focused friend likes it because of the markdown ability.

Feel like that’s the quickest and best way for me and my NUC has a job!

1 Like

Well current state of knowledge repository is distributed between OneNote, Markdown files written using Typora. Currently have most files stored on folders on an Unraid Share. Definitely want to move everything to a unified interface.

My main issue is integrating physical media basically typewritten and written notes.

1 Like

For a base file format, most of my stuff is stored in PDF, but I hate Adobe and working with PDFs. Has anyone used something based on XPS?

1 Like

I use DokuWiki. No MySQL required everything is just in plain and simple files I can back up quickly and easily. It might not really be a secure solution to put on the open internet but for home i can access it local or through a VPN it is just fine.

For storing info about my desktop setup (what’s connected to what using what port), I use my own (open source) app: Connectatron: Open-Source node editor for planning networks of devices

It would work well for home server hardware documentation too! Much better than spreadsheets or pencil and paper.

1 Like

I also use Obsidian. I’m using my own implementation of Zettelkasten.de to organize my notes and link ideas together. I’m paying for sync until I get my self-hosted sync back up.

I also use MyMind to enter quick notes from the browser and to keep track of my bookmarks (I always forget to export them when changing distros lol) and articles I find interesting.

The interesting thing about their service is that they use AI to assist in searching through your bookmarks by adding ‘tags’ to your bookmark. So if i bookmark a spicy chicken recipe called ‘yummy yum recipe’ it, in theory, should be able to add the tag chicken recipe to it. So when im hungry and searching for a chicken recipe, i will find that article when i search ‘chicken recipe’ and forget the name of the article.

I have found it to be a double edged sword when you are looking for a specific article and the search terms bring up a billion results that do not get narrowed down. I may be missing a strict search toggle though… :man_shrugging:

Along with that I use Notion to create ‘manual pages’ for various projects and for Task and Project management using Thomas Frank’s Project Management Template.

1 Like

For “Knowledge Repository” I still use old pen and ink and scan it using a Fujistu SnaSnap Si300i. The ScanSnap Manager software by ABBYY creates searchable PDFs and is worth the price alone.
Caveats. Windows OS only, OCR excellent on typewritten documents but is mostly useless on handwritten items. Have not used the web services that come with it so can’t comment

I use “Okular” for highlighting and adding pop-up notes to PDFs. It has a sidebar that shows you all highlights and notes(unfortunately organized only by the order they were added). You can also add bookmarks that show page numbers and rename the bookmarks. Closest thing I have found to Adobe Acrobat Pro for markup puposes

I… just realized it’s Vi. I litter text files in directories. Generally using SSH for access if I’m not physically around the set of machines I work and store data on. It does tend to keep my notes nestled right up next to the content they’re about.

I didn’t even realize I kept “inline” notes 'till I was watching the video.

I use Google Keep for ephemera. But that’s more along the lines of “buy the eggs” or “replace the serial USB adapter” where I might have a different device handy at the site.

So in school we used Note Taking App - Add Text, Images, Audio, Checklist | Zoho Notebook Now it worked but it wasnt as good as dictating notes. It was AMAZING on ipads and surfaces.

Honestly Ive been doing my best to find or augment and open source solution but its just not holding up to it. I watched your video

I do agree there is a need for this and the market is trash.


For books I wanted to run calibre BUT I ended up just sorting this out in my nextcloud. Now granted Wendell I am curating physical copies of my books.

Now granted I bought these books as well. I do wish to support the author.

All these are just tiny components I think of the vision you want to see where you have a sort of grand concept of a cataloging server. It may have to work in a way where software integrates with an ingress server that can sort oh hey this is a book we should put this pdf on this server with this piece of software etc.

There is this but its literally in early beta

Now yes I realize the way it acquires that information is well very controversial but its a decent ish concept


@wendell @SgtAwesomesauce I guess the first thing to do in this task is something that can feel overwhelming trying to pinpoint. If I were attacking this concept the first thing I would do is a very “engineer” thing to do.

Map out my knowns? What do I need? What do I want? What is the core functionality of such an integrated concept service that I need first. Then expand the diagram only after I finish the core. So to speak

3 Likes

This is something that I am still very much working on, and don’t have a good solution.

In terms of book archival, calibre is awesome. It’s not for your “active” books, like anything that you want to annotate, instead more for a “clean” copies or “finished” copies of books or large documents. It has tons of plugins for things, and since it written in python and has the ability to add custom metadata columns, it is relatively easy to extend. For example, I have a script that lets me get the total wordcount of my library, or a subset of the library.

I’ve tried multiple things for web reading / reading on mobile, and have not found a great solution yet. Both the built in web GUI, and the third party calibre-web are not what I am looking for. The android sync app was incredibly slow (and you have to “connect” to the server EVERY time), especially trying to do searches. The ODPS feed from calibre-web is slightly better, but searching does not handle custom metadata, and ODPS does not sync reading progress afaik. Syncthing for the entire library did not work, as Syncthing + Android file APIs + an SD card is a recipe for never finishing scanning a large directory.


I still need something for book annotations (e.g. highlight, make notes, etc). But, really, I want onenote combined with like three or four other pieces of software.

I should be able to throw PDFs and other documents at it, and have it do OCR on them and autocategorize them. It should have organization, and metadata (both tagging and standard fields for document). It should have both search on titles/metadata, and full text search from the metadata. It should be self-hostable, with both a web GUI, and apps for both desktop and mobile platform, with offline sync. For the mobile end (or touchscreen laptop end), having the capability to do drawing would be good, both for notes and diagrams. And perhaps a bit of stretch, but handwriting recognition would be awesome (actually also on the document ingestion end as well). But I also should be able to use it like a wiki, with a text based formatting (probably markdown) files that can integrate pictures, diagrams, etc. And for editing, it needs to have a preview window, and spellchecking (languagetool ideally), and a way to export to standard markdown, so I can draft in the software and then put the final document into mkdocs.

The individual pieces all exist, and (mostly) exist as open source code (exceptions being good pdf editing and perhaps handwriting ocr?). But putting it all together is the trick. It’s basically the same thing @pfeiferj


For task list, I am using vikunja. Since can act as a caldev server, other stuff should be able to integrate with it in the future. There are some papercuts with it still for me, but given the pace of development, that the author dogfoods (e.g. using an instance as the roadmap), and that there is now a hosted version for financial support, I am hopeful that those will go away.

1 Like

oooof. Do you have a good resource for getting a solid self host of this going?

I have this massive mess of it from my school and learning days. 465 books and counting

Is calibre well suited to this? How does it do in detecting stuff without meta data? Almost all of these are literally references I look back to and highlight (temporarily) and stuff but ive never had a great time with this

Note this is also an almagamation of other friends going through school who needed somewhere to stash their shit too. So it needs work but if I can move it out of nextcloud to something purpose built thats a better first step than where it is at

See ive always supported open source software for these tasks but ive also said I would more than gladly pay for these things.


Wait Readarr and Calibre Integrate? Thats actually nice

I swear the components of what wendell wants, what we want all exists. Just needs repurposing and integration. The thing is each component is kind of clunky and overweight with what its supposed to do so to speak. hmmmm

In note taking, I prefer platforms with markdown compatibility.

  • Outline: For the team’s KB

  • Simplenote: For my personal note taking; Not satisfied with it yet, and I really haven’t found anything that meets what I’m looking for (full markdown support, OSS, version history, clients on all platforms), but it’s doing the job for now.

My “Knowledge Repository” is wiki.js, a wiki software where i write only markdown pages, cause the syntax is easy and if ever necessary i would be able to just move it over to another wiki that supports markdown. It is synced to gitea (git server) so i can always just do a git clone before i do critical maintenance on the server i host wiki.js. On my android phone i use GitJournal, it syncs with gitea, automatically shows a markdown preview and stores the whole wiki on the phone so i can access it even when i have no internet. And the best thing is i can even do edits on the phone and it syncs back to gitea and from there to wiki.js.
I use it for less than 2 years but a recent paper backup was over 130 pages long… I use it to document how my servers are set up, any software/linux tricks and kung-fu i wouldn’t remember cause i need it only occasionally, and life-things like checklists for moving to another place or getting a new car. Whenever i search the internet for a solution to a problem and it takes longer than a few minutes to find the solution, i add it to the wiki so next time i search it again i know exactly where to find it fast. For me this was such a life changer and time saver (in the long run), cause before i had those infos in various text files and couldn’t find the information when i needed it.

Then a password manager is also a must have. I use KeepassXC, synced via Nextcloud to my phone and tablet, with a Yubikey for 2FA. I have way over 300 passwords in there. It makes it so easy to get a long and unique password for every site, that’s so important now that so many user/password combinations are leaked from hacked sites, cause i know a leaked password doesn’t give access to any other site. And KeepassXC can also store notes (like how to log in or how long a premium account lasts) and files (i.e. a geli key for the old zfs geli encryption) and it can even unlock ssh keys.

And third is a document management system. I chose paperless-ng cause it has OCR, a good way to automatically tag documents and good search filters to find documents easy. Before i had just placed my documents on my desk, then i had to clean up cause someone visited and then i would just throw them i a big bag. When i needed a document i couldn’t find it or searched for up to 1-2 hours. Now i get most documents in digital form (when they come via mail they get automatically imported), and the few ones i still get in paper are scanned periodically. So whenever i need a document i find it now in less than a minute.

And the last thing is an RSS reader, at least in tt-rss you can just star an article and it will never get automatically deleted (you could it even set it up to never delete anything but the database might get pretty big and slow). For RSS feeds that include the whole content of an article (some sadly only do headlines or a short introduction) this is a good way to save them practically forever.
And it’s a HUGE timesaver if you want to get the latest news from many sites.
For sites that don’t support RSS (sadly more and more, especially those social media ones) there is rss-bridge which is really good at parsing a website and generating a rss feed out of it, and it supports adding your own bridges for sites that aren’t supported natively.
Pro tip to track github repos: add /.atom to the url of the release or commit site of a project.

Some mentioned calibre here. As i left all my books at my parents house when i moved out, i have all books stored as ebook or pdf. I use calibre-web (instead of installing applications locally i try to run as may things as possible via docker on my server and access it via browser, so i can access them also remotely), but i haven’t found a good way to connect my e-reader (Kobo Aura HD) to it via wifi. The 3rd party reader app koreader can connect to calibre-web but the search feature doesn’t work well and i think there’s no way to sync read progress or notes. So i tend to use the webbrowser of my e-reader to access calibre-web to download a new ebook it hasn’t already stored locally.
But calibre-web just uses the default calibre database, so i could always just run calibre on my pc and sync it to the ebook via usb, but i haven’t done that for years. And i also don’t read that many books so i haven’t invested that much time into my calibre setup yet.

2 Likes

Ya this thread is my jam. I have a reading and writing issue. I have notebooks (Journals, brain dumps, meditations, task management) and 200+ books (only the keepers), all analog, that i would like to track better.

Currently I do it with a Libre Calc sheet. But it would be nice to be able to pull down books stats, ie. title, author, pages, genre, etc.

This person has some stats game, but to much data entry.

It would be nice to integrate with a wiki.


On the computer side of things, Gnote is an awesome place to brain dump. At this time it is mainly linux and computer based information.