The Ultimate Home Server - Component: Knowledge Repository

  • Knowledge Repository
    – perhaps the most important aspect of my home server is that it is my brain outside my brain. I need a wiki to store stuff and keep track of things. I need something to store and keep track of my books BUT ALSO my notes in books.
    – love something that integrates with all my old written notes that have been digitized. Used LiveScribe for almost a decade. The new pens don’t do it for me anymore, though.

What do you use? How well is it working? How do you organize your thoughts? OneNote from microsoft isn’t too bad and with OfficeLens it can work really well.

LiveScribe was amazing for making notes of meetings. You could make notes but also use your written notes as a playback device – touch a written word on the page and jump to the audio of the meeting where that node was made. Automagically Something was lost when it became too tightly integrated into your phone for this functionality to work as well as it had. I haven’t used LiveScribe in 5 years as a result.

First Update:

Do you Zettlekasten?

It would be worth reading the introduction for some more nuanced context for what I’m about to propose.

I have decided to organize the first topic in knowledge management for the level1 series around the Zettelkasten Method, and fancy editors like Obsidian (not fully libre), open source alternatives and EMACS org mode.

What I do could best be described as a modified Zettelkasten Method – what have been the most important parts for me have been capturing notes in my own words and by tagging link collections with keywords more aligned with how I might use the knowledge in the future more than the subject matter itself.

I haven’t seen anything, so far, that suggests one system can manage “everything” I want to do: Manage everything from Books, what I’ve read, bookmarks/notes/annotations, day to day life stuff, personal ToDos and basically everything outside work.

One-system-fits-all is likely not the optimal approach anyway because, taken to its logical conclusion, it becomes a question of does an infinite set contain itself? Meaning that one-system-fits-all can never be as good as one-system-links-to-other-specialist-systems.

OK enough philosophy. We’re here to “blue collar” get stuff done. What can we do today with what we have?

First, the file system. Don’t underestimate the file.

Make a directory, put stuff in there. Make another directory, put stuff in there. Zettlekasten’s original implementation was little more than a physical card catalog of notes. We can get a lot of mileage out of a directory.

Part of the physical process was collecting notes and capturing knowledge as it was generated. Don’t repeat verbatim and think about keywords more in the sense of what you’d need to find it when searching a topic more than the keywords in the material itself.

A second phase was taking the information from ingest area and deliberately organizing it where it needs to go. For me this is something I can do every few days or every week.

But it has to be super easy and low friction to take notes. For books, I want an annotation and a page number. Sometimes a picture is worth a thousand words. I have an easy access document feed scanner and Microsoft Office Lens that I use regularly to capture information.

You can search files across a directory, but just organizing a directory structure leaves a lot to be desired. You can use it just like a card catalog, making a new directory for each drawer you need, and even using sub directories to group things in a given drawer.

That’s how I’ve done it since I was a teenager for everything from my notes to media. But it isn’t enough.

Enter Emacs Org mode. Ooops. I mean Obsidian.md

Imagine if in each one of our directories, we could put a readme file that has any extra notes about what we’ve collected. Well that’s obsidian. It can read across our whole directory structure and show us. What’s more is that we can link directories together to make it much easier to navigate.

With a directory structure you choose one way to organize your notes. But with hypertext links you can have cross-reference wherever you need to.

So with this graph you can see how your documents link to your other documents and so on. This is great to organize and curate your own knowledge in your brain outside your brain.

You aren’t limited to just one way to organize.

Beyond that, you can embed other file types. Media like Movies and Audio? No problem – embed it.

You can even embed PDFs including a specific page number. I wasn’t able to get that to work with epub and other book formats other than PDF, but there is probably a fix for that.

This goes back to the infinite set issue – Obsidian dosen’t need to also be a great ebook reader if we can easily embed in Obsidian links to other programs that do their thing well. And those links are links to data in our local system. Links to things on the internet have a way of not working after 5 to 10 years.

Obsidian has a huge number of plugins, so you can get very fancy with this. It supports git, image processing, and a Zettelkasten style time prefixer.

So as you organize your knowledge you can leverage your own wiring, keyword tags and external files reasonably well.

I’ve found this method useful as I work on a project intensely for a few months, life gets in the way for a few months, then I come back to it. It takes me a bit but I can get up to speed on it again really quickly.

Now the bad news – obsidian is free for personal use, but that’s about it. It seems like a reasonable approach to commercializing their hard work, so I am not going to fault them for that.

Obsidian has been on my radar for a couple of years and I’ve tried it on and off but I personally haven’t clicked with it mainly because my Zettlekasten far predates use of this kind of software. I find myself going back to old habits which wouldn’t be as good as if I’d started with something this good.

What’s great about this, though, is that it lives on top of a file system. If you use this, even though it isn’t free and open source software, you aren’t married to it or locked into proprietary formats.

It’s fast, efficient, compact and a really good product. And no proprietary lockin. I really can’t fault them for commercialization and it seems like software that would be safe to use for a decade.

If you want something that is been there done that, I would point out emacs org mode.

Emacs is a free and extensible editor. It does everything. It is worth learning. It has a mode called org mode which can offer a lot of the same kinds of features of obsidian, but in more of a unix afficionado greybeard kind of way.

A happy medium might be Foam Bubble:

This is a collection of VS Code extensions. VS Code is licensed under the MIT license, more or less, and is open source. VSCodium is a distro of VSCode w/o telemetry if you prefer.)

When you first setup Foam, it’ll pop up and ask you to install a bunch of recommended extensions.

The author says It’s currently about “10% ready” relative to all the features I’ve thought of, so keep that in mind as you use it.

There are some other great packages in the knowledge repository thread on the forum, and other great knowledge in this video series from level1, so be sure to check those out. If you’ve been using Obsidian, Foam or something like it for years and want to share your experience or show off your setup, hit me up on the level1 forums.

26 Likes

I’ve never come to a good solution for this. If anyone can suggest something fast and easy, please do share.

I’ve been using vimwiki hooked up to git for a while and it’s alright, but I keep forgetting to pull on whatever machine I’m on, so I wind up needing to rebase my a wiki. :roll_eyes:

4 Likes

Do people still use Evernote? I had a friend who used it to document everything and also used it to package up his work to provide as documentation for system implementations.

I never really figured it out but it seemed like a decent tool.

3 Likes

I used it a while back, and it was really good, I just hated that I can’t trust it for sensitive notes. I don’t want to store 2fa recovery keys or anything… still don’t fully trust it.

2 Likes

I never really found a great solution, but a good enough option for me is Microsoft OneNote. I’d love to self host something, but it’s very difficult to get a feature set OneNote provides.

While I had my share of “‘the fuck you doing’ Microsoft?”, OneNote is actually well designed, and works on most devices, even the web app version is good.
Very useful for me is their collaboration with Samsung. All office apps work on Galaxy phones and their desktop mode and OneNote integrates really well with s-pen, you can just pull the pen out on the lock screen, jot something down and put it back and voila - it can sync from your Samsung notes to OneNote. It can OCR your notes even if they are not in English, and that it’s very useful for the rest of us :slight_smile:.

I don’t know, plenty to like but I have that anxiety that I always depend on Samsung and Microsoft, and since apparently Samsung is not doing the Note thing anymore a very expensive solution is going to get even more expensive as only option will be a foldable one.

8 Likes

I’m cheating here: there’s a list of software goauthentik has instructions for how to integrate with, so you can have centralized accounts and 2FA and/or oauth and/or AD integration, etc etc.

I found a few things on the list:

  • Bookstack
  • Hedgedoc
  • Wiki.js
  • Vikunja

I think I like the idea of throwing things into hedgedoc on their way to a project related git repo,…

… or just using the hedgedoc for drafting notes

… but I recognize this is very developer centric and not very family friendly… and a very 90s sysadmin/early 2000s approach to doing things… very text oriented.

Also, a wiki might be closer to something like a corporate CMS where non-git users can publish instructions for how to get company logo t-shirts or where Karen and Bob from accounting can put in instructions with example screenshots explaining where you can download your payslips.


Not the same, but, Google Meet supports captions, and random people have built chrome extensions to pipe captions into various places.

edit: just found / remembered Linux unplugged podcast mentioned GitHub - audapolis/audapolis: an editor for spoken-word audio with automatic transcription … need to check it out

3 Likes

I use MediaWiki but not at home, I have used it for my notes publicly at https://lathama.net so that I can access my mess fast from any system.

1 Like

I use onenote as well

2 Likes

I use, besides local text files for temporary notes, https://cryptpad.fr.

It’s an E2E-Encrypted Browser-Based Collaboration Suite.
Basically a GPL-Licensed Google Suite Alternative with some added but also many missing features.
Open-Source and trustable in contrast to Big Tech.

Following Functions I use:

  • Cloud Storage Functionality to Back Up Key Configuration Files
  • Richt Text Files for Documentation
  • Code Files to Store Code Pieces
  • Kanban to Organize Goals / Workflows

Self-Hosting is also an option, though that depens on what ones homelab looks like. I wouldn’t want to store all Documentation on how to Restore Backups on the same server I’m having to restore.
For most purposes the Free Tier with 1GB Storage should provide enough storage for Documents.

2 Likes

This is something that I am actively researching. I currently use NextCloud for simple file storage and sync but that is boring and normal. :stuck_out_tongue:

However I am building something that is inspired by Vannevar Bush‘s idea of a sort of intelligent library. I want something that I can dump PDFs and other documents in and it build a sort of complex network of relationships between the documents using machine learning and AI. This is something that is related to my academic research in cybernetics.

7 Likes

I use https://obsidian.md/ for this. I believe I initially read about it somewhere else on this forum and found myself very impressed, enough to begin a neverending process of switching to it. It’s a tree style markdown editor, meaning it can support including images, internal links and such but gets more powerful.

I migrate from onenote/ plain text files & folders for pdf’s in this regard and don’t feel it’s a downgrade on anything I required, with significant improvements in some e.g. tolerates code segments far better.

For e.g. mobile access I sync the KB directory tree to a cloud service location with unison so can edit/ receive from mobile app.

9 Likes

The only issue I have is that they want you to pay for sync.

2 Likes

I use a mixture of Notion and a locally installed Docuwiki instance in my Homelab. I am struggling to find a solution that works myself. I am finding myself shifting more towards using Notion and using the wiki for things that I don’t believe I will need to use outside my home. I am beginning to believe that even if there was a perfect tool, maintaining my notes and the data seems to become a bigger and bigger task as I take more notes.

2 Likes

i’ve seen good workflows where a “writer” uses https://typora.io/ to create markdown files that can then be automatically parsed for whatever. bash, etc may be needed, but that sounds like just enough anti windows to pursue :slight_smile:

Edit
this does make me think of the previous dream painted by digital asset management systems such as telescope back in the 2000’s. an ingestion system that will tag, and classify your assets to make them easier to find and stitch into future ideas more easily. maybe we should abstract this and look at how we wish to ingest, parse, and ultimately spread the information to whoever may need it as they query or “ask”. either way, totally down for this.

3 Likes

I used to use notepad++, and then made use of zimwiki for a while, but it had some shortcomings that I couldn’t stick with. I now use and enjoy Joplin for my piles and piles of markdown notes. My hope is that one day I’ll finally have enough notes that my ADHD cures itself.

  • I really like the organisation of:
        Notebook
                    ▼ Subnotebook ➞Note1
                                               ➞Note2
    setup that it has. Much easier to jump around. Notebook apps that only have one or two levels are awful. Subnotebooks are optional, and you can keep nesting them.
  • Open Source
  • Works on linux, windows, various Android, ios (this one is broke), and macs.
  • There’s even a terminal version for you freaks: https://joplinapp.org/terminal
  • You can sync with a variety of paid and host-it-yourself ways
  • It’s especially easy to archive files or a webpage with firefox’s “Save Page WE”, and then embed the page so it stays with the note files forevermore. There’s also a webclipper I’ve not bothered to setup, since I’m as likely to embed a file as a I am a page and my way works fast enough.
  • Can display pictures
  • Can easily drag and drop notebooks and subnotebooks around, though it’s a bit finicky until you figure it out
  • Plugin system
  • No hassle themes, various dark modes

Some oddities

  • Note that there are PDF previews on by default, which can be disabled in the settings, which is a must for embedding IMO.
  • Joplin has a phone app, but on my iphone it’s broke as hell. I really need to report this. I’ll make a note…
  • The first start might take a while. On windows it may hide itself hide itself in the little hidden icons

Two other disappointing things which honestly limit the project. If I jump to another project, it’s gonna be these reasons.

One thing that could be a dealbreaker is that the notes are stored in a .sqlite database. So if you have things that depend on folders full of .txt files then you may have a bad time.

7 Likes

So I have made a bit of progress on my project. The general plan is to build complex networks of documents where some the relationships between documents are characterised by machine learning algorithms of different types. I think of these as layers in the network. The first layer I am developing is relationships between shared topics that are discovered by topic modelling. So the first thing I have developed is software that does the topic modelling. Now I have to calculate the distances between the topics of different documents, that will give me a sense of how similar in terms of modelled topics any two documents are.

This will take a bit longer I think.

3 Likes

I still use Evernote… every day. I think it’s overlooked is because there was not much on great new features (that are actually good still present) for a long time. But give it a shot, its really good.

1 Like

I use Evernote as well and think it has a well-rounded feature set. I’m looking for a self-hosted version that checks all those boxes.

2 Likes

So brainstorming on a requirements list, here’s some of the things I personally would like, but I doubt there’s anything out there that meets all of these wants:

  • Self hosted
  • All notes/documents/books/todos/calendar stored in a file system so that I can just open the file if i want instead of going through the various interfaces.
  • Basic text based notes
  • Markdown file support
  • For more advanced formatting ODT file support
  • Handwritten notes, ideally stored in a vector format like SVG
  • Handwriting OCR for converting handwritten notes to text
  • A way to link notes/documents/whatever together
  • A unified search of the data
  • Cross device/platform sync for offline access
  • For spreadsheets ODS file support
  • voice notes
  • speech recognition for converting voice notes to text
  • document detection/flattening for pictures of documents/books
  • OCR for getting text out of pictures of documents/books
  • A web interface for accessing on non-personal devices
  • Native applications for accessing on personal devices (linux, mac, windows, android, ios)
  • Hybrid voice/text/handwritten note taking where the text/handwritten note is time coded to the audio.
11 Likes

I use Org mode in emac and sync the file with nextcloud. But im not convice with it in mobile.

1 Like