🌵 Cactus Comments: Web Comments, Powered by Matrix

Dev Zone Code
, ,
1

Let me get this disclaimer out of the way first:
I am developing this as part of my bachelor’s project. That means I have already been working on it since the beginning of October, and will continue working on it (at least) until the end of 2020.
Also, I am unsure of whether I am allowed to publish any code yet - if I receive PR’s before I submit my report, is that plagiarism? To be on the safe side, I’m going to delay release until January 2021.

Now, lets get into it!

I am building a system for embeddable web comments, which uses the Matrix protocol.
It’s a thing you can add to your static (or not) website, to get a comments section where the back-end is any Matrix server.
It’s essentially a competitor to Disqus or Facebook Comments - but decentralized and open source using Matrix!

If you’re not familiar with Matrix: it’s essentially an instant-messaging protocol, that federates messages between any number of participating Matrix servers.
The architecture is essentially like email, but much better for instant messages, custom multimedia stuff, and group conversations.
It also has a bunch of interesting features, like easily bridging to other protocols (like irc, slack, discord).
Useful links include: https://matrix.org https://element.io

Using Matrix for web comments has some interesting properties:

Client independence
You can participate in the comment conversation from a large number of existing clients

Redundancy
If the default server goes down, you can still access the conversation through any other server (which has previously federated the room).

Privacy
You don’t have to authenticate against any specific server. You only have to authenticate against the Matrix server that you trust.

Freedom from third-party influence
The antithesis to this would be using Facebook Comments, in which case Facebook decides what you can and cannot say. When basing it on an open spec like Matrix, you can point your comments section towards a Matrix server you trust - or just run the Matrix server yourself.

Openness
Matrix is an open specification, governed by a non-profit, with a really acessible and transparent spec-change process. All major clients and server implementations are open source.
I don’t think I need to argue here why this matters.

So yeah… that’s why.
But how?

It’s essentially two parts:

The embeddable web client
This is the thing you put in your website. It lets you view comments while unauthenticated, by making a guest account on a default Matrix server, and fetching the messages from there. The web client allows you to either post comments as this guest user, or to log in with Matrix user on any other homeserver - at which point all Matrix communications happen with the chosen homeserver, instead of the default one.

The web client is written in Elm, because I prefer programming to debugging. :snowboarder:

The server-side appservice
An appservice is like a “Matrix server plug-in”. It lets you do special stuff inside a Matrix server.
The appservice is a non-essential component (the entire system works without it). But it adds a lot of niceties that you might want.
Mainly it supports some moderation stuff, like banning users across a number of Matrix rooms, or provisioning moderation rights across all those rooms.
The appservice also automates room creation, so you don’t have to manually create a new Matrix room, every time you put up a new blog post.

So yeah… that’s roughly it.
There’s of course a bunch more interesting details, but I guess I’ll leave those until later.

Feel free to reach out, if you’re curious about this project or Matrix in general. I’ll be more than happy to chat about it! :man_technologist:

11 Likes
2

This is awesome.

I’m drooling.

I have a website using mkdocs-material, and by default, it has a comment plugin via disqus, which I prefer not to use. I also host a matrix server, so this should be fairly easy to plug in and a much better option.

3 Likes
3

Glad to hear it :sweat_smile:

Thanks for the support.

1 Like
4

I will definitely follow this thread. Best of luck!

1 Like
5

Here’s an update as I’m making my way out of a way too deep yak-shaving rabbit hole.
A yak hole? Hm… :thinking:

Warning:
This is going to be a lot of writing about a days-deep hole I dug myself into, and how I discovered that I was trying to solve problems that didn’t need solving at all.

Spoiler:
I’m going to end up reverting almost everything to how it looked before I started digging this hole.

Background
Matrix message events may contain formatted text (HTML). This is how we can have nice things like markdown rendering. The Matrix spec provides clear guidelines on how to parse and transform this HTML content. It’s pretty important to get this right, since it’s a potential XSS vuln if I don’t.

Solving Moving Creating Problems
The client has supported sanitizing and cleaning HTML content for a while now, but after discovering a CSS injection, I wanted to do more thorough tests to lessen the probability of oversights.

elm-test has some very nice fuzzing tools that I’ve been wanting to try out for a while. So of course I wrote a fuzzer for my HTML sanitization.

Performance (Non-)Issues
My code didn’t exactly provide wrong output on any of the generated tests - but it did crash and burn on memory use when the fuzzer generated larger input.

The Matrix spec says that I should limit HTML messages to nesting 100 levels deep, but my implementation exceeded 2GB of memory with less than 10 levels of nesting - albeit very wide trees.

It’s not acceptible for my client to crash under any circumstances - especially not because of user input. I imagined somebody being able to post a lot of HTML content in a comment, murdering the RAM of visiting users.

Considering this a DoS vulnurability that needed fixing, I rolled up my sleeves and started optimizing.

The Optmization Deliberation
Being a good-boy functional programmer, I immediately recognized that my function is memory intensive, since it makes a copy of almost the entire tree on each recursion level. I also quickly decide that I should solve this by rewriting it to make use of tail-call elminiation in the Elm compiler. This turned out to be harder than expected.

I model HTML trees as a recursive type that looks something like this:

type HTML
    = Element String (List Attribute) (List HTML)
    | Comment String
    | Text String

(don’t worry if you don’t understand this notation)

Essentially it’s a rose tree structure. I spent a while trying to come up with a function that would traverse a rose tree tail-recursively by myself - but this proved to be difficult (see: beyond my ability).

After researching for a day or so, and getting some help from my FP friends and some wizards in the Elm slack community, we arrived at two new implementations:

Both of them performed slightly better than the original. I was able to increase the fuzzer’s max nesting depth a little before hitting memory issues again. It was still managing way less than the depth of 100 I was aiming for.

At this point I was starting to get worried about other parts of the code too - for instance the testing code itself was still traversing the tree in a naive way, so I had to rewrite that in the two new styles as well.

Discovering the Combinatorial
At this point I started looking a bit closer at the output my fuzzer was producing. I knew that the output was huge, but I wanted to figure out just how huge it needed to be.

The answer: way way way smaller that I was testing. Turns out that Matrix only allows events less than 65kb in size.

My fuzzer would regularly make hundreds of child elements at each level, each with potentially hundreds of attributes, all having names and values up to hundreds of chars long… With that large a tree, I can exceed 65k at just the first level.

Ugh. Of course.

  1. There’s an pretty big combinatorial explosion in my fuzzer.
  2. The client will never even get moderately large input from the Matrix server.

I rewrote my fuzzer to never use more than one call to Fuzz.list, in order to generate much more convservative trees - and all of a sudden all three implementations could process narrower trees of nesting depth 100 very quickly.

Picking an Implementation
At this point I’ve spent three days wrestling HTML tree traversals - and have three working implemenations of the same functionality. Which one do I choose?

Thinking like an engineer, I start testing each of the three implementations against ten thousand different realistically small trees generated from the same seed.

And what do I find?

  1. The continuation-passing test performs fine.
  2. The naive recursion runs (negligibly) faster than the continuation-passing approach.
  3. The frame-based approach is buggy, and still hits edge cases where it fails the test. :angry:

When I was debugging the frame-based implementation, it really became clear to me how big the difference in clarity was between the three implementations. The naive solution is obvious and easiy to read and modify, while the two other approaches seemed very obscure to everyone I would show it to.

I will prioritize correctness and clarity over performance any day - especially when it’s security-critical code.

So… we’re almost back to square one. I will be git revert-ing the html cleaning back to the original, naive implementation. At least I have a pretty good testing suite for it now.

Moving Forward
I’m crawling out of the hole.
There are more important features to get to.

Next step is likely going to be one of:

  • Store session data (access token, homeserver url) in HTML localstorage.
  • Handle HTTP Errors better (for instance, retry on HTTP 429).
  • Render more message types (images, file downloads, audio).
2 Likes
6

Short update to cover some steady progress:

Session persistence
Finally! This wasn’t hard to do, but I put it off for a long time since it requires some Javascript interop.
Now the web client will keep your Matrix access token (and some extra stuff) on subsequent loads of the page.

This solves two problems:

  • having to log in again when reloading or visiting another page
  • creating unnecessary amounts of throw-away guest accounts on the Matrix server (b/c now we can re-use the same guest token)

I did it with just one outgoing Elm port, which saves a string to a fixed key in HTML5 localstorage.
Messages are sent through it when a new guest account is registered, or when a log in is successful (basically whenever a new Session is constructed).

The persisted session is loaded into the Elm app at init time as a flag (a nullable JSON value) - if it parses a Session successfully, the client won’t register a guest account at init time. The loaded session might be a logged in user (on any server), or a guest account on the default server.

I ran into some issues with localstorage along the way. First of all, I am kind of shocked that localstorage won’t store regular JS objects. I solve it easily by serializing to strings - it just seems weird that I have to.
Secondly, I had hoped to be able to make localstorage data persist across multiple domains. It would be really sexy to be always logged in (to your own Matrix server!) on whatever Cactus Comments section you come across. Alas, you can’t even share localstorage data between different subdomains…

One way to work around it could be to load an iframe from a specific third party, and then fetch the session data from inside the iframe, and before passing it out to the “parent” element. If the iframe is always loaded from the same domain, it will have access to the same localstorage data. This iframe could even be hosted by the Appservice - which is already a Python+Flask HTTP thing. This would allow all Cactus Comment instances that share the same Appservice to share Matrix auth sessions, and I think it would work!
I just don’t see a way of solving the security problems here… Is it at all possible to prevent a malicious site from embedding the iframe and stealing people’s Matrix tokens? :tired_face: The domain-restricted localstorage access might just be extremely reasonable.
I think persisting the session across multiple sites is just out-of-scoe - but if anyone has a clever idea for this, I’ll gladly hear it! :man_technologist:

Either way, persistence works now - so I’ll mark this one as Closed.

Discovering that I’m not a Web Developer - Part 1: CSS
We tried to do a demo using some off-the-shelf blog template, that uses the Sakura stylesheet - and immediately the default CSS that I wrote for Cactus Comments started messing up.
I discovered three issues that need solving:

  • Login Form shows up way below the page, instead of showing up in front of everything, at the center of the viewport. No idea why.
  • The text inside the buttons in the Login Form float to the top… No idea why.
  • The avatar images attached to each comment are way smaller than expected. No idea why.

None of these issues are present in the minimalist no-content page I have been using for development so far- so something is bad with the interaction between the stylesheets.
This is my first project where I’ve been using just vanilla CSS, and although I feel like I’ve learned a lot , I also definitely sense that there’s a long way to go.

I’m also realizing that there’s this thing called a “CSS pre-processor” that is apparently standard to use? I need to find out whether this is something I need. I definitely need to get some kind of CSS linter soon.

I definitely consider it a requirement that the comments section should look good on whichever page it is plopped into.
So I want to keep doing tests on different blog templates, to make sure it all works.

At some point I might need to have a more experienced front-end designer help me with my stylesheets.
I assume it’s basically a worst-practices atrocity at this point :snowboarder:

Discovering that I’m not a Web Developer - Part 2: Distribution
I’ve also been looking at how to get the compiled JS out in the world.

My favorite strategy is to pin the compiled JS assets to IPFS, and expect users to href some IPFS gatway.

<!-- like this -->
<script type="text/javascript" src="https://gateway.pinata.cloud/ipfs/Qmbnyp39DR96qQJ2E3H2Ms2Y5DSLgaCDq4xGTASdf4MiHg"></script>

I already run unit tests and compile the web client using Gitlab CI, so it was pretty quick to add a step that uploads the compiled asset to a pinning service. Using Pinata.cloud I can do this with just a single HTTP request. Pinata also allows me to pin up to 1GB to IPFS for free, which is quite a lot of releases at 90kb per release.

This works fine for now, and locking a websites imports to a specific hash is nice, but I would still like some way of automatically pointing to the latest release - or latest stable. I could use DNS linking, but that would leave my DNS records as a single point of failure. The better solution is IPNS, which is nice and distributed.

However, Pinata does not support IPNS :frowning: So I will have to set up something else.
Right now I’m looking at Temporal.cloud which supports both pinning and IPNS. They’re very cheap and give me up to 3GB for free.
The catch: they only support pinning a file for up to 2 years. I don’t really want my stuff to disappear just because I don’t push an update for two years. One solution could be to use Gitlab’s scheduled jobs - another could be to pin files using one service, and manage IPNS using another. There are definitely still some decisions to be made.

Another issue is distributing the CSS. Currently I’m counting on distributing that completely separately, so the user might replace or modify the default stylesheet without changing the JS stuff - but I’m wondering if there might be a clever way to bundle it with the JS stuff. I could always allow the user to pass a config flag like overrideDefaltCss: true or smething… food for thought.

…and then there’s the elephant in the room: NPM.
I think most people building JS-heavy web apps prefer to manage dependencies via NPM. Although I don’t love npm, I don’t mind acommodating that.
Using NPM would also have the advantage of easy versioned distribution using a service like jsDeliver.
I don’t really know anything about packaging things for NPM, and I’m still a little intimidated by it. But I figure I need to do this at some point.

Moving forward

Priorities for the next week or so are:

  • Discover and solve more CSS problems, learn about how to write portable CSS.
  • Set up a reliable distribution pipeline.
  • Make prettier, close-able error messages.
3 Likes
7

Okay I did some more things…

Distribution

Still distributing on IPFS! I changed a few things though.l

First off all, I decided that I don’t need DNS linking or IPNS for now. This means that users of my stuff won’t be able to get updates without manually changing the link.
Basically users are forced to pin versions for now, and I think that’s fine.

Secondly, I start pinning a whole directory of built files. This includes:

  • Minified JS
  • Minified CSS
  • Source maps for both

Since it’s a directory, I can refer to the files by human-readable path relative to the IPFS CID hash.

So, the links for v0.1.1 look like this:
https://gateway.pinata.cloud/ipfs/QmeQrWvsPXni3dB5VmFjKwRb2RRoP9LGCPpCbsPHt5Rxdn/0.1.1/cactus.js
https://gateway.pinata.cloud/ipfs/QmeQrWvsPXni3dB5VmFjKwRb2RRoP9LGCPpCbsPHt5Rxdn/0.1.1/cactus.js.map
https://gateway.pinata.cloud/ipfs/QmeQrWvsPXni3dB5VmFjKwRb2RRoP9LGCPpCbsPHt5Rxdn/0.1.1/style.css
https://gateway.pinata.cloud/ipfs/QmeQrWvsPXni3dB5VmFjKwRb2RRoP9LGCPpCbsPHt5Rxdn/0.1.1/style.css.map

Hopefully this will make the resulting HTML just slightly less obscure.
Also, I hope that the browser will just be able to load the source map files for debugging, since they are at the same relative path as they are in my development environment. I haven’t tested this yet, though.

I found out earlier this week that the Brave and Opera browser both have some kind native IPFS support. I’m super curious about what the actual deal is there, and want to spend some time figuring out if I can do something to better support direct (no HTTP-gateway) IPFS loading.

My CD scripts reads the version tag from my package.json file, and runs on every push to master. I think I might want to change this to instead use a version string specified by a git tag release.
The release tagging thing is a feature I never really got into, and would like to learn how to do well.

Also at some point I should look into publishing to NPM as well… For now I think I’ll put it off until someone requests it :sweat_smile:

Minor CSS Advancements

I fixed all the CSS issues that I wrote about in my last post.
Along the way I did a small handful of refactorings / advancements too. Most significantly the buttons are prettier and narrower now.
I also moved some buttons around for improved UX. Nothing really major tbh.

I haven’t yet tested on any other pages, but definitely still want to! (see: need to!)
I think I want to set up a blog using Hugo in any case - that might be a good use case to stress test the styling a bit.

A Better README.md

I wrote a user guide!
It gives a pretty decent overview how to set up the web client, and what the requirements are for the few configuration values it takes.

I also wrote a “quick start” version that is just a minimal working HTML snippet, that fully sets up a working comment section. For the curious, this currently looks like this:

<script type="text/javascript" src="https://gateway.pinata.cloud/ipfs/QmeQrWvsPXni3dB5VmFjKwRb2RRoP9LGCPpCbsPHt5Rxdn/0.1.1/cactus.js"></script>
<link rel="stylesheet" href="https://gateway.pinata.cloud/ipfs/QmeQrWvsPXni3dB5VmFjKwRb2RRoP9LGCPpCbsPHt5Rxdn/0.1.1/style.css">
<div id="comment-section">Loading Comments...</div>
<script>
initComments({
  node: document.getElementById("comment-section"),  // HTML element to make comments section in
  defaultHomeserverUrl: "https://cactus.chat:8448",  // full url of the Matrix server to use as guest
  serverName: "cactus.chat",            // server name of the Matrix server w/ Cactus Appservice
  siteName: "MyBlog",                   // name of your website. used for moderation namespacing 
  commentSectionId: "NovemberBlogpost"  // unique ID for this comments section.
})
</script>

(Note that there isn’t actually a Matrix server on cactus.chat right now, and I don’t care to share our development server at the moment, so you can’t actually use this snippet).
But, I assure you it works! :wink:

Landing Page

A while ago, we bought the domain cactus.chat, which has just been laying around pointing nowhere so far.
So we set up a quick and dirty landing page, that just has the basic concept, our logo (did you see our very sexy logo yet??), and a link to our public Matrix room.

It’s not anything to write home about, but I figured I might as well share it here.

Moving Forward: Experiments

As I mentioned, working on Cactus Comments is part of my bachelor’s project, which I hand in on the 20th of December.
Since I now have a pretty good working PoC / prototype, I am going to transition into designing some experiments to verify that it actually has the properties I assume it does.
I think it’ll be good for both the robustness of the application, and for my final grade :sweat_smile:

A couple assumptions I want to test:

  • When the appservice is down, all existing comment rooms continue to work perfectly.
  • If the default homeserver is down, and a user has already accessed the room from another homeserver, they can continue to interact with the room via their own homeserver.
  • The appservice may run on another homeserver than the default homeserver, without disrupting the guest access functionality. (Key question here: does peeking over federation even work in this bleak pre-MSC2753 world?)
  • If the website itself is hosted on IPFS, and the website includes javascript code to run an IPFS node in-browser, then both the webserver, the Matrix homeserver, and the moderation appservice (all server-side software!) can be turned off without disrupting availability. Of course this requires that the room has been federated and that at least one other browser is currently serving the HTML over IPFS; but I think it’s still a very sexy demo!
2 Likes
8

So I’ve been a bit of a bad boy and loaded up the snippet in mkdocs. It works! Even signing in works! This is REALLY neat.

image
image814Ă—884 28.7 KB

Instructions for how to load a custom comments section in mkdocs-material:

  1. Add
theme:
    custom_dir: overrides

to mkdocs.yml

  1. Create the overrides directory in the root of your mkdocs project.
  2. Create main.html in the overrides directory.
  3. main.html contents:
{% extends "base.html" %}
{% block disqus %}

<html snippet here>

{% endblock %}

So playing with a bit, it does not like it if I remove a comment.

And I need to figure out a way to dynamically change the commentSectionID depending on which page it is on.

9

Hahahah! That’s awesome!

I definitely didn’t expect someone to start using it with basically no documentation. You’re even on an old version right now, that has some bugs.

I think I might just quietly open source it soon, so you don’t have to mess around with reverse engineering my stuff :wink:

Thanks for the heads up! I’ll make an issue for this bug.
There is still a handful of events it does not render properly, for instance message edits are also a bit janky.
I did not think about deletions at all yet.

Did you also deploy the appservice alongside your home server, or are you creating the Matrix rooms manually?
How did you figure out which room alias to create?

Thanks for giving it a go! :grin:
Looking forward to releasing it proper with docs and stuff.

1 Like
10

I am creating the rooms manually. I don’t think you have posted a link to the appservice yet.

Via this error message:
image

The only thing that threw me for a loop was that the error message has the room in uppercase, but the room it was looking for is all lower case. I’m not sure if that is the mkdocs CSS doing, or the error message is originally upper case.

1 Like
11

This is absolutely true - but it is actually available on dockerhub. I was wondering if you had gone all-out OSINT on me :wink:

That’s odd - I think the error message from the homeserver is actually just upper case, unfortunately. I’ll have a look at it :slight_smile:

1 Like
12

I did it! I finished and handed in my bachelor’s project.
Here is the project report, in case anyone is interested.
This means that I am (almost) finished with my B.Sc degree, which feels great! :mortar_board:

It also means that I can get back to writing code, instead of writing about code.
I am far from done with Cactus Comments, and I’m excited to keep working on it.

Oh and I opened up the repository. We open source now!

It’s not ready for production yet, so hold your horses. But if somebody out there is curious, this is what the code looks like.

Moving forward, there is a lot of stuff still to be done. Here are just some of the next things off the top my head

  • Put IPFS links in the gitlab release messages
  • Parse matrix state events like m.room.created and m.room.guest_access
  • Render more message types (images, files, etc)
  • Make prettier css (less distance between message, maybe a separator line)
  • Render messages that relate to other messages (redactions & reactions)
  • Host a Matrix server and cactusbot on cactus.chat
4 Likes
13

I guess I should do a semi-final post on this thing, since I’m only now (way too late) noticing that wendell is asking for something of that nature over in the main thread.

On the academic front: I did my BSc defence, and I ended up getting the highest possible grade for it! Very pleased about that. Soon I will have a bachelor’s degree in cyber technology. Woop!

I’m noticing that some peeps around here are talking about setting up a Level1 Matrix server.
Just for the record: I think that is an excellent idea! https://forum.level1techs.com/t/matrix-server/167203/36

So for my Devember2020 submission: The comment system is fully functional, and is currently in a sort of alpha / early-beta state.
The first users should absolutely be able to pick up and figure out how to use the system.

Before I call anything “1.0”, I want to make the onboarding experience so much smoother. It needs to be closer to Disqus comments’ onboarding that to that of a Matrix bridge. A huge part of this is having a public homeserver and instance of the cactusbot. Once that stuff is in place, the “quick start” guide can become so much shorter. I want prioritize writing extremely accessible and concise docs.

https://cactus.chat is currently just a static GitLab pages site. When I start pointing that domain towards some kind of dedicated VPS in the near future, maybe I’ll be able to use some Linode credits? :crossed_fingers:

Thanks everybody for a super fun Devember this year!

EDIT:

I forgot to mention: there’s a stupid simple demo page over at https://demo.cactus.chat, if anyone wants to try and around with it.

2 Likes
14

We finally released Cactus Comments! I think a bump is justified :cactus: :tada:

The complete documentation, including a friendly Quick-start Guide is all at https://cactus.chat

We’re also hosting a public Matrix server, and a Cactus Comments appservice there. Free as in beer and as in speech, inteded for general public use.

Reception has been really positive so far! The other day I woke up to Cactus Comments being on the front page of Hacker News, we’ve gotten a really positive reception on reddit and the fediverse too. There has even been some talk about implementing it on the matrix.org blog in the latest edition of This Week In Matrix.

:cactus: Go check out our stuff!
Live Demo: https://cactus.chat/demo
Docs: https://cactus.chat/docs
Introductory blog post: https://cactus.chat/blog/hello-cactus-comments
Source code: Cactus Comments · GitLab

4 Likes
15

This is pretty huge, and congratulations on getting it working! I’m super happy to see it working off an existing protocol like Matrix as well. It’d be nice if we could finally ditch stuff like Disqus and have a unified web presence that isn’t personal email associated.

Thanks for sharing this project here, and I look forward to seeing more about it in the future.

1 Like
16

So I did encounter a small bug on the demo site: trying to type the // in https:// for my homeserver when logging in, just kept focusing the text input on the search box, like I was using vim or something.

17

Thanks for reminding me to fix that! A couple of other people have mentioned it to me.

It should be fixed now, although you might need to clear your cache to get the new site.

1 Like