Wasn’t sure if this belonged in this category or Software.
Million dollar company in a garage
Getting to Beta
Myself and two others listened to @wendell too much and decided to start a small business around a chatbot and LLM for a niche industry that I work in that is severely lacking support and good knowledge bases.
We already have several different potential costumers that have expressed interest in the product and that makes it much more motivating to get something across the finish line
Initial Goals
- Conversational chatbot trained on technical documentation.
- Voice and Text interface over a phone number.
- Website interface.
- Send to user relevant documentation
- Website for managing users and files.
Third Party Services
Much of the userbase is boomer construction workers so voice calls were at the top of the list for interfaces and Twilio has a solid integration. Right now to start off we are using chatgpt’s 4o-realtime voice but as time goes on one of the stretch goals is to train our own custom model that we can make lightweight and portable. Depending on how technology progresses we might still be dependent in third-party voice-to-text.
So there has been a fair amount of trial and error as you can imagine. It has taken us about two weeks of working on it on and off to get to what might be our first “Beta” build of our stack.
The Stack
The program started out as a simple mongo db of pre-processed PDF’s that had been converted to MD. FIrst step was getting twilio/chatgpt/mongodb to all work together.
Twilio and chatgpt were easy. Their documentation and integration were great. I was able to get a docker container with a twilio endpoint up and running in no time hooked up to the chatgpt api endpoint.
After that it was introducing the program and logic for querying the mongodb. This required a fair amount of trial and error too get the chatbot logic to ask the right questions and use the query results for the correct context in the conversations.
After that was figured out the next step was sending files to users. We debated just using a google drive or dropbox integration but we are trying to keep as much stuff in house as possible. I already run a Nextcloud-AIO container stack so naturally I started looking into that for housing the original pdf’s and serving them to the user.
I built a tool that the llm program can call to interface with Nextcloud. Initially it was just to find a relevant PDF to the user’s conversation, generate a download link that expires after 72hrs, and then email it to the user. This was pretty easy to get up and running. and taught me a bit about the capabilities that Nextcloud has now.
I will fill in with example code from the python nextcloud tool at a later date.
Current state of the project
We are getting ready to roll out a beta program in September with a local partner to see if we have a viable product. To recap we are currently hosting a PDF library using Nextcloud that has a python program for sending users download links to relevant documentation on demand, a MongoDB instance that is acting as a “Library” for the LLM, and Twilio+ChatGPT as the front end for users to call into and ask questions.
Nextcloud = Everything
This project has been a big learning experience for me so far with lots of new technologies that I haven’t used before. But what I think was really interesting to me was learning how much Nextcloud can do these days and the built in capabilities.
Started to dig into Nextclouds API’s and boy there is a lot it can do. It can house the files, generate download links, send emails, store users, and I might even be able to coax it into a basic authentication directory, and more!
As development has progressed and I learned more and more about what Nextcloud offers these days and boy it does a lot. We now have several integrations with Nextcloud on the dream sheet for future implementation.
-
The Fulltextsearch with ElasticSearch is very powerful. We are considering doing away with the MongoDB and just using the built in Fulltextsearch functionality natively with the LLM tool to recall MD files for context.
-
Proper Vector DB. Right now we are using MongoDB and search queries to provide MD strings that fit into the context window of chatgpt-4o-realtime. Next step is getting a proper vectordb to run alongside the traditional database to provide a smarter chatbot.
-
Unified search API calls for authentication of callers. When someone calls or texts the chatbot the first thing it does is look them up in Nextcloud and see if there is a user associated with the number. Not only authentication but personalized greetings and eventually chat history for the LLM to reference that is tied to each user.
-
User mangment. Nextcloud has easy to use user management tools and also integrates with SSO providers if customers would like that functionality.
-
ExAPPS. Nextcloud now allows you to run microservices coded in python that are directly integrated into the nextcloud-aio instance. This is where it gets really interesting imo. We will be looking into using this functionality to essentially re-skin Nextcloud into a complete solution for the front and backend of our service.
-
Monitoring and metrics. Being able to provide managers with chat histories, usage metrics, etc. for review. Again thinking this will be an app or ExApp built directly into Nextcloud. That way you can see who is using the tool the most, what they are asking about, identify problem equipment or lack of workforce knowledge based on call topics, etc.
The next steps
We are refining current functionality and trying to get it in a suitable state for beta. There are plenty of things to work on and more features to add. I made this post not so much because we are doing anything crazy or super innovative but just to showcase that there is a lot of cool stuff out there for you to work with and maybe make something cool of your own.
I especially want to document some of the Nextcloud custom integrations and code that we come up with. I will probably set up a public github with sanitized versions of the programs. I think Nextcloud might be on it’s way to become a powerhouse software stack for small and medium businesses. It would be very cool if we can get an entire SaaS stack to just run on a custom Nextcloud VM or docker template.