[Devember2021] Hello world 2: The Electric boogaloo

Hello world 2: The Electric boogaloo
…and now we draw the rest of the owl


Recipe Website Preamble:

Problem Statement: a group messaging that is alt discord(text)(I don’t know who they are going to sell to) and me and the boi’s want a place that is off a big platform, but there’s a problem I don’t want to explain what IRC is and matrix seems ify, plus I am software engineer thingy so why don’t I just make my own.

This is really just a study into kube and go with external given public deadline for that pride hit if failure ensues.

This will be more of a Infra and less code, but if I get the infra a good spot(It never will be) I can get some good feature creep going.

Minimum Viable Product

“Users can login and see and post message to the group”

Registered User can login

Registered User can Logout 

Registered User can Edit User Information  

    Registered User Once logged in  

        Can See List of "Servers" they are IN 

            Can "Enter" "Servers" they are part of 

                Once in "Server" They can post messages attached to their User 

                Read Message History of the "Server" 

“Time Allowing”

Invite Users via Email 

    Invited Users can register 

Have "Server" "Admin" group 

    Can Invite New Users to platform 

    Add register Users that are part of that server to Admin Group 

    Can Edit Information about the "Server"  

Messages Have Timestamps 

TLDR; Text Discord, but different, but really just a disguised study into Kube and go

3 Likes

What I got so far(Planned)

Golang – Services

Kubernetes - (Trying Learn)(Yes I know I don’t need it)

Keycloak – I am not writing my own user management system + Postgres + OAuth

Lets Encrypt – TLS + CertManger

Nginx – Ingress

Note: this will all be hosted on the Linode (but I used my free credits up long ago lol).


Next Expected Deliverables:
  • Status Update
  • Git Link
  • Maybe some yaml(this will be changing daily tho)
2 Likes

Some progress on the keycloak and postgres neither are finalized but there are some good parts to this

Git Link: GitHub - Ed-Mar/appalachian-bastion

33MB of yaml later

Okay I have gave up on due this with just kube, and getting some help with from Helm.

I am switching from linode domains handling to cloudflare cause I can’t get the DNS01 to work correctly with the web hook function of cert-manager. There is this Linode Community Post from sub one year. Two personal from that post give two chart Git One Git Two that try and deal with this. I can’t get them to work for some reason or another(I assume I don’t wtf am doing the more I work on this). Cloudflare is a supported DNS01 provider of cert-manager. That transfer is in progress now so i just have that working when that is done.

[85%] helm keycloak/postgres
[55%] ingress with tls
[-20%] actual development

Some Finds along the way

  • Lens a k8 “ide” it just a kube dashboad with some nice options, but I have run into an issue with custom definitions
  • Oliver Coding Blog so found this blog post that outlines at lot of what I am trying to do for the most part
    • Note: the keycloak chart values are little out of date. I will push mine once I get them closer finalized

Thoughts

should have some more time to work on this as this month winds down due to things.

For some reason I really don’t enjoy Helm, Id rather just work with k8 yaml no charts. I know I properly wrong, but just find helm annoying for some reason.

also might try haproxy

Progress 211108_1203

  • ingress controller is operation(haproxy)
  • cert-manger is operational
  • keycloak is operational
    • more testing of full functionally is still needed, but initial smoke test everything seems to as it should

GitHub: has been updated with helm values and cluster issuers

Next items

  • Get a better understanding of the realms and how to best operate keycloak
  • test the email validation for user sign-up
  • plan out the structure of the date of what is need for a user sign up.

Future item (In no particular order)

  • get app up working with some go bones
    • deal with the SAML login
  • plan out the data structures for all the items that I think i can deliver before the time is up
  • pick a backend (relational or not is the real question)
    • once the data is finalized(-ish) get some swagger going
  • frontend work if they exist so i guess i have to deal with that too
  • other(s)

Demo

Notes

I did end up going with cloudflare and haproxy. I spoke to those items in the last post

Looks like we’re getting out of yaml land…

SMTP

I had limited interactions with SMTP really only inputting provided cred to use one that was already set up for some bot related work professionally to integrate with an API

Now working through all the keycloak features when I see this.


But something i did not plan for, and email sever idk why. It would used for users to validate an email. Granted this i not one of the item i have listed as necessary goals, but non the less I will waste only today and try and figure how to add some SMTP to my cluster.

So I go to looking, and i find some open source ones
I find two that the internet seems to like: IRedMail & Postal

IRedMail

doesn’t seem to be production ready or at least the docker version.

Postal

Seems to be more ready for containerization and gives a nod to linode, but when I search on linode for any info on it there isn’t any it seems. Regardless I think will try with this seems a lil more ready plus I already have one redhat item going.

I am going to try and get postal up and working

Postal SMTP

Okay after spending some time looking around at the state of the postal. I have found a few items.

  • Postal Github and it creators and maintainer are making updates and so on, and are focused on containerization.
    • They don’t have a public docker hub from what I can tell, which is odd but I see why so you have to host the image your self if you want use stock version.
  • There is a helm version of this that is of this that was created about 2ish years ago by a Linkyard with consideration to cert-manger and ingress options which is good.
    • There is an issue tho with it being 2ish years old the kube api objects are out of date(apiVersion: apps/v1beta1) which can updated.
    • Also I am not positive but looks like readiness probes wern’t a thing at that time so initcontainers busybox containers are used to wait for the two required backend to be spun up. There also a few other things if left in would bother me so if I do end up going with would require a good amount of changes
  • There also seems to be another option to get this working and its a customized docker version of postal with some spam and anti virius running on alpine instead of using the given Ubuntu. Github tiredofit/docker-postal
    • Its in docker and on the hub which is nice save me some time plus it comes with some addon option out the box
    • So let Kompose it oof that a lot lol
    • yeah that’s is good amount of kube objects and also the those pvc are all in different volumes which i guess would work, but that seems wrong. I had to look into this while writing this,(5 mins later). It seems you can do both so hmm

So what the plan for SMTP

I am going to use both linkyard/postal-kubernetes & tiredofit/docker-postal I am going to mash them together into something that can work on kube. They will just plan kube yaml for no helm. Once I get to a place with all these object I may consider making it into working helm but maybe
I have started on this I got the all the env variables files in config Mappings and respective secrets in to separate files (50+)

breakdown into separate files

There is still a good amount of work to do on converting this into something that works well.

Edit

I will not be doing this Postal is kinda of mess.
The tiredofit/docker-postal uses his own custom alpine image which is eh, but I tired to remove the s6 thing in that wont stop the container even it fails from what I can tell and even it do build its hug for alpine idk

  • They don’t have a dockerhub
    • so you have to build your image. fine
      • but that image will failed then I check the git hub issue board 40+ open issue. I do think i could messaround it get it work with the base image, but have spent about a day on this. SOoooooo

NEW Mail sever time and it this one(for now…)

Docker Mail server

and what’s this some kube support(unofficial)

I will try(again) to get mail sever on the k8…

nvm I gotta find a new way on how to handle mail linode says no mail

fml I have wasted so long on this

1 Like
  1. Configure valid A records and reverse DNS for the Linodes you’d like to use for mailing.
  2. Open a Support ticket and provide us with some basic information (the information we’ll ask for is outlined in our “Running a Mail Server” guide.)

not too much to go through, especially since your are targeting known users that are accepting / waiting for your emails

keep up the fight, its only halfway thru december :slight_smile: (sorry, meant November)

1 Like

Keycloak “Learning”

Preamble: I have play around with keycloak before but I really don’t understand all the in’s and out. I want to learn more about it so I know all the options and constants when planning out the user structures and how the roles would interact with data. I also need some info how plan out the api’s I need to create to interact with it.

Okay I found a book on keycloak that came out this year. It happen to be on O’reilly so that subscription is coming in handy
Book on keycloak

Why is a book needed, cause the documentation on how to best operate with keycloak is some limited info this I was able to stumble through to get some interaction with the keycloak but know that could be better.

2 hours later

Application integration study with keycloak

so the book has some in good info and explains a decent amount more than the basic information . The book has an associated github with some tools for working with openid to get a better understanding of this standard(and the other) Link to Openid Playground


ch4

… Later same day. post workout & oil change

Getting along in this text mite be close to

This book seem pretty good for what I need have really only one or two hiccups. Looks like the keycloak used for this book is 3 major versions behind even tho it was released this year. It seems v11 was release July 2020(current release is 15). so eh.

There is a golang section which was cool, but it just the standard req/response handlers so I will be looking to gorilla options for this after this

even has a golang section
ch8
desgin thought

read time about 8 hours I am not the fasted reader plus I did most of the excises

Post Book time

Okay I did like this book for the most part(or at least for parts I found relevant which was 75%). It did good job in comparison to what is available on the web on how keycloak works and how to get it do what you want. It did go over some design points which i found useful in addition different approaches to problems with some realism through in(i.e don’t make your own Oauth framework). The goal of reading this book was get a better understanding of what I can and need to plan for when designing this app which it did. There is also some relevant information on how to just do some things I had a hard time figuring out my self. tldr book 7/10 would read if getting started with the keycloak.

Unrelated

I am getting back into this after sometime due to getting promoted to customer,social event, and a bender should be able to work on this full time starting 12/6. I am not sure when this contest endings(I assume end of year), but I will be continuing afterwards anyway.

Reply

@paulwratt Yeah i did see that I’m not sure how long that will take to get that thur there CSRs so i am going to fore go it for now, and put in it for it later, cause I’d like to have any emails be hosted by myself(aka Linode) than use of the paid options. I was looking at there TOS for some paid email providers and they are a lil spooky.

Next Item(s)

I am working on plan and design of the app it self now. I got the ol pen and paper out trying to stop my self from making things to complex.

So I am going to let keycloak act as centralized authorization server (crazy i know, but there are other options.) so this will allow me the most flexibility when creating roles and access, Cause I know I am going to change it a good number of time during this initial creation phase.

I am just shooting for the MVP right now( I’ve had to stop my self like 4 times already for trying to plan for the future)
So that means

  • One “Sever”(Discord term(actually in the docs its called a guild))
  • One message channel
  • Only one type of User (So can read message history and post)
1 Like

211106

Clients

clients in keycloak

so planning out the objects/clients etc…

  • I am going with conduit for the app name for now
  • so from my current understanding I am going to need
    • one for the backend(data)(conduit-backend)
      • not super positive on this on if I use a object relation mapper but just in case i think it mite be redundant
    • one for the browser app(conduit-browser)
    • one for the server side interactions api calls (conduit-server-side)

Groups


At least to start i am just going to have these groups
I am going to lean on keycloak to handle what user have access to what servers(discord term) for now I am not postive this best way, but speed is the name of the game right now I need to get some Proof of Concept completed sooner than later.

Users


Just some basic test users with assigned happy path roles

its noon 30. work to cont.

Openid Connect Framework

Okay the book and from last post did have some example code for using go-oidc but it did not have PKCE which seems very important. so i need to find one that does.
enter image description here

There seems to be two options that le google throws back

they both seem to be good options fosite seems to be bit bigger and is not based of the vanilla go lang package so I will be going with odic.

I fell into the rabbit hole of looking in to both of these
Planning next moves

1 Like

211209

I will do the oidc integration with the code later

For now I am starting on the code for the the application now. I have started with some basic apis and middle ware for the server data type. Just gonna start basic stuff and I will build outward. I am going to integrate the keycloak SSO later. I am having trouble getting it work due to lack of understanding on my part( I have already lost like three days on that, I just need to move forward)

By the time you read this should be on github for this(Github Link).

Note: I did remove the infra yaml from the hub for now. I put it back later once I have the sites project structure more thought out. I know I do want everything in one repo

Short term Action items

Services

  • [ ] Get J SON validation integrated
  • [ ] Get Channels data structure complete
    • [ ] Get Channels Service Completed
  • [ ] Get Message data structure complete
    • [ ] Get Message service Completed

Frontend

  • [ ] Flesh out the UI I made the other day in NextJS with bare minimum servers,channels, and message items (just need to display what server/channel you are in plus the associated messages)
  • [ ] Connect the frontend and backend and wiggle them together for now

Containerization

  • [ ] Containerize the services
    • [ ] Server
    • [ ] Channels
    • [ ] Messages
  • [ ] Containerize Next JS frontend
  • [ ] Thrown them on the cluster
    • [ ] Add them to the ha proxy
    • [ ] Check the freaking DNS

Longer Term Action Items

  • [ ] Open API ( Swagger documentation)

    I think i mite do this sooner than later I think It would help me with doing validation plus code gen for the client for the frontend

  • [ ] Integrate oidc (some how)
  • [ ] Add a database (currently using in code data store)
    Server List
1 Like

211210

CRUD operations for Servers

broken down handlers

So I have the basic REST crud operations for the for the server service. For the Create, Update, and delete aren’t necessary for the MVP, but I thought since i was here and for practice. For the future I will have to the create, delete, and update interface with keycloak to add/remove/update the necessary roles, but that is for a later time I really just need the GET right now and will remove them from the client.

Client → Runner → Mux → Handler → Validation → Server CRUD function

Mux to Handler

local testing ListAll

Open 2.0 (Swagger) Meta + Docs

So looking at what needed to be done I needed a client for the frontend so I figure I’d do two things at one documentation and get some of that yummy code gen. so I did that for this primary start.


Powered by ReDocs

Swagger docs

$ server-api/handlers/docs.go

// Package handlers classification of Server API  
//  
// Documentation for Server API  
//  
// Schemes: http  
// BasePath: /  
// Version: 0.0.1  
//  
// Consumes:  
// - application/json  
//  
// services:  
// - application/json  
//  
// swagger:meta  
package handlers  
  
import "server-api/data"  
  
// Generic error message returned as a string  
// swagger:response errorResponse  
type errorResponseWrapper struct {  
   // Description of the error  
 // in: body  Body GenericError
 ...

Serve the redocs page and the swagger.yaml
Serve both the Redocs and the swagger on the base directory

Don’t think I’ll keep the yaml on forever but I think it will help with code gen once it gets on the cluster mite make it easier.

Next.js Frontend

Alright I am getting into frontend and straight up out of my element.

So I followed the Next.js quick-start guide. I have some experience with js but with bots so eh. I followed the guide to get two pages up and still need to table or something for the get call for the table.

basic is basic

Should be on the github when you see this.

1 Like

211212

Added JSON Swagger Def

This package I am going to try and use for next js code gen (Next Swagger doc) needs the json instead of yaml changed the make and runner to do that and host that file with the yaml. Quick addation just have the go runner server that file and have go swagger create a json version in the make file.

Switch to typescript project type for Next js Frontend

For rest of the 12th I just was trying to learn how to best use Next js and the service I got the codegen to gen some code, but need to learn how Next js wants things. God I am bad at the frontend thing. went with openapi-codegen-typescript cause I just wasn’t all the positive witch one to use in swagger editor.
Codegen working

211213

After many hours I have come the understanding that I have no understanding lol.

Ditching Next.js

I thought Next js would be easier, but I was really struggling with it for some reason It just wasn’t clicking for some reason. SO we are going to react. There is more and better documentation for it( yes I know next js has react, let’s just chalk it up to me being dumb and move on)

Gonna also try out this AXIOS which seems to be a lil simpler than the openapi-codegen-typescript and the swagger editor massive file.


Not sure if the updates where I really don’t do anything but fail and learn are useful, but I figure they mite be to someone. Cause I only knew a few of these things beforehand, but professionally I only work with java and services this full stack thing has bloody large amount of moving parts and most them I’ve never work with.

211214

Went with the Just the React and Axios, and I was able to get that to work, but the frontend is gonna need more work as time goes on, but I have completed my goal of wiring the front and back end together. The only change needed to the back end was to enable CORS with gorilla toolkit so that was straight forward

What been completed

From the 211209 POST

Services

  • [x] Get J SON validation integrated (Ongoing)

Frontend

  • [x] Connect the frontend and backend and wiggle them together for now (Ongoing)

Longer Term Action Items

  • [x] Open API ( Swagger documentation) (Ongoing)

There all on going as add more objects they will needed to added, but there is a pattern I can follow moving forward so the big brain thinking can be used later

Onward to Databases

So i have( bad one) link to the frontend so let try and get the preliminary step to get the backend(database up). SO have pick a type wither relational or document style, but I will see that go ORM support and move from there.

So I do have some worries about doing these integrations before data is near finalization which would leave to rework, but i think standing it up before i start can lead me to making better decisions on what i’ll need for the data, plus data structure is never finalized.

211218

Docker image for postures

Bog standard docker compose
boilerplate docker-compose

Note I plan to turn on ssl mode(link) on later when this gets on the cluster

Using GORM for database

Documentation for this and the package it self makes this rather easy. It is extra dependency, but for now its find. I like ORMs but I understand why some don’t and agree for the most part
Automigration is complete for the server struct
Migration of struct done every time the service spins up

.ENV files and being smooth brain

Took me a few (*cough hours) to get the env files to work ending up using the Viper package. Works rather well. There are some random files that need to be cleaned up from some other packages trial and error, but I will make a push by the time this goes up so if you want take a gander


Will cont to work on filling out rest of the CRUD and project files directory layout will be the next items. After those items I will get to start getting to the channels once that

211221

Database CRUD operations integration

Working to get the CRUD operation working correctly with the GORM and the database.

Create (AddServer)

No id provided letting GORM auto++

Auto increment is on for gorm

Update

Update

Get {id}

model it is on for testing for update and delete

Note that the model object has been added to the server model but is only turn on in the json for the update/delete testing

Update

Update

Update time as been updated as well as the object itself

Delete

Delete call


Note that Server is not in that array, but still in the database

Get (Get all)


Get all Servers which will probably be the most used and really the only one that will needed the most at this time, there others are there for future use.


This was stuff I have done a lot of time in other langues, nothing special. I wont do an update like for the rest of the api, but i figure it best to do one.

Channels

Started the work for adding the channels struct to the Server Struct and figuring out the mux, validation,gorm, logging, and error handling when dealing with a nest rest object.

Code has been pushed.

211223

Kinda lost with the channel addition

So In my head its straight forward, cause adding the channel obj to the Server is straight forward but integrating with the existing middleware, and mux is throwing me for a loop.

0415

Native Postman runs like Sh*t

the native app runs like trash it eats up RAM and CPU. Not really related to this project, but I though it was interesting that is not just a me problem I notice while leaving it open.
Link to Postman Performace Issues (Github)

211228

CRUD for Channels is in a operational state.

It been pushed. Github link

Messaging is next

Note on timeline

I am running short on time for this, because at a minimum I have the following things to complete to meet the MVP. Be Straight up I am not going to be able to finish the MVP with amount of time I have left. I will try for the first two just to see a message on the frontend.

  • Messaging CRUD
  • Frontend in some short of form
  • Containerization
  • OIDC from the keycloak

Even after #devemeber2021 is over, I’d still be interested in ongoing updates, even if it ends at “functionally complete” and not “damn this is a fine piece of coding and design”.

Yeah, the fails are also noteworthy, and will help some other people in how they feel about their own failures, especially if its with the same tool or framework. Maybe even one of their devs will come across the post and think “hey maybe we should have a look at this from a noob perspective”, and fix it - meh, you can always live in hope (or phantasyland).

.env: this is one of the most common web probes I see in my firewall endevours - I protect this file (non-exhistant on my setup) by passing back a 4.5Gb DVD ISO image. For fav.ico I 404 it unless there is a HTTP_REFERER.

Anyway, keep up the hard work, it would be nice to see this completed.

Cheers

Paul

1 Like

211230

Circular imports and GORM

GORM leads it self to putting you in a circular import loop by design and I have come to learn that from trying to add this temp user struct due to them being many to many

So each server is going to have many users and have each user is going to have many servers (in theory)

But I did deem them close enough together in scope to be in the same package like channels and servers.

which leads to the user needing the server to define user and vise versa.

Loop de loop
and in go that is a no go.

So what do.
There are few options

How did I not see this coming?

I did know about circular imports, but just didn’t think about it until I had had already done it. The GORM documentation did help, but its still on me but here:

Okay here(above) do you see it? They are in the same file. I just filtered that out in my head.

Thought I just share that. I would like to move away from GORM at some point cause it’s not the fastest performance wise, but until I get closer to a finalized data model i am going to lean on it to save some time

220101

Now What

So I am going to try and break down each of the services that I have created to SQL schemes/Queries with the necessary item so that don’t have go into the database for any reason.

So stuff like this:


Looking at that code you 'll see items that lock me into postgres,(pgx , & “information_schema.table”) which removes one of the advantages of using pure SQL and that it can be database Agnostic. I may end hating this later, but whatever send it.

I am trying find a way to not have to maintain both the model for json and the sql schema together in the same struct so when I need to make changes or additions i can in one place, but there doesn’t seem to be one that can do that with references to other tables

Here is sqlx, but I still have to maintain the schema separate,

This is sqlx

GORM did that, but doesn’t seem to a viable option for micro services or at least at my understanding for what I want to do with.

Next things

Reading another Book

If I am going to do the microservices things I guess I should read their bible things. Cause I think I missing some information on this topic. Going to Cherry pick some chapters that seem relevant to see if any insight can be gained

Get some schema together

With this I have exactly what I want, but I also have know what I want so a double edge sword.

This is not what I going with, but show some things, but is still very wrong
Server Information

Red: UUID this was easier than I thought to get working, but I will need the extension to be enabled for each database it used in. Just another thing to error check for and implement.

Blue: Postgres allows for arrays, but after reading up on them they do append when updating they update the whole array which is a waste, so I think going with inter-between table for many-to-many at least.

Yellow: just being able to do a reference. I think have be able to alter a table with running a query while still using GORM, but doing so makes me just not want to use GORM or any ORM(go-pg did the same thing).

#devemeber2021

So I failed, but I will continue to work on this item for some time, but I need to find a new job so that may take of available brain availability

Response @paulwratt

I will keep going with this. It was always a study first and it lined up with #devemeber2021. Granted I was hoping to get farther than getting to temporary user data model to implement messages and realizing my database interface conflicted with microservices aspirations.

I have no fantasies that anyone would come to this for lessons learned lol, but it helped me get my ideas in some form organization.

For the .env on the back end I will remove write access or if I am really worried about I will pass them as a argument and keep them in a kube secret

For the .env on the Frontend I will look into your SSFW, but Frontend seems so far at this point.

there is not much in SSFW that can offer you a solution. The best I can do is to say, pull up the “known_urls.txt” an search for “.env” to show the sorts of locations are probed.

EDIT: it should be possible to 404 any HTTP attempt to get /.env via webserver configuration, while still being available to the web app (because its a local service/process accessing it)

As for the rest of your project, even your 1st couple of posts pointed to it being (possibly) hard to complete.

On the Postgres lock-in, there should be a solution to that, I cant say what that solution might be, but my previous experience says its possible, but it might be one of those things where you need to allocate some time to “go down a deep dark hole” (so to speak).

Anyway, I think you did alot, learned alot, and exposed alot (to us) - all well worthwhile.

Cheers

Paul