Devember 2022 Project: Golang web overengineered template

Dev Zone Code
, ,
1

So, I really would rather make something thats actually a project, but I am dry of ideas. And I really like to architect and (over) engineer projects well.

Quick precursor

I was never making this for devember just because its not an actual project, just a template which kept groing in complexity. I then decided to actually post it here becuase I have been working on it pretty much full time because I got made redundant from my dev job (dw im fine). So I thought I would just throw it here bc it has gotten out of hand and im pretty much even building a couple of frameworks.

The actual project

There is and will be more info on the github README (I will be making a better write up soon).

This template got started out of the realisation of that html templating does 80% of stuff you need todo when making a website and there is no need for overly complex javascript frameworks (especially as a golang engineer). So I got started by just making a well architected and simple template for setting up a site and some endpoints, database and authentication. So that when I did have an actual useful idea I would be able to use that tempalte to get right into it and not spend too much time engineering and setting up the project.

Aaaaand the scope of this project grew very quickly by trying to setup support for different features, a cli tool to set up the project for you and even a minimal (by design) javascript framework which would work together with the template rendering which would inject each page’s data directly into javascript (I think similar to something like next.js or svelte) so that when the js renders the page does not need to perform another get request for more data.

Whats working right now

  • Database and ORM with gorm
  • Session/Cookie authentication
    • I would like to make a drop in replacement for other standards like JWT and Oauth2 support
  • Package for simple API development
  • Package for Site development
    • Support for a frame
    • Easily create pages with everything a page needs defined in the page file (not including the template itself)
    • Dedicated function to include js in a template
    • Easy preinjected template variables for current auth, data, meta and existing routes
  • Overall easy to read and very extensible design

Features on the horison

  • [x] Typescirpt support for the teamplate script
  • [x] Static file serving
  • [ ] MAAAybe a golang wasm server?
  • [x] Support for easily adding a js framework (React, Vue, Angular, etc…)
    • Maybe see if we can inject the template data into these as well, would need some thinking through
2

The git repo

3

Let me know what you think guys. I’m more than happy to take any suggestions.

4

UPDATE:

So one of the reasons why I started my pseudo (proto) js framework to work in tandem with go’s templating was because I wanted to be able to use js in order todo some DOM manipulation if I wanted to, as there are use cases for js on the web and I wanted to have some simple functions which would work it HTMLElements directly.

However I was not happy because of the lack of type safety.
This might sound like the rambling of someone that had too much meth, but I’ve managed to integrate typescript into all of this and its actually starting to turn into an actual framework?

Pretty much the way it works now is that you would be throwing any .ts files into the templates/ts folder then that would spit out js into the jsdist folder.
On the template side of things you’d use the includeJs function and the path to the transpiled js in the jsdist folder, then the system would inject the compiles js.
In order to make it work with all the js I already have for this proto framework, I’ve made typings in globals.d.ts github link .

My next steps here is to:
One, clean this up a bit. Maybe even add an includeTs template func which would allow you to include the .ts file then have it translate to the jsdist folder automatically.

Two, there is one more problem, the type system is not maintained across the board. When it comes to the templating language it’self, I’m not sure I can do much without using a different language all together… but what I can maybe think of doing is seeing if I can generate typescript types based on golang structs, so I can posibly match the _data passed in to js to an actual type. I could use json tags in the golang structs then use one of the best features that go has to offer reflect.StructTag

Not gunna lie, this makes me feel like a crakhead.

I’ll have to come back to this in a few day’s time and see if I can make a comprehensive doc about all the stuff I have done and maybe fild a vid or something about this.

5

Update on the Go structs to Ts interfaces saga:

I have spent some time trying to figure out what or even if there are any tools I could use to transpile my go structs to ts structs and found a few git repos for that, some worked but not well enough some didnt, etc.

Finally I found this gem of a tool called tygo.
It basically did everything I needed it todo however, there were a couple of issues.

  1. I had to basically just point to package and it would transpile all of the structs in it automatically
    • only options were to use a file based while/black list
  2. If I am importing any outside the package structs I would not work and just default the TS type to any

For point 2 I have a possible work around…but it might not even be a problem. Will need to dig around a bit more.

For point 1…Well I decided to raise an issue about it, then decided to dig around the codebase myself and open up a PR (my very first PR on a public repo :slight_smile: ) and add a feature where we could use golang comments in order to specify which struct to transpile.
Issue

The more I keep working on this the more complex this is getting lol.

6

Update. It’s Done!

Sorry, it’s been a while since last update.

So I’ve had quite a few struggles on the way here…and I’ve even had to fork and butcher another open source tool to make mine work. So lets get into it…

The template itself

This go template is made for creating some sort of web server with api and a site
And it really does come with batteries included…

Template Architecture Features:

On this version of the cli, all of these template features are shipped, and if there is no need for the feature, just remove it. There are details about how to remove each and every feature which is not necessary.

  • Postgres DB with GORM
  • Session authentication and middleware
  • API package
    • “Beautifully” structured layout for easy endpoint definition
    • With the ability for easy sub-routing too
  • Site package
    • More “beautiful” structuring of the code for easy page definition with support for the following:
      • Static hosting
      • SPA hosting
      • SSR/Templating
      • Primitive JS/TS framework meant to be used together with golang’s template rendering
      • Full stack type “safety” with go struct to typescript types generation using a forked version of tygo

Please refer to the git repo for the template itself. I have left detailed documentation there explaining what and how, together with examples of pretty much every feature which has been implemented. (examples and docs are present in any newly initialised project with that the template)

The cli tool to help setup a new project

Can be found here

There really isn’t much to say here. I have played around with making my own cli and to be frank, it was really annoying kind of hard to figure out for a first time cli dev.
It was really annoying to not only figure out how to properly make a cli application but also how to embed te template files into the binary, how to build an distribute the project, version it with releases and specifically make it work perfectly with go install (because stuff like version releasing is trickier to do because go install does not install the release binary but clones the source code and runs it locally).

So I decided to have a look at other cli tools online to see how they were structured and organised. Low and behold I came across go-template which basically was doing EVERYTHING that I wanted to achieve but with a different template.

Everything was very well made and structured, even making it easy to extend the cli system.
And thanks god for open source because I just decided to clone that cli and adapt it for my own template.

This makes me wonder…maybe we could extend this cli into such a way so that one would just point it at a repo with a template config and the cli would be able to use it as the basis for a new project, essentially creating a marketplace of templates.
And I will definitely look into this idea after we’re done with this contest.

Speaking of which, this is a good segway into the next topic…

Coming up…

  • In the future after this I will at some point give the cli some love and see if I can make the system modular and see if we can make that “marketplace” of templates.
  • Look into how I can make the different features of my template modular.
    • I.E. Ask the user if they require an api, or SPA, etc…
  • I would like to extend the JS/TS proto-framework so that it automatically knows which (generated) data type to expect when using the SSR render function
    • I.E. The render function is called on XPage so that the data in the param is of a specific type which is what is passed into the template
  • Think about making some including system where the subsequent child template which is included has it’s own golang data getter function
    • I.E. creating generic components with their own data getters on the server side.
  • To build on top of the previous, look into making the component system into “dynamic islands”
    • I.E. svelte, where if one component needs to update on the screen, its only that component which updates

With all of this laid out, most of these ideas and “todos” are just concepts and ideas. Some which might actually be overdoing it and simply put “dumb”, but would want to play with them and see what I can do with the most minimal amount of js bloat.

Lets make the web great again!

1 Like
7

…ah and some quick instructions

go install github.com/JamesTiberiusKirk/go-template/cmd/gt@latest

gt new

Then follow the isntructions on the screen. After wards, the new project will be in the present directory.

Then the new project will look like the following.

├── api
│   ├── api.go
│   ├── middleware.go
│   ├── README.md
│   └── route
│       ├── hello_world.go
│       ├── route.go
│       ├── user.go
│       └── users.go
├── config.go
├── dev_run.sh
├── docker-compose.yml
├── Dockerfile
├── go.mod
├── go.sum
├── init.go
├── main.go
├── Makefile
├── models
│   ├── models.go
│   └── user.go
├── package.json
├── package-lock.json
├── README.md
├── ROADMAP.md
├── server
│   └── server.go
├── session
│   └── session.go
├── site
│   ├── assets
│   │   └── style.css
│   ├── middleware.go
│   ├── page
│   │   ├── common.go
│   │   ├── homepage.go
│   │   ├── login.go
│   │   ├── not_found.go
│   │   ├── page.go
│   │   ├── README.md
│   │   ├── signup.go
│   │   ├── templates
│   │   │   ├── frame.gohtml
│   │   │   ├── framework
│   │   │   │   ├── framework.gohtml
│   │   │   │   └── framework.js
│   │   │   ├── homepage
│   │   │   │   ├── homepage.gohtml
│   │   │   │   ├── test_include.gohtml
│   │   │   │   └── test_include.js
│   │   │   ├── jsdist
│   │   │   │   ├── ts
│   │   │   │   │   ├── userFetch.d.ts
│   │   │   │   │   └── userFetch.js
│   │   │   │   └── tstypes
│   │   │   │       ├── baseTypes.d.ts
│   │   │   │       ├── baseTypes.js
│   │   │   │       ├── types.d.ts
│   │   │   │       └── types.js
│   │   │   ├── login.gohtml
│   │   │   ├── no_frame.gohtml
│   │   │   ├── not-found.gohtml
│   │   │   ├── signup.gohtml
│   │   │   ├── ts
│   │   │   │   ├── global.d.ts
│   │   │   │   └── userFetch.ts
│   │   │   ├── tstypes
│   │   │   │   ├── baseTypes.ts
│   │   │   │   └── types.ts
│   │   │   ├── user
│   │   │   │   └── user.gohtml
│   │   │   └── user_ssr_example.gohtml
│   │   ├── user.go
│   │   └── user_ssr.go
│   ├── README.md
│   ├── renderer
│   │   └── renderer.go
│   ├── site.go
│   ├── spa
│   │   ├── react_portal
│   │   │   ├── package.json
│   │   │   ├── package-lock.json
│   │   │   ├── public
│   │   │   │   ├── favicon.ico
│   │   │   │   ├── index.html
│   │   │   │   ├── logo192.png
│   │   │   │   ├── logo512.png
│   │   │   │   ├── manifest.json
│   │   │   │   └── robots.txt
│   │   │   ├── README.md
│   │   │   ├── src
│   │   │   │   ├── App.css
│   │   │   │   ├── App.test.tsx
│   │   │   │   ├── App.tsx
│   │   │   │   ├── index.css
│   │   │   │   ├── index.tsx
│   │   │   │   ├── logo.svg
│   │   │   │   ├── react-app-env.d.ts
│   │   │   │   ├── reportWebVitals.ts
│   │   │   │   └── setupTests.ts
│   │   │   └── tsconfig.json
│   │   ├── react_portal_example.go
│   │   ├── README.md
│   │   └── site.go
│   ├── static
│   │   └── index.html
│   └── tmpl_funcs.go
├── tsconfig.json
└── tygo.yaml