Umium run 2 - Blackjack and H00k3rs

ablebit · December 31, 2022, 8:26am

They say an image is worth a thousand words. So here’s 2000 words worth in image form:

standards

Preamble

I’ve never considered myself to be a programmer, or particularly aspired to be one. I’ve done some BASIC programming as a child, but I quickly lost interest. What got me was design and production. I have professional experience as a graphics illustrator, as a web designer and as a video producer, and I have done 3D graphics, music production, web development and CAD work on the side.

Pretty soon, I started gradually acknowledging several pretty much universal problems with software that increasingly impeded me in pursuing my goals, the more software I learned and used the more I found software to be:

too bloated, buggy and inefficient
too expensive and inaccessible
too needlessly fragmented
too complicated and limited

It didn’t take long before my questions went from “how do I do this and that” to “why do I need to do this and that” and “why can’t I do this and that instead”, and the answers have always and only been “If you don’t like it, make your own X”, which I do realize was just a polite way of telling me to go forth and self-multiply, yet since I did not found this possible or practically useful, I decided to take it in the literal sense instead.

So exactly 10 years ago, I set on the seemingly ridiculous journey to get into programming and create better software than what the industry leaders, with their hordes of programmers, mountains of money and decades of experience.

About two steps into that journey, I stumbled in exactly the same as the above problems with programming. And it makes sense, after all, that is still software. I spent like 2 months just figuring what language and framework to start learning. The easy ones were inefficient, and the efficient ones were hard. The general purpose ones were slow to work with, the purpose specific were useless outside their scope.

Programmers appear to almost universally believe that this is not only inevitable, but actually a good thing, and that a universal programming language but a feeble dream, a sort of unattainable Utopia, a pink unicorn that simply cannot be. Maybe programmers see things in a specific way, because even back when I was a complete newb, I could see why those things are, and that there’s no intrinsic reason for it to be that way.

I’ve identified a few key reasons:

The evolution of programming appears to always have been “adding more” to overcome problems and limitations, to the point nowadays it is recommended to have 32 gigs of ram to run a decent IDE on an average sized project. Programmers build on top of the same paradigm, rather than trying something radically different.
The notion of “code” is orthogonal to clarity, software shouldn’t be written in a textual form, and then hope the compiler gets your intent as intended, and then hope the runtime produces the intended results. It is my understanding that software should not be written but designed, and that a good design can accomplish a set goal with but a fraction of the code that goes in a traditional workflow. So far this has been corroborated by my experience with the project.
Math is not essential to programming, or “the language of the universe” for that matter, that would be logic, to which math is just a subset of. This used to be the case back in the days when only scientists had computers, and they used them to quickly solve equations, but computers have gone a long, long way since then. In fact, one can be a perfectly good programmer with like… 3rd grade math, even cutting edge stuff like AI, 3D or multimedia is not mathematically complex, it is all basic arithmetic, the results do not arise from mathematical complexity but computational throughput. I think that what makes a good software designer is above all creativity, and I don’t think math bolsters that, and even suspect it may be impeding it, in short - math is for computers, and pushing a human to be better at a machine thing always comes at the expense of other aspects of human ability. And no matter how much you push a human to be a computer, it will be vastly inferior to even the most basic computer. So Umium tries to do its best to focus more on the human side of productivity and sweep technical nonsense under the rug, without detrimental effects on the end result.

So what is Umium exactly?

I will start with what it is NOT - a programming language.

Umium is not a “language”, it is a graphical tool, it has no syntax, and by extension - no room for syntax errors.

Umium is not strictly about software development, that is just one of the things it is intended to do, and what lies in its very core. But Umium is about much more, it wouldn’t be universal if it was limited in what it can do.

Umium is a high level tool for low level abstract logic design. It is a complete departure from the traditional way in which software development has been done:

Traditional programming: model > code > compilation > execution > result

Umium: model > result

Umium is a real time modeller, it is a tool that directly manipulates raw computer memory, no intermediary representations, only basic safeguards.

Umium is a runtime, compiler, IDE, platform, version control and ecosystem all in one.

Umium is not for just for programmers, or just for computer using creatives. The ultimate goal is that it is useful to everyone. Kinda like don’t need to be a professional orator to get tremendous benefit from being able to produce and understand speech, and you don’t need to be a professional writer to get tremendous benefit from being able to read and write.

It is all about the use of information - speech and writing have played a monumental role in human development, and now that computers are everywhere, the prospect of using, exchanging and processing information has exponential increased. Yet I can’t shake the feeling humanity is not making the optimal use of computers, and even possibly getting a rather short end of the deal.

I remember back when smartphones were becoming a thing. I was rather thrilled by the prospect of having a computer in my pocket, just imagine all he things we could do. But that didn’t really manifest, all due to the lack of software to enable it. I in fact dare say that on the grand scale, people’s personal use of such devices is totally eclipsed by the industry’s use of people through those devices.

Umium is not just about solving technical issues, but all creativity and collaboration related issues as well. Especially problems around licensing, my goal is to make licensing effortless and sort of automatic, so users don’t have to waste time, money and worry on what they can or can’t do. I am still figuring this one out, meaning that my own licensing is not yet finalized, which is also why for the time being I abstain from publishing any code, but for all intents and purposes, all of Umium will be eventually open source.

Umium is insanely scalable and efficient, as a universal tool it is not optimized for any particular usage, but designed to work everywhere, from tiny MCUs to supercomputers. Here’s some meaningless theoretical specs, figures are “up to”:

minimum object size : 1 byte
maximum object size : 1 gigabyte
maximum number of base types: ~ 4.3 billion
maximum number of base objects: ~ 4.6 quintillion
maximum addressable physical memory: 4 exabytes
maximum addressable virtual memory: unlimited*
maximum number of virtual objects: unlimited*

*unlimited - Umium employs a lightweight data compression mechanism that can trade off CPU time for theoretically unlimited amount of memory. This makes it practically feasible for its allocated virtual memory to exceed the used physical memory by many orders of magnitude, while maintaining useful performance levels. So while being scalable beyond the foreseeable limits of physical hardware, it is designed to be extremely efficient in memory constrained scenarios.

Short term goals:

complete the GUI
finalize licensing
release a native client for Windows, Linux and Android
release a web client via WebAssembly
add WebSockets support
add cryptography facilities
add file system facilities
support basic multimedia - images, audio, video

Struggle, failures and lessons learned

When I started this endeavor, I didn’t have the slightest idea what I want to do, much less how, just the gut feeling that I must do it. I’ve done some DOM scripting in JS and AS before, but when it came to writing a full fledged program, I didn’t know even where to start. After learning the gist of several programming languages, I eventually settled for C++, for its power and efficiency, and adopted Qt, as the most comprehensive application framework available for C++.

I spent the next 3 years learning programming and trying smaller and less ambitions things, and then went on with my first real attempt to write Umium. It was a very traditional, by the book approach to software development, the good old “lets start writing some code and see where it goes”, without a carefully designed architecture and strategy. It was slow and tedious process, the pace was slow and demotivating, and clearly not suited to my task, it was a “outside in” approach that wasn’t hard to project would stretch into the hundreds of thousands of lines of code and many many years. Something still feasible for a company with many developers, but totally inapplicable for a one dude project.

So I gave myself 2 more years of learning more before my second attempt, and I did manage to take that one much further with far less efforts. I had an almost complete version, one that made excessive use of Qt and QML facilities, and one that relied on contrived schemes to dynamically describe and generate QML code and execute it on the fly. It turned out, by my own standards, extremely slow and inefficient. Most people in my place would have released that, as it would have been still usable in many scenarios, and it did constitute a significant amount of work, and by doing so, commit to that design indefinitely. But I didn’t, I knew that wasn’t it.

Two failures in a row, and 5 years “wasted” felt a bit discouraging. But what took precedence was that despite this, the lessons that I’ve learned have in no way diminished the feasibility of my undertaking, those were just indications that what I was doing wasn’t an effort in the right direction. This was the first time in my life that I didn’t really know what I need to do to achieve my goal and didn’t have anyone I could ask that, and had to figure it out by process of elimination and wandering into uncharted territory.

I spent another 2 years leaning, but this time I didn’t emphasize on programming. Clearly, more of the same wasn’t going to help, it wasn’t the problem of not being enough, just not being the right way to do it. Instead I took my time to learn lower level concepts, how the actual hardware works and how operating systems work at a kernel level. This led me to eventually commit to my current and from the looks of it, final design, and I went straight for raw memory access, the very thing programming has progressively been moving away from.

Needless to say, it still wasn’t a clear cut path to getting it done. I’ve found the level at which to do it, and I have figured it must be an “inside out” approach, but with still quite a lot of figuring out left to discover what exactly that is, and it took another 2 years and 3 implementations rewritten from scratch to finalize the fundamental design so I can focus my attention on the front end.

I entered last year’s Devember competition under the name of UM. Since then I decided that it is not that original if the name of the thing sounds like “hmm”, so it was promoted to Umium, which does actually sound like a thing.

Last year I couldn’t bring my work to completion, I kinda got lost in case studies and micro-optimizations. Programmers say that premature optimizations are a bad practice, and that’s true if you are on the clock and you need to deliver results. But I do not consider my previous efforts wasted, because those premature optimizations did reveal vital information, which did ultimately help me commit to a fundamental design. Trust me, to change fundamental design later on is about the worst idea one can possibly get.

Another lesson learned - while it is a huge time saver to rely on other people’s code, when you want to accomplish something that no one ever has, relying on code written under a different paradigm turns out to be an extremely bad idea.

I did put a non-trivial amount of effort making my data model work with Qt’s own, fixed roles, abstract model interface, with the intent to leverage its existing model view facilities. I figured it will save me some time, and I can always do my own GUI later on. A HUGE mistake on my behalf, turns out my requirements to present any data in any way is extremely orthogonal to Qt’s own internal design. Different types of views were incompatible with one another, you could not arbitrarily nest or mix and match things like I wanted to. In the months that I struggled to make Qt do what I needed it to, I discovered and reported dozens upon dozens of bugs, arising from unforeseen by its developers “outside the box” use. And with no solution to most of them in sight, I ultimately decided to scrap this effort, and develop my own model / view architecture from scratch (the Blackjack and hookers are implied).

That’s what I spent the last couple of weeks doing, the model part already more or less finalized, and the view work still in its infancy. In addition to that, I took some extra steps to facilitate an eventual complete removal of Qt as a dependency. My current binary is about 25 MB, and some 95% of that is just Qt, so in the midterm, Umium is set for a massive reduction in size.

My “almost production ready” design from 5 years ago never felt right to release and commit to long term, and I dare say the extra 5 years I put into the effort were worth it - yielding around 50x reduction in code size and an even more impressive, in excess of 500x reduction of memory overhead. I can’t give any number for the performance improvement at this stage, but I do expect it to be massive as well.

Looking back at this journey, what I appear to have done is spend 5 years learning “how its done” and another 5 years to shake almost all of it off, before I am left with the few key components that I actually needed all along. And this also possibly explains how come nobody has done this rather trivial in hindsight thing, in the at least a couple of decades it’s been technically feasible to do. It is a huge d’oh moment and real shame it took this long to realize a rather obvious thing - that one simply cannot solve the issues of legacy software development by doing more legacy software development.

Disclaimer

This post does not, in any capacity, endorse the acts of gambling and prostitution.

ablebit · January 7, 2023, 5:59am

I am basing my GUI on only 2 c++ classes, Leaf and Node, which are vanilla qml Items, but with some extra properties and bookkeeping done in c++ in an effort to make things a tad more efficient. Leaf is basically the base type for visual representation of data, and Node is a controller of Leaves.

That being said, Leaf is by no means a dead end in the traditional sense of a Leaf, it is just a dead end for my part of GUI management. So Node can nest other Nodes and Leaves, but Leaf cannot nest neither, it is just a controller for monolithic qml items, so nesting arbitrary levels within a Leaf is possible, but only for pre-coded qml items, where Node trees are fully dynamic, and usable to build things that are not defined at compile time.

The c++ implementation of Leaf just adds several properties for the sake of efficiency, as in qml you can’t do neither bitfield nor even unsigned 32 bit integer properties, but most of all, the Leaf interface is needed on the c++ side for the Node bookkeeping.

Leaf gets some more functionality on the qml side as well, but overall it is kept pretty barebone:

C.Leaf {
  id: leaf

  width: childrenRect.width
  height: childrenRect.height
  item: display.createObject(leaf, { src: leaf })

  signal click(point p, int b)
  function clicker(p, b) { click(p, b) }
  property alias pressed: input.pressed
  property alias point: input.point.position

  TapHandler {
    id: input
    acceptedButtons: Qt.AllButtons
    acceptedDevices: PointerDevice.AllDevices
    acceptedPointerTypes: PointerDevice.AllPointerTypes
    gesturePolicy: TapHandler.ReleaseWithinBounds
    onTapped: (e, b) => clicker(e.position, b)
  }
}

Extending it to fit its dynamically created content, sourcing said content to the Leaf, I am not using a qml stock Loader here, as it doesn’t offer the means to set initial properties, mandating the use additional Binding or Connections element. On the c++ the item property has a custom setter which deletes the previous representation if any. So no Leaf is “married” to any qml representation, it is all dynamic for all leaves.

Then Leaf gets itself user input listening from any and all input devices, and interfacing the required functionality. I am not going straight to the click here, because I want that to be overloadable in the case for Leaves that may instantiate a number of qml items internally, so a single TapHandler can be used with some basic coordinate mapping, rather than having an additional handler in each and every qml item.

A typical Leaf is expected to be managed by a Node, but I don’t want to go exclusively Node and Leaf, because of fragmentation. If for example I need for render a large text, I would need a Leaf for every character, something that already happens internally on engine level in qml, so there’s no need to overburden the scene-graph with multitudes of additional discrete text objects that render a single char each.

Thus PolyLeaf gets “data listening facilities” in order to use a single discrete element to render multiple data points:

Leaf {
  id: pleaf
  
  property var cmd: [
    (i, c, f) => console.log("empty add", pleaf),
    (i, c, f) => console.log("empty upd", pleaf),
    (i, c, f) => console.log("empty rem", pleaf),
    (i, c, f) => console.log("empty rst", pleaf)
  ]
  
  C.Watch { id: _w; onUpdate: (t, i, c, f) => { cmd[t](i, c, f) } }
  function link(c, d) { src = d; return c.watch(_w, d) }
  property alias sys: _w.sys
  property alias size: _w.size
}

Again, keeping it barebone, just adding the data watcher and aliasing some properties on component level. This can then be further specialized by simply overloading the cmd list for the given usage context. For example, here’s what a functional TextLeaf that renders raw data looks like:

PolyLeaf {
  id: txt
  
  display: Text { property Item src; font.pointSize: 24; text: txt.objectName }
  onClick: (p, b) => tctrl.moveto(p.x)
  
  cmd: [
    (i, c, f) => objectName = stradd(objectName, sys.readtext(src, i, c), i),
    (i, c, f) => objectName = strupd(objectName, sys.readtext(src, i, c), i),
    (i, c, f) => objectName = strrem(objectName, i, c),
    (i, c, f) => objectName = c ? sys.readtext(src, i, c) : ""
  ]
  
  function stradd(s, c, i) { return s.substring(0, i) + c + s.substring(i) }
  function strupd(s, c, i) { return stradd(strrem(s, i, c.length), c, i) }
  function strrem(s, i, c) { return s.substring(0, i) + s.substring(i + c) }
}

This uses a basic qml Text element, it is just a static label, that has has no editing facilities, cursor metrics or any of it. I get that functionality by using a single TextControl element that is attached to the selected text element:

umtext

… to be continued

Nic_s · February 1, 2023, 2:30pm

I enjoyed reading this today and I hope you continue working on this as I think it’s quite interesting.

ablebit · May 19, 2023, 1:25pm

It’s been many months since the last update, and that’s another failure to meet a deadline, but there’s some development too.

I had a nasty data loss that extended all the way to backups, so I essentially lost another month or so rewriting stuff that I’ve already written, taking the opportunity to write it slightly better.

I ended up adding ad-hoc Qt model / view support on top of my notification implementation as a convenience. Which also gave me an opportunity to examine the performance impact side by side. And it does offer some early use value until I implement their equivalents, at which point they can be seamlessly replaced.

Oddly enough, between entirely skipping the QModel… interface and issuing notifications with no subscribers, there is almost no performance penalty, maybe 2-3%, although it does increase the noise or inconsistency than overall throughput, I assume given how poorly it performs, it probably touches quite a bit of code and data to do its thing.

I have a direct memory access API on the QML side now, which is great for prototyping without recompiling the C++ code, and despite being 1-3 orders of magnitude slower than C++, it is still way faster than user input, and once prototyped, the same code works with little to no changes in C++ since the API is fully mirrored.

Here’s the three front ends from the video above, the first directly using the model interface, the second using the direct interface to access data via JS functors, and the third one being a C++ version of the second essentially.

readonly property Component _modview: Rectangle {
  width: _data.parent.width; height: _data.parent.height; color: "red"
  Repeater {
    model: _src.source
    delegate: Rectangle {
      width: 3; height: 3; color: "white"
      x: display.ru16() * .02; y: display.offset(2).ru16() * .02
    }
  }
}
readonly property Component _canvas: Canvas {
  id: _canvas
  width: _data.parent.width; height: _data.parent.height
  Connections { target: _src.source; function onUpdate() { _canvas.requestPaint();  } }
  onPaint: {
    var ctx = getContext("2d")
    ctx.fillStyle = Qt.rgba(1, 0, 0, 1)
    ctx.fillRect(0, 0, width, height)
    ctx.fillStyle = Qt.rgba(1, 1, 1, 1)
    core.qmod(0, 0, 0, (d) => { ctx.fillRect(d.ru16() * .02, d.offset(2).ru16() * .02, 3, 3) })
  }
}
readonly property Component _image: Img {
  id: img
  width: _data.parent.width; height: _data.parent.height
  Connections { target: _src.source; function onUpdate() { img.update(core, 0) } }
}

Quite straightforward - the model interface uses the display role to access the underlying data as a generic pointer, crazy - right? It is supposed to never give you a bad pointer, but we shall see. It is the same thing for the manual interface - except you get to optionally write to each individual item. In the specific case, my data item is a random 32 bit integer, so my delegate / functor reads off the x value from that directly, then offsets 2 bytes to read the y.

In order to be able to have my way with data and still get efficient notifications on updates, I came up with this usage format, where I make a selection, which can be an Insert, Remove, Update and so on, then use that anonymous non-copyable explicitly perishable item to inject my will into the whole operation, so it can be performed in the desired order before the operation is finalized and the notification issued.

For example, here’s the “animation” function for the above test, where the entire data resources r is selected, with a lambda macro that calls a convenience method, which casts each data item to a 2d vector.

void anim(uint r) { write(r) << WRITER { dostep(wi.item); }; }
void dostep(u16 * v2) {
    v2[0] += (20 - (int)QRandomGenerator::global()->bounded(40));
    v2[1] += (10 - (int)QRandomGenerator::global()->bounded(80));
}

I also got bulk reader and writer operations, which are handy to the many operations that can be performed in bulk, such as trivially movable data, serialization and deserialization etc… The functors are quite sparse in provided parameters - just a data pointer and a size or index, but auxiliary task-specific parameters can be setup in the context of the lambda and captured. For example:

uint dosum(uint r) { uint sum = 0; read(r) << READER { sum += ri.item.ru32(); }; return sum; }

And just to clarify, those manual pointer offsets are just for testing, obviously in production all that stuff is going to be populated by proper lookup tables and whatnot.

In addition to the Qt model interface, I have a nifty little View component, which can be sourced by a model, with the option to specify sorting and filtering functors on the QML side. This alone is quite useful to implement commonly needed stuff, like say the public view of an object, which simply filters out its privates, or the constant view of a read only object, by hiding all mutable members, or showing an isolated selection of data.

  View {
    id: _dview
    src: _src.source
    filter: (r) => !(r.read_u32() % 7)    
    sorter: (a, b) => a.ru32() < b.ru32() 
  }

ablebit · May 19, 2023, 5:51pm

Licensing

Licensing is a non trivial issue for anyone who wants to get out there. Commercial licenses are pricey, as are license lawyers. Some open source licenses come with strict obligations, and evidently, many companies who make their living on harassing developers into compliance.

With Umium, I want to make licensing effortless - full-proof and fool-proof. The Umium Fair Use License is configurable, and you get to configure each individual project upon its creation. You get to chose the basics:

open source and / or close source
free and / or commercial use
as is or modifyable
generative use

Yes, Umium is so cutting edge it facilitates that crazy new thing - generative use. You can opt in and out on a per project basis, and you can actually monetize generative use of your IP. It’s a bit complex to explain, but it involves the capability to calculate “share of contribution” for each generative work based on tracking what goes in.

All of the options are non-revocable, meaning you cannot publish something open source and then close it, or allow free use then pull the rug from under the users by making it commercial only.

Aside from letting you customize licensing options, Umium automatically checks against the licensing options of the products your product includes internally. Obviously, all of this is facilitated by using Umium as a service / infrastructure, it has no way to magically work for any 3rd party license. And much like with generative use, it is possible to calculate the total share of all work within a product, direct and derived and automatically calculate and distribute revenue accordingly.

That’s pretty much it for now, it is about giving authors options and flexibility, and ensure user licensing choices are upheld and guaranteed, and last but not least - transparently monetize the commercial usage of your IP across the entire product chain.

ablebit · June 4, 2023, 7:14am

Finalized a bit more info:

Umium is a SERVICE!!!

Umium is a service, and it is not “free”. This also means it doesn’t need to come up with indirect ways to monetize user data. In fact, although there are still mechanisms to make certain usage of anonymized user data, the default is OPT OUT, and any such mechanism requires authenticated permission from the end user to opt in explicitly.

There is a “free” tier, which is basically local service only, it enables to do basic work on that local machine, in addition to learning how to use the actual product. Perfectly usable for education or daily office work, but in essentially “offline mode”, although it is possible to pull free apps and tutorials.

The paid tiers charge a service fee. You get out of the box ready to use storage space, bandwidth and cpu time to get your stuff out there, offer or search products and services.

I aim the “Indie” tier at around 7$ per month, as low as around 5$ if prepaid for a year, I’d say that’s affordable enough for everyone who can afford a device capable of running Umium, higher tiers will have more hardware resources and more framework and infrastructure access in turn, plus some extra tools to handle bigger projects and customer bases.

Tier pricing is inclusive - for example, if you prepay for a full year of Indie tier, that amount is also for example the same as the cost of the Pro tier for 2 months, that’s what you get - a couple of months of Pro service to preview. Keep in mind, tier-tied functionality requires active subscription for that tier, so it is technically possible to lock yourself out if tier functionality, by falling out to the previous tier.

Why not free - because ultimately nothing truly is free. I think it is preferable for a service to be FAIR than it is to be “FREE”, that is - the terms of service and operational model are open and honest, and the service is adequately priced.

There needs to be a mechanism to keep off spammers, bots and above all - terminal idiots. So the subscription fee acts as a basic guard against that - people are less likely to bother with fraudulent accounts if each pop costs them money. Plus this gives a payment identity which can be blacklisted for repeated offenses.

Operational model

Instead of a “profit tax” Umium only charges service cost upkeep. The rate is modest, it can be adjusted in the range of 1% to 2% depending on operational costs.

Upkeep counts as community contribution - if your upkeep exceeds your subscription fee, that’s equivalent to paying the next one. Users who generate enough revenue to pass the service tier threshold are automatically promoted to the higher tier. Likewise - if you have generated some upkeep but not enough to cover your next month - you only have to pay the difference.

Service fees are prepaid, you are not redeemed unused resources at the end of the month, and as it is typical for such services, exceeding the limits gets charged.

The upkeep sounds low and it is, but it is not my intent for Umium to be a for profit tool. In case it isn’t clear - I designed Umium to basically be the service that I’d want to use, at the terms I’d want to use it, and at the cost I’d be OK with paying for it. So the growth strategy of Umium is not its own profitability, but its usability. I need a usable tool, not one to milk myself with. I am not creating Umium to get rich off a golden goose, I am just creating a tool that I need for my personal work, and it is just that it doesn’t cost much more to extend that service to other people who are willing to pay for their end of it.

I want to explicitly state that this is not an altruistic endeavor. I think that every rational human action is ultimately self-serving one way or another. For example, people are only charitable because… it is charitable and certain people need that for whatever reason. It is just that there are ways you serve yourself that vastly vary in their effects on the environment. I am not creating Umium out of the kindness of my hearth or indiscriminate unconditional love for humanity, it is just that wider adoption makes the tool even more useful, so it is offered in the way I think will drive a desirable adoption vector that may not necessarily be the much craved “maximum”, which also makes it very balanced - Umium is designed to be just as useful to its users as its users are useful to it, which practically eliminates the prospect of profitability, prioritizing sustainability instead. The sustainable operation theory depends on optimizing and automating just about enough to provide compute infrastructure at the cost of only 1-2% of the revenue it generates, which based on absolutely no hard data study or research I think should be doable by making a tad more creative use of technology.,

As a result, Umium is designed to not have upper or lower management, no employees, no contractors, no customers, no stockholders - it has one interface for user interact, that is subject to the same rule-set for everyone with no exception. Interfacing with any other forms of organization is out of scope and delegated to the users. For example, if company XYZ wants to use Umium as a commercial team, then it is up to XYZs management to create, operate and moderate the respective XYZ group within Umium, each interacting with the group in their capacity of an independent user, under autonomy of the groups’ management.

The teams limits depend on the subscription tier of their founders, but other than that is fully configurable within those limits. An Indie user can have up to 15 active groups with up to 15 users each and be active user in up to 15 groups. You get a founder slot for the groups you found, so it doesn’t eat from the 15 “member of” limit. That limit number increases x10 for every subsequent tier.

Decision making is a collective process, to which everyone is welcome, where good ideas will be noted and bad ones - sanctioned. Decision making is group grained, where the root group is all users, where every branch has autonomy as long as it does not violate the ruleset of its own context. Voting is a collective process as well, where every user’s vote is weighted relative to their share of contribution to the particular group.

Platform support

Umium targets the following platform levels:

0 - base - requires only support for 8 16 and 32 bit integer types, 32 bit can be emulated, so the baseline mode can run on pretty much anything from an 8 bit MCU and up
1 - embedded - requires 32 bit hardware, and at least software FP32 support, that’d be your typical Arm MCU
2 - device - requires FP64 support, SIMD engine, hardware graphics, typically a smartphone / tablet grade platform, it is around the same level of capability as the eventual WASM version
3 - workstation / server - requires an OS with a local compiler toolchain, that would typically be x86 hardware running W or L, although I guess it can be hacked onto an arm system as well, as long as the distro has all the packages.

That’s hardware wise, OS wise, I can support Windows, Linux and Android initially. A MacOS version is feasible, although I don’t have the hardware, if there is enough demand I could try setting up a build instance. As for iOS, I have a gut feeling the app will simply not be allowed on the marketplace. I hear there’s “bootleg iOS” as well, but it is kinda of a dark subject for me, I’ve never owned any of their products.

Other platforms are possible too, Qt already supports quite a few of the more exotic ones, the code should work with all of them.

Installation process

Umium is designed to be self contained, it will not write anything outside of its working directory, and in a way, there is really no installation process. Just extract the archive in any folder of choice, the runtime will initialize the folder and create a unique installation upon first run.

IMPORTANT: Once activated, an installation uses its own unique data format, which is not compatible across installations, the actual transfer of data between installations can only be achieved through a properly authenticated channel. You are not supposed to transfer or in any way modify any of that generated data, even a single bit change will break integrity checks.

Once initialized in a folder, Umium can have multiple independent sessions / users within dedicated sub folders. It is also possible to pass any working folder path to any Umium instance via a command line argument, which has the same effect as running the executable in its current fulder.

And of course, it is possible to have the same or multiple versions running independently side by side on the same system, with very fast IPC.

The memory system

Umium is in a sense a memory management engine, it works with OS heap memory in “device mode”, but its pooling schemas can be used with direct raw memory access in lieu of a heap memory manager on embedded platforms.

The data engine supports full state dumps and loads, which allow to bulk serialize a full state which is way way faster than traversing object trees. It basically happens at the sequential write rate of your storage.

In addition to that, the data engine implements an ad-hoc notification system at the “lowest” level of access, which is then forwarded to platform specific handlers. This is designed to be ad-hoc because it ultimately will be removed and replaced by asynchronous support.

For the time being this is quite useful as it not only gives us full introspection of all allocated memory if necessary, but also a mechanism to automatically notify any data observer of relevant changes. So it can handle GUI and provide a degree of debugability.

Observers can also be used to bind certain types of functionality to certain data resources, in turn providing ad-hoc support for build in platform primitives. This is the “opaque” precompiled layer of static, immutable baseline functionality, which is provided by the environment the data core is being used by rather than being hard-coded in the core itself, making things simpler and more flexible.

Data operations are transactional in nature, using the same functor format regardless of the number of objects in any given location. There are bulk data operations, per item data operations forward and in reverse, and there is support for op sequences:

id newData = createData<Layout>();
insert(newData, 0, count) <<= { createLayout, initLayout } >> activateLayout;

The code example creates a new data resource, then inserts count items, calling createLayout and initLayout for every item forward, then activates every layout in reverse order. The functions only take in the item pointer, and can be anything that implements a callable interface - plain old functions, lambdas and whatnot.

One way in which data differs from the typical data container is it can recompile its content. Obviously, that would break any data that works with a native compiled object, but it is perfectly applicable for user types.

This can be used to splice multiple different data sources into one with multiple fields, or add fields to an existing object layout. On the native side, it uses the same format from above:

spliceIn(userData, 0, 4) << initNewFieldFoo; // splice 4 bytes in at the start of every item and init.

This tool is proving quite useful for QML development even without a fully fledged dynamic type system, as it does allow to use binary data and opaque types that are otherwise not supported from QML, and the storage overhead is negligible compared to using JS for data dynamism. Each data resources has only 16 bytes of internals, which is less than the size of an empty std::vector, which is typically a 3 pointer / 24 bytes implementation. And having the same API on both sides allows to rapidly prototype in QML then simple copy / paste the final solution on the C++ side.

One notable design limitation - the data engine is only supposed to store data. That’s “data” data, not “runtime data” such as say pointers, or if you will - value types only. The state of the data core is supposed to be persistent, and heap allocation is NOT - it produces arbitrary results every time. So it makes absolutely no sense to store anything as a pointer.

So for objects that do have opaque facades provided via a pointer, there is a “dynamic object storage registry” which allows to lookup the opaque pointer from an persistent ID instead. It is possible to add any amount of pre-existing types that do hold such “non data” data to be supported by the registry.

It is possible to define a meta object format reader for a given object group. In this case this is QObject - the base object type in Qt, which does offer a decent amount of introspection. So a fairly trivial translation layer can be established between the Umium and any 3rd party typing system, in turn enabling the facade objects to accurately receive state change notifications from the underlying data they represent.

I don’t want to over-commit to supporting any given 3rd party framework, especially Qt, since it is a huge dependency that I’d rather shed in the future, but then again - the meta object system does allow to create, rig and run QObject based types in any functional order, without having to compile any of it, just include each type in the factory. This does enable quite a lot of functionality early on, but it seems like I should exercise caution and restraint in order to not end up entrenched in dependencies.

It seems possible to synthesize QMetaObjects on the go as well, and even provide facilities for the the Qt meta system to be marshelled through Umium, so Umium objects can present themselves as perfectly compliant and operational QObjects, which is an especially tempting prospect, alas one that not simply introduces but focuses on and emphasizes a heavy dependency, so for the time being this remains just a consideration.

ablebit · June 9, 2023, 2:24pm

Licensing and operational model update

I am still on the fence on whether I should run this thing through a company or a non-profit. I am kinda leaning toward registering a Foundation, because the whole “let’s see how close to flat we can cut it” will not stack particularly well with a commercial entity. I can still conduct commercial activity with a Foundation as long as the proceeds go toward furthering its goals, which is totally cool on my end.

A foundation needs some goals and purposes, so I can go with something corny like “the proliferation of computing and technology” - the free tier of Umium does provide basic training and functionality, so that already should qualify as a free of charge “collective good” enough for a cause, plus providing even more training and a business platform to the “community” - the group of people that pays for the whole thing. Reducing operational costs and environmental impact on technology also sounds like it could be a ringer.

So for the users who’d prefer commercial transactions to community donations, it will be possible to sell subscription service and software licenses. Still most indie users might prefer the donation format, as it does generate higher contribution due to the lower tax on donations and more funds actually reaching operation. All income and spending on said operation will be available as a public log - 100% financial transparency - you can’t even see the damn thing

In order to make the thing useful to people who’d like to use the technology but not the service, I decided to chop up things internally, as the design consists of several fairly well isolated components, which are kinda useful on their own, without the whole service package.

So Umium is going to be available in two “modes” if you will, the full-fledged one will be the service, but it will also be possible to use the core components to embed into your own products that do not use the Umium ecosystem. Here are the 3 new discrete software components:

1 - Umium Data Mainframe - platform independent dynamic layout managed persistent state data access engine that can be embedded and easily ported to any language or framework

2 - Umium Object Model - a dynamic type and logical object management engine that uses UDM

3 - Umium Platform Client - a ready to use platform client reference implementation, instead of rolling out your own, provides a GUI front end and user I/O to design and run UOM based apps

For commercial usage, those 3 are all licensed “a pop”, the licensing fee is applied to every product that uses the licensed work being sold. Revenue, as mentioned above, all goes toward “furthering the cause”. Volume discount licensing - the more you buy the more you save

The remaining 3 sub-products are only available via Umium as service:

4 - Umium Productivity Framework - Umium’s “standard library” if you will, all the stuff it comes with out of the box - core, threading, i/o etc

5 - Umium Server Platform - use subscription cloud resources, run on premise infra, build tools etc

6 - Umium System Modeller - a framework to target embedded devices, device simulation, multi device systems, process control & simulation