Video on youtube: Inside a Language Model on a $19 Graphics Card | VectorFall - Deep Tech with Amir Husain ,
Compelling idea: take an old GPU, make it into an appliance that acts like a background employee, and pass its work off to a larger model or server for final reasoning and output, thoughts?
The A19 in the last gen iPhone is more or less similar to an AMD 890M, around 200% more performance and significantly lower power draw than the Quadro K4000 3gb card he barely talked about, he mostly just rambled without showing any real benefit to such a goofy setup, the model was way too small to consistently give useful text output, from his words.
Especially in this day and age a local on phone model, agent that can access to your own private local AI rig would make a ton more sense, or just on phone models, why do I need another 60-200w always on box with a junk GPU listening or waiting for text input when I can either run a light on phone model, or connect to a capable machine of mine? I can also just run a much better model on kinda affordable hardware that is pretty efficient. The core issue is models that get too small are not good at all, far better to just use normal purpose built software at that point to achieve a goal rather than shrink an “AI” model to run on an old small GPU.
Not to mention a Remote KVM is extremely easy to setup and very useful, as is Tail Scale or similar remote access to my powerful computer at home which I can access from anywhere I am as long as I have 5g or Wifi.