this is great, can I link to this/reference for our video? I have been working on the same but hacking on llama.cpp and applying some out of tree patches from their issue tracker to get slightly less bad perf. out of the box it was like 1 token/s but that was a month+ ago when I started.
I saw your issue on GitHub too, following that to see what and says. therock GitHub is the way to go I think but I worried turning others loose with that may yield inconsistent results because it’s changing so much.
some of the work @ubergarm has done on our desktop systems also benefitted my apu setup here too. there’s some good stuff on the llama.cpp threads here for perf tweaks