Stop Wasting Cloud Tokens - Local AI FTW

the “enterprise” implementation is probably overkill tbh but start here:

This is a proxy that can sit in between your API(s) and model. Which you can use to log what users are doing.

The model router can aggregate so you can go to like

http://ip/v1/models and get a list of models.. where some of those models are cloud (or all of them if you dont have the bandwidth to manage small/local models).

If you really get into it, the flywheel can query multiple models at once and a judge model can judge if the output is any good to help.. but I wouldn’t go for the “full automatic” model selection out of the box as others have rightly pointed out your users might not understand why sometimes the model is really stupid and sometimes not.

1 Like