Thank you! Yes the app has bundled built-in llama.cpp binary with it. The app just launches the executable - nothing fancy. We are a little llama.cpp wrapper :)
Very cool you covered all bases. I'm going through this exact thing in an app, just moved it over from relying on ollama to using node-llama-cpp internally, but maybe I'll copy you and still support ollama too :D
I use the MLCChat iOS app to run Mixtral on my iPhone pretty much exclusively when on flights and am curious about something. Maybe it’s a question about my destination, a fact I’ve forgotten, or if I’m working and haven’t bought WiFi it’s good for quick documentation sythesis.
It’s genuinely useful and fun in a way ChatGPT isn’t- something about the novelty. However outside of that environment I haven’t found a good use.
Also fun to feel the phone heat up as it generates tokens.
The thought of building an interview assistant for coding interviews and selling it for few $k A pop has crossed my mind on more than one occasion. Run audio capture and OCR, along with some finetuning on prompt engineering of how to solve the problem in steps with explanation of each one.
You can start by defining the scope and the tools you want to use.
Both coding or nocode tools can help. The simplest option would be to take one of those "privacy-friendly" options out there for a spin on a free tier and as you work you'll see what limitations you encounter or features you'd want.
And then you can customize it further using coding tools.
I use the 420MB mpnet model. They finally have a T5 that gains a point of performance but is almost 10GB. The SBERT folks already think most people would be happier with a model smaller than mpnet but faster.
I have tried fine-tuning other BERTs and not had one that was really worth using. One of these days I want to train a T5 to do something kinda generative like putting Mastodon tags on articles.
- Formatting/changing text
- Troubleshooting code, esp. new frameworks/libs
- Recipes
- Data entry
- Organizing thoughts: High-level lists, comparison, classification, synonyms, jargon & nomenclature
- Learning esp. by analogy and example
RAG for:
- Website assistants (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)
- Game NPCs (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)
- Discord/Slack/forum bots (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)
- Character-driven storytelling and video game concept art (https://github.com/bennyschmidt/ragdoll-studio/tree/master/r...)