On: Phi-3 Technical a Highly Capable Language Model Locally on Your Phone

[pogue]

Anyone made or working on a UI for this for Android (or iPhone)?

[sottol → pogue]

I've been trying this app but haven't had any luck getting it to actually generate text yet:

https://github.com/Mobile-Artificial-Intelligence/maid

The UI looks nice and includes a native compilation of llama.cpp.

My main phone's screen broke so I'm on an old Pixel 4 until it's repaired but I've had no luck getting 2-3GB models to run so far.

[pogue → sottol]

The APK doesn't include any of the models, so I'm not sure where you get them, where you're supposed to put them on the phone (same directory? Do they need to be compiled?)

When I ask it a question it doesn't respond & when you attempt to switch models it starts looking for nearby devices.

It also appears to be role play for some characters, so it's really not a GenAI chat bot, as far as I can tell. https://ibb.co/DRZhZcH

[sottol → pogue]

You can just get the GGUF off hugging-face. Usually googling for "<modelname> gguf" gets you to the site, then choose a quantization that works for you. But as I said, not working on Pixel 4. Should get a new screen tomorrow and can't wait to try on my Pixel 6 Pro.

Eg "phi-3 gguf" eventually take you here [1]. and you can download the Q4 quantized model on that page.

[1] https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf...

[pogue → sottol]

I see, ty. I'm not familiar with quantization. Can you give me a brief explainer for that or point me in the right direction?

[trao → pogue]

Yeah, it’s on Private LLM.

https://privatellm.app/blog/phi-3-now-available-on-iphone-an...