I wanted a voice assistant that feels realtime but runs completely offline.
This prototype uses MLX + FastAPI on Apple Silicon to hit sub-second latency for speech-to-speech conversations.
Repo: https://github.com/shubhdotai/offline-voice-ai
It’s fast, minimal, and hackable — would love feedback on latency tricks, model swaps, or use-cases you’d like to see next.
Current avg latency: ~850 ms end-to-end (speech → LLM → speech).
Goal: keep it fast, under ~1K LOC, and clean so anyone can swap models or adapt it to their use case.
Feedback welcome on model choices, latency wins, and better UX for barge-in/turn-taking.