"ah, you hesitated" no more so than on every single other question.
the delay for the GPT to process a response is very unnerving. I find it worse than when the news is interviewing a remote site with a delay between responses. maybe if the eyes had LEDs to indicate activity rather than it just sitting there??? waiting for a GPT to do its thing is always going to force a delay especially when pushing the request to the cloud for a response.
also, "GPT-4o continuously listens to speech through the audio stream," is going to be problematic
I wonder how well suited some of the smaller LLMs like Qwen 0.6B would be suited to this... it doesn't sound like a super complicated task.
I also feel like you can train a model on this task by using the zero-shot performance of larger models to create a dataset, making something very zippy.
I wondered similar. Perhaps a local model cached in a 16GB or 24GB graphics card would perform well too. It would have to be a quantized/distilled model, but maybe sufficient, especially with some additional training as you mentioned.
> also, "GPT-4o continuously listens to speech through the audio stream," is going to be problematic
This seems like a good place to leverage a wake word library, perhaps openWakeWord or porcupine. Then the user could wake the device before sending the prompt off to an endpoint.
It could even have a resting or snoozing animation, then have it perk up when the wake word triggers. Eerie to view, I'm sure...
> the delay for the GPT to process a response is very unnerving
I'm not sure I agree. The way the tentacle stops moving and shoots upright when you start talking to it gives me the intuitive impression that it's paying attention and thinking. Pretty cute!
it's the "thinking" frozen state while it uploads and waits for a GPT response that is unnerving. if the eyes did something to indicate progress is being made, then it would remove the desire to ask it if it is working or something. the last thing I want to be is that PM asking for a status update, but some indication it was actually processing the request would be ideal. even if there was a new animation with the tail like having it spinning or twirling like the ubiquitous spinner to show that something is happening
the snap to attention is a good example of it showing you feedback. the frozen state makes me wonder if it is doing anything or not
Back when Anki (the robotics company) was building Cosmo, a *lot* of thought was put into making it expressive about everything that was going on. It really did a good job of making it feel "alive" for lack of a better word.
Kyutai's unmute has great latency, but requires a fast small-ish, non-thinking, non-tooled LLM. What I'm currently working on is merging both worlds. Take the small LLM for instant response, which will basically just be able to repeat what you said, to show it understood. And have a big LLM do stuff in the background, and feeding back infos to the small LLM to explain intermediary steps.
This is the key aspect for future development of models - small instant reasoning, ideally on device that funnels through tho a larger model for reasoning.
beyond the prototyping phase, which hosted models make very easy, there's little reason this couldn't use a very small optimized model on device... it would be significantly faster/safer in an end product (but significantly less flexible for prototyping)
the delay for the GPT to process a response is very unnerving. I find it worse than when the news is interviewing a remote site with a delay between responses. maybe if the eyes had LEDs to indicate activity rather than it just sitting there??? waiting for a GPT to do its thing is always going to force a delay especially when pushing the request to the cloud for a response.
also, "GPT-4o continuously listens to speech through the audio stream," is going to be problematic