Great news! Open-sourcing the CAD files under Creative Commons is a fantastic move by Valve. This will enable the community to create custom modifications and accessibility adaptations for the Steam Controller.
> Is it really as fluid as shown in the video, or are there pauses while the text and audio/video are generated?
Yes the video is as smooth as shown in the video, and there are no pauses while video is generating.
There is alot of moving parts to get it be that fluid but for the video generation we built a NeRF-based framework which allows for realtime synthesis.
> Do they wrap something like gpt-3.5 and HeyGen or D-ID APIs? Did they build their own system?
We didn't use HeyGen or D-ID, we tried these out initially but when we plugged in all the moving parts they were too slow to allow for realtime AI conversations, which drove us to build out our own system. On the LLM side we use GPT-3.5/4 as the brain which is currently probably the most efficient (cost/output wise) at the moment but we also have Llama 2 integration in the works.
> Is there a paper or even a GitHub repo with these models out there somewhere to build off of by any chance?
We're building this out as an API, you can integrate into your own product, to build more complex flows.
It's still in private alpha and we are still polishing the edges but you can join the waitlist (https://diarupt.ai/) to try it out once we start rolling out.
reply