Love this, I’m playing around with the cheapo esp32+servos version of this, super fun.
Something I’m working on is a hardware CLI for agents to run experiments, with a “CICD” pipeline that validates everything and means I can delegate more of the experiments to the agents. I wonder if you have any thoughts on this?
The idea is to allow the coding agent to run the full loop of experiments and validations, with vision, audio, button pressing, speaking etc to interact in place of the human
This seems indeed useful. I haven't seen this for robotics but I'm sure people need this for larger deployments (either for a distributed fleet or for an "arm farm"-like setup where there's many robots in one location for data collection / eval). Interesting idea!
The vision/audio/button loop is the fun part, and the part you'll get working first. What actually decides how much you can delegate is what sits between the agent's decision and the servo moving.
A bad command in software you just roll back. A bad command to a servo can snap the rig, and your CICD ran after the fact. So the gate that earns trust is pre-action, not post. The agent proposes a move, something checks it against limits before power hits the motor. Same idea as a dry-run, except it has to actually hold.
Other thing I'd build early: log every action the agent took and the sensor reading it saw at that step, in order. Physical experiments don't repro cleanly. When one goes sideways at 2am you want the exact sequence back, not a vague "it did something."
I work on this for software agents so take it with that bias, but the physical version is genuinely harder. You can't ctrl-z a servo.
Something I’m working on is a hardware CLI for agents to run experiments, with a “CICD” pipeline that validates everything and means I can delegate more of the experiments to the agents. I wonder if you have any thoughts on this?
The idea is to allow the coding agent to run the full loop of experiments and validations, with vision, audio, button pressing, speaking etc to interact in place of the human