The accessibility-tree approach makes a lot of sense for keeping token usage down.
How do you handle apps where the accessibility tree is incomplete though? Custom controls/canvas-heavy UIs can expose very little semantic information. Do you fall back to vision in those cases, and if so how do you keep the recorded flow deterministic enough to replay?
How do you handle apps where the accessibility tree is incomplete though? Custom controls/canvas-heavy UIs can expose very little semantic information. Do you fall back to vision in those cases, and if so how do you keep the recorded flow deterministic enough to replay?