CLIs, run in a sandbox as tight as your preferred choosing, live in an ecosystem, where, via pipes and redirection, input and output can be easily manipulated. An agent can do similar things but more laboriously (and less token efficiently) via Python or similar but it would still live in a sandbox somewhere.
Going without the sandbox means hobbling the LLM. It can do things directly but is less able to construct ad-hoc programs to deal with looping, conditionality, tame verbosity, connect tools together, and so on.
It's a choice to not give the LLM an environment. As you say, it can be necessary if you're using dumb models. I don't find it particularly worth the trade most of the time.
If not even OpenAI can properly box in their models I definitely won't trust myself to do so with the very limited time available to me, and instead just use a standard that's already defined, and proven to work.
Going without the sandbox means hobbling the LLM. It can do things directly but is less able to construct ad-hoc programs to deal with looping, conditionality, tame verbosity, connect tools together, and so on.
It's a choice to not give the LLM an environment. As you say, it can be necessary if you're using dumb models. I don't find it particularly worth the trade most of the time.