As an OpenAI employee who has worked with dozens of API customers, I mostly agree with the article's tip to break up tasks into smaller, more reliable subtasks.
If each step of your task requires knowledge of the big picture, then yeah it ought to help to put all your context into a single API call.
But if you can decompose your task into relatively independent subtasks, then it helps to use a custom prompt/custom model for each of those steps. Extraneous context and complexity are just opportunities for the model to make mistakes, and the more you can strip those out, the better. 3 steps with 99% reliability are better than 1 step with 90% reliability.
Of course, it all depends on what you're trying to do.
I'd say single, big API calls are better when:
- Much of the information/substeps are interrelated
- You want immediate output for a user-facing app, without having to wait for intermediate steps
Multiple, sequenced API calls are better when:
- You can decompose the task into smaller steps, each of which do not require full context
- There's a tree or graph of steps, and you want to prune irrelevant branches as you proceed from the root
- You want to have some 100% reliabile logic live outside of the LLM in parsing/routing code
- You want to customize the prompts based on results from previous steps
smaller tasks also helps in choosing smaller models to work with, instead of waiting for a large model to respond (really not usable when doing customer facing work)
If each step of your task requires knowledge of the big picture, then yeah it ought to help to put all your context into a single API call.
But if you can decompose your task into relatively independent subtasks, then it helps to use a custom prompt/custom model for each of those steps. Extraneous context and complexity are just opportunities for the model to make mistakes, and the more you can strip those out, the better. 3 steps with 99% reliability are better than 1 step with 90% reliability.
Of course, it all depends on what you're trying to do.
I'd say single, big API calls are better when:
- Much of the information/substeps are interrelated
- You want immediate output for a user-facing app, without having to wait for intermediate steps
Multiple, sequenced API calls are better when:
- You can decompose the task into smaller steps, each of which do not require full context
- There's a tree or graph of steps, and you want to prune irrelevant branches as you proceed from the root
- You want to have some 100% reliabile logic live outside of the LLM in parsing/routing code
- You want to customize the prompts based on results from previous steps