Eh, so in reality there are a lot of AI products people are trying to build and it's very unclear at the outset "if it's possible", where "possible" is a business question that includes factors like:
- How hard is the task? Can it be completed with cheaper/faster models or does it require heavyweight SOTA tier models?
- What's your cost envelope for AI compute?
- How are you going to test/refine the exact prompt and examples you give the AI?
- How much scaffolding (aka, dev time = $$$) do you need to set up to integrate the AI with other systems?
- Is the result reliable enough to productize and show to users?
What you realize when designing these systems is there is a sliding scale where the more scaffolding and domain expertise you put into the system as a whole, the less you need to rely on the AI, but the more expensive it is in terms of man-hours it is to develop and maintain. It looks more and more just like a traditional system. And vice versa, perhaps with the most powerful SOTA models you can just dump 20K tokens of context and get an answer that is highly reliable and accurate with almost no extra work on your end (but costs more to run).
It's very individualized and task-dependent. But we do know from recent history, you can generally assume models are going to get faster/smarter/cheaper pretty quickly. So you try to figure out how close to the latter scenario you can get away with for now, knowing that in 6 months the equation could have completely changed in favor of "let the AI do most of the work".
As an addendum, I think it's completely crazy right now to be in the business of training your own models unless you have HIGHLY specialized needs or like to light money on fire. You are never going to achieve the performance/$ of the big AI labs, and they/their investors are doing all your R&D for FREE. It's like if Ford was releasing a new car every 6 months made out of ever more efficient and stronger carbon nanotubes or whatever, because the carbon nanotube companies were all competing for market share and wanted to win the "carbon nanotube race". It's crazy, never seen anything like it.
- How hard is the task? Can it be completed with cheaper/faster models or does it require heavyweight SOTA tier models?
- What's your cost envelope for AI compute?
- How are you going to test/refine the exact prompt and examples you give the AI?
- How much scaffolding (aka, dev time = $$$) do you need to set up to integrate the AI with other systems?
- Is the result reliable enough to productize and show to users?
What you realize when designing these systems is there is a sliding scale where the more scaffolding and domain expertise you put into the system as a whole, the less you need to rely on the AI, but the more expensive it is in terms of man-hours it is to develop and maintain. It looks more and more just like a traditional system. And vice versa, perhaps with the most powerful SOTA models you can just dump 20K tokens of context and get an answer that is highly reliable and accurate with almost no extra work on your end (but costs more to run).
It's very individualized and task-dependent. But we do know from recent history, you can generally assume models are going to get faster/smarter/cheaper pretty quickly. So you try to figure out how close to the latter scenario you can get away with for now, knowing that in 6 months the equation could have completely changed in favor of "let the AI do most of the work".
As an addendum, I think it's completely crazy right now to be in the business of training your own models unless you have HIGHLY specialized needs or like to light money on fire. You are never going to achieve the performance/$ of the big AI labs, and they/their investors are doing all your R&D for FREE. It's like if Ford was releasing a new car every 6 months made out of ever more efficient and stronger carbon nanotubes or whatever, because the carbon nanotube companies were all competing for market share and wanted to win the "carbon nanotube race". It's crazy, never seen anything like it.