it feels to me most of the real usage of AI is in coding right now, so a small lab that decided to go all in into just code-gen would have at least the differentiator of a narrower field to beat the bigger incumbents doing it all?
I dunno tho.
Big AI labs also have their own agendas and would rather keep scaling and growing than serving a rather smaller real market ?
Once you're into real usage territory, you can't no longer use make up numbers to justify future growth.
Again though, my point was just that it's not actually clear that you can do better than these big models by taking a narrower focus. I'm saying that that the things these big LLMs are learning about other languages probably do have utility when applied even to quite niche languages.
If you take some niche language and build an LLM from scratch that's hyperspecialized on that language, will that LLM actually outperform some big LLM that's trained on all the programming resources out there, and all the blogs, forum conversations, stack overflow posts on all those languages, and then learns to generalize that information and apply it to your niche language?
One of the things that LLMs seem to excel at is taking information from one context, transforming it and applying it to another context.
So how I envision this would be like a dual system, you let the frontier bigger LLM come up with the overall function signature, structure, and reasoning/planning around the specific code, but then have it ask the hyperspecialized fine-tuned model which can only output valid code, to create it.
You get then best of both worlds at the expense of a double-round trip or x2, which for something like coding seems fine, people are OK paying 200 for ChatGPT Plus
This also would solve the context window sizes problem of them getting full and the model starting to generate non-sense, if you have the bigger model using the bigger context window to orchestrate and organize the task calling smaller specialized sub-modules, that seems like it should yield better final code outptus than just one big ass LLM
but we'r'e moving the goalposts from 1 model to multi-agentic system i guess so nvermind
and i agree it seems all the big corps are betting for bigger more data for now
I dunno tho.
Big AI labs also have their own agendas and would rather keep scaling and growing than serving a rather smaller real market ?
Once you're into real usage territory, you can't no longer use make up numbers to justify future growth.