It looks like OpenAI's first mover advantage won't last long, considering the speed at which these models are improving and compacting themselves. Seems like this new 'Moore's law' will be even more steep, at least in the beginning . So steep, that we can hope to be running these in our desktops instead of on someone else's computer.
We saw the same phenomenon with DALL-E. It was state-of-the-art for a blip in time before Midjourney and StableDiffusion surpassed it. And now with ControlNet + the ecosystem of domain-specific models, if you're doing serious generative art, OpenAI isn't even a part of the conversation.
If OpenAI makes their revenue off of charging exorbitant fees to use the LLMs, they'll no longer have incentive to ever open them (even if abuse / misuse concerns are addressed) AND they'll have no incentive to make them more efficient.
OpenAI has yet to show it can have a sustainable advantage. Every other player in the space benefits from an open model ecosystem and efficiency gains.
They do pay for the compute time in some way or another so surely they have the same incentive as everybody else to make it more efficient, at least, to increase their margin.
But yeah, they probably need to evolve from a (very good) two-trick pony into "the microsoft of AI" or something to stay afloat in the long run. That goes for the rest of the smaller AI companies as well though..
Midjourney and StableDiffusion have not surpassed Dall-E for many types of image. They are still well behind in certain critical factors. I often jump to Dall-E when I hit a brick wall with SD.
My money is on Deep Floyd (if and when it gets released)
I don't think Moore's law is the most important factor, instead algorithmic improvements will enable to have smaller models that are as capable as these humongeous models.
Llama 65b is hopefully just the beginning of this trend. It outperforms OPT-175b.
That holds true if scaling laws don’t hold true, otherwise a hypothetical Llama 175b would be even better. So the high end will always be on big clusters.
