> Current top models are mere (mere in big datacenter terms) trillions of parameters right now. ... We'll soon start hitting 10's of trillions and 100's of trillions of parameter models.
Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.
We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).
FTA "The core philosophy there is scaling, and aggressively not research."
This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).
I'm not an AI booster (in a "should" not "can" sense), but in terms of "can" we are going to see next generation hardware (i.e. Vera Rubin GPU architecture) and improvements in model density (i.e. loosely improvements in intelligence per millions of weights), so the models will get bigger, the models will get more dense, and we may see more "expert models" i.e. in the sense that for a while there were human generalists that knew all of science and mathematics until the 1800s, after a certain point we had to specialize since there was too much to know. The models will need to partially specialize after hitting the hardware ceiling.
Bioinspired algorithms may eventually lead to vastly improved intelligence per a weight too.
I don't think we should do this, but I feel doomed in that I suspect that we "can".
> The models will need to partially specialize after hitting the hardware ceiling.
I'm incredibly skeptical of anything that goes "oh, we need to split AI into specialist systems".
I suspect that a lot of the delta between "our best models" and "base models they're trained on" lies in relatively low amounts of parameter changes driven by SFT and RL.
Model capacity is not the limit - the limit is our ability to drive those changes efficiently. Even things like "good research taste" could be represented as tiny in relation to total model capacity, if they could be trained into a model efficiently.
At least for now, those keys haven't been completely unlocked, so parameter scaling is the primary way corporations are achieving increases in ability. If they hit the hardware ceiling, maybe the research direction changes then.
I forget who said it, but a trenchant insight is that current neural networks are generalized function approximators and essentially they are doing through brute force what biology accomplishes elegantly. Our hardware (both in the sense of our brains in terms of elegance and our computer chips in terms of raw throughput) is amazing.
No, parameter scaling still works because it broadly increases sample efficiency. We don't have the raw data to make a Chinchilla optimal 100T, but that doesn't mean there are no gains in building up to 100T.
I don't think "biology" and "elegantly" belongs in the same sentence. Let alone the one you wrote.
I suspect that the "elegant" workings of the human brain are a pile of hacks. Evolution crammed the brain full of patterns that are a good fit for the specific problems it typically solves - the way curve detectors and scale invariance are a good fit for CNNs.
We don't have millions of years of evolution of non-random init, but what we have is enormous datasets and compute at scale.
Not really - today was a rare and large sell off - last time that was close was 2020 - for the pandemic.
The question one should ask is how far down does it go and if history is any indicator it's not gonna be up from here before the new year (and likely down a bunch more). There is a LOT of over valued stocks still to be cleared out - I suspect the cut is gonna be a lot deeper. The history on this is 2018 for Trumps last midterm and a pending rate hike. Sticky oil prices are gonna drive inflation and the fed raise's the market suffers.
> There is a LOT of over valued stocks still to be cleared out
Gigantic amount of debt is contingent on datacenter buildouts and roi on it. All of that suddenly got repriced. This is a catalyst for a multi-year grind-down of the ai-trade. See 2000-2002 for a parallel.
It's going to get hammered into oblivion long before it gets to where it can end up on sp500. The election, and what happens after may see them delaying the IPO (if I was one to bet)
But it also sort of doesn't matter. Anthropic, not in sp500, ends up a fail then the "bubble" pops and the sp500 gets ugly.
> As I grow in my career, on a day to day basis I feel like I'm failing or doing things badly.
This is one thing that we should lament about AI generating all the code. You never get to look back at your own "hand crafted" work with the distance of time. Progress or perspective, it does not matter, one would probably find something they could have done better.
No one designs for users any more. It's all gas no breaks on the path to enshitification. A lot of this is the motivation of the people building software.
Raise your hand if you ever saw a PM get promoted for removing a bunch of bad and/or unused features?
Most giants in highly technological areas don’t evolve, almost by definition. They get to a point where they’re mature companies just operating an existing business, and that’s diametrically opposed to developing new businesses.
Sure, they can spend money on R&D, but then they’re competing with every other new venture in the world - the only benefit they have is money, but collectively, everyone else has more money than they do.
This is why when private equity takes over a mature business, they almost always shut down any R&D the company is doing. Silly mature business, that’s not the reason for your existence!
This appears to have been true for the last few decades, but not before.
From the first World War up to around 1980, a very large number of the most important technological innovations have been made in the research laboratories of a few giant companies, like ATT, IBM, General Electric, Westinghouse Electric.
For some reason, since around 1980 almost all big companies have focused much more on the financial results of the next quarter than on long-term research, so much less innovations were done in them (with a few exceptions, IBM has still produced a few very important innovations until around 2000, but those innovations were less important for IBM products, their impact was due to being licensed to others).
> since around 1980 almost all big companies have focused much more on the financial results of the next quarter than on long-term research
Good point. What I described is not some law of nature, but a function of the current long-term fad of American ultracapitalism.
Happily, the signs are that this is all falling apart in an irreversible way. Whatever rises from the ashes might resemble a civilization again, if we're lucky.
Mature businesses like e.g. tire or appliances manufacturers still have some R&D trying to make their products better. However, it may be more for appearances than for actual improvements, just because consumers like to buy a "new fancy model" over a tire or a dishwasher that have been in the market for 20 years (believing the new thing must be better).
>IBM, GE... how many former American giants failed to evolve.
"How many failed to evolve", on its own, is a bad metric for success, because it misses the how many new giants the US economy managed to create, which IMHO is the more important metric of a healthy economy.
Companies are like people, the old ones die out and are replaced by new young ones, as it should be. Worst you can do is be like Europe where all you do is try to keep ancient dying companies on life support for the sake of "stability", instead of creating new ones to replace them, which is also a fitting metaphor for EU demographics where a third of the state's budget goes to keeping boomers pensions higher than the salary of average worker(in France).
My first program was in 2nd grade on an Apple IIe in basic. It was a program to check if a word was worth a dollar (a=1, b=2, c=3...)
It became a family activity (every one taking turns at the computer) every night to just type in words and see what they came out to. It was all very boring.
I was sure I was going to win.
I lost (well 2nd place), to the kid with 6 older brothers, who had that many years of building out the family list.
> For any application at scale it'd be way more accurate to have a dedicated classifier
"at scale" or the above use case of "specialized domain"
Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.
System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.
Caring about something also means you don't want the issue to be misrepresented.
This is the most effective altruist anti AI headline you could get out of this story.
250k trees seems like a lot from you looking out your window. From the perspective of trees we cut down every year (that we account for) it is close to a rounding error. This is a place where they take forest management seriously - and thats the only reason any one even noticed or had to get a permit.
>250k trees seems like a lot from you looking out your window. From the perspective of trees we cut down every year (that we account for) it is close to a rounding error.
Because there's a bigger bad the smaller bad is totally fine and nobody should care?
And it seems it was 400k in total :
"In addition to the 250,000 trees felled at Muhos, Annala estimates that an additional 150,000 trees were felled at Otanmäki"
I don't care how sick our society is to normalize cutting 400k trees for data centers. It's wrong.
Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.
We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).
FTA "The core philosophy there is scaling, and aggressively not research."
This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).
reply