There's plenty of innovation. Through mass censorship, moderation, and the addiction to free, fueled by VC, it is much harder to get the word out or build a paying customer base on products without venture capital. Venture capital wants their cake and less risk. Founders are often told to only build b2b and forget consumers. Salaries that used to bootstrap innovative businesses find it harder to keep up with the budgets of Silicon Valley, leading to a capital monopoly for the venture backed. Human capital and money capital.
Not to single him out personally (it’s not his fault) but you could say that the golden age of VC (from the beginning of the internet boom) and the rise of Y Combinator funding small startups are the opposite of the kind of innovation that we saw at Xerox Parc (where a big company can dream big dreams about the future of the industry and fail to turn that into a durable competitive advantage.)
Y Co is more of a “moneyball” play than average, it’s not like under Sam’s tenure it was all about one big hit (not like the Softbank story).
Now there is OpenAI and that’s a super unusual case and luck is important in that one. The thing is they managed to get a lot of resources to train a very big model before anybody else did and woke up people to the possibility of LLMs but there just can’t be a GPT-5 which has 50x the parameters of GPT-4 and even if they was nobody could afford the cost to do inference with it.
They can’t afford to go forward the way they’ve gotten to where they have been, sure they can try to find a more efficient approach but everybody else is doing that and nobody else quite feels the same pressure to make a model that performs better and the same way GPT-4 performs so their first mover advantage could make them irrelevant unless they make the right moves and luck strikes a second time.
We have the research divisions of Google and Microsoft. The running joke about MSR is they do really great research (they do) but you would have no idea from the products.
What is missing is the Bob Taylor like visionary in the sense that he was sowing the seeds for something like golden age Parc. We have managed to create a system where the money has flowed to people with little idea of how to use it properly, which is not how capitalism is supposed to work.
I am also reminded of Alan Kay once making a remark about people confusing Linux knowledge for computer science fundamentals. We may be suffering from a Linux monoculture.
I wasn't in CS as an undergrad but I hung out at the computer center where they were teaching it and talked a lot to students in the courses and people who were teaching them.
Before Linux projects for the "operating systems" class were pretty limited, if you were lucky you wrote a bootloader that worked.
Once Linux came around it was night and day better as you could do just about any project that involves writing or modifying a kernel module.
So at one level Linux enabled a lot of learning and innovation in relatively small and marginal ways, but someone could say it killed off major innovations in operating systems.
On the other hand, I think the userspace is just as bad as the kernel in that respect if not worse. Suppose you believed that the ordinary kind of OS is "bloated" and wanted to try something new. Well, you need a userspace and unless you're going to write every single program you want to run on your OS fresh you're going to have to support a POSIX API and if you do that you've put all the bloat back in, even if it is a translation layer over some native API that will likely go unused because most of the software you'll want to run will be existing stuff.
If there is any real alternative to that model it is the "OO API" model seem with COM in Windows and whatever kind of serialization Android uses which is just putting the bloat train into high gear.
I'm not in the valley, so can't comment on that directly, however, I can comment that many VCs (US and abroad) have lost the "venture" part of VC, and that has hurt innovation.
For more than a decade, the focus was on building SaaS businesses that eked out minor efficiencies in the market.
The incentive was not to think big and try to build challenging things, the incentive was to do something simple, that would guarantee a return without much thought.
Don't hate the player, hate the game. (is that the saying?)
For the past 12 years, I've been in deep-tech, and though I am starting to see a change in attitude towards deep-tech companies, I feel like an inordinate amount of that focus is going to anyone who claims to be in AI.
“Focus and attention” is not deal closing. Lots of people will take phone calls and have dinners with people. Signing the contract and getting the money wired to your corporate bank account is much much harder. I think deep tech still has much better close rate as a percentage of meetings than “companies claiming to be in AI”.
Coming from the guy who went to Congress to advocate for using the force of government and licensing, to block the small guys from building large AI models that can compete with his products.
He understands that his products are about to be commoditized with no moat. The price of API calls to foundational models will be akin to the Airline industry. Essentially at cost with a small margin for profit and operating costs. All the value here is in building tailored applications for specific use cases.
Of course he understands. But it's hypocritical to be lamenting the lack of innovation when you are also actively working to block innovation and help create a monopoly for yourself.
He can certainly TRY to create a monopoly for himself. But Facebook, Google, Hugging Face, even NVIDIA won't let him. There are STRONG economic incentives to prevent any player getting a monopoly on a Foundational Model.
'"I hate to say this, because it sounds so arrogant, before OpenAI, what was the last really great scientific breakthrough that came out of a Silicon Valley company?" he said.'
Where does he think they got the Transformer paper from, or even the PyTorch and H100 GPU's they use?
This is the sort of trivialization attitude that I’ve come to associate with people who only care about the “big picture”. It is really irksome if you work on the lower-level stuff that (in some sense) makes the big picture possible.
I think there is a point here that user-facing innovation stagnated and OpenAI helped break that, but it’s wild to me that there is no acknowledgement at all of the giants whose shoulders they stand on. Although I guess that’s what he meant about the arrogance…
> "I hate to say this, because it sounds so arrogant, before OpenAI, what was the last really great scientific breakthrough that came out of a Silicon Valley company?" he said.'
Theranos!
> Where does he think they got the Transformer paper from, or even the PyTorch and H100 GPU's they use?
He's hallucinating. OpenAI is not a scientific breakthrough any more than Theranos was. It's just a commercial service that creates a pipeline out of other people's scientific breakthroughs.
The opacity of it should obviate any claim of it being scientific; for all we know, the backend is a bullpen of librarians chained to workstations. He's also glossing over the ethics of outsourcing moderation to desperate Africans, which would raise some flags in a truly-scientific context.
By each passing day, this pg protégé continues to prove he lucked out through privilege than through acumen. One has to be really myopic to think that GPT4 is God's gift on Earth (where God is, of course, me, sama).
PyTorch was not in any stretch of the imagination a breakthrough, nor is an H100. Those are clear incremental engineering improvements. Notable ones yes, but definitely not breakthroughs.
The transformer paper is the only thing that absolutely can be considered a breakthrough and what commercial success or impact did it have before OpenAI? None.
He is absolutely correct. Silicon Valley had been chasing crap for years now like crypto or various forms of rent seeking "businesses", and making incremental improvements.
On the other hand, put enough incremental improvements together and you've got a revolution. Seen that way, even OpenAI got where it is by standing on the shoulders of giants.
I’ll bite. Then what are the inventions worthy of the term “scientific breakthrough” that Silicon Valley had, but not anymore?
Does the freakin first mass produced EV in the world count? Engineering and manufacturing.
Crypto is also an example of a much more globally distributed phenomena. Vitalik in Singapore, tether founders in Italy?, Binance god knows where, FTX in Bahamas, etc. None of the major tech co’s in the Bay were doing much of anything with crypto. Oh except for Coinbase, you know the YC darling of whom Sam is affiliated?
>"Before OpenAI, what was the last really great scientific breakthrough that came out of a Silicon Valley company?" he said on a Wednesday podcast.
It's as if you're throwing money worth about half of the cost [0] of the entire Manhattan project at a company [1], and then make a big deal over getting results. Nevermind that the transformer wasn't even OpenAI's invention.
every time I see him mentioned now I just think of the first 5 seconds of the Lex Fridman interview YouTube kept pushing on me for a while where he says "Open AI has been a badly mocked and misunderstood org for a long time" in a whiny voice. Naw dude, I'm not misunderstanding that you decided to rug everyone on open source AI. In 2018 I would have never predicted that in 5 years Facebook would be on the top of my list for people driving the space forward in at least a semi ethical way yet here we are.
"I hate to say this, because it sounds so arrogant... " holy shit lol what an absolute tool. What has OpenAI done exactly, invent transformers and GPUs?
Well, OpenAI was the first to bring fire to the masses. Until then, it was a story described behind closed doors. Google wouldn't let you touch their stuff. Meta wouldn't let you touch their stuff. But OpenAI gave us an API. They were open-access first.
Now, of course, Meta and Stability have given us so much more. It's a good time!
I guess they don’t see it as their core business so why not win some goodwill and dilute the moats of their competitors who are more leaned into the current AI hype cycle
When they interviewed him for the New Yorker, I was posting on Facebook criticizing him about how his favorite person was Hyman Rickover, and lo and behold he was mutual friends with a college friend of mine, and added his two cents on the comment thread complaining about the characterization in the article. I reminded him that his best buddy Rickover who made plenty of enemies in his own right thought in the end that nuclear power didn’t belong outside the military for (among other reasons) precisely because of what happened at Shoreham to the backup turbine generators as documented by Greg Palast.
Shoreham was cancelled ostensibly due to evacuation fears, in reality the locals knew what sort of graft, corner-cutting and union crap was going on in the construction. Palast helped leak the test results of the backup generators (sorry if I said turbines). My point is that sort of shit is why Rickover thought civilian nuclear power was an oxymoron.
y'all realize this is PR so that open ai is in the news?
> "I hate to say this, because it sounds so arrogant, before OpenAI, what was the last really great scientific breakthrough that came out of a Silicon Valley company?"
he's literally calling open ai the only innovator. it's that controversial sort of PR that all the VCs/silicon valley types are copying from trump.
Maybe Sam should remind himself that the internet allowed innovation to happen anywhere, at any time, and living within 10 miles of some other smart person who's working on the same ideas is now merely a convenience, not absolutely critical to success.
It's understandable. We have a million new languages, platforms, services, libraries, and tools that we didn't have decades ago. Probably consumes a good bit of talent, all the manpower spent maintaining and modernizing the current ecosystem rather than cooking up novel concepts.
I was looking through the YC companies from the past year or so and struggled to find any that actually sounded like they were doing someone innovative, exciting, and useful.
I agree that foundational technological advancement has slowed.
Most “tech” innovation today, and market cap acceleration, is from innovative use of technology to simplify, scale, and leverage existing business categories.
Not innovative advancement of technology.
—
We now have a company that manages and leverages the biggest network of scrapbook creators the world has ever seen!!! :)
Their impact has been unprecedented in range and unexpected side effects.
But at the end of the day, only a small fraction of their “innovation” has has an a hint of foundational flavor. Some papers and opened up software of fine tuned tech, but nothing ground breaking.
And the product itself, whether you are a creator, advertiser, or audience, isn’t a technology.
It’s just a highly commercialized scrapbook.
—
Google is asymptotically approaching the same. Though a fraction of their impact is new technology in published papers or into products.
—
A small portion of Apple is always focusing on foundational work specific to new device forms and user interfaces.
Their full stack approach to computing devices isn’t as foundational but is unique (in vertical scale) and significant.
Their expansion of services and media, and treating apps like media, is just tech in subservience to regular business.
—
AWS and cloud computing done at Amazon was foundational work.
But now that, and most everything else is just tech optimized for business ends.
—
Microsoft is mostly optimizing tech, for others to use. They do basic research, but it’s not maturing into ground breaking products, seemingly any time soon.
Kudos to them although the impact of their quantum computing and other advanced projects is 99% PR. And 1% hope that Microsoft won’t fumble commercialization.
_
Netflix, optimizing tech for regular media distribution.
Uber? Do I need to say anything?
TikTok, Twitter, YouTube, and so many others, are glorified pinup boards with pinup ads.
Old Silicon Valley wants its valley back. And it’s silicon.
—
AI first companies seem to be the only ones carrying the foundational computing tech torch - that I am aware of.
NVidia is the foundational tech breakout on the big company, hardware side.
And OpenAI, etc. on the small company, algorithm side.
Tesla broke new ground, but progress has slowed, so going forward they are a definite maybe.
