This is precisely why they should really open source their model so that anyone can download and run it on their own infrastructure. Just like google or others have done and one is free to run it on their own laptop (some even without a GPU) , on premise or on any cloud provider infrastructure. They can continue to provide a hosted service for their model but they should allow it to be downloaded just like BERT.
Don't get me wrong, I'd love to be able to run GPT3 and the subsequent finetuned versions thereof myself, but OpenAI has essentially no financial incentive to do so.
A few years ago, we could maybe lean on their open aspirations to get that done, but with the "limited profitibility model" they've since instead adopted, I think that dream is mostly gone.
At least we still get the occasional treat like Whisper out of them.
Why not? Please correct me if I'm wrong but from what I've heard, It's not millions or hundreds of thousands, it's "only" tens of thousands dollars' of hardware we're talking about. There are enthusiasts that spend comparably on their hobbies.
And a few dozen like minded people banding up together, shelling out a couple grands each? I'd say that's a totally realistic scenario.
Open AI's CEO Sam Altman's take is that they will only ever allow API access to their models to avoid misuse.
I don't get HN's take with wanting everything open sourced. Some things are expensive to create and dangerous in the wrong hands. Not everything can and should be open sourced.
Can you think of any non-weapons examples where centralization/gatekeeping of a tech meaningfully and causally benefited society or a technology itself?
Actually, thinking about my own question I'm even inclined to remove the non-weapons qualifier. The most knee jerk response, nuclear weapons, is perhaps the best example of unexpected benefit. The 'decentralization' of nuclear weapons is undoubtedly why the Cold War was the Cold War, and not World War 3. And similarly why we haven't* seen an open war between nations with nuclear weapons. One power to rule over all suddenly turned into "war with this country no longer has a win scenario" effectively ending open warfare between nuclear nations.
There's also the inevitability/optics argument. There are already viable open source alternatives [1], and should this tech ultimately prove viable/useful that will only be the beginning. So there certainly will be "ai" that will be open, it just won't come from OpenAI(tm)(c).
I agree with your view that nuclear weapons on both sides prevent war. However, they’ve only ever been developed by a small number of capable and motivated nations, with considerable resources involved. The later ones (North Korea, Pakistan) developed them while other nations tried to prevent them from doing so.
If ML models continue their exponential growth in size, a similar outcome is possible.
I really don’t like the argument that you should make things free just because it makes the world better. What happened to ownership and respecting the effort it takes to create something?
I see a similar line of reasoning can be used to justify theft from the rich.
I'm not entirely sure whether to praise or condemn them for it, but OpenAI has chosen to keep their initial introduction/company plan publicly available on their site: https://openai.com/blog/introducing-openai/
----
"OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return. Since our research is free from financial obligations, we can better focus on a positive human impact.
...
As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies."
----
My mocking about OpenAI(tm)(c) was not just juvenile "Micro$oft" type nonsense. At some point they discovered they could make a buck, and their ideology suddenly shifted 180. I have no qualms whatsoever about businesses pursuing profit, but the entity currently known as OpenAI couldn't be much further from the principles and values OpenAI was founded on, and their name itself is rapidly trending towards becoming a "Don't Be Evil" type of sardonicism. If this was Microsoft, Google, or other such companies operating in this way - I wouldn't have any expectation of anything besides what OpenAI is doing.
Companies tend to get quite a lot of credit when claiming some socially motivated interest, probably much more than deserved. So when they turn against those ideals, it should be noted - loudly.
> What happened to ownership and respecting the effort it takes to create something?
Ironic taking into consideration that the current generation of AI are more or less copyright laundering for the big corporations. Github Copilot being an extreme example of using GPL projects to generate "proprietary" closed source code. What happened to ownership and respecting the effort it takes to create something?
It sounds almost like a rehashing of Locke's labor theory of property wrt ownership of land that's very popular with classical liberals and libertarians. As that goes, land is initially nobody's, but when some person applies labor to improve or develop it somehow, that labor being "mixed in" makes the whole thing the property of the laborer.
Here, instead of common land, what we have is the common content. And they're saying that, by "developing" that content into a model that can do more useful things, the authors of the model are entitled to full private property rights on it.
I really hope that's not where we're going to end up, legally speaking.
A cluster of 6 year old 24GB NVIDIA Teslas should do the trick...they run for about $100 apiece. Put 12 or so of them together and you have the VRAM for a GPT3 clone.
Amazon has them listed at $200, but still, that's only $2,400 for 12 of them.
Still, adds up once you get the hardware you'd need to NVlink 12 of them, and then on top of that, the price of power/perf you get probably isn't great compared to modern compute.
Wonder what your volume would have to be before getting a box with 8 A100's from Lambdalabs would be the better tradeoff.
If you have time to wait for results then sure, it could work in theory but in practice they are so slow and power inefficient (compared to newer nodes) that no one uses them for LLMs, that's why they cost ~200$ used on ebay.
I just checked ebay and they are shockingly cheap. I can't even get DDR3 memory for the price they're selling 24GB of GDDR5... with a GPU thrown in for free.
Why is this? Did some large cloud vendor just upgrade?
Are there any deals like this on AMD hardware? Not having to deal with proprietary binary drivers is worth a lot of money and reduced performance to me. A lot.
These are pretty old, and all the companies are upgrading. But no one is upgrading from AMD hardware - basically no companies care if they use proprietary drivers. They want a good price-to-performance ratio, so they use NVIDIA stuff.
My take is that it's not good for democracy when the CEO of a private company is the one who decides what constitutes "misuse" and whose hands are "wrong" when it comes to access to a major technological breakthrough.
Yeah, just like the existence of Windows prevented Linux from ever existing.
The wrong hands have the money to seek alternatives. All this policy does is keep it out of the hands of the public, and ensure that whatever open alternatives start up won't be OpenAI's.
Looking at your calculations indicates that you mean RAM but it's 1.5 TB GPU VRAM (but this is assuming they use 64 bit precision, which is likely wrong so it's ~750 GB), not RAM.
I see OpenAI as a 'company' that (eventually) wants to earn money with their models. The 'open' part is trying to commercialize part it in such a way that everyone can use it. Also be open with the research and risks. But not open as in open-source.
