They had to put together ~14 hours of visuals, more than 5x what U2 had to do. There's nothing low effort about that. Some visuals were not as good as others as a result but I can promise you that the audience had their jaws on the floor all four nights.
I think the expectation that the entire consumer market (or even just a majority) is going to collectively become universally informed about all their purchases and shift the market for the better is far less likely then a government intervention being successful.
If you go to countries where there was never any government intervention relating to cigarettes do you know what you'll find? A lot more people smoking cigarettes.
I don't think it has to mean everyone becomes informed and makes educated decisions. We can get to the same end by people simply choosing not to buy products that they don't know much about how they were made.
In other words, the solution can be additive where we only bring in products we're confident in rather than having to learn everything and remove items from there.
As someone else mentioned in this thread, it is not possible to understand how everything you purchase works. That is also incredibly infeasible. What you are asking is to effectively revert back two hundred years of technological progress (a rough estimate for the last time people were actually self sufficient at a local level).
I can't count how many studies I've seen referenced that claim the environment is effectively doomed by 2030 if we don't change course. The same goes for other areas, whether its concerns raised over the risk of chemical exposure or the fragility of our toilet paper and baby formula infrastructure when a pandemic is declared.
If you think that many of the inventions over the last couple centuries are the culprit and would have to be rolled back, that sounds miserable but it also sounds like we at least would have a better chance being proactive rather then waiting for everything to come crumbling down.
I don't personally expect climate change or chemical exposure to destroy us all, life finds a way. But if we can't expect consumers to make purchasing decisions that generally align with what they think is important, why do we even bother with markets or capitalism at all?
Internships are definitely not a waste of time. First of all, they pay well in and of themselves (at least in tech). Second of all, most internships are filled by students on their summer break. What better use of that time than getting an inside view of a company they might want to work for? From the companies perspective it gives them a much better idea of how well the candidate performs and how they fit in at the company, giving much higher confidence in a hiring decision that could lead to significant future impact.
I think you misunderstood. Internships for students during the summer definitely makes sense. However, i was talking about internships/probation as a way to evaluate candidates instead of interviews. Meaning if you are a staff engineer with 10+ years of experience, you’re still going to be hired conditionally for 3-6 months to see if you’re good enough for the team. I personally find that very prone to abuse.
This is the reason why they're not going to move on device anytime soon. You can use compression techniques, sure, but you're not going to get anywhere near the level of performance of GPT-4 at a size that can fit on most consumer devices
I think we’ll see completely new architectures dominate in the near future, ousting the transformer. I am strongly suspicious that, while impressive, transformers use several orders of magnitude more compute than is “needed” for the tasks they perform—if for no other reason because the human brain performs similarly and it only draws 20 watts! And it isn’t even an engineered system, jus the product of a very, very long history of natural selection! I fully anticipate that we’ll see AI in the near future that achieves human-level performance on sub-human power budgets like the ones you’d be constrained by on a phone :)
"neat future" is very ambiguous. At the moment there is nothing even close to transformers in terms of performance. I suspect you are right in general but I'm not sure about the "near future" part, there needs to be a pretty significant paradigm shift for that to happen (which is possible, of course, I just don't see any hints of it yet).
RWKV is an attention-free architecture that's showing promising scaling at a similar level to Transformers right now! There's also recently been Hyena, which uses a new mechanism that's kind of a weird mix of attention, convolution, and implicit modelling all at once. It's shown promise as well. Remains to be seen if these competing methods will truly scale as well as Transformers, but I've got my fingers crossed. Only a matter of time!
I agree that "near future" is quite ambiguous though. If I were to disambiguate my claims, I think I'd personally expect a Transformer-killing architecture to arise in the next 4-5 years.
Scaling up an LM from 2017 would not achieve what GPT-4 does. It's nowhere near that simple. Of course companies saw the potential of natural language interfaces, there has been billions spent on it over the years and a lot of progress was made prior to ChatGPT coming along.
You're making incorrect assumptions. This project wasn't about scaling any published approaches. It was original neural net research that produced excellent results with a new architecture without self-attention, using a new optimizer, new regularization and augmentation ideas, sparsity, but with some NLP feature engineering, etc. Scaling it up to GPT-2 size matched its performance for English (my project was English-only and it was bidirectional unlike GPT so not a perfect comparison), and very likely scaling it up to GPT-3 size would have matched it as well, since GPT-3 wasn't much of an improvement over GPT-2 besides scale. Unclear for GPT-4 since there is very little known about it. Of course, in the meantime, most of these ideas are no longer SOTA and there has been a ton of progress in GPU hardware and frameworks like PyTorch/TF.
What you're saying about companies recognizing the commercial potential is clearly wrong. It's six years later and Siri, Alexa, and Google Home are still nearly as dumb as they were back then. Microsoft is only now working on adding a writing assistant to Word, and that's thanks to OpenAI. Why do you think Google had to have "code red" if they saw the potential? Low-budget startups are also very slow - they should've had their products out when the GPT-3 API was published, not now.
One thing I didn't expect is how well this same approach would work for code. I haven't even tried to do it.
Do you have any publications to back up your claims about your work? They seem more than a bit grandiose. If you're ideas are as novel and useful as you say then you should publish them.
And I'm sorry, but you're completely wrong about companies recognizing commercial potential. I worked on Alexa for five years, it is a far harder problem than you think. It is nowhere near as simple as "we just weren't looking at the right NN architecture or optimizer!" You're acting like it was a novel idea to think LMs would be extremely useful if the performance was better (in 2017). I'm just trying to tell you that isn't the case.
No, I have no plans to openly publish any of it. Some of my researcher employees have published their own stuff. I've previously written about how it was a huge commercial mistake for Google and others to openly publish their research, and they should stop. Indeed, now OpenAI has not published a meaningful GPT-4 paper, and DeepMind has also become more cautious. This mistake has cost them billions, and for what? Recruiting? Now they lost many people to OpenAI and wasted time and effort on publishing. Publishing is fine for university researchers or those looking to get hired. I did record some research ideas in an encrypted format on Twitter for posterity: https://twitter.com/LechMazur/status/1534481734598811650.
If any of the FAANG companies recognized the commercial potential and still accomplished so little, they must be entirely incompetent. When this 2017 deck was created, I had 50k LOC (fewer would be needed now using the frameworks and libraries) plus Word and Chrome plugins. The inference was still too slow and not quite feature-complete, and it was just a writing assistant with several other features in early testing, but it seems more than enough for me to know quite well how difficult is the task.
The fact that you think creating a writing assistant plugged into Word is equivalent to building a general purpose, always-on voice assistant tells me all I need to know.
What? We were talking about making a language model. I mentioned the plugins in relation to the question of commercializing. I'm very clear about what my project was and was not doing. I get that you're bitter because Alexa became a joke with how little progress was made and the struggles of Amazon in getting the top talent are well known. How much did it cost Amazon to fail like this? Is the whole team gone? Is that the billions you mentioned?
"It's six years later and Siri, Alexa, and Google Home are still nearly as dumb as they were back then". You can't even keep a coherent discussion and you are delusional about the significance of your work. You shared a slide with nothing but generic pie-in-the-sky use-cases and you act like it gives you some credibility on the subject ("let's make an AI system that can do the work of your non-professional employees!"). And to top it off you act like you've been successful here! Again, you shared nothing but a slide with generic use-cases that a 12 year old could think up. I don't know what you think you proved. Enjoy your imaginary pedestal.
gptX embeddings aren't even words. Even so, the embedding relationship is useful but not the core of what transformers do to find relationships between words in sequences.
remember the word2vec paper? the surprising bit the authors were trying to show was that putting words in some embedding space with an appropriate loss naturally lends enough structure to those words to be able to draw robust, human-interpretable analogies.
I agree with the sentiment that each individual dimension isn't meaningful, and I also feel like it's misleading for the article to frame it that way. But there's a grain of truth: the last step to predicting the output token is to take the dot product between some embedding and all the possible tokens' embeddings (we can interpret the last layer as just a table of token embeddings). Taking dot products in this space are equivalent to comparing the "distance" between the model's proposal and each possible output token. In that space, words like "apple" and "banana" are closer together than they are to "rotisserie chicken," so there is some coarse structure there.
Doing this, we gave the space meaning by the fact that cosine similarity is meaningful proxy for semantic similarity. Individual dimensions aren't meaningful, but distance in this space is.
A stronger article would attempt to replicate the word2vec analogy experiments (imo one of the more fascinating parts of that paper) with GPT's embeddings. I'd love to see if that property holds.
I wouldn't say the interpretability of word2vec embeddings is suprising - it's just a reflection of words being defined by context/usage, and these embeddings being created based on that assumption.
You can't just wave your hand and tell someone that words are broken up into sub-word tokens that are then transformed into a numerical representation to feed to a transformer and expect people to understand what is happening. How is anyone supposed to understand what a transformer does without understanding what the actual inputs are (e.g. word embeddings)? Plus, those embeddings directly related to the self attention scores calculated in the transformer. Understanding what an embedding is is extremely relevant.
Self sustained and totally independent mental capacity by an IT system... The ability to create and store memory and reasoning on it's own... This definition is not made by me, it's also a lot more vast... If you look up Spielberg's AI or I robot, Terminator, or any of those other films or books ln the matter, the definition is out there.
Use of the word "Intelligence" in Artificial Intelligence implies and indicates that humans are not involved in the equation past the point of initial creation and that it sustains itself and grows on it's own after a point... So far the various GPT models solely rely on human intervention and updates, which is bewildering to some like me why it's being marketed as Ai.
If you read some of the studies of these new LLMs you'll find pretty compelling evidence that they do have a world model. They still get things wrong but they can also correctly identify relationships and real world concepts with startling accuracy.
It fails at _some_ arithmetic. Humans also fail at arithmetic...
In any case, is that the defining characteristic of having a good enough "world model"? What distinguishes your ability understand the world vs. an LLM? From my perspective, you would prove it by explaining it to me, in much the same way an LLM could.
No it wasn't. You have the misconception that an AI system has to achieve anthropomorphic qualities to become dangerous. ML algorithms are goal-oriented, they are optimized to maximize a specific goal. If that goal is not aligned with what a group of people want then it can become a danger to them.
You seem to be dismissing the entire problem of AI alignment due to some people's belief/desire for LLMs to assume a human persona. Those people are uninformed and not the ones seriously thinking about this problem so you shouldn't use their position as a straw man.
> You have the misconception that an AI system has to achieve anthropomorphic qualities to become dangerous.
More than that. It has to achieve super-human abilities.[1] And it might also need to become self-improving, because how else could it spiral out of control, as the hysterics often complain about? (Remember: human intelligence is not self-improving.) It also needs to gain an embodiment that reaches further than server rooms. How does it do that? I guess it social-engineers the humans and then… Some extra steps? And then it gets more and more spooky.
This is more science fiction than technology.
[1] Or else it is no more dangerous than a human, and thus irrational to worry about (relative to what humans do to each other on a daily basis).
> ML algorithms are goal-oriented, they are optimized to maximize a specific goal.
The Paper Clip Machine. I’m more worried about the Profit Making Machine, Capitalism, since it is doing real harm today and not potentially next month/next year/next decade. (Oh yeah, those Paper Clip Machine philosophers kind of missed the last two hundred years of the real-life Paper Clip Machine… but what would the handmaidens of such a monstrosity have against that, to be fair.)
> The embarrassing part. Preaching a religion under professional pretenses.
There are quite a few companies with the explicitly stated goal of developing AGI. You can debate whether or not it's possible as that is an open question but it certainly seems relevant to ask "should we even try?", especially in light of recent developments.
That analogy is nonsense. We're actually making measurable gains in AI and the pace of progress is only increasing. Belief in this isn't based on faith, it is based on empirical evidence.
Elevator- and construction-technology has been improving for centuries. We will surely make a Tower of Babel that reaches the top of the Sky—it’s only a question of when.
Are you suggesting that artificial minds are unattainable in principle, or that the much-hyped language models are elevators for a task that requires rockets?
