
Building AGI Using Language Models - leogao
https://leogao.dev/2020/08/17/Building-AGI-Using-Language-Models/
======
hprotagonist
I remain to be convinced that encoding statistical information about
syntactical manipulation alone will somehow magically convert to semantic
knowledge and agency if you just try really hard and do it a lot.

~~~
mordymoop
How much have you played with GPT3?

~~~
hprotagonist
enough to know it's ELIZA on steroids. Neat party trick, and there's no there
there.

~~~
dwohnitmok
I think that's overly dismissive of it. GPT-3 for me represents the very
beginnings of breaking out of the paradigm of a specialized machine learning
system for each task. GPT-3 is alternatively a (surprisingly good and
consistent) machine translating system, a (disappointingly mediocre)
calculator, a (very uneven with flashes of brilliance) chatbot, etc. All
without special engineering to do any of those things. I can't think of any
other system I have access to that does that.

Regardless of whether you think GPT-3 represents a track towards true AGI,
this is a huge advance! Even OpenAI's API for it is astounding compared to
other APIs. I can't think of any other API that amounts to "provide an English
language description of your task" and returns an answer. Like I said, the
results are still quite uneven, but the fact that it's not extraordinarily
outlandish to provide such an API is absolutely mind-boggling to me.

I don't think I could have ever imagined such an API existing even just 5
years ago!

This is way way beyond even the same qualitative thing as ELIZA.

------
ilaksh
To me GPT-3 excitement is equivalent to when people get hyped about "defeating
aging" after seeing some resveratrol trial or something.

Language is only part of it. And you can't get complete understanding without
integrating spatial information. Take a look at Josh Tenenbaum's work for
explanation of why.

------
msamogh1
It talks about how you can go from a language model that simply generates text
to an agent that is capable of performing actions in the real world

Essentially, the missing pieces in the picture come down to input and output
modules. "How do you formulate any given problem into a form that a language
model can answer?".

~~~
leogao
I don't doubt the input and output modules will need some work, but in the
grand scheme of things probably not that much. The big missing piece imo is
just models with better world models—and it doesn't look like the scaling wars
will stop anytime soon, nor does it look like size will stop helping, at least
not anytime soon. ([https://www.gwern.net/newsletter/2020/05#baking-the-
cake](https://www.gwern.net/newsletter/2020/05#baking-the-cake))

------
bionhoward
If the agent state is symbolic, that’s cool and interesting, but isn’t reality
sub-symbolic?

------
mrfusion
I don’t completely follow this. Can anyone explain?

~~~
rahimnathwani
tl;dr Language models like GPT-3 include incorporate some model of the world.
That's why they can generate plausible-sounding text. Future language models
will be larger, more powerful, and have more complete models of the world. So
we will be able to ask the language model questions, like 'what will happen if
we do X?'. By compiling the answers to many such questions, we can figure out
the best[0] thing to do.

[0] Assuming you have some utility function you can maximize.

~~~
leogao
This is a great summary!

