
Prefrontal cortex as a meta-reinforcement learning system - godelmachine
https://deepmind.com/blog/prefrontal-cortex-meta-reinforcement-learning-system/
======
cs702
_Very_ interesting.

Oversimplifying and ignoring a lot of important details, the key idea proposed
by the authors is that the brain's phasic dopamine system is a model-free
reinforcement-learning system that learns to train the prefrontal cortex as a
more efficient model-based reinforcement-learning sytem -- a form of meta-
learning which the authors accurately refer to as _meta-reinforcement
learning_.

The "Results" section provides compelling evidence that the authors might be
on to something. The authors show and discuss the outcomes of six different
kinds of computer experiments in which a (relatively simple) meta-
reinforcement learning software system is shown to learn and behave in
qualitatively similar ways as, for example, monkeys and rats in equivalent lab
experiments.

I'm still digesting the implications.

Highly recommended reading.

~~~
epberry
One implication for products, if this work pans out, is that the systems which
"adapt to you" (learn your voice, your schedule, what your face looks like)
won't have to change the weights of their network. I believe this could lead
to much better systems because adjusting the weights of an already-deployed
network is dangerous - you could make the system perform poorly. It's also
simpler for engineers because you can scrap all that logic and just deploy one
RL network.

~~~
alex_duf
Sorry my head just exploded, but are you suggesting we have personal
"prefrontal cortexes" used to customise general neural networks?

------
dcre
Full preprint available here:

[https://www.biorxiv.org/content/early/2018/04/13/295964](https://www.biorxiv.org/content/early/2018/04/13/295964)

------
spitfire
Paper is behind a paywall. Sci-Hub doesn't have it. Does anyone have a link to
the paper PDF?

~~~
mindcrime
So here's something interesting to note. The "buy this paper" price for this
article is _almost_ reasonable in and of itself. $22.00. In fact, for somebody
who works in the field or even a tangentially connected field, they could
probably justify spending the $22.00 for it.

But... hilariously, a one year subscription to the journal, is only $59.00
(for individuals, online only).

[https://www.nature.com/neuro/subscribe](https://www.nature.com/neuro/subscribe)

Something seems a bit out of whack when the price of one article is about
1/3rd of the price of the journal for an entire yet.

I'm almost tempted to subscribe. The problem with this model though, is that
it doesn't scale. Even at "only" $59.00, how many journals can one afford to
subscribe to before the aggregate cost breaks the bank? _sigh_

~~~
keithwhor
It scales because nobody subscribes individually. If you can't access a
journal, you get your department head to forward things up the chain so your
institution can just add another line item to the millions of dollars they
spend annually on journal subscriptions for every researcher and student.

AFAIK almost all of that money goes directly to Elsevier [1] who, well, they
pretty much run a monopoly.

[1]
[https://en.wikipedia.org/wiki/Elsevier](https://en.wikipedia.org/wiki/Elsevier)

~~~
mindcrime
What I mean is, for those people (like me) who don't _have_ a "department
head" (or a university at all) to take care of journal subscriptions, and who
_might_ be willing to subscri be individually - it doesn't scale beyond a
journal to two.

At most, in years past, I subscribed to maybe a total of 10 journals, but
that's when I was maintaining an IEEE membership and a lot of their journals
are actually not very expensive, in terms of the marginal cost once you are
already a dues paying member.

These days though, I find it hard to justify. I find myself wondering if the
publishers would do a better job of providing a model that works for
individuals, if they might not actually make more money in the end.

~~~
davidjnelson
That is so darn insightful! So check it out, if there were a way to pay $20/mo
for access to ~5 publications/mo from a list of hundreds/thousands, that would
be fantastic! Netflix for research. I'd pay for it.

It wouldn't be terribly challenging to build. The tricks are in making the
deals with the content owners, and getting the word out to users.

The total market is 137 Billion/year ( [https://www.ibisworld.com/industry-
trends/market-research-re...](https://www.ibisworld.com/industry-
trends/market-research-reports/professional-scientific-technical-
services/professional-scientific-technical-services/scientific-research-
development.html) ) Not bad.

Looks like half the market is the government. Might be enough there to build a
business on. Hard to say. Requires more research.

~~~
keithwhor
The trick is that you're making a deal with Elsevier who is either going to
kill you or buy you, making this a non-starter for most VCs.

It's tremendously difficult to make money in a space this entrenched. If you
go a layer down and try to cut out Elsevier (go to researchers directly)
you're asking scientists to basically ignore career-defining opportunities in
high-impact publications... so that you can make money and a small market of
hobbyists or professionals without a research budget can subscribe to
journals. Also, the amount of work that needs to be done around peer-review is
obscene and Elsevier (and others) have built a ridiculous moat around related
volunteer work that is _hugely_ inefficient but constantly socially reinforced
(reviewing can be a status symbol). At that point, the alternative is open
access and free for everybody: why not publish there?

I'd be of the opinion that the only way to reasonably attack this space is to
build an adjacent content management and distribution platform _not focused on
journals_ and edge your way in, like, say, classroom management or MOOCs.

Which, well, there's Top Hat [1] who raised at a $185M valuation last year
[2]. Not sure if they care about journals all that much yet though, not a
whole lot of insight into their business beyond the pop business news.

[1] [https://tophat.com/](https://tophat.com/)

[2] [https://www.bloomberg.com/news/articles/2017-02-15/top-
hat-r...](https://www.bloomberg.com/news/articles/2017-02-15/top-hat-
raises-22-5-million-to-go-after-pearson-mcgraw-hill)

~~~
davidjnelson
Wow, you have a lot of knowledge in this area. Thanks! I wonder if because
Elsevier is likely a bigger company it could be a good business to create by
partnering with them, then let them buy it. So they don't have to take on the
risk of developing it in house, but can still reap the rewards.

~~~
TeMPOraL
With continued pressure from SciHub, I'm thinking this might not be a bad bet
to make. Elsevier et al. might just be willing to jump on the occasion, in
order to save some of their business.

That said, I still think it's time to double-down on supporting SciHub. Both
scientists and the interested public like it.

------
mr_overalls
A more accessible discussion by the paper's authors is here at DeepMind:

[https://deepmind.com/blog/prefrontal-cortex-meta-
reinforceme...](https://deepmind.com/blog/prefrontal-cortex-meta-
reinforcement-learning-system/)

~~~
dang
We've changed to that from
[https://www.nature.com/articles/s41593-018-0147-8](https://www.nature.com/articles/s41593-018-0147-8).
Thanks!

