
Deep image reconstruction from human brain activity [pdf] - grzm
https://www.biorxiv.org/content/biorxiv/early/2017/12/30/240317.full.pdf
======
jostmey
They trained a deep neural network to reconstruct what a patient was seeing
from fMRI signals and were mildly successful. Cool!

Things get really awesome in Figure 4. They had patients imagine an
image/symbol that that they would then try to reconstruct from fMRI signals.
They had a few successes and were able to operate at better than chance when
humans evaluated the reconstructed shapes. That's awesome!

~~~
pedrosorio
[http://news.berkeley.edu/2011/09/22/brain-
movies/](http://news.berkeley.edu/2011/09/22/brain-movies/)

~~~
caycep
Guess the kyoto group are competing w/ Jack Gallant's lab...

btw, pycortex from his lab is out on GitHub...neat to play around w/ if you
have the fMRI data

------
paulsutter
Cool results. Just want to point out that there's a time lag between
"thoughts" and the appearance of a signal through fMRI, which explains why the
image needed to be imagined for a long period of time (8 seconds).

Most thoughts are more fleeting than this, and couldn't be read with fMRI. So
there's no need to get /too/ concerned about the privacy of our thoughts. Not
yet :)

> Each imagery block consisted of a 4-s cue period, an 8-s imagery period, a
> 3-s evaluation period and a 1-s rest period.

~~~
jcims
Strap someone to a gurney and you have all the time in the world. Would an
fMRI require a search warrant?

~~~
anigbrowl
You can't make them hold the image in their head for you to pick it out,
though.

~~~
TeMPOraL
Don't think of the pink elephant.

~~~
v_lisivka
I can't imagine pictures in my head, so regardless of how long I will think
about pink elephant, you will see nothing.

~~~
TeMPOraL
For people like you, they can always fall back to the old "$5 wrench"
method...

~~~
v_lisivka
IMHO, thermorectal cryptoanalysis with soldering iron is much more effective
for similar price.

------
andy_ppp
I wonder what would happen if you could feed back the output into the visual
system using VR in near real time. Would you be able “see” anything you could
think of?

~~~
ttul
And then feed the signal from that back into... oh fuck it

------
partycoder
Very impressive. Please note this was done with magnetic resonance imaging.

A more invasive approach was this: "Reconstruction of Natural Scenes from
Ensemble Responses in the Lateral Geniculate Nucleus" (1999).

\-
[http://www.jneurosci.org/content/19/18/8036](http://www.jneurosci.org/content/19/18/8036)

\-
[http://www.jneurosci.org/content/jneuro/19/18/8036/F2.large....](http://www.jneurosci.org/content/jneuro/19/18/8036/F2.large.jpg)

What is a bit concerning are the applications, e.g: forcibly extracting
thoughts from non-collaborating people.

------
stanfordkid
Seems like a weird form of overfitting.

You give the brain a set of images to fit/train the original model. Then you
show the brain a set of images similar to the ones used to train the model.

... you will never really understand what the brain sees in novel situations
unrelated to the training task.

------
romdev
Here's a related article:
[http://www.sciencedirect.com/science/article/pii/S0960982211...](http://www.sciencedirect.com/science/article/pii/S0960982211009377)
...and a few other interesting ones from my link archive

Brain map [http://www.extremetech.com/extreme/227498-uc-berkeley-
team-b...](http://www.extremetech.com/extreme/227498-uc-berkeley-team-built-a-
semantic-atlas-of-the-human-brain)

3d-printing by thought [http://www.bbc.com/future/story/20130613-3d-printing-
your-th...](http://www.bbc.com/future/story/20130613-3d-printing-your-
thoughts)

Brain computer interface for controlling a Hex Bug and painting tools
[https://en.wikipedia.org/wiki/OpenBCI](https://en.wikipedia.org/wiki/OpenBCI)

Typing by thought [https://spectrum.ieee.org/the-human-
os/biomedical/bionics/fa...](https://spectrum.ieee.org/the-human-
os/biomedical/bionics/facebooks-director-of-typing-by-brain-project-discusses-
the-plan)

------
frlnBorg
What is up with "These authors contributed equally to this work."?

~~~
SubiculumCode
Typically most credit goes to first and last author. However, there are cases
where one person needs to be first author, but multiple people are equally
deserving.

~~~
Y_Y
This strongly depends on what field you are in and can vary even between minor
sub-disciplines. Also that statement is frequently meaningless and authors get
added for politics target than contribution.

------
pashariger
One step closer to Black Mirror Season 4: Crocodile, among other things.

------
m3kw9
I would definitely buy a play back machine in the future so I don’t have to
take so many pictures on trips.

------
cma
[https://www.youtube.com/watch?v=SjbSEjOJL3U](https://www.youtube.com/watch?v=SjbSEjOJL3U)

------
senatorobama
Now if only the information from EEG was half as useful from fMRI.

~~~
cinquemb
Theoretically, you could do the same thing with EEG via source localization
techniques[0] on the EM spectrum.

In my experience, barely any labs have the know how to implement them in a
performative manor to pull something like this off whereas a lot of the fmri
software is made from the manufacture (GE, Siemens, etc) that transforms the
raw data into the voxels that were used in this setup.

[0]
[http://www.scholarpedia.org/article/Source_localization](http://www.scholarpedia.org/article/Source_localization)

~~~
senatorobama
Sounds like you still need too many electrodes !

~~~
cinquemb
Depends on which brain regions you want to acquire signals from (usually based
on the literature), but generally yes, you need to sample from a "sufficient"
number of places (one of my fmr colleges has a lead reduction paper coming out
this year that will talk about this).

But keep in mind here, using eeg (and beamforming) is still way cheaper and
accessible than any fmri machine, though as far as having the skills to
implement such, I think it is out of the reach of most labs who do this sort
of research.

Electrodes are even getting better[0] wrt cost and actual ease of real world
usage (dry/non gel needed/comfort).

[0] [http://doc.ml.tu-
berlin.de/bbci/publications/GroVoiFaz11.pdf](http://doc.ml.tu-
berlin.de/bbci/publications/GroVoiFaz11.pdf)

