This is deep in uncanny valley. Can it reconstruct images from our imagination as well? What about images from dreams? How far are we away from video?
If you know the series Max Headroom you probably know what I'm getting at.
There is an episode about dream-harvesting which Wikipedia summarizes like this[1]:
"In an attempt to get an edge over the major networks, a subscription cable channel turns to airing recorded dream sequences. When Carter begins researching a story on dream recording, he learns that the process can have fatal side effects for the donors."
This is just picking up the imprint of a direct perception; it is essentially equivalent to just photographing the reflection of the TV in your eye.
This isn't "decoding" any internal information actually present anywhere in the brain. It is showing, at best, that visual input "lands somewhere" in the brain -- but is not showing any of the brain's processing, nor any of the impact of the visual.
It's just, essentially, looking at the terminal node of the eye.
The model can't guess your thoughts unless it is trained with your brain activity (fMRI, EEG). After training, you feed it more brain imaging and see what comes out.
I don't know if this could be applied to, say a prisoner in Guantanamo, strapped to a fMRI and forced to look at images while the model trains on his brain activity. The subjects in these experiments are cooperative, while a uncooperative subject could try to blur his sight, meditate, think of loud songs, a movie, food, etc., to dissociate their vision from their thoughts.
This is impressive work but it is based on an existing library of trained imagery, so it is not able to actually pull out contents of the imagination that haven't been specifically trained.
One thing I don't understand is that the dataset already comes with a "train dataset" (120 recordings) and "test dataset" ("Recordings of 22 repetitions of 7 narrative test set videos"). Why are they not just setting aside a randomly selected subset of the training data for testing?
I'm unclear on whether the testing data contains some of the same video clips that were shown in the training data, repeated literally? It seems it would be best to test the model on a previously unseen image, to ensure that it really learnt the mapping and is not just "memorizing" the responses.
If I were a billionaire, I'd finance a project where this kind of technology is used on witnesses of historical events (WWII battles, marching with MLK, lost rock concerts, etc). Ask them to visualize the event, and record their visualizations. Obviously any individual witness would be unreliable, but data recorded from multiple witnesses could be valuable.
Time is running out. I don't know how many WWII veterans will still be alive by the time I'm a billionaire.
That's pretty impressive. This new work seems like a step forward because it's getting the information from slightly further downstream - the visual cortex which is what the LGN feeds into. But it's still not magic because it was already known that there's a mapping between points in the 2D field of view and points in the 2D layers of the visual cortex. That mapping is already known to be very simple for the V1 layer which is one of the 3 layers of the visual cortex that this technique uses. It also uses the V2 and V3 layers that seem to me to have a higher level representation and doesn't perform as well on them.
Still, this seems like a step towards reading people's imaginations, and they hint at that in the conclusion with "While admittedly the promise of algorithms that reconstruct internally generated or externally induced percepts is yet to be fully achieved".
Not that I know what I'm talking about. Feel free to point out any minunderstandings I've shown.
>While admittedly the promise of algorithms that reconstruct internally generated or externally induced percepts is yet to be fully achieved
I find this kind of writing kind of sad. It seems to me that the authors clearly wanted to write something about reading minds, and its quite logical. But they have to write in this obtuse, dry way because even in speculation they need to be precise. I am not sure if this is good or bad practice, but its pervasive in academic writing and I don't like it. A paper should not lose credibility because they went on a flight of fancy for a sentence or two.
A lot of things in academic writing are awful. Passive voice, for instance. Who didn't fully achieve it? These authors or all of humanity? And yea, the sentence is so safe that it really has no meaning because "not fully achieved" is consistent with both "not achieved at all" and "almost perfect".
The first reconstruction example in the preview PDF is going to haunt my nightmares for eternity. The original stimulus is a female face and the reconstruction looks like a horrible probe-wielding alien abductor.
That is not probe, but a prosthetic replacement for their exobrain. The Ood are generally good in Doctor Who, it's humans who are horrible abductors for them. If that image is from "Pond Life" mini series as suggested by the title on the first two frames on Figure 2, that's the butler of the eponymous family.
I suggest you to just watch the relevant episodes as an exposure therapy:
So this model takes as input an fMRI recording of the brain watching a movie live. Interesting work, but I am curious how much crossover there will be to remembering the film after watching it.
I'm more inclined to assume this will be exploited by police, imagine being convicted of a crime they claim you remembered while in pre trial detention.
Of course it could be as simple as swapping your video tape with one they created in order to incriminate you. As much as I really want this technology, it could do wonders for multitudes of disabled people, I'm more concerned about it being abused.
But it's an early version of mind-reading that requires expensive fMRI machines to get data from the brain and uses deep learning models that don't work that well... yet. There are other early examples of mind-reading, e.g., Neuralink implants.
From here on, it's all about improving the technology -- making it better, faster, cheaper, smaller, etc.
That doesn't answer the question of how far we are from possible abuses. I'd say we need to start talking now about the legal aspects of this tech, rather than rely on governments to figure it out when it's already being abused
Yes, I agree, though I'm less worried about use by criminals than about use by large technology-driven companies and bureaucracies that are already abusing their ability to "read minds indirectly" by collecting behavioral data. I shudder to think what those organizations could do if they start getting data directly from people's brains.
Hopefully I'll have a bullet pick my brain apart before a Facebook Neuralink Headset™ or a Palantir FREETHINK standard issue FBI neural interface kit get to do it.
I think there's a really interesting topic here. What does it mean to understand? Many phenomena that humans can work with in practice are beyond individual understanding.
For instance, I doubt any single human fully groks the design and manufacture of modern CPUs, for instance, there are mathematical proofs that can only be verified by machine, etc.
We frequently say we understand some physical phenomenon when we have math that explains it. But complex systems seem difficult to describe in less than a simulation and all of those numeric details can't fit in one human head.
I suspect that as we explore complex phenomena like brains, ecosystems, economies and weather systems, that we'll have to get used to a 'proxy' understanding where we can work with them only through the use of models that don't quite fit in human brains.
If you know the series Max Headroom you probably know what I'm getting at. There is an episode about dream-harvesting which Wikipedia summarizes like this[1]:
"In an attempt to get an edge over the major networks, a subscription cable channel turns to airing recorded dream sequences. When Carter begins researching a story on dream recording, he learns that the process can have fatal side effects for the donors."
[1] https://en.m.wikipedia.org/wiki/Max_Headroom_(TV_series)