

The man who hears colour - reacocard
http://www.bbc.co.uk/news/magazine-16681630

======
noonespecial
_Last year I was attacked by three policemen at a demonstration who thought I
was filming them. I told them I was listening to colours, but they thought I
was mocking them and tried to pull the camera off my head._

For some reason, this strikes me as particularly awful. Not that police don't
want to be filmed, that's predictably repugnant for its own clear reasons. Its
that they had no problem ripping a prosthesis off someone before even
bothering to try to understand it because it looked different. What's next,
wrestling old folk to the ground and ripping out their hearing aids because
they might be recording devices? Will there be an unwritten "normalcy code"
that disabled people will have to follow to avoid assault?

~~~
owenmarshall
>Will there be an unwritten "normalcy code" that disabled people will have to
follow to avoid assault?

You're lying to yourself if you honestly believe that doesn't exist today.

------
evincarofautumn
I expected yet another “synesthesia—isn’t it interesting!” article, and was
pleasantly surprised. As a synesthete (grapheme→colour and sound→colour), I
find that the topic has been done to death, and non-synesthetic writers tend
to romanticise it to the point of outright misrepresentation. Anyway, the
brain’s peculiar propensity for conflating senses seems to have proved useful
for once. Props to this guy for hacking his brain to get around a stroke of
bad genetic luck.

~~~
corin_
I agree that the topic gets written about often, and usually pretty terribly -
but because of that I was hoping it was another "synesthesia - isn't it
interesting!" article, just a rare good one.

I don't have it myself, but did study music with someone who had it, and since
then it's always fascinated me.

------
got2surf
Very cool!

Somewhat related: my main research area is actually in sonification
(representing data through non-speech sound) - imagine listening to changes in
the stock market through changes in pitch, or loudness, or tempo. We can use
sonification for the visually impaired, communicating data and patterns in new
places, as this guy has done. But we can also use it to revolutionize how we
interact with computers - we can be mobile, multitasking, visually overloaded,
and still process data through sonification. IMO a potentially revolutionary
technology!

------
fab1an
Some context: there is an entire neuroscientific field of study devoted to
substituting one sensory modality with another:
<http://en.wikipedia.org/wiki/Sensory_substitution>

The field was pioneered by Paul Bach-y-Rita
(<http://en.wikipedia.org/wiki/Paul_Bach-y-Rita>) who most notably invented a
setup that allowed blind people to "see" via a camera connected to a vibrating
grid attached to their their backs, effectively substituting visual with
haptic input.

In a nutshell, there is nothing intrinsically "visual" about neurons in the
visual cortex, nor are neurons in the, e.g., auditory cortex exclusively tuned
towards sound - the brain is plastic enough to "make sense" of a new type of
input signal, which typically takes a couple of weeks.

My co-founder Peter König at EyeQuant.com - a neuroscience professor at the
University of Osnabrueck - is working on similar projects with his feelspace
group, where they created a compass-belt that vibrates whereever north is,
taking sensory substitution a step further by effectively creating a _new_
sensory modality of direction (Wired article:
<http://www.wired.com/wired/archive/15.04/esp.html>)

As an excellent philosophical take on this I would recommend Alva Noe's
"Action in Perception": <http://www.amazon.com/dp/0262140888/>

~~~
Lockyy
I'm curious if for blind people the back stimulation system for seeing things
would start to produce visuals in a way similar to normal sight. Similar to
the possiblity that ultrasound used by bats might allow them to "see."

Also, I _really_ want to get one of those compass-belts. It seems like an
incredible experiencee. I wonder how it feels to not have it on though. Losing
a sense mustn't be the nicest experience.

------
KennyCason
The part that I particularly thought was interesting is when he mentioned
expanding it to Infrared and Ultraviolet :)

~~~
troymc
Yeah, the idea of "seeing" things normally invisible reminded me of Geordi La
Forge's visor in Star Trek TNG.

------
pharrington
This is some really cool tech. But while light frequencies can obviously be
translated into other media we can perceive, "color" as such always comes
attached with a ton of spacial information, so I wonder how well the eyeborg
conveys that? It seems like this would feel like being _extremely_
nearsighted, which is suggested when he mentions getting close to peoples'
faces when doing portraits. I also wonder how much this is a constraint of the
technology, and how much are the limits of our sense perception? For example,
if the device were able to encode arbitrarily specific spacial information,
could one train themselves to be able to instantly distinguish among 100s of
unique, simultaneous sounds (like we do with sight), or would the experience
always be din?

------
unimpressive
I have fully working eyes, but the concept of self-induced synesthesia is
interesting. Especially for the purposes of getting extra-human senses. (Even
if they're not that useful in practice.) Of course, if your eyes can already
see color; glasses with a screen filter might be more efficient.

~~~
camtarn
This one's come up on HN a couple of times before - sensing EM fields via a
fingertip-embedded magnet:
[http://www.wired.com/gadgets/mods/news/2006/06/71087?current...](http://www.wired.com/gadgets/mods/news/2006/06/71087?currentPage=all)

More sensitive EM field detection could be vaguely useful, but to be useful it
would probably require a bit of pre-processing (e.g. scaling the entire
frequency range of the currently used EM spectrum into a range we can hear,
see or feel) and maybe some protocol-specific hardware (decoding radio, video,
wireless etc).

Other interesting candidates for extra-human senses beyond just increasing the
range and sensitivity of existing senses would be EM/light polarization
(insects can see polarization in the sky, so they can navigate by the sun's
direction even when it's hidden by clouds) and magnetic field direction (which
exists in bacteria, invertebrates, and birds, and may exist in some mammals.)

------
bobby07
This does look really cool, however, surly his resolution is only 1 pixel?

He is limited to hearing one note at a time, therefore he can only perceive
one color at a time?

Am I missing the point?

~~~
Retric
Your ears decode multiple frequency's at the same time so nothing limits
things to a single color at a time.

But, this is for someone who literally could not perceive color at all but can
still see. If he waves it around he can probably quickly tell that the wall is
blue or white and then focus on extracting details from things he finds
interesting.

~~~
bobby07
Ok, really good points.

Taking onboard what you said, a system could work to match sound volume with
direction to see in a higher resolution.

I.e. It would superimpose all color in range together with those closer to the
center having a higher amplitude. So the colors directly ahead would be player
louder and peripheral colors played quietly.

Or perhaps this is how it works already....

Roughly how many different notes could you hear at a time?

------
Martijn
Is the color->note map arbitrary or is there some logic behind it? I imagine
it greatly affects the associations (including emotions) he has built up over
the years.

~~~
setrofim_
>Is the color->note map arbitrary or is there some logic behind it?

One way of doing this would be to map from the electromagnetic wavelength of
the color to a corresponding audible frequency.

Audible range is something like 32 to 32768 Hrz. Assuming speed of sound in
air of 343 m/s, this translates to wavelength range of 0.010437 to 10.6875 m .
This can then be mapped onto the visible spectrum of 390 to 750 nm.

light_range = (750 * 10^-9) - (390 * 10^-9)

sound_range = 10.6875 - 0.010437

light_wavelength = (light_range * ((343 / sound_freq) - 0.010437) /
sound_range) + (390 * 10^-9)

------
michaelcampbell
I would love to see a sample of his art pre- and post- prosthesis to see the
evolution.

------
sukuriant
So ... can I make one/buy one?

~~~
wccrawford
Sure, grab an Arduino, a color sensor, and a speaker.

Here's something to get you started.

<http://itp.nyu.edu/physcomp/sensors/Reports/DIYColorSensor>

------
dmoy
Wow that's really cool tech.

Not gonna like though, I thought this was going to be about
<http://en.wikipedia.org/wiki/Aphex_Twin>

~~~
BasDirks
If you want to get un-downvoted I think it would help your case if you provide
us with more info than a wikipedia-page about dear Richard.

~~~
corin_
Aphex Twin does have synesthesia, so there is at least logic to thinking that
a BBC article on the subject _could_ be about him, though maybe not that it
_would_ be.

