
The Humane Representation of Thought [video] - jashkenas
http://vimeo.com/115154289
======
rayalez
The interesting part is that a lot of what he is talking about is already
possible.

\- You can use augmented reality or VR with something like Oculus to display
things in 3D space.

\- You can use Motion Capture technologies, plus maybe some convenient remote-
control/keyboard-like devices to put a human in that virtual space and
manipulate things.

Only 2 technologies(Oculus + Mocap) would be enough to accomplish something
like this: [http://youtu.be/VzFpg271sm8](http://youtu.be/VzFpg271sm8)

Give a bunch of people oculus rifts and put them in the same virtual reality -
and we have the dynamic stage he is talking about.

They don't even have to be in the same room of course, we have 3D realtime
MMORPGs and second-life that already solved that problem!! You could attend a
virtual talk by sitting in your apartment with oculus rift on your head.

And of course "downloading a room" instead of a book is already very doable.
We already have computer games and engines. That room is like a video game
level. Downloading it is not a problem, to explore it all you need is oculus
and a simple game controller or a keyboard/mouse.

And in terms of creating such content, the closest thing I can think of is 3D
editors, like Maya or Houdini, especially
houdini([http://www.sidefx.com](http://www.sidefx.com)), which is the
brilliant combination of computer graphics and programming, that allows you to
model and program dynamic things visually.

------
javajosh
I've also been thinking about the concrete representation of symbols and the
negative "inhumane" impact of screens, but my conclusion was much different
than Victor's: screens take us away from the fullest experience of world, and
so their use should be minimized. This is basically Victor's conclusion minus
the possibility of what I might call "somatic programs" at scale.

That screens are problematic asserts a moral imperative for programmers to
avoid writing applications that are designed to be experiential cul-de-sacs.
This, in turn, goes very much against the grain of most technologists and the
people who fund them, who are all looking to make the most potent screens they
can to pull people out of real life as much as possible. This makes great
economic sense, but it is _unethical_.

Most of us who are so fundamentally tied to a screen for so many years at a
time would do well to systematically disengage and interact with the real-
world as much as possible. What I wouldn't give at this point in my life to
put on a hard-hat and help build a bridge!

------
cusack
I'd push back on the dog in a cage analogy. I think it could be argued that
we've narrowed our communication mediums for purpose of effectiveness and
efficiency. Also affordable ubiquity. I would argue it's more like limiting a
fish to water rather than a dog to a cage. Not to say it can't be improved in
light of today's tools though.

Great talk overall, a lot of interesting insights!

~~~
indrax
The analogy was an example of something that is inhumane for analogous reasons
that restricted media is inhumane. It doesn't allow for full dog-life or full
intellectual-life, some critical and important experiences and behaviors are
excluded.

Changing the analogy to a fish in a tank suggests an assertion that these
other intellectual modes are not important for human experience and wouldn't
make communication more effective and efficient.

To tweak the analogy further: Otters are very good at swimming, but their
habitats should include land.

~~~
resu_nimda
Putting a dog in a cage is inhumane because one actor has taken away liberties
that the other actor is known to have "by default." We know that dogs are
naturally able and inclined to run around and play, and we have intentionally
restricted that.

Personally I don't think his usage of the term "[in]humane" to describe human
media is appropriate. Who is responsible for this inhumanity? Our entire race
is collectively treating itself inhumanely by not inventing entirely new forms
of media and expression? That seems like a stretch. Are we inhumane to dogs
because we haven't yet helped them discover their fullest and richest
capabilities as a species, whatever those may be? I love Bret and his vision
and talks, but I feel like this particular usage is heavy-handed shame
rhetoric (I get that he is trying to light a fire under people, but it still
seems forced).

 _some critical and important experiences and behaviors are excluded._

What defines experiences as critical and important, such that to exclude them
would be inhumane? Is it inhumane that I haven't traveled on a spaceship to
Mars? That seems like a pretty important experience. But of course that's
ridiculous, because you can't include the infinite landscape of experiences
that might be possible. Personally I don't think either analogy (dog or fish)
is particularly apt, but I think the fish one is better because a) our
generally known and accepted capabilities are not being forcibly restricted by
another agent, and b) the stuff Bret is talking about is entirely new and not
possible within our natural environment (i.e. there are many natural and
accessible ways to write symbols on a surface, but nothing like that for
arbitrary-dynamic-physical-hologram-things).

~~~
indrax
I don't know why anyone has to be 'responsible' for an inhumanity to exist.
One actor taking the liberties of another actor is not required.

A) Our capabilities are being restricted by the circumstances we find
ourselves in. A dog happens to be in a cage built hundreds of years ago. There
must be some words to describe this situation, I think inhumane applies.

B) Now we can start opening the cage.

Edit: our natural environment is inhumane and awful. Your commentary seems to
heavily favor the status quo in a way I can't relate to.

The additional inhumanity of media is from 2 things: the fact that information
processing has been mostly limited, and that information processing has become
really important lately, so a lot of people are spending a lot of their time
in that limited environment.

------
chewxy
Call me obtuse and change my view please:

I'm actually quite interested in this spatial learning thing. I had an
aboriginal acquaintance who explained to me how his culture would attach
meaning to places and even physical orientation (North South East West etc).
However, it has been so far very very difficult for me to wrap my head around
that. Visual notation imo, has been the most efficient way of communicating
between humans. It also helps a lot in understanding.

A lot of what he says harks back to a 1980s/1990s/early 2000s style of
teaching with an increasing use of "aids". Using props to teach math is a bit
silly (well, asides from geometry, which is inherently spatial). I really
cannot see how we can represent the learning of linear algebra in a dynamic
spatial manner.

How for example, would one represent higher dimensional spaces spatially, when
you really are limited to only 3 dimensions in real life? I am aware that Iron
Man style projection tech may actually help in visualizing it, but again,
that's a visual aspect of learning, not spatial. The most efficient way of
doing so unfortunately is still in algebraic notation.

Perhaps I'm doubtful because the use of props, spaces, songs, moving your body
etc has never worked for me in school. Ever. Can anyone share any studies done
on the use of these cheesy 1980s-1990s educational aids (the closest analogue
to all the things Brett mentioned) in aiding learning?

EDIT: If you're interested, there's been a good book that I read, in attempt
to wrap my head around the concept (however to no avail) - Wisdom Sits in
Places.

~~~
schoen
One of the educational techniques you're alluding to is probably TPR (total
physical response), for foreign language teaching.

[https://en.wikipedia.org/wiki/Total_physical_response](https://en.wikipedia.org/wiki/Total_physical_response)

I had a Spanish teacher and later a Latin teacher (teaching adults) who
favored TPR and would tell people to jump a specified number of times, or to
touch their heads, and so on. I don't have a clear opinion about the
effectiveness of TPR but I thought I'd mention the name so you could use it to
search for research if you like.

There is also a century-old approach to using colors to visualize 4D geometry
(I think the color represents the offset in the 4th spatial dimension),
proposed by Charles H. Hinton; they're called "Hinton cubes". Martin Gardner
describes them and reports that some people have found them _too_ effective, I
guess becoming mildly obsessed with them or having the learned associations
interfere somehow with day-to-day visual perception.

------
Detrus
First a few nitpicks

1\. Walking around a gallery and waving your arms is not enough exercise 2\.
Incremental progress is giving us VR through Tactical Haptics, Oculus Rift,
etc. He says his thing is not VR but R or dynamic R. The distinction is
unclear, sounds like marketing.

I also had the idea of talking about systems, particularly political systems
through models and simulations. Having spent too much time talking about
politics, economics, and computer holy wars in text, it feels like a few
interactive models could have saved all that time for millions of people.

That said, simulations are widely used to communicate about financial topics,
mainly spreadsheets. And the effects of using spreadsheets are pretty
interesting [https://medium.com/backchannel/a-spreadsheet-way-of-
knowledg...](https://medium.com/backchannel/a-spreadsheet-way-of-
knowledge-8de60af7146e)

People start thinking in spreadsheets. A lot of the weird merger acquisitions
since the 1980's were basically caused by them. Economists already use fancier
models. And as with spreadsheets, garbage in, garbage out. Still I feel they'd
be useful in casual conversation.

~~~
jamesrom
1\. If you are doing it all day, it would almost certainly be sufficient.

2\. He's talking beyond VR. He never says VR isn't a stepping stone. But he
does make a good point about it being dehumanising.

------
31reasons
Great talk and ideas. But the idea of representing mathematical equations in
the physical form is like supporting the same medium of symbols and just
representing it in a different way. But fundamentally math was invented using
pen and paper, so the centuries of mathematical thoughts were already slaves
to the 2D medium. Maybe it would be interesting to think about, what would
happen if we take away 2D surfaces and ask people to reinvent math in a medium
that Bret is talking about. How would someone go about discovering, solving,
proving mathematical systems without pen and paper? How would you represent 4,
5 or 20 multi-dimentional spaces in a 3D environment ? Maybe pen&paper was
kind of abstracting out limitations of physical world so that mind can explore
even further.

~~~
orbifold
The progress in n-category theory is largely about that. Instead of just
studying objects, operations and equations, like "a group is a set g, together
with multiplication + certain axioms" one studies objects, relations between
objects, relations between relations etc. To study this up to level two, you
visualize objects as regions in 2d space, relations between objects as lines
between them and relations between relations as points where lines meet.
Equations are then equalities between such pictures. If you look at those
equalities as a dynamic process, which starts at time t=0 with one picture and
time t=1 with the other, you get a picture in three dimensional space, which
you can visualize as a movie (to be mathematically precise after the choice of
a morse function you get a movie). One of the easiest examples would be the
zig-zag identity. See for example
[https://golem.ph.utexas.edu/category/2006/10/classical_vs_qu...](https://golem.ph.utexas.edu/category/2006/10/classical_vs_quantum_computati_4.html)
and scroll down.

One person who has completely gone overboard with that is Jacob Lurie, he has
practically single handedly written down a foundational work of higher algebra
and algebraic topology, which essentially precisely corresponds to what you
are asking. How does mathematics look like if we weren't constrained to 2d
paper to write down equations. Of course to actually write down those things
you have to somehow project the equations to several essentially 1d sections
or invoke very abstract methods to reason about them. Of course algebraic
topology already has tools for that.

The early impetus came from Grothendieck I believe, who began to create a
theory of "higher groupoids" in several thousand pages, ordinary groupoids
being categories in which every morphism is an isomorphism.

------
teekert
Extremely nice, very out of the box.

On TV in the Netherlands we have a program that aims to educate the public
(state sponsored tv), every couple of months there is a new episode, they are
listed here (but they are in Dutch): [http://dewerelddraaitdoor.vara.nl/DWDD-
University.2772.0.htm...](http://dewerelddraaitdoor.vara.nl/DWDD-
University.2772.0.html)

So far they were about the big bang, the quantum world, Einstein (All from
Robbert Dijkgraaf, now director of the Institute for Advanced Study in
Princeton), The cow and silicon valley. They were always set in a large
theater (with live public) with a stage full of props (Inflating balloons to
represent the inflating universe, buckets of sand to represent the amount of
stars in a galaxy, cameras on rails to represent relativity, living cows as
well as all their nice parts of meat on display.) These "presentations" are
very nice to watch and my wife who is not a scientist always enjoys them as
well.

There are also museums like Nemo in Amsterdam
([https://www.e-nemo.nl/nl/ontdek/brein/](https://www.e-nemo.nl/nl/ontdek/brein/))
and Naturalis in Leiden
([http://www.naturalis.nl/en/](http://www.naturalis.nl/en/)), these places are
filled with rooms with stuff to touch and experiments to perform. I'm sure the
US is laced with those as well so it is not that ground-breaking of an idea.

A holo-deck would be the killer feature for this way of communicating.

~~~
JeroenRansijn
I thought about Robbert Dijkgraaf's TV lectures as well. A must see! I have no
idea if there are any english subs though.

------
tmerr
On the topic of how representations relate to programming... if you have ever
tried to express some tricky math in code you might have had a sense that it
would be more clear if it were written out on paper with the proper notation.
That's because mathematicians have streamlined their reasoning by deciding one
day, well, I use the sum function a lot, let's call it Σ. Or I'd like to be
able to recognize integrals from a mile away so let's call it ∫. Not to
mention those symbols come along with a spatial organization: the lower limit
of integration goes below, the upper limit goes on top.

Programmers can't easily streamline like this. Maybe there are multiple forces
locking us in: the keyboard, text editor, and programming language. Sure,
maybe the optimal way to write code is with a string of ascii characters
arranged left to right top to bottom but I doubt it. So even in the visual-
symbolic space there is unexplored potential.

I'm glad someone is out there tackling these assumptions and much more, I
should keep an eye on his work.

On an unrelated note, his website is entertaining:
[http://worrydream.com/](http://worrydream.com/)

~~~
joshkpeterson
His website is legend among UX designers because it's such a great example of
how to do a portfolio creatively.

~~~
theoh
Really? I thought he was a celebrity among programmers because of his
imaginative approach. Surely any UX designer worth their salt would find his
home page to be moronically simplistic in the way it tries to shake up graphic
design conventions. He is interesting, but not, I think, in a "design" way.
Also, "worrydream", "beast of burden"? A bit too neurotic to be considered
appropriate by, well, anyone grown up.

~~~
seanmcdirmid
A website meant to be usable is a bit different from one meant to be a
portfolio. From that perspective, I think his website works well, but it is a
bit dated in terms of aesthetics (not as flat as the current fashion goes).

I don't think Bret Victor's work is really about design (most of the UX
designers I know have never heard of him), it is about invention and is a bit
more on the programmer side. Putting him in the designer bucket is too
inaccurate (but buckets are always inaccurate).

I saw this talk in person at SPLASH; the puppy in cage for life really pulls
at some heart strings. But it didn't really do it for me like the previous
talks did (though he didn't have any prototypes to show off as with the
previous talks).

------
arikrak
Looks interesting, going to watch rest later. I wonder if the graph of trade
with England is really the 'basis' for all scientific graphs of data. Didn't
Descartes develop the idea of graphing Y in relation to X?

[http://en.wikipedia.org/wiki/Cartesian_coordinate_system](http://en.wikipedia.org/wiki/Cartesian_coordinate_system)

------
the_cat_kittles
great talk! one touchstone i wish he would have embraced is woodworking.
having listened to the talk, i can see how woodworking is experienced through
many channels: tactile (hand planes, saws), visual, spatial, symbolic (plans,
designs), auditory (sonic feedback tells you a lot of about the wood, and how
well a tool is working), environmental (the workshop)... i feel like adopting
his channels perspective helps me understand why that activity is so engaging.
certainly gives credence to his claims, too.

------
niels_olson
I think something like ipython can get to where he wants to go. I have already
used the RISE profile and MDTraj so I can show people my notebook in slide
layout, and pull protein models directly from PDB (Brad talks about the
provenance of the data). So I can run code live, manipulate meaningful models
live. I think some solid navigation wouldn't be too far off. Now we just need
holograms. Cheap blue lasers, people. We need cheap blue lasers.

------
GraffitiTim
Still watching the presentation, but virtual reality is the medium that will
eventually allow for a lot of the things he's talking about.

Edit: he later dismisses this, but I don't think he's right to do so.

~~~
bchjam
VR still lacks good tactile, kinesthetic representations. Not to say it's not
a step in the right direction, but it is arguably taking a deeper step into
visual-only representation.

(edit: relevant link, Victor's rant on hands in interaction design.
[http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...](http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/))

~~~
corysama
While it's neither perfect nor complete, I can say from multiple hands-on
experiences that the Tactical Haptics grip works much better than you might
expect.

[https://www.youtube.com/watch?v=XBB_OFMJ-
Go](https://www.youtube.com/watch?v=XBB_OFMJ-Go)

~~~
bchjam
I've never used them personally but I'm sure they do improve the experience. I
suppose there's a point at which that's still tricking the human nervous
system in ways that a physical model doesn't have to, but haptic VR is also
more realistic with today's technology than dynamically self-assembling
nanobots.

------
pokpokpok
exciting to be young and engaged in technology at a time like this

