
Show HN: Paper Programs – Run JavaScript on pieces of paper - janpaul123
https://paperprograms.org/
======
janpaul123
Hi! I was asked to repost this (a la
[https://news.ycombinator.com/item?id=11662380](https://news.ycombinator.com/item?id=11662380)),
so here goes. I'm JP, and I created this project after visiting Dynamicland a
couple of times. I wanted to capture some of the magic there in my living
room, and decided to implement the "programs running on pieces of paper" part.

It was super fun implementing this. I ended up using OpenCV.js, which has only
just been released, reimplementing an OpenCV algorithm and learning about
computer vision, learning about OffscreenCanvas, which is so experimental that
it crashes periodically (a bug I was able to isolate and submit to the Chrome
dev team), learning about how to mount a projector facing downward, and so on.

Few things are as satisfying as a piece of paper rolling out of a printer
literally onto the floor, where it starts executing. And then cutting it up
with a pair of scissors to change its shape.

I hope you have as much fun with this as I do!!

PS: If you want more inspiration for building cool stuff with projectors and
cameras, check this out:
[http://tangible.media.mit.edu/project/pico/](http://tangible.media.mit.edu/project/pico/)
:) They have a ton more interesting projects like that!

~~~
nni
I've been seeing folks get excited about this, and haven't yet been able to
see why it is so interesting. However, the people behind it (Bret Victor et
al) and who I've seen playing with it are too interesting for me to ignore
this...

~~~
theon144
It's a very different kind of interaction than with code on the screen, and
fits the way one thinks in a much more concrete way. It's like an embodied
metaphor, one that people are intimately familiar with, and the immediacy of
it allows you to iterate quickly as you have direct feedback, and the whole
structure is made visible for everyone to see, enabling a kind of
collaboration unusual for computer programming (image the dynamic of a group
of people trying to build a tent together compared to a group of people all
staring at a static view of an IDE).

Of course, it's nothing groundbreaking just yet, but I can't help but get
excited at the possibilities of this sort of medium.

~~~
TeMPOraL
I still don't see how this is true, either in this case or in Dynamicland -
neither of those seem to OCR programs from the piece of paper. So you still do
classical programming alone, and only get to play with the _results_. A piece
of paper may contain the actual program behind system's reaction to it, or it
may contain a picture of a cat - the contents are irrelevant; it's the dots
that matter.

~~~
janpaul123
If you write small programs that interact with other programs, then
rearranging them on the floor/table becomes a form of programming (similar to
how combining UNIX-style programs in shell scripts is a high-level form of
programming).

So even when playing just with the papers, you can be programming.

~~~
meemoo
There's also the interplay of arranging papers, and live-editing the code in a
particular piece of paper. In Dynamicland, the projector highlights printed
code in green/red to visualize diffs.

You can sketch a UI element on the paper (eg a rectangle and token for a
numerical slider), then go back to code the interactivity. Units in the code
are inches on the table. Subtle things like that break you free from the
laptop.

When you want to "commit" your code changes, you print a new version.

------
ChicagoBoy11
I work in an elementary school and I'm trying hard to think about how to
possibly take advantage of this in my environment. The interaction model is
really rich and perfect for the children, but I'm so far coming up empty on
the sorts of things we could/should do with it.

~~~
unoti
how About a space game where you move cargo ships around and manufacture and
trade goods. You could teach about coordinate systems and math. Players need
to figure out optimum things to haul for their amount of available funds.
Different players could have different capabilities and teams need to figure
out how to work together to accomplish more together as a team than they could
on their own. Game space shows what the cargo is and how far a ship can move
with its current cargo.

~~~
Pulcinella
It would be great if it could be connected to something like the MIT Media LAb
inFORM table and/or some kind of under the table system of magnets. That way
physical objects could be moved instead of just the projector displaying an
image of the cargo ship. Right now the physical manipulation seems one way,
from the physical world to the digital. It is “easy” for the the system to
motion track an object as a student moves it, but the system doesn’t really
have a way for it to move the object itself.

Imagine students exploring gravity by placing spheres on a “magic” table and
then the table causes the spheres to orbit around eachother. Students can
adjust mass or gravitational constant and see how that affects the system, or
even come up with their own laws of gravity.

------
camtarn
Very interesting interaction model. I love the idea of a program being able to
read the shape of the paper - lots of cool possibilities.

However - slightly disappointed that the programs are not literally controlled
by symbols on the pieces of paper, but rather the paper just holds identifiers
who point to a program on a server.

I wonder if you could extend the computer vision system to pick up actual
programs from the paper - either by OCRing actual text (perhaps limited to
capital letters only, and/or particular shapes for letters such as I-for-
India, to make it easier?) or a combination of symbols (each corresponding to
a function in the API) and numbers (parameters).

That way you could do really interesting things: modifying somebody's program
by literally pasting a scrap of paper on it containing different symbols or
parameters, or having programs which interact with other programs (either by
treating their program code as input data, or by piping data between the
programs).

On the other hand, it's probably easier to write interesting programs with an
actual keyboard :)

~~~
vitovito
This is how Dynamicland currently works, also, but their goal is what you've
stated. It's a hardware limitation.

~~~
yorwba
I'm not sure how large the paper needs to be to be recognized. Maybe you could
have snippets that hold only a single identifier or syntax element and
programs would be arranged by puzzling them together. That should be an
appropriate granularity for manually placing (no fidgeting with tiny
individual letters) while still allowing full programmability with minimal
trips to the printer (only to create some new identifier).

~~~
camtarn
Something like Scrabble tiles, perhaps?

I wonder if you could create a set of symbols which were sufficiently
distinctive enough that you could hand-draw them and stand a chance of the
computer recognising them. Difficulty level increasing again...

------
osrec
Are the dots essentially a "barcode" that map to a particular program on the
server? How does the projector factor in? Do you adjust the projected image to
ensure the result falls in the respective paper's area? Interesting project -
did you have a use case in mind for it?

~~~
janpaul123
Yeah, exactly what you said.

I work on software for public transit agencies, and Dynamicland immediately
made me think of a workshop that Jarrett Walker (humantransit.org) sometimes
runs, which involves physical paper and pieces of coloured strings. So I tried
to build something inspired by that
([https://twitter.com/JanPaul123/status/923037721760735232](https://twitter.com/JanPaul123/status/923037721760735232)).

But that experiment didn't involve having programs interact which each other,
which is way more powerful as then anyone looking at the table can rearrange
the programs differently, which itself becomes a form of programming (when
your programs do small enough things).

So anyway, no specific use case yet, I think it's too early for that.

------
throwaway2016a
This is awesome work. Great job.

Dynamicland looks awesome. I wish I could visit. I both live on the wrong
coast and it doesn't seem to be open to the public yet.

One fascinating aspect I found on this project was compiling OpenCV[1] to web
assembly. I find it difficult and frustrating to compile OpenCV for my Mac
never-mind compiling it to web assembly and running it in a browser. I love
the idea of doing that, I'll have to try it.

[1] Open Computer Vision to save a few people a google

~~~
meemoo
Good docs and demos:
[https://docs.opencv.org/master/d5/d10/tutorial_js_root.html](https://docs.opencv.org/master/d5/d10/tutorial_js_root.html)

------
seanmcdirmid
Wondering if this could be done with AR (e.g. Hololens or Glass)?

~~~
jf
The benefit of physical objects (like paper) is that you don't need to
simulate reality in AR. Using physical objects lets you discover things that
you wouldn't have otherwise with simulation. One of the neat things about the
paper at Dynamicland is that can you "stretch" paper by just tearing it in
half (or into quarters) and move those out.

Doing all of that is easy and natural, sometimes you even discover cool things
by mistake.

~~~
seanmcdirmid
Augmented reality headsets can theoretically project onto real piece paper
just like a projector (actually, a projector is a form of AR). They don't mask
out the current reality like a VR headset does, they have no need (and indeed
no capability) to simulate a different reality.

~~~
jf
Ah. I see what you're saying. I thought you were proposing to replace paper
with AR ("this desk is now a screen", etc)

Trying to channel the researchers at Dynamicland, I think the answer would be
that they aren't investigating AR because doing so isn't in line with their
design principles. The mission statement for Dynamicland includes this phrase
"incubating a humane dynamic medium" \- and since most humans aren't equipped
with AR hardware, I'm guessing that is why AR isn't an area of research for
this particular technology.

~~~
seanmcdirmid
I'm pretty sure Bret Victor has been asked this before and responded that it
wasn't tangible or collaborative enough. For their purposes, it wouldn't work,
but it seems like something that could work for another project.

------
theon144
Yes!

I've been similarly enchanted when I first saw Dynamicland (well, on twitter,
the actual place being on a different continent) and I've actually started my
own experiments (but never finished them :( ). Glad to see someone follow
through, I'm definitely going to be watching you closely!

------
nestorherre
One of the coolest things I've seen in a while. GJ!

------
Outpox
Really interesting, thanks for sharing!

