
Browserball - ABNWZ
http://weareinstrument.com/ball/#
======
markmahoney
I built this in like 2009 or 2010. Super curious how it managed to resurface
now.

------
tambourine_man
This is amazing.

Kind of like when I first dragged a window across multiple monitors in the
90s. We are so used to content being stuck in artificial containers that it's
kind of crazy when it transcends them.

~~~
Pxtl
There was a clever game called Metal Marines back then that exploited this
idea - the game was a 1v1 base-building RTS where each player's "base" was on
its own map in its own window. When you fired missiles or launched APCs from
one base into the other, the missile would fly out of your base window, _over
the desktop_ and into the other window to hit his base. The effect was jaw-
dropping even though it was a simple and cartoony game.

~~~
viraptor
Thank you! I was looking for that game a few days ago, but couldn't remember
the name at all! And it looks like there's a reboot of it on Steam:
[http://store.steampowered.com/app/335940/](http://store.steampowered.com/app/335940/)

------
21
What's the reason for allowing web pages to get absolute screen coordinates?

This is a privacy leak. I have a 24" screen, and I don't keep the browser
window maximized because it would be too big. I presume other people do to,
and I'm pretty sure most have a preferred size and position.

~~~
sleepychu
How do you know the coordinates are absolute?

It seems like the child windows are 'special', perhaps the web page can obtain
the relative coordinates of these child panels?

~~~
Cpoll
It's using window.screenY (and screenX, of course,) which gives absolute
coordinates. [https://developer.mozilla.org/en-
US/docs/Web/API/Window/scre...](https://developer.mozilla.org/en-
US/docs/Web/API/Window/screenY)

I don't think the method you're describing exists. If you want child
coordinates relative to parent coordinates, you would use both the parent and
child's absolute coordinates.

~~~
sleepychu
Thanks to you (and others who've provided this information), so I guess my
question now is what is the privacy leak described. I understand that the
browser is confessing information about the machine to the web page but I
don't understand why that represents a private piece of information. I feel
like the information about where the window is located on the screen is useful
for implementation and I can't think what it says about me as a user.

~~~
nommm-nommm
Nothing on its own. However it is one of the pieces of information used to
create a unique fingerprint for your browser which can be used to track you as
you travel across the web. IIRC Tor browser recommends never resizing the
window from the browser's default window size.

------
ethana
I remember this demo when Chrome first came out. I thought that was the
coolest thing.

edit: I found the original site that I linked to this. Courtesy of
archive.org:
[http://web.archive.org/web/20120214090814/http://blog.insicd...](http://web.archive.org/web/20120214090814/http://blog.insicdesigns.com/2010/02/mind-
blowing-javascript-experiments/)

------
bshimmin
I think the thing I like most about this is how it's evidently very imperfect
- the ball sometimes goes missing, the rules for what velocity it needs to
bounce from one window to another seem hazy, etc - but none of that really
matters: it's fun and cool to play about with for a minute or two, it doesn't
at all take itself seriously, and you know that any real implementation of
this sort of thing, if there were an actual need for it, would be implemented
more robustly anyway.

------
kirkbackus
I made a project in college like this that utilized the Windows API to
enumerate the rectangles of the windows and you could create shapes and other
things that you can throw around your desktop

[https://www.youtube.com/watch?v=38qWXZ0hDv4](https://www.youtube.com/watch?v=38qWXZ0hDv4)

------
DonHopkins
I am frustrated that it's 2016, and we still can't tilt windows diagonally.

~~~
agumonkey
Meaning ?

~~~
DonHopkins
Then you could make the ball roll down towards the corner and drop into
another window, of course!

The NeWS window system let you reshape the entire framebuffer to any
orientation or clipping, as well as individual windows and sub-windows, in the
late 1980's.

Why shouldn't I be able to lay down on my side next to a laptop and read a web
page off the screen sideways, or adjust the rotation of the window to match
the inclination of my pillow?

The other thing I want the window manager to support (which NeWS couldn't do
since PostScript only supports 2D affine transforms) is estimating my head
position relative to the screen, and projecting the window in perspective so
it looks rectangular from an oblique viewing angle, so I can watch a movie on
an extra screen off to the side.

If you could rotate windows around on the screen, Browser Ball should be able
to use the laptop accelerometer to detect the true direction of gravity, and
bounce the ball accordingly as you turned the windows and the laptop itself
around.

Unfortunately Apple stopped putting accelerometers in laptops with SSD drives,
since they were only used to retract the heads when falling.

You can do a lot of fun things by modeling and tracking the positions of
multiple people's heads and multiple devices in augmented reality! Here are
two people using two tablets, two laptops, and a desktop computer together
with Pantomime: [1]

[1] [https://vimeo.com/149319403](https://vimeo.com/149319403)

~~~
agumonkey
Why did I not read your nickname .. :)

The pseudo billboard idea is cute, but I wonder how it fares in readability on
non HD screen (font rendering in 3D space may impede it, then viewing
angles.). I'd love to see it done anyway, it's cute idea.

Another 'this is <year>' rant, we should have way more sensors on usb or i2c
and have our laptops integrate with the real world more.

~~~
DonHopkins
Here are some ideas and discussion about aQuery -- like jQuery for
accessibility [1] -- which would be useful for implementing this kind of stuff
out of of existing window systems and desktop applications.

[1] aQuery:
[http://donhopkins.com/mediawiki/index.php/AQuery](http://donhopkins.com/mediawiki/index.php/AQuery)

Morgan Dixon's work is truly breathtaking and eye opening, and I would love
for that to be a core part of a scriptable hybrid Screen Scraping /
Accessibility API approach.

Screen scraping techniques are very powerful, but have limitations.
Accessibility APIs are very powerful, but have different limitations. But
using both approaches together, screencasting and re-composing visual
elements, and tightly integrating it with JavaScript, enables a much wider and
interesting range of possibilities.

Think of it like augmented reality for virtualizing desktop user interfaces.
The beauty of Morgan's Prefab is how it works across different platforms and
web browsers, over virtual desktops, and how it can control, sample, measure,
modify, augment and recompose guis of existing unmodified applications, even
dynamic language translation, so they're much more accessible and easier to
use!

\----

James Landay replies:

Don,

This is right up the alley of UW CSE grad student Morgan Dixon. You might want
to also look at his papers.

Don emails Morgan Dixon:

Morgan, your work is brilliant, and it really impresses me how far you've gone
with it, how well it works, and how many things you can do with it!

I checked out your web site and videos, and they provoked a lot of thought so
I have lots of questions and comments.

I really like the UI Customization stuff, and also the sideviews!

Combining your work with everything you can do with native accessibility APIs,
in an HTML/JavaScript based, user-customizable, scriptable, cross platform
user interface builder like (but transcending) HyperCard would be awesome!

I would like to discuss how we could integrate Prefab with a Javascriptable,
extensible API like aQuery, so you could write "selectors" that used prefab's
pattern recognition techniques, bind those to JavaScript event handlers, and
write high level widgets on top of that in JavaScript, and implement the
graphical overlays and gui enhancements in HTML/Canvas/etc like I've done with
Slate and the WebView overlay.

Users could literally drag controls out of live applications, plug them
together into their own "stacks", configure and train and graphically
customize them, and hook them together with other desktop apps, web apps and
services!

For example, I'd like to make a direct manipulation pie menu editor, that let
you just drag controls out of apps and drop them into your own pie menus, that
you can inject into any application, or use in your own guis. If you dragged a
slider out of an app into the slice of a pie menu, it could rotate it around
to the slice direction, so that the distance you moved from the menu center
controlled the slider!

While I'm at it, here's some stuff I'm writing about the jQuery Pie Menus.
[http://donhopkins.com/mediawiki/index.php/JQuery_Pie_Menus](http://donhopkins.com/mediawiki/index.php/JQuery_Pie_Menus)

\----

Web Site: Morgan Dixon's Home Page.
[http://morgandixon.net/](http://morgandixon.net/)

Web Site: Prefab: The Pixel-Based Reverse Engineering Toolkit.
[https://web.archive.org/web/20130104165553/http://homes.cs.w...](https://web.archive.org/web/20130104165553/http://homes.cs.washington.edu/~mdixon/research/prefab/)

Video: Prefab: What if We Could Modify Any Interface? Target aware pointing
techniques, bubble cursor, sticky icons, adding advanced behaviors to existing
interfaces, independent of the tools used to implement those interfaces,
platform agnostic enhancements, same Prefab code works on Windows and Mac, and
across remote desktops, widget state awareness, widget transition tracking,
side views, parameter preview spectrums for multi-parameter space exploration,
prefab implements parameter spectrum preview interfaces for both unmodified
Gimp and Photoshop:
[http://www.youtube.com/watch?v=lju6IIteg9Q](http://www.youtube.com/watch?v=lju6IIteg9Q)

PDF: A General-Purpose Target-Aware Pointing Enhancement Using Pixel-Level
Analysis of Graphical Interfaces. Morgan Dixon, James Fogarty, and Jacob O.
Wobbrock. (2012). Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems. CHI '12\. ACM, New York, NY, 3167-3176. 23%.
[https://web.archive.org/web/20150714010941/http://homes.cs.w...](https://web.archive.org/web/20150714010941/http://homes.cs.washington.edu/~mdixon/publications/mdixon-
prefab-chi2010-final.pdf)

Video: Content and Hierarchy in Prefab: What if anybody could modify any
interface? Reverse engineering guis from their pixels, addresses hierarchy and
content, identifying hierarchical tree structure, recognizing text, stencil
based tutorials, adaptive gui visualization, ephemeral adaptation technique
for arbitrary desktop interfaces, dynamic interface language translation, UI
customization, re-rendering widgets, Skype favorite widgets tab:
[http://www.youtube.com/watch?v=w4S5ZtnaUKE](http://www.youtube.com/watch?v=w4S5ZtnaUKE)

PDF: Content and Hierarchy in Pixel-Based Methods for Reverse-Engineering
Interface Structure. Morgan Dixon, Daniel Leventhal, and James Fogarty.
(2011). Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems. CHI '11\. ACM, New York, NY, 969-978. 26%.
[https://web.archive.org/web/20150714010931/http://homes.cs.w...](https://web.archive.org/web/20150714010931/http://homes.cs.washington.edu/~mdixon/publications/mdixon-
content-hierarchy-chi2011-final.pdf)

Video: Sliding Widgets, States, and Styles in Prefab. Adapting desktop
interfaces for touch screen use, with sliding widgets, slow fine tuned
pointing with magnification, simulating rollover to reveal tooltips:
[https://www.youtube.com/watch?v=8LMSYI4i7wk](https://www.youtube.com/watch?v=8LMSYI4i7wk)

Video: A General-Purpose Bubble Cursor. A general purpose target aware
pointing enhancement, target editor:
[http://www.youtube.com/watch?v=46EopD_2K_4](http://www.youtube.com/watch?v=46EopD_2K_4)

PDF: Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse
Engineering of Interface Structure. Morgan Dixon and James Fogarty. (2010).
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
CHI '10\. ACM, New York, NY, 1525-1534. 22%
[https://web.archive.org/web/20150714010936/http://homes.cs.w...](https://web.archive.org/web/20150714010936/http://homes.cs.washington.edu/~mdixon/publications/mdixon-
general-purpose-target-chi2012-final.pdf)

PDF: Prefab: What if Every GUI Were Open-Source? Morgan Dixon and James
Fogarty. (2010). Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems. CHI '10\. ACM, New York, NY, 851-854.
[https://web.archive.org/web/20141024012013/http://homes.cs.w...](https://web.archive.org/web/20141024012013/http://homes.cs.washington.edu/~mdixon/publications/mdixon-
prefab-workshop-chi2010-final.pdf)

Morgan Dixon's Research Statement: [http://morgandixon.net/morgan-dixon-
research-statement.pdf](http://morgandixon.net/morgan-dixon-research-
statement.pdf)

Community-Driven Interface Tools

Today, most interfaces are designed by teams of people who are collocated and
highly skilled. Moreover, any changes to an interface are implemented by the
original developers and designers who own the source code. In contrast, I
envision a future where distributed online communities rapidly construct and
improve interfaces. Similar to the Wikipedia editing process, I hope to
explore new interface design tools that fully democratize the design of
interfaces. Wikipedia provides static content, and so people can collectively
author articles using a very basic Wiki editor. However, community-driven
interface tools will require a combination of sophisticated programming-by-
demonstration techniques, crowdsourcing and social systems, interaction
design, software engineering strategies, and interactive machine learning.

~~~
agumonkey
Small temporary answer while I unwind all the linked content, Dixon's target
aware pointing is already missing so much. I wonder how on earth nobody in
smartphone land thought to implement something similar. I'm already hooked :)

~~~
DonHopkins
It's missing from many contexts where it would be very useful, including
mobile. It's related in many ways to those mobile gui, web browser and desktop
app testing harnesses. It could be implemented as a smart scriptable "double
buffered" VNC server (for maximum efficiency and native Accessibility API
access) or client (for maximum flexibility but less efficient).

The way jQuery widgets can encapsulate native and browser specific widgets
with a platform agnostic api, you could develop high level aQuery widgets like
"video player" that knew how to control and adapt many different video player
apps across different platforms (youtube or vimeo in browser, vlc on windows
or mac desktop, quicktime on mac, windows media player on windows, etc). Then
you can build much higher level apps out of widgets like that.

Target aware pointing is one of many great techniques he shows can be layered
on top of existing interfaces, without modifying them.

I'd like to integrating all those capabilities plus the native Accessibility
API of each platform into a JavaScript engine, and write jQuery-like selectors
for recognizing patterns of pixels and widgets, creating aQuery widgets that
tracked input, drew overlays, implemented text to speech and voice control
interfaces, etc.

His research statement sums up where it's leading: Imagine wikipedia for
sharing gui mods!

Berkeley Systems (the flying toaster screen saver company) made one of the
first screen readers for the Mac in 1989 and Windows in 1994.
[https://en.wikipedia.org/wiki/OutSpoken](https://en.wikipedia.org/wiki/OutSpoken)

Richard Potter, Ben Shneiderman and Ben Benderson wrote a paper called Pixel
Data Access for End-User Programming and graphical Macros, that references a
lot of earlier work.
[https://www.cs.umd.edu/~ben/papers/Potter1999Pixel.pdf](https://www.cs.umd.edu/~ben/papers/Potter1999Pixel.pdf)

------
neogodless
Everything was made to be broken. Got this neat effect where the ball was
spinning rapidly in the corner of each spawned window!
[http://imgur.com/gmwQOCb](http://imgur.com/gmwQOCb)

------
junto
This is one of those awesomely pointless conceptual experiments. Love it.

~~~
visakanv
The internet Ig Nobel!

------
cyphar
I remember playing with this at least 8 years ago. It's really scary that
there's enough information disclosure in browsers that it's possible to do
this.

~~~
softawre
> really scary

This is really scary to you? I get that you can do fingerprints, and honestly
there's a LOT more than just browser window position/size in them, but "really
scary"?

Maybe we need to stop exaggerating on this sort of stuff if we want people to
take us seriously.

~~~
cyphar
Fine. It's scary that browsers allow for so much fingerprinting. This toy
webapp is an example of a larger picture of "things that we really shouldn't
allow to happen". So "really scary" was referring to the issue of browser
fingerprinting as a whole, not just one toy webapp.

------
nxzero
YouTube Demo:
[https://m.youtube.com/watch?v=XmH2QF92Jwo](https://m.youtube.com/watch?v=XmH2QF92Jwo)

~~~
mavci
why so serious?

------
tjtate
they did this with the birds in the wilderness downtown experiment like 5
years ago right?
[http://www.thewildernessdowntown.com/](http://www.thewildernessdowntown.com/)

~~~
tokenizerrr
I just wanted to try that since it seems interesting. Sadly it tells me to
enter the address of the home where I grew up, but then promptly refuses to
recognize anything that I put in. It also says I can try schools, but does not
recognize any of those either.

Maybe it only recognizes American addresses or something, but I don't know any
of those.

edit: Ok I got it working by putting in Amsterdam, which happens to be a city
and not an address.

------
mjs7231
Is the full ball rendered behind an overlapping window? The world may never
know.

------
ptibbetts
I can't remember the last time I was so happy to have 3 screens

------
stevep98
Reminds me of the Amiga Boing Ball demo from 1984.

[https://www.youtube.com/watch?v=-ga41edXw3A](https://www.youtube.com/watch?v=-ga41edXw3A)

------
ahabman
Security issues aside, has anyone tackled how to intentionally span multiple
monitors using similar techniques? e.g. A dedicated four monitor web-based
media dashboard.

------
krolow
remembers one project that i did 4 years ago
[https://www.youtube.com/watch?v=NgA5EvOryU0](https://www.youtube.com/watch?v=NgA5EvOryU0)

