
iPhone 5 web teardown: How Apple compresses video using JPEG, JSON, and canvas - dbloom
https://docs.google.com/document/pub?id=1GWTMLjqQsQS45FWwqNG9ztQTdGF48hQYpjQHR_d1WsI
======
jasonkester
Nice to see they're getting bit by their own decisions.

They've gone out of their way to ensure that you can't ever play an
Audio/Video clip automatically on page load in iOS Safari. Every new iOS
release for the first few years included a patch to kill any new workarounds
that let you do so. (Curse you iOS 4.3 for taking away our simulated clicks.)

But now they have a use case of their own that needs it, so they invent the
mother of all workarounds. And now the rest of us will start using it. And it
will become the "standard" way to run video on iOS Safari.

Then they'll kill it off for iOS 7. Then they'll have to come up with an even
crazier workaround so that they can render their own website a few months
later.

Fascinating to watch this play out.

~~~
batista
> _They've gone out of their way to ensure that you can't ever play an
> Audio/Video clip automatically on page load in iOS Safari_

Which sounds very wise from a user perspective.

Imagine every crappy web advertiser auto playing videos on page load on Mobile
Safari. Yuck.

I don't want videos to auto-play on my Laptop, why would I want them on my
phone?

> _Curse you iOS 4.3 for taking away our simulated clicks_

Simulated clicks? I rest my case...

~~~
maayank
What do you do when that happens on your laptop? You close the page and move
on. Why does it need to be any different on your mobile/tablet? I assume the
battery cost of 2 seconds of video loading/playing once in a long time is
negligible.

~~~
Uchikoma
I usually do not use my laptop in a restaurant.

------
codeka
I'm not sure if I should be impressed by this. From a technical standpoint, it
seems quite clever and unique, but it also seems like a lot of work just to
avoid using a <video> tag.

~~~
borplk
Yeah it's weird. My web developer side is impressed by the technical detail
that go behind it. But my other web developer side asks why the f*ck should
achieving a simple video effect on a web page should be this hard in 2012?

There is a whole pattern of similar things like this in web that is killing me
a little bit every day.

Want to center a div on the page? No worries just wrap it in these 5 divs and
apply these 20 css properties and you are good to go. Oh btw, IE doesn't like
that, you need this.

Honestly I think we developers should stop putting up with crap like this, and
accepting the burden of broken technologies using hacks, etc...

If we become less tolerant of crap, there will be less crap. As long as we
accept it by using workarounds, etc... there will be more shitty things to
come.

~~~
rglover
What's difficult about centering a box? Was that just sarcasm? A couple of
approaches:

Margin Auto w/ a fixed width

Absolute 50% w/ negative margins

Text-align center parent w/ inline-block display on children

If it was sarcasm, disregard my stupidity.

~~~
borplk
That's still non-intuitive. Just do a search about centering (or X problem in
CSS) and see the thousands of threads and questions and corner cases, hacks
and exceptions.

The whole thing feels like you are stretching something beyond what it was
intended to be used for.

Things like element positioning should be so so simple that if you search
about it, you get nothing back because no one had a problem to ask for
solutions.

Although I must say I'm not mocking CSS in particular, most web technologies
feel this way because they were designed many years ago for a web that was
supposed to have simple 'documents' like an article, text and paragraphs and
today each component of it is expected to do a lot more.

It's not getting all that much better either because each new version and
improvement is building on top of the previous decisions, etc...

~~~
rimantas

      > most web technologies feel this way 
    

Most technologies you don't know feel this way. And the reason—you don't know
them.

    
    
      > It's not getting all that much better either because each
      > new version and improvement is building on top of the
      > previous decisions
    

In this case: no.

------
dangrossman
> Apple's website needs to work on all major browsers

That doesn't seem true. The second-most-prominent links on the iPhone and iPad
sites are to videos that only play in Quicktime. If they wanted the site to
work everywhere, they'd use a format those browsers can natively play.

~~~
quink
The videos linked to from the iPhone and iPad pages, then they're video tags
these days. If you're talking about the movie trailers or whatnot, then Apple
isn't trying to sell you Quicktime, but it's damn well trying to sell you an
iPhone or iPad.

~~~
josteink
_If you're talking about the movie trailers or whatnot, then Apple isn't
trying to sell you Quicktime, but it's damn well trying to sell you an iPhone
or iPad._

Using a technology you probably already have to own an iPad or an iPhone to
have installed no less.

------
jp_sc
Sublime Text was already doing the same with the big animation of its home
page: <http://www.sublimetext.com/> (you can actually see three animations
there, depending if you use Windows, Linux or Mac).

What I'm really curious about is how those images are being generated. Is
there a tool already available for that?

~~~
dbloom
It doesn't look like Sublime is using the same technique as Apple, but it does
look really interesting. It looks like their system allows for better reuse of
images, and it is a lot simpler too. It might be less optimal though for
frames with a large number of small differences, since they just use plaintext
encoding for that stuff.

Sublime's example does highlight a solid usecase for this technique that
Apple's example doesn't really emphasize: <canvas> \+ JS video compression is
still the only way to do this stuff losslessly, which is pretty important for
what Sublime is doing.

------
Jayasimhan
This whole implementation could just have been someone's weekend hobby that
made it to production. Remember how Apple got its intel based macs?
<http://qr.ae/8eDNG>

Just want to mention it before we blurt out the obvious over-engineering
argument. ;)

------
kevinburke
Speaking of Apple website updates, I am surprised that they have never offered
a responsive design for their apple.com site where you buy iPhones and iPads.
Every app they make is designed differently for the phone or the iPad, but not
the website.

~~~
spullara
Since they have an Apple Store App that works better than the website, why
would they do that?

~~~
justinph
You're 100% right, of course, it is in apple's best interest to have you use
their app.

It's just that downloading a bulky app for something you'll use infrequently
is a crappy way to to business.

~~~
r00fus
The App does more than just let you buy product. You can reserve for pickup,
use EasyPay, set appointments at the Genius bar, etc.

That said, perhaps they could integrate the two a bit better.

------
dflock
It's a shame webkit never implemented the Animated PNG extensions and that no
browser supports MNG:

<http://en.wikipedia.org/wiki/APNG> <http://caniuse.com/apng>

This would probably some of what people want here.

------
tisme
Maybe they should have used flash instead. That would have given them a higher
percentage of successful deliveries than this solution.

~~~
chimppp
Wow, took like 200 comments for someone to mention Flash, the thing that's
been doing this exact same effect in a 20kb file since the year 2000 without
having to write your own codec. Shows how well the Apple anti-Flash campaign
has brainwashed the crowd.

------
fmntf
If someone is interested in creating animations in this way, I created a
simple script which converts videos in PNG/JSON/canvas:

<http://blog.webmatters.it/2012/09/canvas-rocks-video-sucks/>

------
wullon
Very interesting. Looks like some oldschool video game developers got into
Apple webdev team.

About the last paragraph, we (Adinpsz) tried the PNG compression technique for
JS demomaking (actually it's even a self-loading PNG-HTML ;)). You can learn
more here with the JsExe tool: <http://creativejs.com/2012/06/jsexe-
javascript-compressor/> And see it in action here:
<http://pouet.net/prod.php?which=59071> (<http://adinpsz.org/online/fabrik/>)

But HTTP compression should be better anyway for "real world" usecases?

------
mmmmax
Keep in mind that Apple outsources a lot of this front-end work to their
agency, TBWA/Chiat/Day Media Arts Lab. As a services firm they have less
incentive to use standards created by Apple, though I think they do really
great work.

~~~
rlu
I was just about to comment asking whether Apple engineers actually did this.
For sure this is some cool stuff, but I think it'd be a bummer to say "I work
at Apple" and then in response to "oh cool, what do you work on?" say "I work
on the iPhone -webpage-"

~~~
alanfalcon
Why?

I worked at Blizzard. On the Blizzard web pages.

~~~
trafnar
I've always been impressed with Blizzard's web pages. It's the type of style I
typically don't like, but it is done really well and appropriately. I often
used it as an example in my classes to show how to do very graphical websites
the right way.

So, good work! I think you're cool :)

------
zachwill
Nice teardown. Knew they were using a <canvas> tag, but didn't dive deep
enough see how they were using Base64 and decoding the frames. Thanks for
writing this up.

------
xyz_ak
Very interesting and kind of absurd. Why didn't they simply recreate the
animation with moving sprites instead of taking the "video" approach?

------
Flenser
Isn't this the same as Clipstream[1]? The company that developed it has filed
for patent[2]. I wonder if Apple licensed the technology or came up with it
themselves independently.

[1] <http://news.ycombinator.com/item?id=4419345>

[2] <http://www.dsny.com/HTML5>

------
firefoxman1
After reading this article I still don't have a clear idea of how they did the
animation. It seems pretty in-depth technically, but the overall execution is
still a mystery.

For example, how does the _frames array of base64 data related to the two
unlock_00X.jpg files? All it says is "The "unlock_manifest.json" file
specifies how the updated parts are positioned."

~~~
dbloom
The image data is read from the JPEG files in order as a series of 8x8 pixel
blocks (in left-to-right then top-to-bottom order). Each instruction is
basically "draw the next {{length}} blocks from the JPEG files to location
{{location}} on the canvas".

Surprisingly, there is no mechanism to explicitly state which blocks to get
from the JPEG. Instead, it always just uses the next unused blocks. (This
means that blocks from the JPEG cannot be reused.)

------
n-gauge
Regarding the retina loupe effect demo this has been done a while back - using
css and an image for the loupe edge. Picks up mouse and touch events. Even
works on older browsers.

[http://homepage.ntlworld.com/infinnerty/differences/attic/at...](http://homepage.ntlworld.com/infinnerty/differences/attic/attic.html)

------
bencevans
This idea is also in use at <http://www.sublimetext.com/> The demo is using
this technique because streaming video would actually take more bandwidth in
this case as only bits of the image are actually changing at one
time/incremental changes. It works a treat!

~~~
Jabbles
Highly unlikely that a real video format would take more bandwidth if encoded
properly. Video engineers have thought of this problem.

H.264 would emit "skip" macroblocks for everything except the area that was
changing (probably near the cursor). The repetition of "skip" would be easily
compressed.

------
nachteilig
I really like that Apple is willing to experiment and push the envelope on
their corporate site. Very refreshing.

------
xentronium
> You can see this in action on the Retina Macbook Pro "Features" page --
> which loads about 5MB of JPEG images (using lots of separate HTTP requests)
> just for that 2 second effect.

I immediately thought of web1.0 sites of 1990-s, with lots of animated gifs
floating around. Weird times.

------
BigBadaboom
I see a lot of comments describing this as new and unique. Hardly. Video
compression by using image diffs and run-length encoding has been around since
the 80s. In fact this is more-or-less the technique that Quicktime's first
codec, "Cinepak", used.

------
squarecat
OK, so this would be one of those instances where I support a headline rewrite
for clarification. The iPhone connection is incidental and I expected
something entirely different.

This was more along the lines of, "How Apple reinvented/over-engineered the
animated GIF"

------
brown9-2
As a side note, publishing this on Google Docs is an interesting choice by the
author - alleviates any need to worry about load from the piece becoming
popular, but it almost makes it an anonymous article.

~~~
dbloom
I was going to use Blogger, but multi-login bit me. I had already written the
draft in Docs, so I just published that. (I also like how plain published
Google Docs look)

I went ahead and added a really short byline though.

------
aniketawati
There could another simpler way. NoVnc + tightpng encoding. Create and record
a high framerate VNC session to desired video. Tweak no vnc to receive data
from local callbacks and replay the recorded data.

------
federicoweber
This is brilliant, maybe overpowered but brilliant. But the thing I cannot
stand is the use of images for text. I would have preferred to se the use of
@font-face with fallback to images for old browsers.

------
bkorte
Man, I wish there was a library to help us normal folk do similar things. The
ability to do small video on mobile would rock.

~~~
Kell
You know, that this isn't exactly a video... If you really need to play a
small video like that... try using gifs. It's easier, and there is only two
drawbacks : 1- The stumbling loading of frames. 2- You can't do the thing they
do with the earphones, going back and forth in the "video". If you're fine
with these two drawbacks... You probably don't need all that over-engineering.

~~~
hwyfour
Well, there is a really hackish way to do this, actually. You could preload
the gif and then use something like <http://slbkbs.org/jsgif/> to jump around
the gif frames.

------
tambourine_man
I doubt iOS is the reason they went through all this trouble. The performance
even on the iPhone 4S is miserable.

------
thomasfl
Still none of the pages on apple.com are made for small mobile screens. Only
exception is the iPhone manual.

------
mihaipocorschi
Apple has a way of hiding new web tech in plain sight. Otherwise very good
write-up. Thumbs up.

------
foooooooo
when you browse istore from an android device they added programmatic wiggle
and jerk when you scroll the screenshots for an app. It's hilarious because
android jerking wasn't that consistent. to make it authentic they needed to
add randomness to it.

------
droithomme
Great analysis, very interesting.

Obviously this is absurd overengineering.

And, despite the assumed total cross-browser compatibility claimed, I found
that the video wouldn't play in any browser available to me, despite pinning
the CPUs on my computer while just sitting there doing nothing.

------
effinjames
this is so useless

------
fidz
In short, Apple don't want their demo videos to be easily downloaded and
viewed?

~~~
lnanek2
Article claims the reason is so that the videos will play on iPhone. If Apple
considers people who already own an iPhone most likely to view the page and
buy the newer one, it makes sense that they'd go through a lot of work to show
them animations they couldn't through the video tag.

Preventing download and viewing doesn't seem very important to me. Anyone
could just use one of many screen recording apps to get good enough copies
anyway if they wanted them for some reason, and who cares if they do have
them.

