
SPDY performance on mobile networks - Garbage
http://googledevelopers.blogspot.in/2012/05/spdy-performance-on-mobile-networks.html
======
jsnell
Measuring the speed of web page loading on mobile networks is something that
we wrestle with weekly (we make a TCP accelerator product for mobile
operators), so I always love reading about new approaches and results.
Unfortunately this particular one seems to have particularly dodgy
methodology.

Most importantly, they did in fact not measure anything on mobile networks or
even simulations of mobile networks. Instead it was a just a crudely throttled
wired connection, which is a poor approximation for how a 3G network behaves.
The right way to deal with changing network conditions is to actually do lots
of measurements. This is particularly problematic since one effect of parallel
HTTP connections is to isolate the congestion control effect of a random
network event to just one of the 6 connections to a server.

Second, they are not fetching the actual web pages, but getting all the copies
from a local cache. This totally removes the point of testing against multiple
web sites. In the real world there's huge variability in things like web
servers and TCP stacks / options.

Also, if I understand the replay tool correctly, it's not accounting for the
network latency on the inet side of the test setup at all. I'm not completely
sure whether this is benefiting SPDY or HTTP though (on one hand SPDY suffers
from the latency at the start since it can't accelerate the single flow as
quickly during slow start, while HTTP suffers from the latency). But it
certainly removes the very common effect of there being a one or two outlier
resources being loaded from slow servers.

Removing DNS overhead completely is a bit strange too, and would have the
effect of magnifying any relative speedups / slowdowns.

~~~
zokier
> Second, they are not fetching the actual web pages, but getting all the
> copies from a local cache. This totally removes the point of testing against
> multiple web sites.

Disagree. Different websites have different structures of content. eg some
sites may rely heavily on external resources, while another has everything
inlined, etc.

~~~
jsnell
Fair enough. Let's just say that this method removes some major sources of
real-world variation between sites.

------
chrisacky
Incidentally, for anyone interested in support for nginx, it should hopefully
be available by the end of this month.

<https://twitter.com/#!/nginxorg/status/192301063934705665>

------
zokier
I don't see the point of using a phone for measurements if the phone is
tethered to computer. Also more interesting research would be measuring
performance improvement vs degradation of network.

~~~
mdwelsh
We wanted to use a real phone to accurately capture the performance of the
phone hardware, browser (Chrome), and OS (Android). For example, JavaScript
runs much more slowly on most mobile platforms than it does on desktop
browsers, and we wanted to ensure our results were not skewed by that.

