
“SPDY does not clearly outperform HTTP over cellular networks” [pdf] - jsnell
http://conferences.sigcomm.org/co-next/2013/program/p303.pdf
======
mdwelsh
I did the original study at Google on SPDY over mobile networks, which was a
small-scale lab study. Since then, we have collected data from millions of
Chrome Mobile users which shows that SPDY does tend to outperform HTTP on
cellular, but of course there are many factors involved - the content of the
site, the network conditions, etc. I haven't had a chance to get this data
into a form that can be shared, which I really need to do. Still, the authors
of this CoNext paper did a really nice study. I hope we have more to share on
this in the future.

~~~
igrigorik
Some recent SPDYv3 vs HTTPS performance numbers from various Google
properties: [http://blog.chromium.org/2013/11/making-web-faster-with-
spdy...](http://blog.chromium.org/2013/11/making-web-faster-with-spdy-and-
http2.html)

------
jbb555
I'm not a fan of SPDY at all. HTTP was simple to implement and understand.
This is neither. And it matters. The small gains in performance are not worth
the complexity.

~~~
FooBarWidget
SPDY is just HTTP, just like TLS is. TLS does not replace HTTP: it wraps HTTP.
Same thing for SPDY.

As a developer you never deal with SPDY. You just deal with HTTP and HTTP
headers. SPDY is just a configuration option in your web server. There is no
complexity to deal with, it just works.

~~~
stormbrew
TLS is not HTTP. TLS is a layer below HTTP and largely a black box to a client
or server implementer (as it should be, as it's very hard to get right).

SPDY is a replacement (or, in the inevitable future where http/2.0 effectively
is spdy, an upgrade) for http. There is no way to write an http client or
server without wildly divergent code paths.

* Note: When I say client or server I mean at a low level. I don't mean an app server running on a library that implements the http part.

~~~
nostrademons
How many people write Internet-facing HTTP servers, though? Usually even if
you're writing a webapp framework, appserver host, or HTTP library, you'll
stick Nginx in front of it.

There are a lot of security and DOS concerns that you have to worry about when
exposing a server to the public Internet - it's usually best to let mature
software handle that and proxy to your custom stuff, particularly when the
mature software is already open-source.

~~~
stormbrew
How many people write anything? This kind of reductive reasoning would
basically leave everyone in the world writing software that can fit nicely in
a UML model generator.

It's entirely reasonable to ask if complexity is worth it on every level. Many
of the greatest successes of internet protocols have come through making
things as simple and modular as possible. This doesn't mean SPDY is terrible
necessarily, but "no one ever needs to work with it" is a clearly wrong answer
to a criticism that it's overly complex.

~~~
nostrademons
I guess my point is that as projects mature, there are usually fewer serious
options on the market, and those options do more, and so it makes sense to
capture a lot of the complexity inherent in the problem within those few
pieces of software rather than force all users of that software (which are
numerous) to deal with it.

Perhaps there are simpler solutions to the problem, but if they make things
slower for end-users, that's inconveniencing millions to save effort for
dozens, which is not a particularly good trade-off.

------
lxyu
Twitter seems to have an opposite opinion:
[https://blog.twitter.com/2013/cocoaspdy-spdy-for-ios-
os-x](https://blog.twitter.com/2013/cocoaspdy-spdy-for-ios-os-x)

> However, we have measured as much as a 30% decrease in latency in the wild
> for API requests carried over SPDY relative to those carried over HTTP.

> In particular, we’ve observed SPDY helping more as a user’s network
> conditions get worse.

------
salient
Isn't SPDY encrypted by default? Why would you assume it _should_ outperform
the unencrypted HTTP? Is it because Google made that claim, or why?

Either way, security at the application level on the Internet seems quite
broken. We need easy to implement strong security at the Transport and IP
levels. Google (or IETF, rather) should be experimenting with a CurveCP-like
protocols to encrypt all the packets on the Internet to replace TCP. That's
what I'd really like to see.

~~~
gsnedders
The problem is plenty of boxes (esp. consumer-grade NAT stuff!) drops any IP
packet that is not TCP or UDP, hence introducing anything at that layer is as
hard as introducing IPv6 (i.e., it might happen, eventually, maybe). This is
why Google's QUIC is layered _on top of_ UDP — as UDP is a thin enough layer
(on top of IP) it doesn't add too much bloat.

------
blueskin_
I really don't see the point of stuff like SPDY and WEBP/WEBM. They're just
google doing what Microsoft did in the 1990s, setting their own proprietary
'standards' instead of working to improve things.

~~~
matthewmacleod
_I really don 't see the point of stuff like SPDY and WEBP/WEBM._

The former is generally more performant that HTTP, and the latter also offer
benefits versus other options.

 _They 're just google doing what Microsoft did in the 1990s, setting their
own proprietary 'standards' instead of working to improve things._

This is dangerously wrong, for two reasons:

1\. These are open standards. They're not proprietary. 2\. What is "improving
things" if not proposing open, alternative standards which solve existing
problems?

~~~
est
you know what else offered more benifits?

WMA over MP3, WMV over AVI.

~~~
BitMastro
And both are proprietary.. moreover you're mixing formats and containers

------
justinsb
I think QUIC is supposed to address this:
[http://blog.chromium.org/2013/06/experimenting-with-
quic.htm...](http://blog.chromium.org/2013/06/experimenting-with-quic.html)

~~~
youngtaff
My understanding is QUIC is being used as a proof of concept for protocol
experiments - it may be pushed towards standardisation or the the lessons from
it may be pushed into other protocols

~~~
dragonwriter
Isn't that how SPDY started out?

~~~
youngtaff
It is but talking to some of the Dev Advocates at Google, they're pretty open
that it could go either way - some of the things the QUIC guys have learnt
about speeding up TLS negotiations is going information HTTPS stuff (can't
remember the exact details)

Where QUIC may come into it's own is as as a replacement protocol for things
like WebRTC.

------
josteink
It doesn't matter if SPDY offers and technical improvements or not to our
good, robust and lasting protcols HTTP.

It's Google who is promoting it and all the "hip" hackers will rally for its
cause.

~~~
matthewmacleod
That's a _non sequitur_ \- SPDY shows markedly improved performance over wired
and 802.11 networks. The underlying problem discussed in the paper is down to
TCP's interaction with cellular networks, and just happens to particularly
punish SPDY somewhat more than HTTP.

That's something that can be worked on and indeed there's a recommendation for
improving it. I'd hope something like this would be included in HTTP/2.0, or
at least in a subsequent revision.

~~~
josteink
Since SPDY was created with DSL MTUs in mind[1] and optimized for them _only_
(unlike plain HTTP which was transport neutral and lasted almost 20 years) I
don't find it hard to believe that if you look _anywhere_ outside Google's
test-matrix you will see the benefits of SPDY tapering away almost instantly.

[1] Basically, the test was: will one web-request (including all of Google's
tracking headers) to google.com or any of its adwords/analytics beacons exceed
one TCP packet and cause possible TCP reassembly-issues or not when using
regular consumer-grade DSL MTUs?

~~~
username223
If they want to save a bit of bandwidth, how about removing this HTTP header?

> P3P: CP="This is not a P3P policy! See
> [http://www.google.com/support/accounts/bin/answer.py?hl=en&a...](http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657)
> for more info."

I can't believe they're still adding that...

------
d0ugie
Hey Ilya: Is it possibly in the works to merge QUIC's UDP and magical error
correction into whatever SPDY doesn't have that QUIC could complement it with,
maybe eventually abandoning QUIC and just going full-force on a further-
supercharged SPDY?

Thanks for lowering my bounce by the way..

~~~
igrigorik
Short answer, yes. Longer answer, there are two ways QUIC can succeed:

a) QUIC makes headway on its own, delivers significant perf win, moves to IETF
and becomes a standalone protocol in the long run. b) TCP and TLS leverage
techniques & lessons learned from QUIC.

Either way, the users will win. It's too early to tell which of these paths
it'll take.. But I do know that both TLS and TCP groups are paying close
attention to QUIC and there are already discussions on how to improve
performance based on some of the ideas being prototyped in QUIC.

------
tinco
Why did they take the effort to produce an entire paper, over a research
question that has a simple logical answer?

The research question (sort of): Is SPDY, which was designed to make HTTP work
better on connections with more bandwidth and not worse on less bandwidth,
increase performance on connections with less bandwidth (and not really
related more latency)?

Their answer (sort of): no, as designed SPDY only increases performance on
connections with more bandwidth, and performs similar to HTTP on smaller
bandwidths.

Their conclusion (sort of): TCP is not very suitable for transmitting data on
high bandwidth (relatively) high latency connections, we should use a
different transport layer protocol.

What your reaction should be (sort of): No shit sherlocks, that's exactly what
everyone thought 15 years ago and only now the idea is picking up steam, with
HTML5 including SCTP in the spec, and Google working on the QUIC effort.

~~~
jsnell
You're wrong on all counts.

First, the results aren't obvious (nor the underlying reasons). In fact Google
published results a couple of years ago that got massive amounts of coverage
suggesting the opposite ([http://googledevelopers.blogspot.com/2012/05/spdy-
performanc...](http://googledevelopers.blogspot.com/2012/05/spdy-performance-
on-mobile-networks.html)). I've worked on TCP optimization of mobile networks
for the past 3 years, publicly criticized Google's study when it came out, and
still the results of this paper are somewhat surprising to me.

Second, the underlying issue isn't the latency or the bandwidth of the
cellular connection. It's the unpredictability. In fact, the Google study was
using dodgy methodology where they assumed that a cellular connection could be
modeled simply by bandwidth throttling + delay.

Third, it's absolutely not true that everyone has been thinking that replacing
TCP is the solution, even if its problems in the cellular context have been
acknowledged. If anything, the opposite, nobody has been thinking of it as a
viable solution before QUIC. The deployment problems have just been too big --
but Google is in a unique position to fix that.

~~~
tinco
Ah, I did not know Google was actively positioning SPDY as a good solution for
mobile networks, I missed that.

I sort of want to defend that I counted the unreliableness of cellular as lack
of bandwidth, but you're right if that's really wanted to say then I should
have said it much differently. But do you really find it surprising that in
the light of unreliability SPDY would not perform much better than plain HTTP?

To your third point: I wasn't saying everyone said it was a viable idea.
Instead I say that only recently, with SCTP in HTML5 and QUIC people are
beginning to think it might be a viable idea. But everyone has known for years
that TCP just isn't suitable for the modern web, and if only we could, we
would have substituted it with SCTP long ago.

~~~
jsnell
There are factors favoring SPDY, such as fewer roundtrips thanks to
pipelining, or header compression removing the uplink as a bottleneck. There
are factors favoring HTTP, such as being less vulnerable to head of line
blocking, higher effective congestion/receive windows, and a smaller reduction
on the effective congestion window from a single packet loss.

Without testing, it's not at all obvious which set of benefits is actually
more significant in the real world. But if I'd had to guess, I would have
expected a reduction in roundtrips to be the dominant factor. So yes, I'm a
bit surprised.

~~~
tinco
Alright, makes sense :) Thanks for responding.

------
kdot
Anyone else find it strange that they tested 3G (UMTS) instead of LTE?

~~~
rprospero
I'm in an American city with a population equivalent to Reykjavik. I spend
about 80% of my time on 2G and the remainder on 3G. The number of times I've
seen LTE I can count on my fingers.

~~~
dded
That'd be ~120,000 (according to Wikipedia). In case anyone else didn't know
that off the top of their head.

A better analogy for Americans would be Springfield. You can almost just pick
one: IL, MA, MO (OH is a bit too small).

