
Caching the uncacheable: CloudFlare's Railgun - jgrahamc
http://blog.cloudflare.com/cacheing-the-uncacheable-cloudflares-railgun-73454
======
mmaunder
Exec summary: This is not caching. It is reduced latency for your visitors by
removing connection setup over a large geographical distance. It also reduces
the latency your web server has to deal with to effectively zero, but you can
do that with Nginx. It also will reduce your bandwidth consumption from your
providers perspective to a fraction of what it was.

I think describing this as caching hides the real benefits. My startup pushes
150 Mbps on average so I care about this stuff.

Rather than being caching, this introduces a proxy server that is
geographically close to your visitors that then communicates with your server
using an efficient compression protocol. So, much of the network path that
would transfer a full payload is replaced by a compressed connection that does
not have to build up and tear down a TCP connection with a three way
handshake. The most important benefit I see here for site visitors is reducing
latency by removing the connection setup. They will spend a few ms setting up
the connection with a server a few hundred miles from them instead of on the
other side of the planet. That server then serves a cached page or uses an
established connection to only get the changes, which could mean as little as
a single send and single receive packet.

Another benefit of this is that your local web server will be talking to a
local cloudflare client which means there is practically zero latency from
your perspective for each request. This means that each of your app server
instances spends less time waiting for it's client to send or receive data and
more time serving app requests. It's why people put Nginx in front of Apache.

I think the most important cost benefit here is reducing your bandwidth
consumption. We're constantly negotiating our colo deal based on 95th
percentile and getting your throughput from 1 Gbps down to 50 Mbps which I
think this may do will drastically reduce your real hosting costs. Of course
Cloudflare need to maintain their servers and will be serving 1Gbps to your
customers but those cloudflare servers will be geographically closer to your
customers. However because data centers bill based on your throughput at the
switch and not how far your customers are away from you, I don't see that
there are any cost savings they (cloudflare) can pass on to you. They're going
to be billed what you were being billed for bandwidth, but they'll mark it up.
I suppose you could argue there are economies of scale they benefit from, but
that doesn't seem like a compelling argument for reduced costs.

~~~
jgrahamc
CloudFlare does not bill customers for bandwidth consumed. It's a flat fee:
<https://www.cloudflare.com/plans>

~~~
mmaunder
Holy cupcakes. I might use this. So I can use your free package which includes
the CDN to serve 100 Mbps of static JS and images?

~~~
jgrahamc
Yes. And we'll also send you some cupcakes.

~~~
mmaunder
I'm curious if I'm going to run into a catch. It seems unsustainable. Just to
be clear, that's 30.8 Terrabytes of data I'll be transferring per month on
your network for $0. Can someone verify this?

~~~
jgrahamc
I'm not on the business side, but looking at our global traffic 30TB per month
is a tiny percentage of what we are doing in terms of traffic.

I doubt it would go unnoticed, though, so I expect someone will be interested
in persuading you to get a Business account with us which is $200/month
because at that point we give you an SLA.

------
alt_
Delta encoding for HTTP has been proposed since 2002[0], but seems to have
been lukewarmly received.

A fully server-side solution will probably bypass most problem cases, but
might solve some, so I hope CloudFlare looks into contributing to a
distributed standard solution.

[0] <http://tools.ietf.org/html/rfc3229>

~~~
obtu
This deals with the protocol and leaves the diff tool pluggable (RFC 3284's
vcdiff is suggested), so it looks like something railgun could be using
internally.

Google's SDCH (mentioned in CloudFlare's post) is an alternative that isn't
tied to a single URL, but involves prefetching some data that may or may not
be needed, so it's a bit hacky.

An interesting approach would be to hook into a template engine to generate a
page skeleton with dictionary references. That would allow reordering the
small, newly generated varying snippets before the large common ones; both
could be preemptively pushed using SPDY to avoid round-trips.

~~~
jgrahamc
The really big problem with SDCH is that is places a burden on the web site
owner to generate dictionaries and to know when to generate them and how. The
protocol is one thing, the reality is another and that's why it has not been
widely adopted.

------
jgrahamc
And for those that care about this sort of thing: Railgun is about 4,000 lines
of Go.

~~~
wickedchicken
> Railgun is about 4,000 lines of Go.

Probably going to get downvoted, but [http://www.reactiongifs.com/wp-
content/gallery/yes/shaq_YES....](http://www.reactiongifs.com/wp-
content/gallery/yes/shaq_YES.gif)

------
reitzensteinm
I'm having a hard time wrapping my head around the benefit of this - maybe
John can elaborate some more.

Taking the CNN front page for example. If you set a TTL of 60 seconds, and you
have 14 edge locations (taken from the CloudFlare about page), you've got to
satisfy 1440 * 14 = 20160 requests a day. The CNN page is currently 97527
bytes, which gzips down to 20,346 bytes. That's 391 megabytes per day. Serving
the edge locations even with this relatively short TTL is trivial.

Now, the TCP connection means the content can be pushed and not pulled, so
latency is better. It also means that caching a lot of pages will become
cheaper (though still expensive).

But it doesn't seem like a lot of benefit for essentially replacing HTTP
(between the content servers and the edge nodes), for all the proprietary
software and vendor lockin that entails. For each byte you're sending to edge
nodes, it's going to be served up orders of magnitude more from the edge nodes
to end users, which seems like where almost all the cost would lay.

I'm sure they know what they're doing, and that their customers have asked for
this, but think some real world case studies would help make it click. That
the blog post is going on about 'caching the uncacheable' really doesn't help.

~~~
jgrahamc
Firstly, we're not 'replacing HTTP'. HTTP runs over Railgun in the same way
that it does over SPDY and the content servers are still running HTTP.

The big benefit is not in terms of bandwidth saved (for us) it's in terms of
total time to get the page. That's partly driven by latency and partly by
bandwidth. Because we have worldwide data centers we can see high latency from
say the data center in Miami and a web server located in Sydney. Railgun helps
with that problem.

Also, CNN has a TTL of 60s but many, many web sites have a TTL of 0 because
they want no caching at all (see New York Times web site) or because the page
is totally personalized.

~~~
marcusf
I'm still not sure how this solves the personalization use case. Usually (I
think) the main problem is rendering the page, not pushing it down the wire.
So if you still need to render for each request for a personalized page to
make sure it's not been updated, where's the gain?

I think I might be slow, but I'm not sure where the wins are here and how you
overcome the personalized rendering problem? I'd love for you to talk a bit
about it.

~~~
StavrosK
It seems that all the gains are at the point between the upstream server and
the CDN node. It has to send fewer packets for the entire page. However, I'm
not sure this isn't pretty much as efficient as a large TCP window. Of course,
I'm not the one whose job it is to worry about these things, so take this with
a grain of salt.

------
jacques_chester
If understand this correctly:

1\. You install "Railgun" on your publishing web server

2\. It pushes deltas of your webpage to CloudFlare

3\. Who then update their cache of your webpage across their CDN.

It seems as though folk are starting to see that in most environments caching
ought to be driven by POSTs, not GETs + timeouts.

~~~
judofyr
> It seems as though folk are starting to see that in most environments
> caching ought to be driven by POSTs, not GETs + timeouts.

HTTP supports that too (through ETag and 204 Not Modified).

~~~
jacques_chester
I meant in terms of server-side caching.

------
robryan
Would love to see some real world figures on the difference in page loads.
Intuitively it feels like a lot of overhead outside of the packets being saved
between the host server and cloud flare.

~~~
jgrahamc
I do have some numbers on load times based on testing we did with a hosting
partner where we took a small number of their sites and tested using Railgun.
We saw a speedup of between 2.77x and 4.78x on the page download time.

With a different hosting partner (and a different set of sites) we saw a page
download time speedup of between 2.94x and 8.12x.

------
beaker52
Pushing 0.5kb opposed to 93kb to a downstream caching proxy isn't going to set
the world on fire if you consider the data transfer speeds we typically
achieve these days, especially if you also consider the processing overhead on
both your network and the cache provider.

They're not really caching the uncacheable, either.

~~~
rdl
If you're on the wrong side of a really long high latency link, it should make
a big difference. Slow start, etc.

Leaving a persistent connection between the cache and the server obviously
does most of it (especially with latency mitigation tricks/tcp acceleration).
It would be interesting to calculate the benefits of nothing vs. a cache with
an accelerated persistent tcp connection vs. deltas. I suspect it's something
like 500 vs. 50 vs. 45, but every bit helps.

~~~
beaker52
The time to first byte latency due to long distance is going to be the same
whether the payload is 2kb or 100kb.

If you have servers with a very low throughput outbound connection to the
cache, then reducing the data transfered in this manner could be worth it. But
100kb over the wire at todays throughput is not going to add all that much
latency to the whole transaction. As you suggested, some figures would be nice
though.

~~~
rdl
First byte, yes, but won't you get the first 100 KB a lot faster due to being
effectively 20ms away from it, vs. 320ms, thanks to slow start? Maybe little
impact at 1KB, but for a 4MB abortion like some of the web pages I've seen,
...

(This all from TCP acceleration, not deltas, though. Deltas might give you 1ms
on a 1G link if it saves you sending 100KB. BFD. Deltas to the _edge_ , where
you might be constrained by bandwidth on a mediocre 3G connection, is where
deltas would rock - coupled with SPDY and tcp acceleration and caching and
we'd be living in 2015.)

~~~
beaker52
Remember that we're talking about content provider to content cache here.
Server infrastructure to server infrastructure, usually connected by huge
pipes.

Passing deltas to a client over a mobile data network would be an awesome
development, I agree. With some additional specification to HTTP and vendor
implementation on clients, it'd definitely be possible.

~~~
rdl
For some values of "huge". I was using 1Gbps as the link size, since that's
almost certainly the uplink on the server, and thus an upper bound on the
smallest link.

It would possibly be fair to use something closer to 155Mbps in a lot of
places (the most constrained part of the link; we're not even talking about
congestion/packet loss, which would exponentially favor this technique, and
which does happen on SP transit links sometimes). At that point, 4MB could
actually matter:

155Mbps = 20 MB/sec. 4MB takes 200ms to transfer; 100KB takes 5ms. If you
assume a single packet for the delta instead, I'd be happy to save 5-200ms.

~~~
rdl
Assume for the sake of argument that processing is free and instantaneous, and
that the cache is in-line with the normal network path (and much nearer to the
end user than the content server). This is arbitrarily close to true as
Cloudflare expands.

So you're actually saving these 5-200ms vs. non-cloudflare delivery. It might
still be 305-505ms total page load time, or easily greater (I used GEO sat for
a while, feeding cell; it was hell), but if it were 305 instead of 505 I'd be
quite happy actually.

(I'm assuming a 4MB page with 100KB which actually changes between loads. A
4MB all dynamic page, frequently reloaded, which can't be cached but where a
tiny delta is possible would be pretty pathological.)

With CDNs you can also get benefits from prefetch/prefill, either
opportunistically (due to multiple users hitting the same thing, or through
actual scheduled fill). Works for many media assets but not for dynamically
generated pages, which is what Railgun is supposed to address.

------
omh
As a small company, using CloudFlare as a cloud-based CDN/firewall/DDoS
protection looks very attractive.

The thing that has stopped me from trying it is that you have to move your
whole DNS to them though, and they have had downtime for DNS as well as their
hosting. I'm comfortable with serving www.example.com through them and dealing
with the occasional downtime. But I'm not so comfortable with downtime for MX
records - I really want mail@example.com to be super reliable.

~~~
luigi
I launched a startup in late March, using Cloudflare, and had many similar
thoughts. But it was an awful experience.

Because Cloudflare acts as a proxy, it gets in your way in subtle but
devastating ways. First, the SSL support wasn't stable and I turned it off a
few days into launch. That probably killed some traffic.

Then, I realized that caching was an all-or-nothing proposition. You can have
Cloudflare cache your assets, but it doesn't seem to respect expires headers.
You can set a cache time in the web interface, but the minimum is 2 hours! So
you end up creating a rule to have it cache no assets, at which point you
might as well use AWS Cloudfront for a CDN.

I advise you not to use Cloudflare. It's a great idea, but the execution just
isn't there.

~~~
luigi
Here's a thread with the bad experiences other HNers have had:

<http://news.ycombinator.com/item?id=3856004>

------
sirwitti
This looks like an interesting option for static content. But what still is
not cachable of course is every request that _has_ to hit a database (or at
least clear parts of a cache).

In my experience (e.g. a site with 50.000-100.000 pageviews/day) it's mostly
the requests hitting the db and not server throughput that are the bottleneck.

So for some people this could solve the wrong problem...

------
davedx
Pretty cool tech. I guess this can work in parallel with SPDY too. The
Internet is evolving a lot lately.

------
kierank
So basically this is Edge Side Includes reinvented?

~~~
eastdakota
Kinda, but without the pain of having to rewrite your pages, and with a much
higher efficiency. Actually have a blog post about this queued up to go out
tomorrow.

------
mycodebreaks
Do any other CDN providers have similar technology as Railgun?

~~~
cbsmith
Good question. The answer should be obvious, as CloudFlare presents this for
the innovative and ground breaking technology it is...

[end sarcasm]

So yeah, basically almost all of them have it. Most have had it for years. CDN
and various other business models (POP's, etc.) have been converging for a
while now, so this has become pretty standard for CDN's to provide.

~~~
ksec
Really? Well may be they are for the big boys, none of the commonly used CDN
advertise it, All Hosting providers reselling Level 3, Akamai, EdgeCast,
NetDNA or even Amazon Cloudfront dont show it either.

I wonder why these aren't available to normal users.

~~~
cbsmith
Not sure about Amazon and NetDNA, but the rest certainly offer it.

------
bigiain
I wonder if there's a way to replicate/simulate this with a longer ttl on the
pages served out of your cdn, and a bit of javascript that pings a lighter
weight server only asking for deltas, and updates the page in place (or
reloads the entire page from the cdn when the deltas get too big).

------
bluelu
So how does this work if you want to count the number of visitors, delivering
customer specific versions of the site etc...? Does the bbc example even work
because I can imagine that there multiple versions of the site depending on
which IP is accessing BBC

~~~
StavrosK
I'm guessing the visitor IP still gets passed. This looks to be entirely
synchronous, so everything would still work. The CDN just only has to fetch a
delta of the page rather than the whole.

------
zobzu
so railgun is internally sending diffs. I would have thought caching companies
were doing that already, but it seems I'm wrong.

That being said, I'd have used another name ;à

------
hucker
This looks rather interesting... Does any fully server-side solution such as
varnish have any similar functionality? Seems like a no-brainer.

~~~
gizzlon
Well, since those caches run on the same LAN as the webserver, the cost would
probably outweigh the benefits.

You can still break up your page into pieces and cache all but the once that
change. Believe this is one of the reasons you would use ESI:
<http://en.wikipedia.org/wiki/Edge_Side_Includes>

------
machbio
hope they make it available for CloudFlare Pro accounts too.. wait to go
cloudflare ..

------
ta12121
Differential compression? Isn't this what Opera Turbo and other web
accelerators do?

------
ldawoodjee
Tl

