
BBC Digital Media Distribution: How we improved throughput by 4x - woolie
http://www.bbc.co.uk/blogs/internet/entries/17d22fb8-cea2-49d5-be14-86e7a1dcde04
======
phkamp
Varnish author here.

I'm several sets of information short of being able to say anything
particularly precise about their results.

Their diagnosis is very detailed and useful, but I think the main thing we can
learn from it is that the Linux kernels VM system is horrible when memory is
overcommitted. At least to me that's not news.

I will also say that we in the Varnish project have not worked much on
performance the last couple of years, because people have asked much more for
features than performance.

Some of the stuff we have done have traded CPU cycles for flexibility, but if
the Varnish users change their mind and decide performance needs a boost, we
can do that too.

The reason we dropped sendfile(2) in Varnish was that nobody seemed to be able
to show any credible performance improvement, and dropping it allowed us to
implement on-the-fly processing of the HTTP bodies in a cleaner way.

We can certainly bring sendfile(2) back for "the straight" path if that's
important to people.

The "file" storage was our first rather primitive storage engine and it has
never been optimized to any extent. Varnish Software Inc. has a commercial
offering that includes a "massive storage engine", but most people seem to
just use the 'malloc' storage.

And no, Varnish is not dying just because BBC doesn't use it in the absolute
far corner of high-end usage doesn't use it: If NGINX is the better tool for
them, they should absolutely be running NGNIX.

------
gpvos
Direct link without two tracking redirects:
[http://www.bbc.co.uk/blogs/internet/entries/17d22fb8-cea2-49...](http://www.bbc.co.uk/blogs/internet/entries/17d22fb8-cea2-49d5-be14-86e7a1dcde04)

------
olavgg
This is really interesting! But keep in mind, Varnish is blazing fast if you
manage to fit everything in memory by using malloc. Also Varnish is more
configurable than the Nginx cache. For example if you need to serve different
content on the same url based on the Accept-Language header or a cookie, you
can do this easily with Varnish by creating a lookup hash of the url + header
value in VCL.

~~~
poooogles
Came here to say this, feels as though they maybe didn't do Varnish justice in
some ways.

Kudos on the huge improvement though, it's the results that matter not the
means.

~~~
Veus
Seems to be that they did, but Varnish didn't suit their needs as they
couldn't fit all the data in RAM. It's OK to change what you use because you
outgrow it.

------
ris
Not that it's a bad article (lots of nice detail in there), but the short
version of this is: if you're web-serving disk-scale things, use a web server
that's designed to serve from disk. If you're web-serving memory-scale things,
you can use a memory-based web server. Even if you think you're just
"caching".

------
eddieroger
Interesting bookend to this [1], which felt like the best possible endorsement
for Varnish I'd seen, and was a driving factor in me building it out where I
work. As others have pointed out, it seems more like an issue of being the
wrong tool for the current needs, and not so much a problem with Varnish
itself.

1\. [http://www.slideshare.net/grahamlyons/varnish-at-the-
bbc](http://www.slideshare.net/grahamlyons/varnish-at-the-bbc)

~~~
tomcart
Its worth pointing out that the BBC continues to use Varnish in the way Graham
described in that presentation.

As you say, its about finding the right tool for the job.

------
anc84
Actual URL:
[http://www.bbc.co.uk/blogs/internet/entries/17d22fb8-cea2-49...](http://www.bbc.co.uk/blogs/internet/entries/17d22fb8-cea2-49d5-be14-86e7a1dcde04)

------
majke
Interesting line of reasoning. Definitely mmaping 8.6TiB into user space
process virtual memory is gonna take a toll.

I tried to find why varnish doesn't use sendfile: [https://www.varnish-
cache.org/lists/pipermail/varnish-dev/20...](https://www.varnish-
cache.org/lists/pipermail/varnish-dev/2008-May/005891.html)

    
    
      By default [Varnish uses sendfile] never; there are problems with all sendfile() implementations

~~~
jonhohle
In the past I've found that sendfile is not portable. FreeBSD and Mac OS X
require the file that is being sent be a regular file[0] while Linux will take
any file descriptor[1]. That being said, I can't imagine what benefit
memmapping would have in the context over the file system page cache.

0 -
[https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2](https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2)

1 - [http://linux.die.net/man/2/sendfile](http://linux.die.net/man/2/sendfile)

------
toopok4k3
I'm surprised about the 6-year-old kernel on a server with those specs.
Apparently its still maintained until early next year. Anyone know if that
might play a factor on the benchmarks?

EDIT: nvm they did try other kernels but it only made minor changes in
results.

------
darkr
> The RAID 50 array appears to be the current bottleneck.

Any form of parity RAID is always going to be a bottleneck, and should never
be deployed into any production environments this side of 2010.

If your application can gracefully handle IO balancing (and potentially
replication) between individual block devices, and handle them dying in all of
the weird and wonderful ways that they do, then go with JBOD. If not, then
your choices are a mirror (E.G RAID1), or striped mirrors (E.G RAID10).

~~~
woolie
Yep, we went JBOD and saw the performance win as expected :)

> dying in all of the weird and wonderful ways that they do

It's only been a few months, but there have been plenty of great surprises
thrown out by those disks. Both in the ways that they've failed and how the
kernel has handled those failures!

------
elcct
That sounds like having wrong organisation of data and then using wrong tools
to do the job. Interesting article though.

------
visarga
What I want to hear from BBC is not how fast they can make a web server, but
what are they going to do about those 1 million hours of archive TV and radio
content they are sitting on. It's a shame for that treasure to be locked in
copyright limbo, while YouTube collects 100 hours of cat videos per hour.

~~~
Ntrails
I used to record Jools Holland onto VCR every week, and then make mix tapes of
my favourites. _I would happily pay 99p a track for my favourites_ \- and feel
like a winner.

Simlarly with live lounge sessions. We get a selection of what some pratt at
the BBC thinks were the best of the year in a compilation CD. WHY NOT JUST
SELL THEM ALL?

I have been bitching about this for years, how has it not been made a thing?

~~~
johneth
I think the problem with music broadcasts (TV or radio) is that they'd need
permission of the artists and/or publishers before they could sell it
commercially. It's probably for similar reasons that they don't show all the
Glastonbury stuff (mixed in with revenue sharing - some artists/publishers
might want too much, for example):

> "On occasion an act may not wish to be filmed or recorded. Artists may also
> agree to be recorded but only allow a limited number of songs to be aired.
> This could be for a number of reasons for instance: the quality of some
> parts of the performance, because they do not wish to broadcast new or
> unreleased material, or they do not want to broadcast their entire live
> set."

I do agree that it'd be a good idea to do, though.

~~~
Ntrails
I mean, obviously artists could opt out (and tracking them all down for back
catalog perms would be a nightmare - I'm not without sympathy for the effort
involved) but going forward could they really not set up a default agreement?
The fact that they DO release some stuff commercially seems indicative it's
entirely possible.

Glasto is indeed another great example of audio tracks I'd gladly pay for -
funnily enough though by the time the BBC takes it's coverage down some
valiant internet person has stuck what I want up on torrents.

