
How to build your own CDN using BIND, GeoIP, Nginx, Varnish - nreece
http://blog.unixy.net/2010/07/how-to-build-your-own-cdn-using-bind-geoip-nginx-and-varnish/
======
pquerna
If you want a non-patched server, built for this task, consider pgeodns:
<http://geo.bitnames.com/>

It is used by svn.apache.org, CPAN, the NTP pool, etc.

I personally have my doubts about any value created for static pages doing
this though, there are relatively cheap CDNs on the market now, rather than
trying to find VMs in a dozen countries.

In addition you have to be careful about services like OpenDNS which for
example, made svn.apache.org always resolve to the EU even for users in the
US, due to how their DNS anycast is setup.

~~~
buro9
> I personally have my doubts about any value created for static pages doing
> this though, there are relatively cheap CDNs on the market now, rather than
> trying to find VMs in a dozen countries.

Please share the options.

My scenario is that I transfer around 800MB in static content (images, CSS and
JS) per month. I'd simply want that part CDN'd.

I used to use SimpleCDN but found them too expensive after they changed their
pricing model.

<http://www.simplecdn.com/pricing>

<http://www.simplecdn.com/savings>

The cost of 2 additional Linodes would be (2 x $19.95) = $39.90 per month.

My bandwidth usage isn't so great as to make the cost of an off-the-shelf CDN
solution beneficial, yet it's great enough that a couple of Linodes would make
a visible speed difference to the 100,000 unique visitors per month that visit
my site from North America.

To my mind there is a gap between small site and large site into which running
a couple of VMs to provide a cheap CDN is an advantage.

Although if you can show me cheaper drop-in (pass-through) alternatives to
SimpleCDN then I'm very eager to hear about them.

~~~
buro9
I got my numbers wrong thanks to an early morning typo... 800GB bandwidth per
month, not 800MB.

Of the options recommended:

Max CDN = $39.95 per month deal (1TB), usually $99.95 = Same price as DIY on
the deal, or more expensive normally.

SoftLayer = $140 per month (1TB) = More expensive.

CacheFly = $299 per month (2TB) = More expensive and have to upload files to
server.

Rackspace CloudFiles = $176 per month (based on current 800GB bandwidth alone
- other costs not yet factored in) = More expensive.

Amazon CloudFront = $131 per month (based on current 800GB bandwidth alone -
other costs not yet factored in) = More expensive and you have to upload files
to the server.

So yeah... DIY looks substantially cheaper. There's still a big gap between
small site and large site. For a few days of my time I can easily save myself
a lot of potential cost as well as speed up my website. I can't really see why
I wouldn't set up my own CDN.

~~~
adamt
But the $19.95 linodes have a 200GB/month limit - so you'd need at an absolute
minimum of 4 to do 800GB/month, and given you won't have perfect geographical
spread, about 6-8 is perhaps more realistic. So you'd be looking at least $120
for the VPS, plus the time/hassle of setting this up and managing/monitoring
it.

So it's just as expensive, and there's no way the DIY model would be cheaper
in terms of TCO.

~~~
buro9
I already host with Linode and they pool the bandwidth allowance for all
Linodes in your account.

As I've already got nicely layered VMs some have a totally unused allowance.
For example my MySQL nodes only transfer traffic on the private network at
Linode and so both the master server and slave use virtually zilch of their
bandwidth allowance.

I'm fairly sure that most Linode users with more than 1 node will be using
only a fraction of their allowance.

My total pooled bandwidth allowance with Linode is close to 4TB, of which in
total I'm using only 1TB with 800GB being static content.

My point was simple: The DIY solution with Linode looks to be a really good
idea as I already am with them and placing CDN servers in other datacenters
should mean good routing internally within their network (or peers) as well as
being extremely cheap.

I have yet to be shown that for my scenario there is a cheaper option and that
this is solved. I am very happy to be proven wrong as it will save me some
work, but so far DIY is winning.

------
papaf
This is an interesting article but one thing I don't yet understand (which may
become clear in part 2) is why both Nginx and Varnish are needed. Nginx can
certainly do this on its own, and as far as I know, so can Varnish which has
been designed to handle slow backends.

~~~
jjoe
1) Nginx is a fully fledged Web server and has some performance penalty
compared to Varnish. The latter is lightening fast.

2) Nginx is no match for Varnish's VCL for header/cache manipulation

Regards

~~~
hphp
1- yes but at the very highest loads 2- you don't need that much manip. for
simple load balancing. 3- can you backup your claims with reference(s)?

~~~
jjoe
1 - not necessarily 2 - it depends on app complexity. Most people that read Y
need it. 3 - Benchmarks? sure :) <http://www.unixy.net/apache-vs-litespeed>

~~~
zepolen
What does an Apache v Litespeed benchmark have to do with anything?

~~~
jjoe
The above post proposed that Varnish is not needed because Nginx could handle
the task well with the best performance. But Nginx is a bit slower than
Varnish. It's been shown in benchmarks that Litespeed is a bit faster than
Nginx. By inductive reasoning to show that Varnish is faster than Nginx, I
posted a link to show that Varnish is faster than Litespeed, which in turn is
faster than Nginx. So Varnish is faster than Nginx.

Let me know if you have any question/comment.

Regards

~~~
zepolen
My question is how you manage to stay in business with such flawed
'reasonings'.

That benchmark in particular is so full of crap that people will become dumber
after reading it.

~~~
jjoe
I'm not going to respond to your personal attacks; just the factual content of
you post(s).

I understand that you perceived my posting of the benchmark link to be
inappropriate and I agree with you to an extent. It remains to say that the
link represents my proof within the context of the conversation chain from
earlier.

You seem to be a long-term contributor here and I'm sure it's appreciated. I
would be interested if you addressed the flaws or issues with the benchmark
results. I do make mistakes just like anybody but own up to them at the end.

Let me know if you have any question and I'll be happy to discuss.

Regards Joe

~~~
zepolen
Flaws with the benchmark:

1\. Misrepresentation

The title says 'Apache vs Litespeed', when in reality it is 'Varnish vs
Litespeed', Apache has nothing to do with this particular benchmark.

2\. Questionable results

Varnish serving 217.05 req/sec for a static page at 100 concurrency level?
Something is seriously messed up in the setup; Varnish can easily handle
thousands of req/s for something as simple as looking up a cache entry and
returning a static file for 100 clients. Oh wait, 11177.99 [Kbytes/sec] -
looks like you saturated the pipe, making it a test of how fast your network
is, and nothing to do with Varnish.

3\. Flawed benchmark testing procedure

You can't expect a simple `ab -c 100 -n 10000` to provide any insight
whatsoever towards the general handling of a server. The only thing obvious is
that Litespeed has been saturated with 100 concurrent connections. If you're
going to do a benchmark, you must post varying concurrency levels at least
(along with exact setup configurations). Likewise, cpu and memory usage are
just as important to providing valuable insight.

4\. Comparing apples to oranges

I'm not entirely sure what Litespeed is doing (eg. does it cache statically?)
since you haven't specified it exactly, but considering that almost 70% of the
requests failed and almost all requests after the first few had a different
content length, I would assume that Litespeed is returning a dynamic request
each time, whereas Varnish is returning a static file. I have no doubt in my
mind that if you ran the same scenario (ie. Litespeed serving a static file as
well) the results would be much different.

5\. Irrelevant to the article benchmark

Saying "But Nginx is a bit slower than Varnish", then linking to a terrible
_irrelevant_ benchmark and using circular logic (based on another completely
crap Litespeed vs Nginx for serving _php!?_ benchmark) to infer that Varnish
is faster than nginx hurts my brain.

I ran a really simple benchmark on Varnish 2.1.3 and nginx 0.8.49's
proxy_cache directive on a single url; nginx is faster.

However, what if we test a heavy config with a bunch of complicated rules?
Varnish's VCL compiles straight to C and might give it the edge in those
cases. What if we use test a million different urls and really test how each
one handles their cache hit/miss procedures. How do they fare under real world
scenarios where thousands of different clients on various connection speeds
are connected? I know nginx is damn efficient memory wise when dealing with a
load of keepalives.

6\. Conclusion

Good benchmarks are difficult.

~~~
jjoe
1\. All is disclosed in the page. Varnish is indeed mentioned on the page as
being the crucial piece that made is all happen. But mea culpa, the title
"Apache vs Litespeed" does need the Varnish bit added. I'll correct it
shortly.

2\. You're making the assumption that varnish/VCL is caching the first server-
side result indefinitely (ttl). That is not the case. So apache is still doing
work behind the scenes, albeit less work than it's used to. Whatever hosting
conditions we imposed on Varnish/apache are the same imposed on lsws. It's
fair game (for better or worse).

3\. Actually Litespeed is able to handle more than 100 concurrent connections.
What choked it apparently is its inability to manage the 100 PHP requests
well. It actually ran out of memory during the experiments. Again, Litespeed
was given the same amount of memory and was under the same conditions as
Apache/Varnish. While the environment/conditions are not optimal, both
varnish/httpd and Litespeed started equal. At first, we started with -c 10 and
-n 10000 but both setups performed fairly well. So we upped it to -c 100.
Checkout the load average from the Litespeed node below.

4\. One of the goals of the experiment is to install stock Litespeed and stock
Apache (RPM). No configuration changes were made to either. Litespeed has
static object caching (20MB) enabled by default. Varnishd was given a 16MB
cache. Stock installs are common in the Web hosting business, which is the
scope/interest here.

5\. Based on experience with client portals, Varnish as a cache seems to
perform better than Nginx as pure cache. It could a perception. We use Nginx
and/or Varnish where it makes sense though. They're both free and open so why
not!

6\. Can't agree more. Just for the record, we didn't select the hardware and
environment to favor either web server. We had a vacant server, network, and
resources and went from there. The fact that this environment did not produce
good results for Litespeed is purely coincidental.

FYI, Litespeed Tech noticed the benchmark and mentioned that they're working
on developing a cache module that will ship with Litespeed web server. We'll
see.

[root@lsws_node /]# while [ true ]; do uptime; sleep 3; done 19:45:37 up 0
min, 0 users, load average: 0.00, 0.00, 0.00 19:45:40 up 0 min, 0 users, load
average: 0.00, 0.00, 0.00 19:45:43 up 0 min, 0 users, load average: 0.00,
0.00, 0.00 19:45:46 up 0 min, 0 users, load average: 0.00, 0.00, 0.00 19:45:49
up 0 min, 0 users, load average: 0.00, 0.00, 0.00 19:45:52 up 0 min, 0 users,
load average: 0.00, 0.00, 0.00 19:45:55 up 0 min, 0 users, load average: 0.00,
0.00, 0.00 19:45:58 up 0 min, 0 users, load average: 0.00, 0.00, 0.00 19:46:01
up 0 min, 0 users, load average: 0.00, 0.00, 0.00 19:46:05 up 0 min, 0 users,
load average: 2.88, 0.60, 0.19 19:46:09 up 0 min, 0 users, load average: 5.45,
1.17, 0.38 19:46:12 up 0 min, 0 users, load average: 7.82, 1.73, 0.57 19:46:15
up 0 min, 0 users, load average: 7.82, 1.73, 0.57 19:46:18 up 1 min, 0 users,
load average: 10.08, 2.30, 0.76 19:46:21 up 1 min, 0 users, load average:
10.08, 2.30, 0.76 Segmentation fault Segmentation fault Segmentation fault
-bash: /usr/bin/uptime: Cannot allocate memory -bash: fork: Cannot allocate
memory [root@lsws_node /]# while [ true ]; do uptime; sleep 3; done -bash:
fork: Cannot allocate memory [root@lsws_node /]# while [ true ]; do uptime;
sleep 3; done -bash: fork: Cannot allocate memory [root@lsws_node /]# while [
true ]; do uptime; sleep 3; done -bash: fork: Cannot allocate memory
[root@lsws_node /]# while [ true ]; do uptime; sleep 3; done -bash: fork:
Cannot allocate memory [root@lsws_node /]# while [ true ]; do uptime; sleep 3;
done -bash: fork: Cannot allocate memory [root@lsws_node /]# while [ true ];
do uptime; sleep 3; done 19:46:40 up 1 min, 0 users, load average: 18.36,
4.66, 1.56 19:46:43 up 1 min, 0 users, load average: 17.53, 4.72, 1.60
19:46:46 up 1 min, 0 users, load average: 17.53, 4.72, 1.60 19:46:49 up 1 min,
0 users, load average: 16.13, 4.64, 1.59 19:46:52 up 1 min, 0 users, load
average: 14.84, 4.56, 1.58 19:46:55 up 1 min, 0 users, load average: 14.84,
4.56, 1.58 19:46:58 up 1 min, 0 users, load average: 13.65, 4.49, 1.57
19:47:01 up 1 min, 0 users, load average: 13.65, 4.49, 1.57 19:47:04 up 1 min,
0 users, load average: 12.56, 4.41, 1.56 19:47:07 up 1 min, 0 users, load
average: 11.55, 4.34, 1.55 19:47:10 up 1 min, 0 users, load average: 11.55,
4.34, 1.55 19:47:13 up 1 min, 0 users, load average: 10.62, 4.27, 1.55
19:47:16 up 1 min, 0 users, load average: 10.62, 4.27, 1.55 19:47:19 up 2 min,
0 users, load average: 9.77, 4.20, 1.54 19:47:22 up 2 min, 0 users, load
average: 8.99, 4.13, 1.53 19:47:25 up 2 min, 0 users, load average: 8.99,
4.13, 1.53 19:47:28 up 2 min, 0 users, load average: 8.27, 4.06, 1.52 19:47:31
up 2 min, 0 users, load average: 8.27, 4.06, 1.52 19:47:34 up 2 min, 0 users,
load average: 7.61, 3.99, 1.51 19:47:37 up 2 min, 0 users, load average: 7.00,
3.92, 1.50

------
buro9
Nice.

The instructions assume some familiarity with BIND and Varnish, what I'd love
to see would be a Linode article explaining how to use their datacenters to do
this.

Specifically I'd love to put noes in 2 US datacenters and the London one to
help reduce my page load speed for the US visitors, whilst leaving the dynamic
nodes all in London where most of my users are.

------
rmoriz
I would use Powerdns instead of BIND. It's opensource and already comes with a
geo backend (among others).

<http://doc.powerdns.com/geo.html#AEN6738>

The doc says, Wikimedia uses PowerDNS' geo backend.

------
komplex
[http://poramin.com/post/818944318/install-bind-with-
support-...](http://poramin.com/post/818944318/install-bind-with-support-
geoip-on-ubuntu)

------
davidu
Relying on MaxMind for your CDN location targetting is not how I would
recommend doing this.

Anycast is better these days as it will taking routing topology into
consideration over a hardcoded list of IP addresses. And routing topology can
then be adjusted to account for congestion and other variables.

------
Goosey
Great article to help me understand a bit more on how CDNs work! As others
have pointed out the cost-effectiveness of deploying your own CDN is
questionable in this climate, but the KNOWLEDGE of how your leased CDN works
is certainly valuable!

------
js4all
Great article. I always wondered how this is done. I was aware of varnish and
nginx, the missing link was the patched bind9 version. Many thanks.

(btw. a little typo: varnish forwards to nginx on port 90 -> should be port
91)

