
How and why I run my own DNS servers - zrail
http://bugsplat.info/2012-12-31-how-i-run-my-own-dns.html
======
druiid
For many many years we ran our own DNS servers using PowerDNS with MySQL
backends. That's all fine and well and it's a very powerful architecture and
relatively reliable and strong to even high amounts of traffic... that was
until we started regularly seeing 10Gbps+ DDoS attacks, so we put a DDoS
protection service in front of the sites, but there wasn't really anything at
the time capable of protecting the DNS servers as well.

So a few months go by, everything has been fine and then another DDoS hits.
Looking through the web-servers, not hitting there, all is fine. Study
firewall traffic through the network and note that the majority of incoming
traffic (that was making it through to the network that is), was headed to the
DNS servers. The attackers were sending the DDoS to the DNS servers,
requesting some of our root domains and given that it appeared as valid
traffic there was not much to be done to filter any of it. The easy solution
in this case happened to be blocking all Chinese and Russian IP's for a couple
days which mitigated enough of it to solve. After that, we stopped hosting our
own DNS.

Moral of the story: DNS services are a great learning tool to use and PowerDNS
is probably where you want to start, but at scale you may or may not want to
actually run your own.

~~~
dsl
DDoS risk is based on the type of business you're running (online casino,
ecommerece, etc) and not really much to do with if you are running
infrastructure in house or not.

For that matter, I don't know of a reputable DDoS mitigation service that
_doesn't_ provide DNS hosting as part of the package. It's pretty core to the
whole thing.

~~~
druiid
Indeed, but what I was mostly getting at is that you have another possible
attack vector if your run your own DNS, while services that provide it for you
are basically pennies on the dollar compared to potential losses.

------
larrys
"To host your own DNS servers the registrars require you to list two IP
addresses "

Registrar here. This may be a requirement of some registrars but not a
requirement of all registrars. And in any case you can fake this (see warning)
by simply using the same IP address with a different host name or by using a
different IP address with a made up IP address.

Warning: Obviously you should have two valid DNS servers no doubt. I am simply
pointing out that there are work arounds to this especially if you are trying
to get up and running and won't have the 2nd dns operational for a short
period of time you can at least get started with DNS that should work (since
it only takes one and assuming that one is working you will be serving up
records.)

~~~
dsl
As a registrar you should know this is actually a requirement of the registry
and varies based on TLD policy. (maybe you meant you're a reseller?)

~~~
larrys
The registry has one requirement.

The registrar could have another one.

.com .net "require" only 1 nameserver (Verisign). PIR/Afilias require two
(hence the need to fake or use a real one).

Please note also that my parent comment was with regards to the statement "the
registrars require you to list two IP addresses" not why that may or may not
have been required by the registry.

It's possible of course that even though .com .net only require one
nameserver, a registrar might require two for some reason. It's also possible
(but unlikely) that even though .org requires two the registrar could take
care of this by putting in a dummy record if they wanted and only requiring
one of the customer.

End users don't deal with the registry. They deal with the registrar.

We are an ICANN accredited registrar.

------
thaumaturgy
I also run my own domain name servers. I use PowerDNS with a MySQL backend;
setting up replication between the two MySQL servers wasn't too hard, which
gives me redundant name servers in two data centers. I have a few more than 32
zones, but the name services stuff has almost zero impact on server resources.

Updating and querying the name records is a piece of cake thanks to the MySQL
database, and if I ever get around to finishing it, I can have my own web-
based front-end for it.

Best of all, I can have a nice, short ttl on all of my entries, and PowerDNS
is configured to always check the database, so I have a two minute refresh
period on any changes made to entries on my name servers -- that alone has
been well worth running my own. (Client: "Don't I have to wait like a few
hours or a day or something for this to spread over the internet or
something?" "Nope. 'Bout two minutes in most cases, little bit more for AT&T's
customers.")

I definitely didn't do it right the first time, though. DNS has a few gotchas,
like making sure you disable axfr requests, making sure you have an SOA record
for appropriate zones, and so on. I'm probably still doing something wrong,
but I don't know what it is at this point. Also, the default recommended
PowerDNS MySQL setup is needlessly complex, as far as I can tell.

~~~
larrys
"and if I ever get around to finishing it, I can have my own web-based front-
end for it."

If you finish it let me know. I would be interested in possibly buying the
front end from you (even if it's rough wouldn't be customer facing.)

~~~
druiid
There's several existing MySQL management solutions for powerDNS, including:
<https://www.poweradmin.org/trac/> <https://github.com/averna-
syd/PowerdnsTango>

PowerDNS Tango is the prettiest looking one.

~~~
basilgohar
Looking at the featureset of these frontends, my own implementation pales in
comparison. It would need a lot of work before matching the features these
offer.

------
fleitz
As for the SSH attempts just move it to a different port, disable password
logins. SSH shouldn't really be accessible on the same IP as any publicly
facing service for your domain. It invites this kind of automated scanning,
which is actually a huge risk unless you're on top of your patches. Also
change your SSH config to misreport the OS/SSH version.

The IP for SSH access should then be heavily filtered to only SSH and the
requisite ICMP packets.

Have you considered using Route53 / Cloudflare?

~~~
ChuckMcM
Or fail2ban which puts in iptables rules to ban IPs that fail to login. The
ssh scrapers try a variety of user accounts, and it is pretty easy to spot
them (for example root can't login on my system from anywhere except the
physical console, so trying to login as root,bin,games,demo,Etc from ssh is an
instant ban.

Of course you need to not run an open DNS resolver which means 'don't do
recursive lookups for anything _except_ your trusted addresses'. I saw one
setup that worked really well which was to only do recursive lookups from
loopback, then client would create a vpn tunnel to the dns server, get their
dns service that way. Seemed a bit extreme but it allows off site DNS service.

~~~
jstalin
I use fail2ban as well. I honestly don't get why people get so worked up about
SSH. As long as you have a long, secure password, how is it a security
vulnerability?

~~~
jacquesm
Because if there is an exploit that allows an attacker to recover your
password for instance from a session replay then you're instantly open.

~~~
ChuckMcM
So split the conversation, there are little threats and big threats. I don't
have 6 locks on a steel door at my house, the big threats are going to break
the picture window and walk in that way. So if you're running one of the 100s
of millions of non-descript servers out there then fail2ban and putting ssh on
a non-standard port will keep the opportunists out of your server (at least
via ssh). If they are using a zero day against you, or are waiting to capture
traffic to do a plaintext attack they aren't opportunists, they are gunning
for you. That is a different threat model.

~~~
jacquesm
Unfortunately I have to deal with both, but you're 100% right, those are
definitely different categories.

------
tom_fitzhenry
To further improve reliability, consider using a secondary-as-a-service (as
well as your self-hosted secondary name service). This will only work if
tinydns supports zone transfers (AXFR/IXFR), which as far as I can tell, it
doesn't.

I don't use a self-hosted secondary. I use a secondary-as-a-service as my
secondary /and primary/, which has the advantage of my VPS going down (for
less than 4 weeks) wouldn't impact my DNS's uptime.

I blogged about it this month: [https://tom-fitzhenry.me.uk/blog/2012/12/host-
your-own-dns-w...](https://tom-fitzhenry.me.uk/blog/2012/12/host-your-own-dns-
without-sacrificing-reliability.html)

~~~
zrail
Before I picked up the Ramnode server I was using BuddyNS as my primary with
the prgmr node as the stealth primary. It worked fine but it was less
straightforward than I wanted.

~~~
tom_fitzhenry
Though the stealth primary solution involves more moving pieces, and
introduces zone transfers, which are more complex than provisioning via
puppet, I've had no noticeable issues.

I say "noticeable issues", because although my DNS is monitored, the
monitoring isn't granular enough to detect sub-minute outages, for example.

Did you have any issues in particular with the stealth primary solution?

~~~
zrail
Not any technical issues, it was just more moving pieces than I want. BuddyNS
is a fine service but having to force an update with an email seemed pretty
hacky.

~~~
tom_fitzhenry
You can use DNS NOTIFY rather than sending an email. When the primary is
updated, DNS NOTIFY will tell each secondary about the change.

DNS NOTIFY is standardised in: <https://tools.ietf.org/rfc/rfc1996.txt>

BIND 8+ supports it by default, according to
[http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch10_03.h...](http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch10_03.htm)

There is a perl script to achieve the same thing in djbdns, according to
<http://www.fefe.de/djbdns/#notify> (which joe_bleau alluded to).

------
rachelbythebay
Those "machines" seem huge for what they're doing (DNS services for 32 zones).
They add up to more resources than all of my machines put together from just a
few years ago.

I guess this is progress. It just seems strange to me.

~~~
zrail
Yeah they're way too beefy for what I'm using them for at the moment. My
ongoing plan is to move various services onto these machines but it's been
slow going.

------
chernevik
I do this myself off a Linode. It's neat to understand how this stuff actually
works, it expands the imagination of what's possible. I like being able to
quickly stand up a subdomain for development / testing / staging. I really
want to get an email server running, I think the ability to spark up email
addresses and servers for automated reporting or file distribution opens up
some useful possibilities.

~~~
dotBen
As it goes, Linode has an amazing DNS service bundled with their instances -
replicated between all their international datacenters.

If you host with Linode I can't see why you wouldn't use it, unless you are
just futzing/curious to roll your own.

~~~
vivab0rg
+1 for Linode's DNS service. The best part is that you can access it via API
too.

------
Sami_Lehtinen
Why bother using Postfix for mail forwarding? Why not to use Gmail MX
directly?

I run my own servers with postfix, dovecot and roundcube. Therefore I don't
need Gmail for anything. (It's also huge privacy issue.)

~~~
zrail
I don't use google apps for my domains so Gmail MX wouldn't really work, plus
I have some aliases that I think are maybe too complicated for Gmail. What do
you think of roundcube? I haven't tried it yet, since php puts me off.

~~~
aioprisan
What aliases could be too complicated for Gmail? Roundcube is an OK web email
client, but let's be honest, it doesn't even match up what Gmail can offer
you, in terms of integrated services like Google Docs, free phone dialing,
video conferencing, etc.

~~~
zrail
I have some stupid aliases that go to multiple people set up as virtual
addresses in Postfix. I'm sure I could convince google to do that for me.

~~~
IgorPartola
Yes, Google Apps let you create mailing lists.

------
jstanley
"To host your own DNS servers the registrars require you to list two IP
addresses with the idea that you'll be providing redundant service. The one
thing you don't want is downtime with DNS, it screws everything up."

I've got around this by having two hostnames for one machine.

The way I see it, since all my services are on this one machine anyway, I'll
have bigger problems than DNS if the machine is down: there will be nothing at
the address the names are pointing to, so who cares if they resolve properly?

~~~
zrail
Fair point. I've got email and DNS distributed onto both machines so I don't
lose anything, and prgmr was pretty clear that I couldn't have a second IP
just to fake out DNS so I went with a whole second server.

~~~
lsc
I wasn't trying to sell you another server. conventional wisdom is that you
should have your secondary DNS on a different server, on a different network,
run by a different provider, not another server with me.

~~~
zrail
Oh I know, that's why I've got what I have :)

------
kjackson2012
I ran my own DNS server from my own home DSL static IP address 10+ years ago.
It's fairly innocuous as long as you only respond to DNS requests for your own
domain and don't forward requests. But it ends up being more hassle than its
worth.

As recently as a few months ago, I hosted my own DNS server on a Linksys SLUG,
but after I suffered a power outage and realized that there's no reason for
hosting this stuff myself, I just decided to use namecheap.com for their Free
DNS service.

------
iSloth
Personally I would have gone with a dedicated server from somewhere like OVH,
about the same price as those VPS's with similar resources (but they are
dedicated to you...)

As for SSH, just move it on to a port other than 22, that will fix 99% of the
bots trying to guess your password.

~~~
MidnighToker
been using ovh for years and really enjoy their self-manage tools (though I
actually work for a different provider).

fail2ban || denyhosts are also good ways of stopping ssh brute force :)

------
aioprisan
Route53 is a much better alternative to running your own service. I used to
have a similar setup for taskup.com and other sites that I'm hosting but for
the $0.20 that I pay every month to Amazon, I get weighted DNS responses based
on the region that the request comes from and requests get served to the local
server for that region, redundancy in 12+ zones, etc. The latency alone for a
DNS lookup from India was about 300ms to our server in PA, but now it's under
30ms. Having said that, it's a good learning opportunity.

~~~
lesterbuck
My reading of the Route53 prices is that you pay $.50/month/domain, plus query
charges after a million. How do you pay $0.20/month? I find it very irritating
that Amazon says "Pay only for what you use. There is no minimum fee." when
there obviously _is_ a minimum fee of $.50/month, even if zero queries are
resolved. There are entire low end VPS servers that cost only marginally more
($14/yr) than what Route53 costs for one lousy domain, i.e. one entry in a
database table.

~~~
aioprisan
with Route53, it's not just another entry in a DNS zone file, you actually get
multiple-zone IP addresses for each domain hosted, by default. Also, you get
WRR and anycast, as well as latency based routing. Not to mention that you can
create aliases that are not visible to resolvers but point to internal Amazon
resources that you have access to.

------
lvh
Twisted[1] comes with a DNS server built-in.

For non-authorative local DNS caches:

    
    
        twistd dns --recursive --cache
    

For authorative servers, to serve a BIND-style zonefile:

    
    
        twistd dns --bindzone ZONEFILE
    

But, of course, that's pretty boring. The fun stuff is when you have a Python
source file as your zonefile! See:
[https://twistedmatrix.com/documents/current/names/howto/name...](https://twistedmatrix.com/documents/current/names/howto/names.html)

twisted.names (that's the name of the DNS package) is simple enough to use for
DNS serving (demonstrated above) and flexible enough to get it to do pretty
much whatever you want.

------
spc476
I run my own DNS, but in a rather unique way. The registrar record points to
my hosting company's DNS servers, but they are slaved off my DNS server; the
only IPs that can hit my DNS server are the hosting company's. I make the
changes I want, they're pushed out to the actual Internet facing DNS servers.

------
Nick_C
Has anyone tried the PaperTrail service? I'm tempted except for the tiny
window for queries, only a week. I'd like, say, a month at least.

~~~
zrail
Once you sign up you can customize your plan as much as you want, setting
search retention to a month is a possibility. It's not cheap, though.

