DNS is a wonderful distributed system, with plenty of safeguards and fallbacks to ensure continuous operation.
But still, screwups happen. Here’s some tips on what to do to try to ensure you aren’t caught out in the cold.
Tip 1: Have multiple servers.
Without a doubt, this is the biggest tip about DNS. Designed in from the beginning was an assumption that you’d have multiple nameservers for a given zone. So… have them!
Put them as far apart as you reasonably can – different hosts, different networks, different power. The more they share, the more risk you’re in.
Countertip: Hosting your DNS server only over your ADSL link.
Tip 2: Do backups.
Pretty standard sysadmin fare. RAID isn’t a backup, and neither is a slaved nameserver.
Tip 3: Nameservers must all agree.
You know how kids will sometimes ask their parents the same question independently, hoping for a different answer? It’s important that the parents always give the same answer, and it’s downright vital that your nameservers do too. Don’t let them get out of sync!
Typically, zone transfers fix all your woes here, but do make sure they’re working.
Tip 4: Test your changes directly against all nameservers.
It’s just a small change, right? What could go wrong? Lots! So test each server individually. If one doesn’t update, maybe you have a problem that you need to fix. (Or maybe it’s just a bit laggy – it happens). “dig” is your friend.
Countertip: Not realising until too late that you’re breaking Tip 3
Tip 5: Make your NS records match your glue.
If you’ve told your domain registrar that your nameservers are ns1.example.org and ns2.example.org, then make sure you put that in your zone file too – all sorts of wacky caching issues can ensue when you don’t.
Tip 6: If you use a CNAME record, don’t use anything else.
CNAMEs are a really convenient way of saying “www.example.org is really webserver.example.org”. You can’t then say “But www has an MX of foo.example.org” or “www is also a subdomain with nameservers at …”.
That’d be contradictory, because you’ve already said with the CNAME that it’s really webserver.example.org. It can’t be both, if it’s both then it’s actually something different altogether and needs its own records.
Relatedly, don’t point a CNAME at anything other than a plain hostname – Don’t try to CNAME www.example.org to example.org, it’ll just break stuff.
Tip 7: Don’t firewall out DNS queries to your nameserver.
No, really. The whole internet needs to be able to look up domain names, not just some of it, not just most of it. (You’re excused if it’s a private nameserver, of course!).
Counterpoint: Using bogon filters on nameservers and ignoring genuine queries.
Bonus Tip: Monitor your servers.
If you’re running DNS servers in production, monitor them so you know that you haven’t lost one. Once it’s all set up right, you really can lose one without noticing.