Load Balancing DNS Using Cisco's IP SLA Feature

It's generally accepted that using any sort of stateful load-balancer in front of a set of DNS servers is a bad idea. There are several reasons for this, but my favourites are that:

1. it means adding an unnecessary potential point of failure
2. the state engines in load-balancers aren't scaled for DNS, and will be the first thing to fail under heavy load or a DoS attack

The issue with the state engine in a load-balancer is that it is scaled for average Internet traffic, which DNS is not. Each DNS connection (each flow) is typically one UDP packet each way, and well under 1KB of data. By contrast, with most other common protocols (smtp, http, etc.) each flow is hundreds to millions of packets, and probably tens, hundreds or even millions of KB of data. The key metric here is the flows:bandwidth ratio. Devices are built so that when they reach their maximum bandwidth capability, there's room in memory to track all of the flows. The problem is, they're typically scaled for average traffic. Since the flows:bandwidth ratio for DNS is so very much higher than other types of traffic, you can expect that a load-balancer in front of busy DNS servers will exhaust their memory in trying to track all the flows long before the maximum bandwidth of the device is reached. To put it another way, by putting a load-balancer scaled for 1Gb of traffic in front of DNS servers scaled for the same amount of traffic, you actually drastically reduce the amount of DNS traffic those servers can handle.

There are better ways.

ISC, the maker of BIND, has an excellent technote which describes using OSPF Equal Cost Multi-Path (ECMP) routing to distribute load between a set of DNS servers. In effect, it's a scheme for doing anycast on a LAN scale, rather than WAN. Put simply, it involves using Quagga or some other software routing daemon on each DNS server to announce a route to the DNS service address. A wrapper script around the DNS process adds a route just before the process starts, and removes it just after the process exits. The approach works quite well as long as the local router can handle OSPF ECMP, and as long as it uses a route hashing algorithm to maintain a consistent route choice for each source address without needing a state table. For example, the Cisco Express Forwarding (CEF) algorithm uses a hash of source address, destination address, and number of available routes to produce a route selection.

The down sides to the ISC method are that there's a small amount of complexity added to the management of the DNS server itself (for example, you can no longer use the standard application start/stop mechanisms of your OS for the DNS software) and the risk that a failure may occur which causes the DNS software to stop answering queries, but not exit. If the latter occurs, the route to that server will not be removed. This is pretty safe with BIND, as its designed to exit on any critical error, however that's not necessarily the case with all DNS server applications.

There's another method available (that I'm going to describe here) which, while being very similar to the ISC methodology, does not have these particular flaws. I should point out here that the method I'm about to describe is not my invention. It was pieced together from the ISC technote and some suggestions that came from Tony Kapella while chatting about this stuff in the hallway at a NANOG meeting a while back. After confirming how easy it is to get this method to work I've been singing its praises to anyone who will listen.

At a high level it's quite similar to the OSPF method. The DNS service address is bound to a clone of the loopback interface on each server, and ECMP routing is used, but rather than populating the routes with OSPF and running routing protocols on the DNS servers, route management is done with static routes on the local router linked to service checks which verify the functionality of the DNS service.

Setting It All Up

In this example, we'll use the RFC 3330 TEST-NET. The service address for the DNS service will be 192.0.2.253. This is the address that would be associated with a name server in a delegation for authoritative DNS service, or would be listed as the local recursive DNS server in a DHCP configuration or desktop network config. The network between the local router and the DNS servers will be numbered out of 192.0.2.0/28 (or 192.0.2.0 through 192.0.2.15). The server-facing side of the router will be 192.0.2.1, and that will be the default route for each of the DNS servers, which will be 192.0.2.10, 192.0.2.11 and 192.0.2.12. This network will be the administrative interfaces for the DNS servers.

Once the servers are reachable via their administrative addresses, make a clone of the loopback interface on all three servers. Configure the second loopback interface with the DNS service address.

On FreeBSD, the rc.conf entries for the network should look something like this:

defaultrouter="192.0.2.1"
cloned_interfaces="lo1"
ifconfig_em0="192.0.2.10 netmask 255.255.255.240"
ifconfig_lo1="192.0.2.253 netmask 255.255.255.255"

It's a little more difficult to represent the configuration under Linux since it's spread across several config files, but the above should give you a pretty good idea of where to start.

Once the network setup is finished, configure your DNS server software to listen to both the administrative address and the service address. So, on the first DNS server, it should listen to 192.0.2.10 and 192.0.2.253.

That's all that needs to be done on the servers. Note that doing this was far simpler than configuring the servers to run OSPF and automatically add and remove routes as the DNS service is started or stopped.

The last few steps need to be taken on the local router. The first of these is to configure the router to check up on the DNS service on each of the three servers and make sure it's running; this is where Cisco's IP SLA feature comes into play. Configure three service monitors, and then set up three "tracks" which will provide the link to the service monitors.

ip sla monitor 1
type dns target-addr www.example.ca name-server 192.0.2.10
timeout 500
frequency 1
ip sla monitor schedule 1 life forever start-time now
!
ip sla monitor 2
type dns target-addr www.example.ca name-server 192.0.2.11
timeout 500
frequency 1
ip sla monitor schedule 2 life forever start-time now
!
ip sla monitor 3
type dns target-addr www.example.ca name-server 192.0.2.12
timeout 500
frequency 1
ip sla monitor schedule 3 life forever start-time now
!
track 1 rtr 1
track 2 rtr 2
track 3 rtr 3

This sets up three IP SLA Monitors which repeatedly query the administrative address on each server for the A record www.example.ca. The DNS server must respond with an A record for the QNAME you use; if it is unable to respond, or responds with a different record type, the monitor fails. In the example above the monitor attempts the lookup every second (frequency) and fails if it doesn't receive a valid A record within 500ms (timeout). You may need to experiment with the timeout value, depending on how responsive your DNS servers are. If you find individual servers appear to be going out of service when the daemon is still operating fine you might have the timeout value set too low.

With the monitors in place, turn on CEF and then configure three static routes to the service address via each server's administrative address. The routes are linked to the service monitors using the track argument:

ip cef
!
ip route 192.0.2.253 255.255.255.255 192.0.2.10 track 1
ip route 192.0.2.253 255.255.255.255 192.0.2.11 track 2
ip route 192.0.2.253 255.255.255.255 192.0.2.12 track 3

And that should be it. DNS queries arriving at the external interface of the router bound for 192.0.2.253 should now be routed to one of the DNS servers behind it, with a fairly equal load distribution. Since the router is using a hashing algorithm to select routes the load distribution can't be perfect, but in practise I've found that it's incredibly even. The only likely reason to see an imbalance is if your DNS servers receive an unusually high percentage of their queries from just one or two source addresses.

It's important to point out that most of the cautions in the ISC technote, particularly in reference to zone transfers and TCP DNS, apply equally here. I highly recommend reviewing the ISC document before implementing this in production.

Of course, there is still one big downside to this particular method of load balancing: it's depedant on one particular vendor. I have not yet found a way to reproduce this configuration using non-Cisco routers. If anyone is aware of a similar feature available from other major routing vendors please let me know and I'll integrate instructions for those routers here.

Not Your Best Interview, Mr. Tovar

I've been thinking for some time about starting to regularly post thoughts about random things somewhere visible — blogging, if you will. This afternoon I was thinking about it somewhat more earnestly, trying to choose between several topics for a first post, when a friend pointed out this interview. Seeing as the subject (the DNS) is what I spend most of my days doing, it seemed like an excellent place to start.

First let me say that I disagree with Tom Tovar's basic position on open source DNS software. As a general class of software, there is absolutely nothing wrong with using it for mission critical applications — not even for critical infrastructure. The various arguments about "security through obscurity" vs. open analysis and vetting have been made so many times that I won't bother with the details here. Suffice to say that, all other things being equal, I'd be far more inclined to trust the security of my DNS to open source software than closed, proprietary commercial software. Not that his position is that big a surprise. After all, he is the CEO of a company that sells some extremely expensive DNS software.

Mr. Tovar's early statements about DNS security, Kaminsky, and the current concerns of the DNS industry as a whole are hard to argue with. However, as one gets further into the interview, some serious problems with what Mr. Tovar has to say crop up including one glaring error in a basic statement of fact. I'll approach those in turn as I work my way through the article, addressing flawed points in the order they come.

I'll approach one particular point here as a pair of answers, since there's tightly related information in both.

GCN: The fix that was issued for this vulnerability has been acknowledged as a stopgap. What are its strengths and weaknesses?

TOVAR: Even to say that it is pretty good is a scary proposition. There was a Russian security researcher who published an article within 24 hours of the release of the [User Datagram Protocol] source port randomization patch that was able to crack the fix in under 10 hours using two laptops. The strength of the patch is that it adds a probabilistic hurdle to an attack. The downside is it is a probabilistic defense and therefore a patient hacker with two laptops or a determined hacker with a data center can eventually overcome that defense. The danger of referring to it as a fix is that it allows administrators and owners of major networks to have a false sense of security.

GCN: Are there other problems in DNS that are as serious as this vulnerability?

TOVAR: I think there are a lot of others that are just as bad or worse. One of the challenges is that there is no notification mechanism in most DNS solutions, no gauntlet that the attacker has to run so that the administrator can see that some malicious code or individual is trying to access the server in an inappropriate way. If UDP source port randomization were overcome and the network owner or operator were running an open-source server, there would be no way to know that was happening. This has been a wake-up call for any network that is relying on open source for this function.

Mr Tovar's last point in the first question is right on the money: the patches released for dozens of DNS implementations this summer do not constitute "a fix." Collectively they are an improvement in the use of the protocol that makes a security problem tougher for the bad guys to crack, but does not make that problem go away.

As for the rest of what he has to say here, on the surface it seems perfectly reasonable, if you make a couple of assumptions.

First, you have to assume that commercial DNS software, or at least his commercial DNS software, has some special sauce which prevents (or at least identifies) Kaminsky-style attacks against a DNS server, which cannot be available to operators of open source DNS software. It may be reasonable to take the position that it is beneficial for DNS software to notice and identify when it is under attack; however, it is not reasonable to suggest that this is the only way to detect attacks against the software. When the details of Kaminsky's exploit were eventually leaked, almost immediately tools began to spring up to help operators detect poisoning attempts, such as the Snort rule discussed in this Sourcefire Vulnerability Research Team white paper [PDF].

The second assumption you must make to consider this point reasonable is that a network operator is going to fail to notice a concerted effort to attack one or more of her DNS servers. Overcoming source port randomization, the "fix" under discussion, requires such a concerted effort — so much bandwidth consumed for so long a period — that it is generally considered unlikely that a competent network operator is going to fail to notice a cache poisoning attempt in progress when it is directed at a patched server. It's a bit like suggesting that it takes 10 hours of someone banging on your front door with a sledgehammer to break through it, and that it's unlikely you'll notice this going on.

This is the only reason informed DNS operators will allow anyone to come anywhere near implying that the patch is a "fix" of some kind. In fact, the only currently available fix for this vulnerability in the protocol is securing the protocol, which is where DNSSEC comes in.

GCN: Is open-source DNS software inherently less secure than proprietary software?

TOVAR: The challenge of an open-source solution is that you cannot put anything other than a probabilistic defense mechanism in open source. If you put deterministic protections in, you are going to make them widely available because it is open source, so you essentially give the hacker a road map on how to obviate or avoid those layers of protection. The whole point of open source is that it is open and its code is widely available. It offers the promise of ease of deployment, but it is likely having a complex lock system on your house and then handing out the keys.

There actually is a widely available deterministic defense mechanism which is implemented in open source, as well as commercial software. It's known as DNSSEC, and it comes up later in the interview. Putting that aside though, I'd be curious to know what other deterministic defense mechanisms Tovar is implying are available in commercial software, but are not available in open source software. Obviously he doesn't say; he can't. His own point addresses why non-revealable deterministic countermeasures are insufficient for securing DNS software: should they be revealed (and let's face it, corporate espionage happens) then they are immediately invalidated.

GCN: BIND, which is the most widely used DNS server, is open source. How safe are the latest versions of it?

TOVAR: For a lot of environments, it is perfectly suitable. But in any mission-critical network in the government sector, any financial institution, anything that has the specter of identity theft or impact on national security, I think using open source is just folly.

He is, of course, welcome to this opinion. I do not share this opinion, and I would bet real money that neither do a majority of the other operators who manage critical DNS infrastructure. Refuting the basis for this opinion is what this post is all about, so I'll move on.

GCN: Why is BIND so widely used if it should not be used in critical areas?

TOVAR: The Internet is still relatively young. We don’t think poorly of open source.

The implication being... as the Internet ages and we gain more experience, we should think more poorly of open source?

In fact, many of our engineers wrote BIND versions 8 and 9, so we do believe there is a role for that software. But the proliferation of DNS has occurred in the background as the Internet has exploded. DNS commonly is thought of as just a translation protocol for browsing behavior, and that belies the complexity of the networks that DNS supports. Everything from e-mail to [voice over IP] to anything IP hits the DNS multiple times. Security applications access it, anti-spam applications access it, firewalls access it. When administrators are building out networks it is very easy to think of DNS as a background technology that you just throw in and then go on to think about the applications.

The rest of this statement is about how people deploy DNS, not about how the software is designed or works. Nothing here explains why he feels open source DNS software can't be used in mission critical situations. All he explains is how he thinks it can be deployed in sloppy ways. Commercial software can be deployed sloppily just as easily as open source software. Nothing inherent in the software prevents or aids the lazy administrator in being lazy.

GCN: Why is DNSsec not more widely deployed?

TOVAR: Not all DNS implementations support it today, and getting vendors to upgrade their systems is critical. Just signing one side of the equation, the authoritative side, is a good step, but it’s only half the battle. You need to have both sides of the DNS traffic signed. And there is no centralized authority for .com. It is a widely distributed database and you have to have every DNS server speaking the same version of DNSsec. So the obstacle to DNSsec deployment is fairly huge. It is going to take government intervention and wide-scale industry acknowledgment that this is needed.

And here, at the end, is where I think Mr Tovar shows his ignorance of DNS operations in general, and in particular the operation of critical infrastructure. There is, in fact, a central authority for the .COM zone. That authority is Verisign; they manage and publish the .COM zone from one central database. Mr Tovar is, perhaps, thinking of the .COM WHOIS database. The WHOIS database holds the details of who holds each registered domain, and is indeed distributed among the .COM registrars.

This sort of fundamental misunderstanding of the DNS infrastructure is a good indicator of just how much weight should be given to Mr. Tovar's opinions on how that infrastructure is managed.

UPDATE: Florian Weimer raises an interesting question, one that I'm embarassed to say didn't occur to me when I was originally writing this. Even though the question itself is facetious, the point is valid. The data sheets [PDF] for the main product that Mr. Tovar's company markets say that it only runs on open source operating systems. How can he keep a straight face while on the one hand claiming that his product is suitable for mission critical applications but open source isn't, while on the other hand his software only runs in open source environments? It's quite baffling.

UPDATE UPDATE: At some point the dns-operations archive index got regenerated and the link to Florian Weimer's email ended up pointing to the wrong post. Fixed 2025/06/03.