Not Your Best Interview, Mr. Tovar

I've been thinking for some time about starting to regularly post thoughts about random things somewhere visible — blogging, if you will. This afternoon I was thinking about it somewhat more earnestly, trying to choose between several topics for a first post, when a friend pointed out this interview. Seeing as the subject (the DNS) is what I spend most of my days doing, it seemed like an excellent place to start.

First let me say that I disagree with Tom Tovar's basic position on open source DNS software. As a general class of software, there is absolutely nothing wrong with using it for mission critical applications — not even for critical infrastructure. The various arguments about "security through obscurity" vs. open analysis and vetting have been made so many times that I won't bother with the details here. Suffice to say that, all other things being equal, I'd be far more inclined to trust the security of my DNS to open source software than closed, proprietary commercial software. Not that his position is that big a surprise. After all, he is the CEO of a company that sells some extremely expensive DNS software.

Mr. Tovar's early statements about DNS security, Kaminsky, and the current concerns of the DNS industry as a whole are hard to argue with. However, as one gets further into the interview, some serious problems with what Mr. Tovar has to say crop up including one glaring error in a basic statement of fact. I'll approach those in turn as I work my way through the article, addressing flawed points in the order they come.

I'll approach one particular point here as a pair of answers, since there's tightly related information in both.

GCN: The fix that was issued for this vulnerability has been acknowledged as a stopgap. What are its strengths and weaknesses?

TOVAR: Even to say that it is pretty good is a scary proposition. There was a Russian security researcher who published an article within 24 hours of the release of the [User Datagram Protocol] source port randomization patch that was able to crack the fix in under 10 hours using two laptops. The strength of the patch is that it adds a probabilistic hurdle to an attack. The downside is it is a probabilistic defense and therefore a patient hacker with two laptops or a determined hacker with a data center can eventually overcome that defense. The danger of referring to it as a fix is that it allows administrators and owners of major networks to have a false sense of security.

GCN: Are there other problems in DNS that are as serious as this vulnerability?

TOVAR: I think there are a lot of others that are just as bad or worse. One of the challenges is that there is no notification mechanism in most DNS solutions, no gauntlet that the attacker has to run so that the administrator can see that some malicious code or individual is trying to access the server in an inappropriate way. If UDP source port randomization were overcome and the network owner or operator were running an open-source server, there would be no way to know that was happening. This has been a wake-up call for any network that is relying on open source for this function.

Mr Tovar's last point in the first question is right on the money: the patches released for dozens of DNS implementations this summer do not constitute "a fix." Collectively they are an improvement in the use of the protocol that makes a security problem tougher for the bad guys to crack, but does not make that problem go away.

As for the rest of what he has to say here, on the surface it seems perfectly reasonable, if you make a couple of assumptions.

First, you have to assume that commercial DNS software, or at least his commercial DNS software, has some special sauce which prevents (or at least identifies) Kaminsky-style attacks against a DNS server, which cannot be available to operators of open source DNS software. It may be reasonable to take the position that it is beneficial for DNS software to notice and identify when it is under attack; however, it is not reasonable to suggest that this is the only way to detect attacks against the software. When the details of Kaminsky's exploit were eventually leaked, almost immediately tools began to spring up to help operators detect poisoning attempts, such as the Snort rule discussed in this Sourcefire Vulnerability Research Team white paper [PDF].

The second assumption you must make to consider this point reasonable is that a network operator is going to fail to notice a concerted effort to attack one or more of her DNS servers. Overcoming source port randomization, the "fix" under discussion, requires such a concerted effort — so much bandwidth consumed for so long a period — that it is generally considered unlikely that a competent network operator is going to fail to notice a cache poisoning attempt in progress when it is directed at a patched server. It's a bit like suggesting that it takes 10 hours of someone banging on your front door with a sledgehammer to break through it, and that it's unlikely you'll notice this going on.

This is the only reason informed DNS operators will allow anyone to come anywhere near implying that the patch is a "fix" of some kind. In fact, the only currently available fix for this vulnerability in the protocol is securing the protocol, which is where DNSSEC comes in.

GCN: Is open-source DNS software inherently less secure than proprietary software?

TOVAR: The challenge of an open-source solution is that you cannot put anything other than a probabilistic defense mechanism in open source. If you put deterministic protections in, you are going to make them widely available because it is open source, so you essentially give the hacker a road map on how to obviate or avoid those layers of protection. The whole point of open source is that it is open and its code is widely available. It offers the promise of ease of deployment, but it is likely having a complex lock system on your house and then handing out the keys.

There actually is a widely available deterministic defense mechanism which is implemented in open source, as well as commercial software. It's known as DNSSEC, and it comes up later in the interview. Putting that aside though, I'd be curious to know what other deterministic defense mechanisms Tovar is implying are available in commercial software, but are not available in open source software. Obviously he doesn't say; he can't. His own point addresses why non-revealable deterministic countermeasures are insufficient for securing DNS software: should they be revealed (and let's face it, corporate espionage happens) then they are immediately invalidated.

GCN: BIND, which is the most widely used DNS server, is open source. How safe are the latest versions of it?

TOVAR: For a lot of environments, it is perfectly suitable. But in any mission-critical network in the government sector, any financial institution, anything that has the specter of identity theft or impact on national security, I think using open source is just folly.

He is, of course, welcome to this opinion. I do not share this opinion, and I would bet real money that neither do a majority of the other operators who manage critical DNS infrastructure. Refuting the basis for this opinion is what this post is all about, so I'll move on.

GCN: Why is BIND so widely used if it should not be used in critical areas?

TOVAR: The Internet is still relatively young. We don’t think poorly of open source.

The implication being... as the Internet ages and we gain more experience, we should think more poorly of open source?

In fact, many of our engineers wrote BIND versions 8 and 9, so we do believe there is a role for that software. But the proliferation of DNS has occurred in the background as the Internet has exploded. DNS commonly is thought of as just a translation protocol for browsing behavior, and that belies the complexity of the networks that DNS supports. Everything from e-mail to [voice over IP] to anything IP hits the DNS multiple times. Security applications access it, anti-spam applications access it, firewalls access it. When administrators are building out networks it is very easy to think of DNS as a background technology that you just throw in and then go on to think about the applications.

The rest of this statement is about how people deploy DNS, not about how the software is designed or works. Nothing here explains why he feels open source DNS software can't be used in mission critical situations. All he explains is how he thinks it can be deployed in sloppy ways. Commercial software can be deployed sloppily just as easily as open source software. Nothing inherent in the software prevents or aids the lazy administrator in being lazy.

GCN: Why is DNSsec not more widely deployed?

TOVAR: Not all DNS implementations support it today, and getting vendors to upgrade their systems is critical. Just signing one side of the equation, the authoritative side, is a good step, but it’s only half the battle. You need to have both sides of the DNS traffic signed. And there is no centralized authority for .com. It is a widely distributed database and you have to have every DNS server speaking the same version of DNSsec. So the obstacle to DNSsec deployment is fairly huge. It is going to take government intervention and wide-scale industry acknowledgment that this is needed.

And here, at the end, is where I think Mr Tovar shows his ignorance of DNS operations in general, and in particular the operation of critical infrastructure. There is, in fact, a central authority for the .COM zone. That authority is Verisign; they manage and publish the .COM zone from one central database. Mr Tovar is, perhaps, thinking of the .COM WHOIS database. The WHOIS database holds the details of who holds each registered domain, and is indeed distributed among the .COM registrars.

This sort of fundamental misunderstanding of the DNS infrastructure is a good indicator of just how much weight should be given to Mr. Tovar's opinions on how that infrastructure is managed.

UPDATE: Florian Weimer raises an interesting question, one that I'm embarassed to say didn't occur to me when I was originally writing this. Even though the question itself is facetious, the point is valid. The data sheets [PDF] for the main product that Mr. Tovar's company markets say that it only runs on open source operating systems. How can he keep a straight face while on the one hand claiming that his product is suitable for mission critical applications but open source isn't, while on the other hand his software only runs in open source environments? It's quite baffling.