In Which I Discuss Getting Properly Motivated

I have a confession to make.

For some time now, I've been a heavy TED user.

It started off innocently enough: a bit of cutting edge physics here, a new idea in ergonomics there. Before I knew it, I was spending hours at a time mainlining new talks... as many as I could get my grubby little hands on.

For those who are not familiar with TED, I'll say only that it's a highly addictive source of fascinating information, compelling thought, and inspired discourse. Those already familiar with TED know what I'm talking about. For the rest of you, begin viewing it at your peril because, like Lays, you can't stop at just one.

Anyway, all of that is beside the point. For now I'd like to talk about a particular talk I just saw given by Dan Pink, on incentives and motivation.

You can go watch it later, if you want to risk visiting the site. For now, I'll sum up to say that most of what he talks about is extrinsic motivators ("I'll give you a bonus if you get this done faster") and their effects on tasks that require creative thought. He references several studies which state, put simply, that typical carrot and stick motivators work great for simple mechanical or procedural tasks that have a clear path and end result, but that for virtually everything else these sorts of motivators either do not work, or in many cases actually harm productivity. Extrinsic motivators narrow our focus, and restrict creativity, reducing productivity in areas that require us to be creative.

The solution he proposes, and again he references studies to support the idea, is to use intrinsic motivators... incentives built into the work itself. Specifically, the incentives he refers to are autonomy (our desire to direct our own lives), mastery (our desire to improve at something that matters to us) and purpose (the desire to work as part of something larger than ourselves). Autonomy is key to his talk, and he references several cases where high levels of autonomy result in even higher levels of productivity from creative workers.

My first reaction to this talk was, "okay here we go, more studies into the obvious." Like the recent study by UK Music that states that lots of kids download music. But the more that I thought about it, the more I realized that this isn't such an obvious conclusion for most people. I happen to work in an industry where the most productive people tend to be those quirky loners, wearing all black, with an unusually high tendency toward Asberger's. There's a certain percentage of us that are treated just a bit differently, partly because we're all socially stunted to wildly varying degrees, and partly because big business depends on us so much to keep the lights on and you don't want to upset the emotionally fragile guy in the basement that could cripple your entire business. And as we progress in our careers, and we can make more demands of our employers, that special treatment only grows.

In the last ten years, I don't think there's been more than two days in any given week that I've shown up at the office before 10am. I often work long hours, especially compared to most nine-to-fivers, but I've tended to work for employers that recognize that my job puts odd time demands on me. For example, it's not unusual for me to have to do scheduled maintenance outside of regular work hours, or to be paged in the middle of the night. And so, I've experienced an incredible degree of autonomy over the years, culminating in my current position where I can work pretty much where and when I want, as long as the work gets done. This is how I'm able to be sitting on my balcony at 4am, with a beer, writing, instead of sleeping so that I can be at work at 8:30 in the morning.

I occasionally forget that most of the workforce doesn't experience this degree of self-determination, and thus my flawed first reaction that Dan Pink is telling us things we all already know. So maybe it's not so obvious to everyone, but I think the fact that it seemed obvious to me, due to my experience, shows what truth there is to what he has to say. And anyway, as Pink says in his talk, this isn't a feeling, or philosophy, it's science.

So if science tells us that virtually the entire industrialized world is doing it wrong, why are we still doing it that way? My answer is, the momentum and dogma of middle management. I've seen both represented in several managers that I've seen in action.

The dogma is that employees will not work unless someone is watching them. This is perhaps true for some employees.. but if you have these employees you already have a problem. It's best to let them be lazy, and fail, so that they can be discovered and removed, rather than keep them on a tight leash to make sure they're engaged in some passable, minimum effort.

The momentum is the managers' own work styles. I've had this conflict with a couple of managers in the past: due to the fact that my career is all about managing servers on networks in many locations, there is no requirement that I do my job from any one location, however some of my managers have been incapable of handling a remote employee who they cannot see, or speak to in person. I use the metaphor of momentum to describe this because I believe it will taper off over time as the older managers, who are not used to the online world with dozens of methods of instant communication slowly retire. As younger workers who are used to communicating in ways that do not provide the additional bandwidth of face-to-face communication take over, this will be less of an issue.

I'm quite happy that my current managers do not have either of these problems, but I have had to deal with them in the past. And, most people deal with them on a day to day basis, though they might not realize it.

So then how do we solve the problem? Is it simply a waiting game, where we hold our breath and wait for a slow evolution in the way businesses manage their people, or is there some revolutionary step we can take to change the minds of hundreds of thousands of managers convinced that this is the right way, as well as the millions of employees sold on the idea of bonuses and stock options? I've only been thinking about this for a few hours, so I don't have a solution yet, but I'd bet than Dan Pink has some ideas. After all, he has a book on this very subject due out soon.

I, for one, will be looking forward to the rest of what he has to say.

On Securing the DNS

I don't plan to make a habit of talking about things specific to my job here... in fact, I will almost never do that. It's just easier than always having to disclaim any relationship to the views of my employer, and so forth. However, today I can't help but toot our own horn.

A few hours ago, Public Internet Registry (PIR – the manager of the .ORG Internet domain name) announced that the .ORG zone has been secured with DNSSEC, the DNS Security Extensions. This makes ORG the largest Top Level Domain that has been signed to date, and the only open registry to implement DNSSEC (open in the sense that all of the other signed TLDs are at registries which have restricted registration policies: six national TLDs, and .GOV).

Undead is the New Chic

I'm not sure what it is... maybe it's the economy... maybe it's part of the overreaction to the swine flu (more on that later)... or maybe these things just come in cycles like cicada broods.. but whatever the reason, zombies seem to be on the rise right now.

Just this morning I was tweeting that in 28 days it will be the six year anniversary of my move to Ottawa. Of course, that led to an obvious joke. Almost immediately I heard back from @bestswineflu about his posted defense tips for the coming swine-flu-fed zombie apocalypse. It turns out that right about the same time, a friend of mine was announcing the grand opening of his new web site, The Daily Zombie – a news site for the informed zombie. Not to be left out of the zombie action today, @thinkgeek chose this afternoon to pass on this video showing the zombie defense training being inflicted on two Japanese kids. And that's just today.

Does the collective unconscious know something about the future that us poor individuals are missing?

Wild Thing, I Think I Love You

So it's out. The first trailer for the Where The Wild Things Are movie was released this morning, and it makes the movie look like everything I hope it will be. I'm still a little afraid though... it really is an iconic book from my childhood, and I have to wonder if any movie, especially a live-action film with people in Wild Thing suits, will be able to live up.

A few friends that I've directed my excitement at have responded with weird looks and statements like, "I have no idea what you're talking about." To those people I say, "your childhood is incomplete. Go back and get your Sendak credit."

Thru You: Kutiman Remixes Youtube

A couple of days ago a link started going around for Thru You, a musical project by Kutiman, an Israeli musician and producer. What he's done is take several dozen Youtube videos, chosen for their audio content, and cut and splice them together into a collection of seven brand new, original pieces of music. It's an amazing technical feat to begin with, but the music he's created is incredible in its own right.

The first and last time I saw anyone do something like this was in 1998 when Coldcut and Hexstatic got together to release the Timber EP (Timber remixes video of logging operations, and has a strong anti-clearcutting message). But even Coldcut/Hexstatic didn't quite commit to the concept the way Kutiman has; Timber contains several audio tracks that clearly aren't from the video sources, including some synthesizer sounds.

Thru You is also an incredible mix of styles; funk, dub, R&B, and even big-beat electronic. All seven tracks are excellent, and if I can get my hands on some clean mp3s they'll be going into high rotation on the home stereo. The first track, The Mother of all Funk Chords, is the best demonstration of the video remix concept, but my favourites are probably This is What it Became, an awesome dub track, Babylon Band, which is a bit like an Eastern-European Prodigy meets Nusrat Fateh Ali Khan, and Just a Lady, a nice slow R&B tune.

The Thru You web site has gone down at least once, probably due to its popularity, so Katiman has posted some alternates on Youtube itself. The videos on the main site seem to be better quality, so it's probably best to view them there (you can also see the original video sources that way by clicking on the Credits link). But, just in case, I'm including links below to the Youtube postings.

The Mother of all Funk Chords
This is What it Became
I M New
Babylon Band
Someday
Wait For Me
Just a Lady

Load Balancing DNS Using Cisco's IP SLA Feature

It's generally accepted that using any sort of stateful load-balancer in front of a set of DNS servers is a bad idea. There are several reasons for this, but my favourites are that:

1. it means adding an unnecessary potential point of failure
2. the state engines in load-balancers aren't scaled for DNS, and will be the first thing to fail under heavy load or a DoS attack

The issue with the state engine in a load-balancer is that it is scaled for average Internet traffic, which DNS is not. Each DNS connection (each flow) is typically one UDP packet each way, and well under 1KB of data. By contrast, with most other common protocols (smtp, http, etc.) each flow is hundreds to millions of packets, and probably tens, hundreds or even millions of KB of data. The key metric here is the flows:bandwidth ratio. Devices are built so that when they reach their maximum bandwidth capability, there's room in memory to track all of the flows. The problem is, they're typically scaled for average traffic. Since the flows:bandwidth ratio for DNS is so very much higher than other types of traffic, you can expect that a load-balancer in front of busy DNS servers will exhaust their memory in trying to track all the flows long before the maximum bandwidth of the device is reached. To put it another way, by putting a load-balancer scaled for 1Gb of traffic in front of DNS servers scaled for the same amount of traffic, you actually drastically reduce the amount of DNS traffic those servers can handle.

There are better ways.

ISC, the maker of BIND, has an excellent technote which describes using OSPF Equal Cost Multi-Path (ECMP) routing to distribute load between a set of DNS servers. In effect, it's a scheme for doing anycast on a LAN scale, rather than WAN. Put simply, it involves using Quagga or some other software routing daemon on each DNS server to announce a route to the DNS service address. A wrapper script around the DNS process adds a route just before the process starts, and removes it just after the process exits. The approach works quite well as long as the local router can handle OSPF ECMP, and as long as it uses a route hashing algorithm to maintain a consistent route choice for each source address without needing a state table. For example, the Cisco Express Forwarding (CEF) algorithm uses a hash of source address, destination address, and number of available routes to produce a route selection.

The down sides to the ISC method are that there's a small amount of complexity added to the management of the DNS server itself (for example, you can no longer use the standard application start/stop mechanisms of your OS for the DNS software) and the risk that a failure may occur which causes the DNS software to stop answering queries, but not exit. If the latter occurs, the route to that server will not be removed. This is pretty safe with BIND, as its designed to exit on any critical error, however that's not necessarily the case with all DNS server applications.

There's another method available (that I'm going to describe here) which, while being very similar to the ISC methodology, does not have these particular flaws. I should point out here that the method I'm about to describe is not my invention. It was pieced together from the ISC technote and some suggestions that came from Tony Kapella while chatting about this stuff in the hallway at a NANOG meeting a while back. After confirming how easy it is to get this method to work I've been singing its praises to anyone who will listen.

At a high level it's quite similar to the OSPF method. The DNS service address is bound to a clone of the loopback interface on each server, and ECMP routing is used, but rather than populating the routes with OSPF and running routing protocols on the DNS servers, route management is done with static routes on the local router linked to service checks which verify the functionality of the DNS service.

Setting It All Up

In this example, we'll use the RFC 3330 TEST-NET. The service address for the DNS service will be 192.0.2.253. This is the address that would be associated with a name server in a delegation for authoritative DNS service, or would be listed as the local recursive DNS server in a DHCP configuration or desktop network config. The network between the local router and the DNS servers will be numbered out of 192.0.2.0/28 (or 192.0.2.0 through 192.0.2.15). The server-facing side of the router will be 192.0.2.1, and that will be the default route for each of the DNS servers, which will be 192.0.2.10, 192.0.2.11 and 192.0.2.12. This network will be the administrative interfaces for the DNS servers.

Once the servers are reachable via their administrative addresses, make a clone of the loopback interface on all three servers. Configure the second loopback interface with the DNS service address.

On FreeBSD, the rc.conf entries for the network should look something like this:

defaultrouter="192.0.2.1"
cloned_interfaces="lo1"
ifconfig_em0="192.0.2.10 netmask 255.255.255.240"
ifconfig_lo1="192.0.2.253 netmask 255.255.255.255"

It's a little more difficult to represent the configuration under Linux since it's spread across several config files, but the above should give you a pretty good idea of where to start.

Once the network setup is finished, configure your DNS server software to listen to both the administrative address and the service address. So, on the first DNS server, it should listen to 192.0.2.10 and 192.0.2.253.

That's all that needs to be done on the servers. Note that doing this was far simpler than configuring the servers to run OSPF and automatically add and remove routes as the DNS service is started or stopped.

The last few steps need to be taken on the local router. The first of these is to configure the router to check up on the DNS service on each of the three servers and make sure it's running; this is where Cisco's IP SLA feature comes into play. Configure three service monitors, and then set up three "tracks" which will provide the link to the service monitors.

ip sla monitor 1
type dns target-addr www.example.ca name-server 192.0.2.10
timeout 500
frequency 1
ip sla monitor schedule 1 life forever start-time now
!
ip sla monitor 2
type dns target-addr www.example.ca name-server 192.0.2.11
timeout 500
frequency 1
ip sla monitor schedule 2 life forever start-time now
!
ip sla monitor 3
type dns target-addr www.example.ca name-server 192.0.2.12
timeout 500
frequency 1
ip sla monitor schedule 3 life forever start-time now
!
track 1 rtr 1
track 2 rtr 2
track 3 rtr 3

This sets up three IP SLA Monitors which repeatedly query the administrative address on each server for the A record www.example.ca. The DNS server must respond with an A record for the QNAME you use; if it is unable to respond, or responds with a different record type, the monitor fails. In the example above the monitor attempts the lookup every second (frequency) and fails if it doesn't receive a valid A record within 500ms (timeout). You may need to experiment with the timeout value, depending on how responsive your DNS servers are. If you find individual servers appear to be going out of service when the daemon is still operating fine you might have the timeout value set too low.

With the monitors in place, turn on CEF and then configure three static routes to the service address via each server's administrative address. The routes are linked to the service monitors using the track argument:

ip cef
!
ip route 192.0.2.253 255.255.255.255 192.0.2.10 track 1
ip route 192.0.2.253 255.255.255.255 192.0.2.11 track 2
ip route 192.0.2.253 255.255.255.255 192.0.2.12 track 3

And that should be it. DNS queries arriving at the external interface of the router bound for 192.0.2.253 should now be routed to one of the DNS servers behind it, with a fairly equal load distribution. Since the router is using a hashing algorithm to select routes the load distribution can't be perfect, but in practise I've found that it's incredibly even. The only likely reason to see an imbalance is if your DNS servers receive an unusually high percentage of their queries from just one or two source addresses.

It's important to point out that most of the cautions in the ISC technote, particularly in reference to zone transfers and TCP DNS, apply equally here. I highly recommend reviewing the ISC document before implementing this in production.

Of course, there is still one big downside to this particular method of load balancing: it's depedant on one particular vendor. I have not yet found a way to reproduce this configuration using non-Cisco routers. If anyone is aware of a similar feature available from other major routing vendors please let me know and I'll integrate instructions for those routers here.

The Pledge

I think it's amazing the degree to which Obama has managed to inspire a new kind of patriotism among American citizens. For too long, American patriotism has been about how the US is cool just for being there. But really, what has the country done lately that Americans can be proud of? In the last few years a lot of people have been waking up to this, and attitudes are starting to change.. slowly... and I think Obama will be the catalyst to cause a new attitude of doing something about it to spread like wildfire.

People who criticize artists for speaking out about issues should love this video. Personally, I love the idea that there are people willing to try to use their celebrity to educate and affect important issues. But, for those who don't like to hear what celebrities think you should do, here is what they pledge to do themselves, and a challenge to find your own.

For the other geeks out there, this seems like a great start. Are there other service projects geeks can get involved in?