Waaaay back when I ran hp.com, I had what I only now realize was an enviable position:  I was HP’s hostmaster (the somewhat-ceremonial title given to the person responsible for a zone) but not much else.  I dabbled in NTP and ran a big mail relay, but the bulk of my responsibility was DNS.  From when I got to work in the morning to when I left in the evening, I could concentrate on DNS.

At the time, I didn’t realize what a luxury that was.  I figured every big company probably had a person dedicated to DNS.  And in those days, some did. Partly, this was because we hostmasters could get away with it.  DNS was such a black art that you could simply assert that it took up most of your time and your management wouldn’t know any better.

How the times have changed.  I’ve had the opportunity to meet the folks responsible for DNS at many big companies, but I hesitate to call them “hostmasters”—not because they don’t deserve the customary title, but because it sells them short.  These people run routers, switches, firewalls, mail servers, and more.  Almost no one has the luxury of specializing in DNS any more.  The economic climate dictates that we all take on more responsibilities to make our employers more competitive.

Read more...

An hour or so ago, I tried to check a Wikipedia entry and my browser told me it couldn't find en.wikipedia.org.  Surely that's wrong, I thought, but pushed "Check Wikipedia" onto the stack and went on to something else.  Then, coincidentally, while searching for DNS-related news articles to inspire my next blog entry, I ran across this one from PC Magazine.  Turns out Wikipedia's European data center had an overheating problem that caused many of their servers to shut down in an act of self-preservation.  To shunt European traffic to their servers in Florida, they enacted their failure procedure, which modifies their DNS records.

Unfortunately, that failover mechanism was broken (they didn't specify how), and broken so badly that it interrupted DNS resolution for all Wikimedia sites globally.  While they quickly recognized and fixed the problem, it took as long as an hour for the corrected data to propagate because of TTLs.

Read more...

...especially at the expense of the excellent folks who run .SE, but wasn't I just writing last month about everything that can go wrong in a manually administered DNS environment?  In fact, didn't I specifically say:

"Use a trailing dot to prevent the origin from being appended to a domain name.  After editing a zone data file, increment the serial number and reload. Forget any one of those and you've caused an operational issue, maybe even an outage."

Well, it looks like .SE had one of those very problems.

Read more...

Over the past few years--I can't remember exactly when, which is part of the problem--I've become alarmingly forgetful.  I'll get up, walk across the building to do something, and forget completely what it was that I intended to do.  Talk to Julie about upcoming roundtables?  Ask Eric or Arlen a question about UI design?

That's a nuisance for me around the office, but it would be downright dangerous if anyone still let me manage a production zone or name server.

Even in the simplest DNS environments, there's a lot to remember:  An SOA record has seven RDATA fields.  Use a trailing dot to prevent the origin from being appended to a domain name.  After editing a zone data file, increment the serial number and reload. Forget any one of those and you've caused an operational issue, maybe even an outage.

Read more...