An hour or so ago, I tried to check a Wikipedia entry and my browser told me it couldn't find en.wikipedia.org.  Surely that's wrong, I thought, but pushed "Check Wikipedia" onto the stack and went on to something else.  Then, coincidentally, while searching for DNS-related news articles to inspire my next blog entry, I ran across this one from PC Magazine.  Turns out Wikipedia's European data center had an overheating problem that caused many of their servers to shut down in an act of self-preservation.  To shunt European traffic to their servers in Florida, they enacted their failure procedure, which modifies their DNS records.

Unfortunately, that failover mechanism was broken (they didn't specify how), and broken so badly that it interrupted DNS resolution for all Wikimedia sites globally.  While they quickly recognized and fixed the problem, it took as long as an hour for the corrected data to propagate because of TTLs.

Read more...

...especially at the expense of the excellent folks who run .SE, but wasn't I just writing last month about everything that can go wrong in a manually administered DNS environment?  In fact, didn't I specifically say:

"Use a trailing dot to prevent the origin from being appended to a domain name.  After editing a zone data file, increment the serial number and reload. Forget any one of those and you've caused an operational issue, maybe even an outage."

Well, it looks like .SE had one of those very problems.

Read more...

Over the past few years--I can't remember exactly when, which is part of the problem--I've become alarmingly forgetful.  I'll get up, walk across the building to do something, and forget completely what it was that I intended to do.  Talk to Julie about upcoming roundtables?  Ask Eric or Arlen a question about UI design?

That's a nuisance for me around the office, but it would be downright dangerous if anyone still let me manage a production zone or name server.

Even in the simplest DNS environments, there's a lot to remember:  An SOA record has seven RDATA fields.  Use a trailing dot to prevent the origin from being appended to a domain name.  After editing a zone data file, increment the serial number and reload. Forget any one of those and you've caused an operational issue, maybe even an outage.

Read more...

According to a DHS report cited in FT's Tech Blog, DNS is "the part of U.S. information technology most at risk from a serious attack." The report lists several "mitigations" of the threats against DNS, including monitoring, infrastructure diversity, anycast (called out by name!) and DNSSEC.

Read more...

The slides from this morning's webinars are now available for downloading!  Enjoy!

Read more...