

Tumblr just dropped out of DNS - state_machine
https://www.tumblr.com/

======
sbierwagen

      @tumblr
        Tumblr has taken the site down in order to resolve a 
        network issue. We will update as we know more. 
    

<https://twitter.com/tumblr/status/279000741706878976>

~~~
emeraldd
Wow! Killing DNS is a nasty way to take down the site. That could take hours
to come back.

~~~
Silhouette
Given that they've been having can't-see-blogs-at-all scale problems for
something like 24 hours already (I was looking for one last night, and it's
currently nearly midnight UK time) I suspect a few more hours to clear up
whatever fundamental screw-up is behind this could be a good investment.

~~~
nwh
Seeing as they have engineers deploying code directly into production with
`vim`, a fundamental screwup was inevitable.

~~~
IgorPartola
No build process? In 2012?

Do you have a reference for deploying code with vim?

~~~
nwh
The source of their index.php, and the subsequent discussion on HN. Just
imagine someone franticly trying to get into insert mode, and saving on a
production server. It stayed like that for a reasonable period before someone
noticed.

<http://pastebin.com/raw.php?i=aPQJUh1Q>

<http://news.ycombinator.com/item?id=2343330>

~~~
fryguy
That doesn't necessarily mean that they don't use source control or build
control. The index file linked has tons of production-specific configuration
information (database ip/keys), which wouldn't be checked into source control
under best practices. That would require manual editing on the server to set
up correctly.

~~~
bashtoni
s/manual editing/mangement from configuration management like Puppet or Chef/

------
state_machine
Incidentally, this means every startup (and some of the bigger kids) that
hosts its status page on Tumblr is missing a status page at the moment... eg:
<http://status.twitter.com>

~~~
munificent
There's something beautiful about:

    
    
        * Tumbler reports its outage status using twitter.
        * Twitter reports its outage status using tumbler.

~~~
TallboyOne
Beautiful

------
emeraldd

      dig +trace www.tumblr.com
    
      *snip*
    
      tumblr.com.             300     IN      SOA     pdns1.ultradns.net. hostmaster.tumblr.com. 2012121602 86400 7200 604800 300
      ;; Received 108 bytes from 204.74.108.1#53(204.74.108.1) in 21 ms
    

Looks like their dns is down alright. You could try in your hosts file:

    
    
      72.32.231.8 www.tumblr.com tumblr.com
    

The ip is from their whois info and appears to be giving the We're sorry error
message.

~~~
rtkwe
Huh the only way you can tell 72.32.231.8 redirects to tumblr's crash page is
the link under find out why. Completely unbranded.

------
Pr0
Seems it started with just a normal outage:
[http://thenextweb.com/insider/2012/12/12/tumblr-confirms-
use...](http://thenextweb.com/insider/2012/12/12/tumblr-confirms-users-are-
experiencing-slow-loading-and-intermittent-errors-engineers-are-on-it/)

------
brokentone
Remember when we all decided Tumblr wasn't stable to host a professional site?
Well... these people didn't.

<http://theatlantic.tumblr.com/> <http://fox411.blogs.foxnews.com/>
<http://motherjones.tumblr.com/> <http://gq.tumblr.com/>
<http://tumblr.elle.com/>

~~~
blake8086
If the cost of hosting elsewhere exceeds the cost of the downtime, they made
the right choice.

~~~
brokentone
_Used to work at Fox, helped write the theme_

Lot lot lot more costs involved, including an inane templating system that
took way longer to work with than to build normally. Hosting would have
happened on the existing platform adding little to no cost. Also, the cost
(lost revenue, people running around pointing fingers) of downtime for a site
that large (yes, even the entertainment blog) quickly DWARFS any possible
hosting costs.

edit:formatting

------
bbuffone
Visit this page to see the length of downtime ->
<http://www.websitetest.com/ui/tests/50c922b17a6c8757bb000005>.

The test will run every 10 minutes for the next 10 hours. Testing is only good
for diagnosing issues like downtime and performance issues.

------
rdl
They get hundreds of extra points for hosting their status page on their own
service.

That's basically the main (already widely accepted) lesson people should take
from this -- people want twitter updates as well as an outside-hosted blog and
monitor for service availability.

~~~
Achshar
Doesn't serving status page form the site itself defeat the purpose of status
page? In what case would that make any sense? I am now intrigued, this is not
a small thing that can get over looked.

~~~
oh_sigh
No, it does not defeat the purpose. Instead, it just lessens the effectiveness
of the status page. It really depends on what tumblrs most common error
scenario is. 95% of errors may manifest themselves in manners which do not
affect the status of the status page.

------
el_cuadrado
Looks like they have MX and TXT, but missing A. Weird.

------
vhost-
Speculation: Gmail, Facebook, then Tumblr? Kill switch?

Half kidding...

~~~
beagle3
I was having the same thought.

But until proven otherwise, I'll assume strange coincidence. Although, when
microsoft.com falls of the face of the earth tomorrow, I'm calling it a
conspiracy.

------
cllns
The main site/webapp is still down (3 hours later) but individual sites are
up. <http://theparisreview.tumblr.com/>

They're returning 66.6.36.7 for DNS.

------
kondro
Could someone refresh my memory on how Tumblr makes its revenue?

~~~
citricsquid
Sponsored posts, premium themes, promoted posts. Pretty sure they're far from
profitable, but they're bringing some revenue in.

<http://www.quora.com/Tumblr/How-does-Tumblr-make-money>

<http://www.businessinsider.com/tumblr-revenues-2012-9>

~~~
cft
If they are not profitable, it means that the downtime is actually improving
their financials...;)

~~~
ChuckMcM
Chuckled at that, given their scale I'm sure they are already at fixed rates
for colo space and bandwidth.

------
tmx
That's pretty bad.

Admittedly, they update their zone very frequently (every time a user signs
up/changes name/deletes themselves), but you'd think they would have an
independent secondary DNS provider somewhere.

~~~
cmelbye
Do they really? It seems more likely that they'd simply use a wildcard.

~~~
tmx
That'll teach me to post late at night. I forgot wildcards existed.

I was thinking that (if I built tumblr) they would do subdomain searching at
the DNS level to avoid hitting their database.

Makes it even worse if they are using wildcards.

~~~
X-Istence
But if you don't publish an A record you get an ugly browser error message,
whereas publishing the wildcard gives the user a notice saying that no such
blog exists.

------
kordless
CloudFlare. Seriously guys.

------
onetwothreefour
Looks like UltraDNS disabled their zones. Someone probably forgot to pay
invoices or something.

You're pretty much screwed in this case.

~~~
emeraldd
What are you basing the "disabled zone" statement on? A 'dig ANY tumblr.com'
shows MX, TXT, and NS records . . .

