
Spamhaus Nightmare: Domain Shut Down, No Notice, Over A Million Pages Down - jpadvo
I am writing this here on HN, because at the moment you cannot access our blog. Our domain name was shut down this morning, and I'm trying to get it back. Here's what happened...<p>Our company provides tools to help people put together pages for their businesses. Our free tool has been used to create over million page tabs on Facebook. Unfortunately but predictably, sometimes bad people use our app. Like spammers.<p>Overnight, our domain was blacklisted by Spamhaus because <i>one</i> of our pages contained spam. (Anybody want a free iPad?)<p>We run our infrastructure on Heroku, and use Bluehost for domain names. Well, as soon as Bluehost recieved notice from Spamhaus, they shut off the DNS for our domain. All million plus pages, gone in the blink of a DNS propagation.<p>Thankfully we were able to switch over to [appname].heroku.com for now and most of the pages are back, but we have paying customers who are in the dark because they rely on our custom domain name.<p>Our product, that over a million people rely on, suddenly ceased to exist. No advance notice. Nothing we could have done to stop it. Because of ONE bad apple.<p>This kind of thing will happen in SOPA world, if we let ourselves get there. But instead of being able to call my registrar and yell at them, I would have had to call the government, and oh-by-the-way they might fine or imprison me for having hosted spam.<p>Let me end with a practical, really-important-to-me-right-now question: is there any possible way to <i>not</i> get randomly nuked by Spamhaus?
======
pg
The Spamhaus people are bad guys. I gradually realized that during the time I
worked on spam filters. They presumably started out with good intentions, but
the position they're in has corrupted them.

It's true of a lot of the guys running blacklists. And more generally, of a
lot of people in the position of police. You tend to become a mirror of
whatever bad guys you're fighting. Your tactics have to match theirs, and
pretty soon your principles start to as well. I suspect this tendency is so
universal that you have to make a conscious effort to avoid it.

~~~
pg22
Oh my, same old Paul. Just can't stop holding a grudge. Seems he forgot to
mention the reasons why he still feels the need to badmouth Spamhaus (other
than perhaps nothing better to do these years, lack of new ideas?).

Firstly, Paul's grand "A Plan for Spam" method of using Bayesian filters to
stop all spam ("I think it's possible to stop spam, and that content-based
filters are the way to do it."). Uh, so, how'd that work out? Spammers quickly
figured out how to make a mockery of Bayes based solutions. And who is still
out there filtering spam using IP addresses & domain names? Spamhaus.

Then, what really got his goat was back in 2005 (yes, long grudge holding, one
wonders what he feels about the mail-carrier who lost a letter of his back in
'78 ;-) when his vanity site, shared-IP-hosted at Viaweb which had become
Yahoo! Stores was blocklisted at Spamhaus. Back then, Yahoo, and Yahoo Stores
were a spammer-hosting cesspool and Paul's page was wallowing in the center of
it. Rather than get to the bottom of it, Paul just got on a high-horse and
ranted about the evils of Spamhaus. A good take on the rant can be read here:
[http://www.circleid.com/posts/we_hate_spam_except_of_course_...](http://www.circleid.com/posts/we_hate_spam_except_of_course_when_its_inconvenient_to_do_so/)

So, multiple biases. How often people forget to mention those when they post
attacks. Now one must ask, who is the "bad guy" and is "corrupt" here?

But the Spamhaus people should be happy with the irony in Paul's hypocrisy.
How so? Well, his paulgraham.com's email is filtered by Spamhaus, as is his
ycombinator.com's email. As are the emails of most of the social/blog sites
he's on (posterous.com, etc.) One wonder how many of these still use "A Plan
for Spam"? Okay, that was rhetorical.

Lastly, the pop-psychology in his posting attests that Paul's degrees are in
philosophy, not psychology.

~~~
click170
You could have pointed out said hypocrisy without the ad hominem attacks, in
fact I wouldn't have felt compelled to down-vote you if you had. Instead your
comment reads more like a petty tantrum based on some grudge you (apparently,
continue to) hold against Paul.

~~~
frankwong
Totally off topic, but the markup for a down voted comment actually drew my
attention to it. Kind of a "nothing to see here" sign.

I do agree that the rant is quite unnecessary.

------
freejack
My recommendation would be to run your own DNS on your own IP addresses. Even
with the IP shortage, you should be able to get a small block delegated to you
that you can use for your mission critical apps. Once you've got that arranged
for, its a fairly trivial task to find a registrar with policies more
complimentary to your business.

If its mission critical for your business, then you can't afford to think like
a victim. Take charge of your infrastructure where you have to. Relying on
third parties is lean, but not always effective - a small amount of fat in the
right areas can give you a lot of flexibility (and insurance) that you might
not get when you rely on a third party.

~~~
dholowiski
Correct me if I'm wrong, but even if you run your own DNS servers, can't your
domain name registrar still decide to take away your domain name?

~~~
FreebytesSector
This is unusual, and it would likely only happen with a respectable registrar
if there was a dispute with ICANN.

~~~
nitrogen
...or an ICE/DHS seizure conducted independently of your registrar.

------
elliottcarlson
"is there any possible way to not get randomly nuked by Spamhaus?"

I guess the first step is to set up better monitoring services to prevent your
system from being abused by even one bad apple. Try to catch the abuse as
quick as possible so you won't raise red flags.

Additionally you should possibly work on segmenting out your customers. If
your paying customers are important to you, use a different system for them.
If this has the possibility of happening again you don't want to hurt those
customers from a similar thing happening again.

~~~
jpadvo
> I guess the first step is to set up better monitoring services to prevent
> your system from being abused by even one bad apple.

I'd love to, but when you have the volume we do there are going to be false
negatives. Bad apples will slip through. And if somebody slips through, we're
vulnerable for getting blacklisted.

> If your paying customers are important to you, use a different system for
> them.

This is a very good plan. I'm definitely looking into that...

~~~
elliottcarlson
> I'd love to, but when you have the volume we do there are going to be false
> negatives. Bad apples will slip through. And if somebody slips through,
> we're vulnerable for getting blacklisted.

This doesn't mean you have to auto-ban people - but you could easily setup
listings that you can quickly glance at to see what your monitoring found. If
you are picking up too many false positives, then you can refine your
monitoring. Yes - you can't prevent all of them, but once your system starts
getting abused you have to assume that others will do it as well.
Additionally, you could very well be losing money in service costs due to
these people (I don't know your business model so it's just an assumption) -
you want to protect that as well.

~~~
bitmonk
Right, that works really well for YouTube. The great problem that is created
when one tries to apply rules in this way is that, look, big companies won't
suffer, and small ones will unduly.

The internet is full of user-generated-content sites and the core objection to
SOPA is that we cannot police every posting. We do not currently have a legal
obligation to do anything other than respond to complaints.

Further, as a veteran of the hosting industry, I'm really disappointed in
BlueHost for taking action against a paying customer's domain name. Be sure to
read the SLA and TOS when you sign up for services.

~~~
elliottcarlson
I am against SOPA just like everyone else; however that does not mean we don't
have a duty to police our own sites. In this case, someone misusing a service
caused the whole service to suffer - including paying customers. This has
nothing to do with SOPA but with making sure your business runs properly, and
won't affect or cast a bad light on your service.

As for the disappointment at BlueHost - they should have probably let the
customer know before taking action - but other than that I think they did the
right thing. As a veteran of the hosting industry, if one of your customers
wordpress blog was hacked and hosting a phishing site, would you not disable
the site and let the customer know right away that they need to clear things
up? That's just a random example, but any sane company will protect their
servers via a TOS - if they didn't I would be quite concerned about the
service they are offering. Just my $0.02 on this..

~~~
symkat
I'm sorry, DNS/Registration is NOT the same as an exploited website.

There is nothing intrinsically bad about DNS that it needs to be turned off;
the OP has already said they were using different hosting.

BlueHost was in no way vulnerable, and in no way needed to protect itself, as
the only traffic was DNS requests. GoDaddy tries to pull the same thing with
disabling DNS[1].

I feel like we need a Chris Crocker video about DNS systems this month.

1:
[http://en.wikipedia.org/wiki/Go_Daddy#Suspension_of_Seclists...](http://en.wikipedia.org/wiki/Go_Daddy#Suspension_of_Seclists.org)

~~~
elliottcarlson
You are correct - this is different since they were only providing DNS, too
many hours had passed when I replied that I forgot that part of the story.

------
battaile
While I'm sorry this happened to you, and I'm as anti-SOPA as anyone (have
called my congress(wo)man, called Boehner and Canter when it looked like they
were going to sneak the vote through last week), this has nothing to do with
SOPA, and trying to invoke the name for something that you should've been
better prepared for is kind of a discredit to the cause.

~~~
glombus
I agree. I am anti-SOPA, but this is not censorship so much as an overzealous
attempt to stop spam, executed poorly. Did they do a wildcard
block/hold/whatever on a top level domain of yours *.mydomain.com so all your
sub domains got blocked? I'm guessing some goober at bluehost just went one
step farther than he should have in just removing the one DNS entry, and they
definitely should have contacted you. Sounds like poor customer service.

~~~
fr0sty
And if SOPA passes you will see suspiciously simil overzealuos attempts to
stop piracy. Same song, different verse but this time you get the federal
government involved which is a whole new level of fun.

~~~
FreebytesSector
This is true, and it makes me thing of all of the private sector solutions
that are already available. The market creates the solutions and does not need
government interference slowing it down or making dispute resolutions more
complicated.

------
jimbobimbo
Unfortunately there's no guarantee that anyone would escape Spamhaus' "love" -
they and other RBLs do more damage than spammers, in my opinion. The real WTF
in this story is Bluehost's reaction: shutting down DNS on one notice from
Spamhaus, really!?

~~~
whortleberry
More damage than spammers? I see you've never run a mailserver. I have and I
know that Spamhaus are one of the good guys doing a hard thankless job,
risking lawsuits and threats, in order to keep email as a useful tool.
Spamhaus' RBL is the most reputable of all of them, thanks to years of hard
work and sacrifice.

The only people who don't like Spamhaus, in my view, are those ISPs who were
happy to make money from selling connectivity to spammers while pretending in
public that they hated spam. Them, and people who don't understand what
Spamhaus do, like the author of this article, and who think Spamhaus are to
blame for their troubles.

~~~
SageRaven
The GP is correct.

As one who has worked in the trenches as a mail admin (small potatoes,
granted: a few small clients and a couple of small hosting companies), my
observation has been than customers bitch _way_ more about the MX servers
which reject mail from our servers than the amount of spam in their own in-
boxes. They don't give a shit that the _recipient_ is rejecting _legitimate_
mail -- they blame _us_ for their problems. All because some asshat with a
copy of TheBat! signed up and managed to send out a couple hundred "Russian
bride" spams before we were alerted and nuked the account. I could probably
fund a semester of college for some random kid with the time I've been paid to
waste on de-listing and convincing idiot admins that one of their customers
_really_ wants to get mail from one of mine.

Sure, 99.9% of email hitting the typical in-bound relay is spam, but CPU, RAM,
and disk I/O are _cheap_. Do per-inbox statistical filtering and let the
_user_ decide what spam is. Better yet, let client-side filters do the work.
Do you think any person would stand to allow a US Postal carrier decide what
was junk mail and then not deliver it? People just need to buck up and put in
a little of their own effort.

I haven't used an RBL (even if its just one in a battery of weighted tests,
such as with Spam Assassin) due to my loathing for the vigilante nature of the
RBL scene as a whole. If you operate an RBL -- fuck you. If you are an admin
that rejects mail based solely on being listed in RBLs, then fuck you, too. I
know I sound like an asshole myself here, but the existence of RBLs has caused
me and various mail end-users way more pain than any spammer has.

Bitter? Nah.

As a mail admin, I want to throw SMTP out the window. It wasn't spammers that
killed the protocol, but rather the growth of use of RBLs.

Rant aside, I do have a question to contribute to the discussion: Has one of
the larger RBLs _ever_ listed one of the huge mail providers (Gmail,
MSN/Hotmail, Yahoo?) for any length of time? I know I've gotten spams and
scams from all three.

------
ianlevesque
So its time to add Bluehost to the list of companies too unreliable to do
business with.

~~~
Nick_C
Funnily enough, I just got my Dreamhost bill for the next year and was
thinking of switching to Bluehost due to cost. Not any more.

------
jodrellblank
_Overnight, our domain was blacklisted by Spamhaus. Nothing we could have done
to stop it. Because of ONE bad apple._

Because of major internet infrastructure run at whim by 3rd party blacklists,
you mean.

 _is there any possible way to not get randomly nuked by Spamhaus?_

Spamhaus _and every service like them_.

------
yaix
>Bluehost I'd sue them for damages. WTF do they delete your domain from their
name server?! Get a more reliable registrar/name server. Spamhaus or similar
black listers can always accidentially list you. Go to their site and remove
your domain. No sane person/comapny should immediately assume anything but a
accidential listing.

------
dbe
Have you tried using Spamhaus's Blocklist Removal?
<http://www.spamhaus.org/lookup.lasso>

------
subway
This doesn't sound right. Since when does Spamhaus police site content? I'm
pretty sure they primarily go after folks sending out spam email, not after
websites containing spammy pages.

~~~
elliottcarlson
They also go after places that enable spammers - such as when they blocked
Google Docs [1]

[http://news.softpedia.com/news/Spamhaus-We-Blocked-Google-
Do...](http://news.softpedia.com/news/Spamhaus-We-Blocked-Google-Docs-Not-
Gmail-153093.shtml)

~~~
whortleberry
They added Google Doc ip addresses to their RBL, so that SMTP traffic from
those IPs would be blocked by those who chose to run Spamhaus' blacklist.

------
kfcm
Of course, you could just go to Spamhaus itself and attempt to remove your
domain from the DBL: <http://www.spamhaus.org/lookup.lasso?dnsbl=domain>

It could be your registrar is just running an automated process based upon
that.

------
jamespo
No one besides the particularly clueless should use spamhaus and similar
services as a black or white answer on whether to block, as they don't care
about friendly fire and are run by neckbeards.

Spamhas should be used as part of a body of evidence, like in spamassassin
scores.

------
Isofarro
Spamhaus normally collect evidence of abusive activity on their site. Look
there first at the accumulated evidence. I'd have a look myself, but I don't
know who you are, what domain you are using, what domain is being used to
spamvertise. Perhaps you can post the spamhaus evidence file and we can take a
look?

Also, Spamhaus makes recommendations. Third parties use their lists to filter
spam. It sounds unusual for a Spamhaus listing to result in a domain name
shutdown, unless the DNS provider did that based on a listing. So this is not
really Spamhaus' mistake (if indeed their evidence listing shows a history of
hosting spamvertised websites - then there is no mistake on the listing. You
could be listed either because your site/host/network has a solid history of
not dealing with spam/abuse reports quickly, or because a big spam operator
has landed on using your services. Are you sure it was just one site (and just
advertising a free ipad)?)

Yes, I understand you run a facebook static html tab content site. But that
isn't a million miles away from bog standard cheap/free hosting solutions that
form the bulk of spamvertised websites. Might be worth investing some time
looking at the parallels and how good cheap webhosts approach dealing with
spamvertised websites and spammers.

So I'd suggest finding the evidence file, dealing with the problem(s) listed,
then contacting Spamhaus with details of what you've done, and what's in place
to reduce future abusive activity (if it's more than one site offering a free
ipad). Then do something about your web hosting solution - that seems like a
very weak link - either build up a better relationship with them, or move.

------
dholowiski
"is there any possible way to not get randomly nuked by Spamhaus?"

As any email administrator will tell you, "no". The best you can do is take
measures to prevent abuse coming from your domain name/IP, but bad things
still do happen. You are still at the mercy of spamhaus (and other rbl
providers).

------
brador
If my host did that then reversed the decision I would still be moving out of
there as fast as possible.

There's absolutely no reason to be giving second chances to online services
with so much competition about, on what is, essentially, a commodity.

------
RealGeek
Boycott Bluehost?

~~~
sp332
No need for a boycott. If they're so unreliable, customers will move away
themselves.

~~~
sycren
I'm up for renewal in february with them, could you suggest a better host?

~~~
zabraxias
I use linode but I was happy with slicehost before also. Granted I've never
run popular apps/websites on these so I am not sure how they'd treat abuse
complaints.

------
giberti
As a fellow Facebook tab provider (My Tab) I feel your pain. I'm concerned
about how SOPA and ProtectIP will impact this class of service as it would be
impossible to police all content added via tools like ours. It's already been
said, but you can run your own DNS or even contract for DNS services from a
wide variety of places. I would move your name and SSL certificates to a
trusted registrar ASAP. Glad you were at least able to work around the issue
by pointing directly to the app.

------
conductor
"is there any possible way to not get randomly nuked by Spamhaus?"

Get the list (like the level1) of "evil" corporations/governments ip ranges
and show a picture of a pink elephant to them instead of your real content.

------
gasull
For the next time, have your blog at blog.example.com, and point it to another
hosting provider different than the one for example.com. That way at least you
prevent the blog from going down.

~~~
dangrossman
Doesn't work if the person pulling the site down is the registrar, and they do
it by hijacking your DNS. Your pointer to the other host goes away too.

------
snowwrestler
So do we just invoke "SOPA" for any little hiccup now? Spam blacklists are not
exactly a new issue on the Internet.

------
xsmasher
Can we get your app name, and a link to the spamhaus listing? It'd be nice to
hear the other side of the story.

------
whortleberry
I cannot ever recall seeing a more misleading and manipulative posting
attempting to garner undeserved sympathy by falsely trying to associate one's
case with bad legislation. This has nothing to do with SOPA, it is not
remotely related to anything SOPA, and at worst, these kinds of false
analogies only serve to weaken the case against the very real harm that SOPA
will do.

Spamhaus are not the villains here. First of all, you make the absurd
complaint that Spamhaus "blacklisted" your domain. That is a lie. Spamhaus
runs an SMTP blacklist of ip addresses that some other SMTP providers use, not
all. There is no way for Spamhaus to blacklist anyone's domain.

So what actually happened? Spamhaus detected a spammer website hosted on your
company's ip addresses, and they did the responsible thing. They reported the
spam website to the ISP hosting it.

As for your claim that Bluehost shut off the DNS, why aren't you ringing up
Bluehost to demand that they restore it? You might find that a better use of
your time than making these absurd allegations and trying to win sympathy by
making comparisons to SOPA where none exist.

~~~
Gigablah
The point here is the removal of due process. His DNS was shut off by Bluehost
without any warning -- in this case there was actually abuse, but what if it
were a false positive?

------
fleitz
Use a reputable host, you may have to pay more than a few dollars a month.
I've dealt with numerous spamhaus complaints, they generally result from
idiotic users who send messages to spamhaus instead of clicking unsubscribe.
If you spend an hour creating a really detailed form letter response it makes
the AUP tickets go away quickly. When I sign up for hosting I detail exactly
what we do and pay appropriately, most 'cheap' hosting places exist solely to
pick up the remainder of the months service fee from a dubious spam complaint.
If you spend $100 - $200 per month it's pretty easy to find a hosting provider
that will let you run a single opt-in list, especially if you detail this up
front in writing and refer to this in your response to any spam complaints.
Web marketing shouldn't be a problem for any real hosting provider, unless you
have extremely dynamic load I'm not sure why you'd bother with heroku. It's
only a couple hours work to setup your own infrastructure. A quad-core server
for $130 a month will run circles around what heroku provides for $130 per
month.

------
bmnbug
Is this your problem? Seems you do have a spammer plastering crap on facebook.

<body> <form name="redirect_form"
action="<https://statichtmlapp.heroku.com/tab/1/show> method="post"> <input
type='hidden' name='signed_request'
value='fJSfey7ELpgNY4r3gZFT5DyXp0MoW4TF2DsNQWwcoTY.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImlzc3VlZF9hdCI6MTMyNTM1ODI1NywicGFnZSI6eyJpZCI6IjI4MjczNjc1ODQxMjg4MCIsImxpa2VkIjpmYWxzZSwiYWRtaW4iOmZhbHNlfSwidXNlciI6eyJjb3VudHJ5IjoidXMiLCJsb2NhbGUiOiJlbl9VUyIsImFnZSI6eyJtaW4iOjIxfX19'
></input> </form>

~~~
blhack
Would you mid deleting (or editing) this comment? You're breaking the page.

~~~
bmnbug
Not sure how, can't find an edit or delete button. (except for this reply, it
has both. But the higher level one does not)

------
bmnbug
You mean this crap being shot all over folks comments?

<https://www.facebook.com/FreeiPAdd2>

action="<https://statichtmlapp.heroku.com/tab/1/show> method="post"

value='fJSfey7ELpgNY4r3gZFT5DyXp0MoW4TF2DsNQWwcoTY.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImlzc3VlZF9hdCI6MTMyNTM1ODI1NywicGFnZSI6eyJpZCI6IjI4MjczNjc1ODQxMjg4MCIsImxpa2VkIjpmYWxzZSwiYWRtaW4iOmZhbHNlfSwidXNlciI6eyJjb3VudHJ5IjoidXMiLCJsb2NhbGUiOiJlbl9VUyIsImFnZSI6eyJtaW4iOjIxfX19'

~~~
blhack
Would you mid deleting (or editing) this comment? You're breaking the page.

------
bmnbug
www. facebook. com/ FreeiPAdd2

Like that spam site being plastered all over the place?

------
zackzackzack
First suggestion: get an ip address people can remember. Not very practical I
know, but I guess that is the only way to get by without DNS at the moment.

Also: you've emailed your customers the new address yes? Even if it is only
temporary? Maybe buy a new domain and point them towards that:"Please use
[FINGSOPA].com while we get everything back to normal."

