
Sendgrid is down - sunils34
http://support.sendgrid.com/entries/23417077-21-Mar-Website-Down-Refusing-Mail
======
grey-area
I'm sad to see that HN is being used as a platform for this sort of demagogy,
and sad that stories about this tawdry little tale are indiscriminately voted
up above other stories which are far more interesting.

DDOS of a mailing service that lots of websites rely on because a completely
unrelated company decided to fire someone is not an occasion for _lol_ and
_schadenfreude_ as some posters here would have it. As a method of justice it
has more in common with a lynch mob than a court of law - this isn't going to
get the guy's job back, and it certainly isn't going to teach anyone a lesson,
apart from that the internet is fickle, and monumentally stupid. But I very
much doubt the people behind this attack are interested in justice or truly
care about the man who lost his job, they're just doing it for the lolz and
are punishing the internet at large over a silly little dispute at a tech
conference.

Congratulations to the mob, I guess; it has shown its power, if not any sense
of discrimination or proportionality.

~~~
kybernetyk
Now you can argue as much as you want - but the (sad) truth is that mob
mentality exists and we will have to deal with it for at least the next few
centuries.

This should be a lesson for companies that hire highly confrontational people
as their official community representatives.

Yes, in a perfect world there shouldn't be DDOS' and other attacks because of
a tweet about an immature joke but it's not a perfect world and we should be
very wary what personalities we hire to represent us.

As much as I don't like it - the "right" thing for sengrid would be to replace
their 'developer evangelist' with someone who's less confrontational. Yes, it
sucks. But as businesses we have to deal with reality.

~~~
lopatin
If she was confrontational she would have confronted them. Replace
'confrontational' with 'passive aggressive' I totally agree.

------
aculver
This was a Heroku win for us. I just visited their add-ons page, identified a
comparable provider, and restored mail in our application with the following:

    
    
        heroku addons:add mailgun:basic
        heroku addons:docs mailgun:basic
        vim config/initializers/mail.rb
        git add config/initializers/mail.rb
        git commit -m "Switching to back-up mail provider."
        git push heroku master
    

Since the account creation is all automatic and billing is all through Heroku,
I never even had to visit Mailgun's website.

~~~
adambenayoun
How do you handle SPF/DKIM and domain keys records? Do you send out a lot of
emails? Have you had a mailgun account ready?

~~~
aculver
I don't know anything about this. If someone could provide some
details/resources around this I'm sure it would be valuable to folks right now
who need to evaluate the risks of switching providers on a short-term basis.

(For us in particular, SendGrid only represents a small amount of the email we
send and it's mostly internal emails. The problem for us was that the
ActionMailer emails aren't sent in a background process, so this caused a
couple request timeouts.)

~~~
swampthing
If memory serves, if you can log into Mailgun, it will have some text on how
to set up all those things on your dashboard. Congrats on switching - Mailgun
has vastly superior deliverability, at least in my observation.

~~~
adambenayoun
I'd love to hear why you think mailgun is more superior than sendgrid. I'm
currently using sendgrid.

~~~
swampthing
We started off using Sendgrid too but found that our emails were often ending
up flagged as spam, even though we had the SPF records and all that jazz. I
should note that our emails were not spammy at all. Switched to Mailgun and
haven't had any issues since.

I have also noticed that false positives from other startups in my own spam
folder are often sent through Sendgrid (though I suppose that's not
dispositive since it could just be that more startups use Sendgrid).

------
slig
Even though I'm a SG customer, I can't help but feel a little schadenfreude.

edit: when I made this comment I thought this was a random service failure
that would last couple of minutes. More than 1 hour later, I don't think it's
that funny anymore as I'm being affected as well.

~~~
efa
SG customer as well. What's all the negativity towards them? Is there
something I should know about them? I wasn't aware of any incidents.

~~~
adambenayoun
There's not much negativity toward them just a bit of criticism on how they're
handling this outage. As I mentioned in another comment of mine on that thread
- they should have reached out to customers and let them know there's an
outage (especially if this is an outage that is > 10 minutes).

P.S: I still think Sendgrid are awesome and fortunately they'll listen to what
we have to say and next outage will be handled differently.

~~~
JPKab
A female Sendgrid employee overheard a joke mentioning a dongle looking like a
phallus, took a picture of the guy, posted it to Twitter, and he got fired
from his job without anyone hearing his side of the story. His apologies were
ignored.

Now the mob is mad at Sendgrid.

~~~
thisone
and by mob we mean possibly one person acting like a dongle and knocking out a
website.

~~~
gregd
Can I have a picture of you please for using the word dongle in a fashion that
I may or may not be upset about? You know, for internet justice..

------
adambenayoun
Sendgrid customer here.

Since many services rely on SMTP providers like Sendgrid, they should have a
way to notify customers when their server go down and transactional
notifications may be disrupted.

I shouldn't be notified by someone who know we're using Sendgrid and happen to
read HN.

~~~
mryan
They've been tweeting about it for an hour or so:
<https://twitter.com/SendGrid>

I would expect them to send emails to customers in the event of serious issues
though, expecting your customers to learn about problems through Twitter isn't
that much better than learning about them from HN.

~~~
nwh
> I would expect them to send emails

~~~
pytrin
Preferably not with their own SMTP service ;)

------
lifeisstillgood
OK, I'll bite - is this a coincidence - or are they being DDOS'd over the
PyCon incident?

~~~
joshrotenberg
The dreaded FDDOS: Forking Dongle Denial of Service.

~~~
rmrfrmrf
Still unfunny and tasteless. Imagine that.

~~~
kevincrane
Well I laughed at his joke. Maybe it's time to grow up and understand
different people like different jokes?

------
timjahn
I'm just tuning in now, as I happened to see this here on HN and my company is
a SendGrid customer (used for transactional emails).

Am I understanding this right - transactional emails for my company may be
interrupted because of some random personal argument between two people?

~~~
thomasvendetta
Not confirmed, but definitely possible:
[http://venturebeat.com/2013/03/20/playhaven-developer-
fired-...](http://venturebeat.com/2013/03/20/playhaven-developer-fired-for-
making-sexual-jokes-after-sendgrids-developer-evangelist-outs-him-on-twitter/)

------
uptown
I'm surprised PlayHaven hasn't shared the blame for their role in this whole
thing. Adria, and by-extension, SendGrid shoulder some of the responsibility
for what happened, but PlayHaven chose not to back their own employee.

------
whaevr
It seems that Adria's personal blog is having similar problems..

<http://butyoureagirl.com/>

~~~
canttestthis
Yes she mentioned that her website was getting DDOSed and that she was going
to put it behind Cloudflare. I'm not sure its a DDOS though, it could just be
the amount of traffic she's getting from HN/Reddit/Twitter/etc.

<https://news.ycombinator.com/item?id=5400134>

EDIT: It seems that her personal website (adriarichards.com) is down too.

~~~
Hopka
How is putting it behind Cloudflare going to help now that the (alleged)
DDoSers have the IP address of the actual server?

~~~
beedogs
mirror the content to a different host. point cloudflare at it. something like
that i guess. could be a pain in the ass if you can't get to the original
server and you have no other copies of the content, though.

------
jneal
As a SendGrid customer, I'm certainly glad I visit HN frequently and was able
to find out about this. I'm surprised we weren't notified by them directly,
though.

~~~
markeganfuller
We've never been directly notified by them despite their SMTP breaking a
number of times.

~~~
jneal
I'll just start following them on Twitter then. I've recently started working
somewhere new where we use SendGrid, and this is the first outage I've been
aware of.

~~~
luney
I did the same thing. Followed them on twitter just to be aware of the outage
status without having to go look for it.

~~~
dasil003
Since the service is timing out, I think we'd all be best served in this case
by having robust internal monitoring.

------
entropyneur
First we replaced SMTP that was distributed and designed to be resilient to
outages (i.e. not lose messages) with proprietary HTTP APIs and now we
complain that they don't notify us via email when they are down. Nice.

I am guilty of using SES myself, but it's sad to see email becoming
increasingly centralized.

~~~
symfoniq
We use SendGrid for transactional email, but handle (solicited) bulk email in-
house. The centralization of email is at least partly due to the fact that
ensuring email gets delivered is difficult and time-consuming.

While I'm sure a lot of SendGrid's customers don't want (or don't know how) to
configure a mail server, there are other customers who know that delivering
email isn't as simple as installing Postfix. The rise of centralized email
services is the inevitable byproduct.

~~~
entropyneur
Yeah, I know. Not blaming SendGrid in the slightest. Spammers have spoiled it
for us all. Freedom disappearing because of a minority of abusers seems to be
a common pattern...

------
uptown
Ironic:

<https://twitter.com/adriarichards/status/312265091791847425>

~~~
andyakb
That doesn't belong in this discussion. This isn't reddit.

~~~
uptown
Gah! I totally forgot to ask for your permission and editorial insight before
I posted. Please accept my apology, and thanks for staying on top of things.

------
plusbryan
FWIW, we use Amazon SES and enjoy it immensely.

------
efa
Looks like she got the boot. <http://www.facebook.com/SendGrid>

------
andyl
Great for Adria: more blog hits. Not so great for SendGrid: a PR disaster.

------
unhe
all thnx to adria and her cyber bullying

~~~
Kudos
All thanks to a dickhead with a botnet _

