Is anyone else here having problems today?
As these IP groups warm up, you may see some deferrals if you are a Free or Essentials customer. However don't worry, this warm up period won't last long. Maybe a few days at most, or until major email receivers have enough data to determine the legitimacy of email being sent from these new IPs.
Keep in mind SendGrid will continue to attempt to deliver these throttled emails on your behalf for up to 72 hours (it rarely takes the full 72 hours to deliver an email throttled in this way).
If you wish to avoid disruptions like this in the future, considering upgrading your account to a Pro or higher plan (https://app.sendgrid.com/settings/billing), which includes a dedicated IP address as opposed to sending from a shared IP group. Dedicated IP addresses are great because instead of many different users sending from the same IP or group of IPs, you are in complete control of your sending reputation.
Customer feedback is extremely important us here at SendGrid, and we have made these changes as a result of that feedback. We know in the long run, this will immensely help your sending.
This will go away, just hang in there with us! If you have additional questions, please feel free to contact us by going to support.sendgrid.com.
Suggesting that it's ok to wait a few days for our transactional email to deliver is frankly offensive. By definition transactional email is expected to be delivered in minutes if not seconds - days? That's got to be a joke?
This response is enough in its own to tip us into make a switch to an alternative supplier. It may be of no interest to SendGrid if an "essentials" customer leaves but if this is how you treat them you may find that more than a few do.
Frankly this entire situation has been mismanaged. As email "experts" you should have been able to migrate you customers over to the new IPs one by one as the IPs send level increases.
Finally suggesting that as this only effects some of your customers and so doesn't warrant being on the status page is just you making excuses to lie to your customers and hide the fact you have a serious issue. If this was a planed move with any possibility of delay you should have notified the effected customers before hand. We only discovered the problem when our customers stopped receiving emails and our support team saw an increase in support requests.
P.S. Your competitor Postmark has a very different oppinion about dedicated IPs https://postmarkapp.com/why/delivery
The proper way to transition IPs is to split traffic between them, so when one IP is throttled, traffic can continue. These accounts should have been assigned the new IP in addition to the old one. Once warming completed after a couple of days, the old removed. I would strongly recommend all affected customers to open support tickets with Sendgrid to get put on good IPs.
Another SendGrid customer here, in exactly the same situation (low-volume, purely transactional, paid "Essential" plan). But even if I were sending marketing blasts, a few days delivery delay would be unacceptable. It's hard not to hear SendGrid's response as "we don't really support anything below the Pro plan".
SendGrid has put up a support article about this: https://sendgrid.com/docs/Classroom/Deliver/shared_ip_thrott...
FWIW, a helpful SendGrid chat agent did move us back to a warmed-up IP pool once I complained, and our mails are going through again.
[Edit: clarify we're on a paid plan -- just not the Pro one.]
I opened a support ticket 12 hours ago and have just been told it will eventually be sorted.
I'm astounded that an email company couldn't do us the decency to email their customers to tell us what's going on. We've wasted valuable time and lost business because of them.
We're switching over to Postmark now. This year has been a nightmare in transactional email land from Mandrill to SendGrid to now Postmark, ugh. Hoping Postmark can do both you and I well.
> However don't worry, this warm up period won't last long.
How long is "not long"? Your customers can't fix support problems this causes by saying "Well golly gee our vendor says it'll be over in not long"--in fact, thanks to you, they may not be able to email their own users at all.
> Keep in mind SendGrid will continue to attempt to deliver these throttled emails on your behalf for up to 72 hours
After which time...what? You'll summarily delete the emails? Refund the customers for failing to do your jobs? What exactly?
> Customer feedback is extremely important us here at SendGrid, and we have made these changes as a result of that feedback.
Sure, but you didn't warn the effected customers. Some of whom were paying you.
You all fucked up here, and this is probably going to lose you a lot of accounts. Good work.
The tone of this message is pretty terrible.
However, this message warns me not to sign up unless for a Pro account (don't know what that means).
You are essentially telling your Free or Essentials customers to go and f*ck off. Not nice.
The point I was trying to make is this: in the example you gave ("waiting to install software") or the one I gave (warming up a car on the driveway), you can get the system ready for action without creating any impact in a production environment.
The case of email servers is an unusual example where you have to do real world tests in order to make the system work. In most other situations, you can do everything before the rubber hits the road.
So, whilst this could have been handled much better, it's not a simple situation.
The response to my support ticket on the subject was "things are working".
SG is incompetent with regards to email delivery. This should not come as a surprise to anyone that uses their service.
It took a month for them to confirm what we told them: users will still receive previously-queued emails even after they unsubscribe (!!!), meaning it's no surprise that they will flag future messages they receive as spam. A month or so later, SG acknowledged the bug but stated they had no intention of fixing that behavior at this time!
Their _core_ selling point is email delivery with low rejection and you don't need to build it yourself. They totally failed there as users are rightly marking their messages as spam _and_ we are therefore required to write, run, and maintain our own mail queuing service to check outgoing emails against _their own unsubscribe list_ before forwarding the email to SG to deliver for us. At this point, we use them only for the metrics, which is just not worth supporting such a sleazy operation.
They have a whitelisting feature, in case you are using your own domain you need to go through whitelisting steps. You need to configure your domain appropriately. It goes on and on. Then there are multiple ways to go about all of those things.
If you don't do it correctly then sometimes it doesn't work or a high amount of email ends up in spam. By default they enable features which some countries tightly control like tracking which all needs to be tinkered with to get right.
Then once it's all working perfectly I have seen it take up to 10 minutes for an email to be actually sent. I went with Sendgrid because they billed themselves as easy to use. You might be better off using a SMTP server of your own. You've got to configure it anyway.
Talked to chat support and they moved us to a new IP group and emails started being Delivered, however this lasted only 10 minutes.
After that all of our emails started being Deferred again. Contacted chat support again and they moved us again to a new IP group.
Let's see how much it lasts this time.
Chat support said having all of our emails Deferred for days with none being Delivered is normal and expected. We send transactional emails so having our customers wait days for the emails is not feasible for us.
We had recently switched from Mandrill, we'll have to start searching for a new provider.
We switched over to sendgrid after being pissed of with Mandrill due to their sudden change but switching to sendgrid was a bad decision in hindsight. Mandrill was rock solid and we never had issues. Sendgrid continues to have deliverability issues every once in a while and not to mention the blacklisting of their shared IPs (which I understand is a common problem with all providers but never happened with Mandrill for our business).
We have switched back to Mandrill (yes, got a paying mailchimp account just to use mandrill).
Not only sendgrid's UI is confusing, they don't even show the actual email content in their dashboard.
We use AWS' SES on a few projects, and, while it seems to work effortlessly, I have absolutely no idea how to get any logs out of it (like, anything beyond the absolute number of emails sent), which is a fairly major limitation for me.
Support was very helpful and migrated our traffic to already warmed up IPs.
https://news.ycombinator.com/item?id=11203056 (Mandrill's Betrayal)
A email delivery service isn't able to send those, and it considers that it isn't worth posting on their status page!
Are these messages really from SendGrid support? Looks like a desperate bid to prevent any future takeover attempts.
Haha that's a great idea for a "dirty tricks" campaign against a competitor. Every time they have an outage, a greenbean HN user starts posting less and less plausible excuses with the aim of pissing everyone off. Eventually the real CEO has to get on and say, "We don't know who this is but please ignore them but no I don't want to say anything about our outage." Repeat for added lulz...
Overall, I think Sendgrid does a fine job and it's unlikely this mistake will be repeated. These gaffes don't represent the majority of the company.
Maybe a disgruntled support person then.
If using yourselves is going to result in problems like this then I fail to see what we'd be paying for.
All emails are listed under 'Activity' as being deferred.
Waiting for AWS to approve my SES access so I can switch away.
Now we're using Postmark; much better service and no issues with deliverability.
Then I switched to AWS. Nobody got fired for using AWS S3.
Hence, no one can get fired. How can you get fired, if your fired email is just dumped in an s3 file?
Wish they would have communicated this in advance, or even update their status page.
(read using Homer voice)
"Uhhh, I love donuts! The status page is used only when something wrong happens. In your case, we decide to give you an outage by purpose, without telling you - haha - duh.
(Where is my beer, Marge?)
Oh, the customer is still waiting for my answer:
Dooonn't wooorrryyyy so much, we always do this on low plan customers - you know, we need some dumb guys to warm up our new IP range - hohoho - meeeryyyy x-mas!"
We detached this comment from https://news.ycombinator.com/item?id=12144124 and marked it off-topic.
SendGrid under cover agent here...
Don't believe the conspiracy theorists out there, that this wasn't a push to the PRO plan for free tier usage account holders.
The free tier customers don't actually earn any money for send-grid, so I wouldn't be surprised if this little stunt increased subscriptions for the pro plan!
Remember, this wasn't an outage for anyone other than free tier account holders and that's why it wasn't on the outage page!
SendGrid does want your business, as long as you don't spend too long on the free-tier plan and ultimately upgrade!
Keep on Trucking...
Secret Agent... out!
-----END PGP PUBLIC KEY BLOCK-----