Hacker News new | past | comments | ask | show | jobs | submit login

Running an email server became a real pain about 4 years ago. Until then, we could run our own exchange server, control everything and it was awesome. Then came the email security products from the large companies.

It became a self fulfilling prophecy. Our email would trigger a rule in big co's third party security app, which would then report to a centralized rule breaker clearing house automatically. Clients couldn't receive our emails, we would dive in, submit appeal to clearing house, get clear and a week later do it all again.

Everything was configured properly on our end. We passed all the online validation engines. IPs were our own personally owned block and pristine.

It became too much.

Switched to office365 and all problems magically disappeared. Sent emails to same big cos with third party email security and haven't had a single issue.

At my last job at an email security provider, this was a huge pain in the ass. We would provision new mail servers, and because the new servers' IP reputation to Office365, Google, etc. was poor initially, someone (until it became a shell script) had to spend the better part of a week taking the new server out of rotation, putting it back in, and so forth, so not too many of our client's emails got stuck in the new server's reputation blackhole.

I was there for 10 years, and this was only a problem in the final ~2 years of my employment, which roughly lines up with your 4 years remark.

Yup. As more and more domains centralize email in the handful of mega-corp hosted solutions the hosts have less and less reason to care about accepting mail from outside the walled gardens.

It's not about setting up perfect signed email and all the new techs. It's not about having the same clean IP for a decade. It's just the same old network effect combined with profit motive. The obvious change in the last few years (I've run a mailserver for 9) is managerial not technical.

For this reason I keep running my email server[1] even if it take some time and increase my attack surface.

In my humble opinion, leaving email to mega-corp alone can be a freedom risk, even if a small one.

[1] https://gioorgi.com/2020/mail-server-checks/

I know I've said this before here but.. it goes the other way too.

People make a big deal of SPF/DKIM etc. I think you should, for the sake of best practice. But you can set yourself up with gsuite or Office 365 and pay no attention to these things, and you can fully expect mail to get through to everyone. You can even setup a broken SPF record that disallows office 365 email and for the most part, everyone will still get your email.

That angers me so much. I had same IP address and domain for 15 years. Never a spam was sent from it, since the same address was used for NAT in home network I also blocked outgoing SMTP (forcing to use my SMTP instead) for cases that a PC got a virus and starts sending spam.

Things worked fine, but I think 2 years ago I realized that my e-mail started going to a spam folder. Fortunately adding SPF/DKIM appeared to fix it, but it feels like it is getting harder and harder now to have own mail server.

I would probably gave up, but it's infuriating to me that from courts rulings if your mail is handled by 3rd party, the 4th amendment no longer applies.

Not that I'm expecting it or anything like it, but running my own mail server means I'm the one who gets the subpoena for anything that might be on it. Google/Microsoft giving Third-Party Doctrine-driven open or nearly-open access to every email on their servers for any purpose is not my idea of a good business decision.

Yep, you can do everything outlined in the article and have an IP that has not send spam in 10+ years and gmail will still spambox every single one of your mails.

However if a domain signs up for Google Apps, magically you get whitelisted, even if you are handling your own email, funny thing that.

I guess Google's customers are just more trustworthy...

> I guess Google's customers are just more trustworthy...

I wish. I've been having issues with being sent to spam when replying to a colleague on the same domain which is on Google. The email never left their servers, everybody was authenticated, and the emails were within an hour of each other, so it's pretty obvious that it's a reply. Still: off to spam it went.

We now have a filter rule to always consider our own domain as not spam, because it's just too unreliable otherwise.

Note that your filter might be breaking an important security feature:

If you get an incoming email “from” yourself and it is not marked as spam, it will be put into your sent label.

At least one person claims to have been incorrectly fired because of this. Their employer found incriminating emails in the person’s sent box, and considered it a closed case.

That's interesting, do you have a link for that case? I don't see why it would get put in the Sent folder, but if it does, that might certainly lead to confusion.

This is a really annoying Gmail+Gsuite bug that has affected us as well. Surprising that it seems to happen this often.

Counterpoint. I run my own mail server, and have done for 10+ years on the same IP.

In that time I've had a single incident (earlier this year) where spammers got it and sent 20k spam mails in less that 24h mails. Luckily I noticed this and stopped it pretty quickly.

I had mails to some destinations bounce for around a week after that.

Beyond that incident, it's very rare to have trouble delivering mail - I think maybe once in the past 5 years there was a company who I had trouble with.

Having said all that, the spam incident this year highlighted how fragile hosting your own mail server can be if things go wrong, and I felt really bad about the spam mails sent from my server - continuing to run it almost feels like a liability, and when time permits I plan to move our mailboxes to O365

Unless you have substantial volume to gmail (hundreds of mails a day), I can virtually guarantee gmail is spam boxing your mails, try mailing an account you have never sent mail before for instance.

They don't refuse delivery, but that is entirely meaningless when nobody ever checks the Spam box anyways (which is also hidden by default)

Our email volume is pretty low - average is probably 5-10 a day.

Hmm, I just tried what you suggested, with 2 different accounts never sent to, and the emails got through fine.

We use SPF and DKIM, but otherwise it's a fairly standard Postfix/Dovecot install (albeit a very old one).

Would you mind sharing where you host your email server?

I have a pet theory that big email platforms care deeply about the IP "neighborhood" the sending email server lives in. So a small email server set up perfectly in Digital Ocean IP space would struggle, while an email server set up OK in (for example) CenturyLink Enterprise IP space would get through.

It's with a European (Austria-based?) VPS provider, Edis (edis.at), and my server is hosted in a UK DC.

Is the implication that this is an intentional practice to drive out small players?

It is, at least if you ask me.

People here are speaking about how gmail will "effectively spambox your mails by default". That's not been my experience (from setting up multiple small customers). Anyway, at least I've never heard of gmail just eating your e-mails. They either get rejected or accepted and put somewhere (maybe the spam folder, but at least somewhere).

Office365 / hosted Exchange / Outlook Protection or whatever is it called these days... they should be routed to /dev/null by everyone.

They just won't track your reputation unless you send them more than 100 mails per day. Does this sound bad? There's more: if they don't have reputation info from you (because they refuse to track it due to the low volume), your mails will go to spam inboxes even when their filters indicate that the message is not spam.

And there's more! Don't dare to ever get a bad reputation (i.e.: a user managing to get hacked and their account used to send a couple hundred spam e-mails before triggering your countermeasures). If this happens you are 100% fucked. Now they will DROP your e-mails. Hear, hear: their servers will accept your mails and just DELETE them. No spam folder, nothing.

You will try every possible thing: set up everything for their feedback loop, sign up in their "Smart Network Data Services" to track your reputation (it will be empty except for that day)... and finally contacting them at their sender support.

Do you want to know what they will reply? That you should be patient and let your reputation build up over time. What a joke! How on earth can your reputation improve when users cannot mark your mails as "not spam" because they (outlook, not the users) are simply DELETING them without a trace?

Oh, there's a way out of all this though: obtain a "Return Path Certification" [1]. That is, pay them an absurd amount of money and your mails are guaranteed to get to the users' inbox unless you are clearly spamming (all of the above assumed you are NOT).

Up to this final point you could think they just do their best and all that I've explained is collateral damage. That last "pay and we let you off the hook" is what clearly signals to me that this is an elaborate scheme to get small players to either pay them anyway or just give up and use a big-provider service.

[1] https://sendersupport.olc.protection.outlook.com/pm/services...

> People here are speaking about how gmail will "effectively spambox your mails by default". That's not been my experience

It is mine. I used to have to take a proactive approach to email - send an email, and if I don't hear back in a timely manner, I check their MX records. If it's Google, I'd contact them out of band - yep... spamholed.

These days, if it's important I'll contact out of band. If not, meh... I'm not going to bother just because of Google Knows Best. I'm over pandering to Google.

> a user managing to get hacked and their account used to send a couple hundred spam e-mails

For this reason one of the necessities of survival as an email sender is to classify your own outbound messages, to see if recipients might think they are spammy. If they just look spammy but you still want to send them, you need sacrificial IP addresses on separate netblocks that are dedicated to spam, so the reputation of your main IPs isn't polluted.

If there's any possibility that any of your users could get their account hijacked, or there could be malware on any device permitted to send messages, you need outbound classification.

Has anyone tried pursuing a defamation lawsuit over systemic false spam labeling?

Edit: or, alternatively, tried fighting this using anti-racketeering laws? The payment aspect sounds like a protection racket scheme to me...

Antitrust, too. Making a market impossible to enter for new entrants is a blatant abuse of market power, and EU antitrust law would probably care about it.

Your experience tracks with mine as well.

I run VoIP PBXes and they occasionally need to send an email, usually for voicemail to email but occasionally for other alerts.

By default they just send straight to the internet, sending from pbx@pbx.domain.name, forward and reverse DNS mapping back to their own addresses, etc. but no other setup.

This works perfectly fine with Gmail and Gsuite accounts, but Office 365 occasionally decides it hates one of our servers. Even if the client has the sending address whitelisted, it still just gets hard blocked for no apparent reason.

Gmail, I can fire up a telnet session right now and send myself an email from my home IP address just typing raw SMTP commands in to a console. It's going to work, and unless I'm spoofing a real email address it's not even going to end up in spam.

> Oh, there's a way out of all this though: obtain a "Return Path Certification" [1]. That is, pay them an absurd amount of money and your mails are guaranteed to get to the users' inbox unless you are clearly spamming (all of the above assumed you are NOT).

Could you name the cost?

They are a "contact us" kind of organization, but if you look hard enough you'll find prices from a while back [1].

TLDR: Minimum tier was $400 signup + $1,375/year for up to 100,000 mails/year.

[1] https://returnpath.com/wp-content/uploads/2015/06/Return-Pat...

Bonus: They straight up declined my business because it was too marketing-y for them.

What was too marketing for them? A 100% opt-in list that reminded customers when a product was in stock. Fully made in-house, with captcha, privacy policies, opt-out, and with the user typing their email and name, the complete works for a one-time reminder email.

Nope far too marketing related for them.

I don't think it is an attempt to drive out small players, it's a natural consequnce of domains being almost free to stand up. They just don't provide any level of assurance that the owner isn't a spammer. So the blackhole systems treat them as "probably spammer" and fire up a block at the first whiff of malfeasance.

It's harder to sign up for a Gmail account than it is to register a domain and get some hosting. And Google has their own protections against sending large amounts of mail built-in. The system is as it is now because spammers will abuse it otherwise.

I disagree.

The big email players are accepting mail from universities and large companies with misconfigured email servers. If you use the same rules that Google and Microsoft use to refuse emails, you will be bouncing emails from valid domains. If you don't apply them, you end up accepting spam. The big players are able to deal with the spam using specialised teams and I suspect advanced algos including machine learning.

IF the big players applied their famous rules to everybody equally, everybody, including universities, big companies etc... Would quickly configure thier mail servers properly, which WOULD reduce spam, possibly eliminate it.

Being able to keep inboxes fairly free of spam in a world full of spammers is what distinguis big email providers and enable them to sell their services.

> The big players are able to deal with the spam using specialised teams and I suspect advanced algos including machine learning.

I remember when I imagined big companies might use their considerable resources to put together something like that. Then I found out it's just a handful of random engineers cobbling together crap and hiding it behind a fancy domain name.

It is probable that Office365's entire spam filtering system is a very large and ugly spreadsheet that gets passed around between a few teams using a ticketing system.

I used to work on the O365 antispam team, and I'm not sure how much I'm allowed to say in public, but I'm pretty sure I can publicly say "a very large and ugly spreadsheet that gets passed around between a few teams using a ticketing system" is not accurate.

Relevant wikipedia page: https://en.wikipedia.org/wiki/Exchange_Online_Protection

Originally, the product started out as an acquisition, and there's still some legacy code that references the old Forefront name. There's about 50 people on the team.

DKIM/DMARC/ARC is worked on by two engineers.

gmail is effectively block by default these days, unless Google has some reason to not block your mail, it will be spamboxed.

True. It's a mistake to do a job search using gmail. Interview invitations and job offers are going to go straight to spam.

Sending email to Google is always a crap shoot and I still find to this day that sometimes when I send email contains logs to my own account using my own account credentials it'll end up being classified as spam.

Right now, Google is refusing to deliver email from logwatch - giving me the "Message rejected. See https://support.google.com/mail/answer/69585 for more information." so those are going to /dev/null with no recourse to fix it because Google literally doesn't like the content of the email.

The fact that I can't hit a button and tell Google that yes indeed I'm not trying to spam myself is ridiculous.

That was the case for me few years ago. But I decided to set up mailserver once more recently and Gmail to my surprise accepted my mails. I configured everything mentioned in article.

That said, my outcoming mail traffic is almost non-existent outside of few test emails, LoL.

I have an old, but frequently used email address that got force-moved from a pass through relay to gmail.

After setting gmail to forward everything, I noticed it was sending 33% of my legitimate email to spam.

They don’t let you disable the spam filter, but you can set up an escoteric filter that prevents it from actually putting stuff in spam.

I honestly don’t understand how people cope with gmail. Fastmail is cheap and orders of magnitude better.

I created my gmail account in 2004 I think. Things were different back then. Now there's just 16 years of momentum preventing me from switching.

I like the idea of hosting my own email but even if I get around to doing that I will still need a backup. Fastmail is probably where I will go.

With Outlook you can't unfortunately do that. Create rules, which would un-spam mail. Because filters aren't applied to spam emails at all.

I don't think it's done intentionally, but I also don't think the big players go out of their way not to harm the small ones. All the big email players would love to take your money to "outsource" your email.

No, I think that there are just too many spammers, so big players have no choice except to make it hard to own your mail server until your server gains positive reputation.

I don't think there are many spammers compared to, say, 2005.

In 2005 you would get 100 spams to an unfiltered, widely published address, these days it is more like 2-5.

The continued lockdowns and protocol changes of gmail and yahoo are a sign of being overstaffed, "feature" oriented and yes, the attempt to shut down competition and the free exchange of mail.

You're lucky if you only get 2 spams a day. My parents are getting more like 20-50 a day on the email provided by a national ISP.

It's getting impossible to read emails, too many. If it were not for them being used to their email address and me being lazy, I would move them off to gmail.

I do not think so, I think the spammers are still out there, armed with cheap cloud email providers, but the "block by default" is putting a nice dent on them.

This is not true. They're not even making an effort and are quite probably actively malicious. There is nothing you can do to build up your reputation if you're a small email server. There are no introspection tools to give you a hint on what's wrong and no one to contact. Besides, what are those fancy ML antispam algorithms for if the only cure for spam is "reputation"? It's clearly an undefined term meant to be exclusionary.

A person from Gmail even posted on HN a while ago, stating they'll look into this and do better. That was about a year ago and the situation is exactly the same or worse.

I really think so. They are forcing everybody into their services.

A middle-ground that can be worth considering is mailgun.org or something similar - you can host the rest of the stack yourself and just use them for the last hop of outbound SMTP from your own MTA.

You don't have to throw out the baby with the bathwater (:

that is really not a middle ground, you are handing full control of your email over to a third party.

That's not really true. You're using a third party to send out messages, and if you don't like it, you can switch pretty painlessly. You could even use you're own local MTA at the same time, if you really wanted to.

The IMAP/POP3 server is yours, and all the email is stored on something you control.

This is what I do but I use Amazon SES to send out mails from my selfhosted email infra. All emails pass through and I have the satisfaction of knowing I can just switch to another provider if SES shitcans me.

I disagree. The last couple places I've worked at have all had their mail hosted internally, meaning on a dedicated IP from one of the big ISP's on a business plan.

No real issues. Sure, you make it on a spam blacklist once in a while, but at least you know when it's going to be fixed, instead of dealing with zero response (or denial) from a big hosted mail provider.

Besides, not all of "our email" comes out of our office, our CRM and ERP products send mail on our behalf, and they have problems delivering mail more than we do.

Running an email server has been a pain forEVER. Running one's own personal mail server is a terrible idea unless you are a professional sysadmin with time to kill.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact