
The Hostile Email Landscape - jodyribton
http://liminality.xyz/the-hostile-email-landscape/
======
tacon
I've managed my own mail server since 1993, and my email address has been the
same that entire time. Here are some tips for maintaining sanity:

Greylisting still works amazingly well. With a long, long whitelist and
greylisting plus DNSBL, I don't even bother running a spam filter, since the
little bit of spam and emails from new senders ends up in its own directory as
it came from a non-whitelisted sender.

Comcast finally started blocking residential mail server ports inbound a few
years ago, so I had to migrate to a smarthost environment using a VPS as email
server for $15/yr.[1]

Last year for a few months, Gmail was dropping everything I sent into the spam
folder, even after recipients were marking it not spam. I eventually
discovered the "Authentication-Results:" header that Gmail adds to every
inbound message. It is under the "Show Original" dropdown menu. That showed
that I "hadn't changed anything"(!) on my mail server, but suddenly Gmail was
connecting to my mail server over an IPv6 interface, and I had never bothered
to put the IPv6 block into the SPF record. Gmail was nice enough to explain
exactly what it didn't like about those emails.

[1] [http://lowendbox.com/blog/top-provider-poll-2014-q3-the-
resu...](http://lowendbox.com/blog/top-provider-poll-2014-q3-the-results/)

~~~
niravshah
> Greylisting still works amazingly well. With a long, long whitelist and
> greylisting plus DNSBL, I don't even bother running a spam filter, since the
> little bit of spam and emails from new senders ends up in its own directory
> as it came from a non-whitelisted sender.

Any good tips on this section in particular? If I'm running my own mail
server, how would I get started making sure this is in order?

~~~
tacon
I have been running exim4 for years, but I'm in the process of moving to
postfix, as postfix is considerably easier to set up all the DKIM, etc.,
machinery that is now required. Inbound email comes through procmail and is
mainly read in emacs (mh-e), which is kind of old fashioned. I have a small
script that makes a new email address within my domain for each new use. I
sign up for a lot of mailing lists and groups, and my /etc/aliases is more
than 5300 lines. I can track if domainA's address starts coming from domainB
and disable that address, but that doesn't happen very often, which is a
pleasant surprise.

I also have a small script that puts a new sender on my whitelist of sender
email addresses. My whitelist is 12000+ lines right now, collected over many
years. Procmail sorts to mailing lists and vendor folders, and finally puts
things that are not on the whitelist into a "possible spam" folder. From the
five or ten items a day, it is easy to spot legitimate emails and I add those
to the whitelist. The majority of spam is blocked by the combination of
greylisting and DNSBL lists, as the delay of greylisting (ten minutes for me)
is enough for them to make the blackhole list, if they happen to ever attempt
delivery again.

I was thinking recently that I should be collecting statistics on the use of a
lot of those aliases and whitelisted emails, and maybe start garbage
collecting my lists.

There are various reputation reports and services that can tell you how your
mail is doing in the major ISPs, but a lot of those require higher traffic
than a personal or small business generates. There is one service, DMARC[1],
that is free and can give you some visibility into how email from your domain
is being processed. I put the txt record in my DNS, and Google, Facebook,
Comcast, Yahoo, Fastmail, and a few others send me reports about email they
have processed from my domain. It's not that interesting at the moment because
things are working, but it might help to debug issues if your email was being
rejected. At least I see a few spammers are trying to use my domain from their
servers.

[1] [https://dmarc.org/](https://dmarc.org/)

~~~
chei0aiV
If you are using emacs, you might want to try the emacs UI for notmuch:

[http://notmuchmail.org/](http://notmuchmail.org/)

------
sjackso
I've run into similar issues with a similar setup. It's frustrating. You can
convince gmail user A to whitelist your messages, and so they'll get through
to user A, but gmail user B probably still won't see messages from you unless
you tell him to dig them out of the spam trap. And your messages to A might
still be classified as spam if they have attachments or hyperlinks in them.
(Even if you've been corresponding with A for several years!)

Once upon a time, to send email, you needed to use SMTP. Now, you must use
SMTP, from a IP block that isn't categorized as residential, and which has
never before had any association with outgoing spam, and you also must
implement several ad-hoc identification protocols like SPF and reverse DNS.
You should also use a domain name that you've owned for some time and which is
not expiring soon. Every system to which you want to send mail will give
different weights to all these signals. If they don't like you, their behavior
is to report successful delivery and then silently hide messages from their
intended recipients.

Spam is no fun, but the present situation is pretty weak too.

~~~
nabla9
[http://cr.yp.to/im2000.html](http://cr.yp.to/im2000.html)

Internet Mail 2000 IM2000 is a project to design a new Internet mail
infrastructure around the following concept: Mail storage is the sender's
responsibility.

~~~
lsc
Bernstein is an incredibly smart guy. But he's also an example of how the
social stuff matters. His software would run the internet if he put a little
more effort into the social/political side of things.

The big problem with IM2000 is that it doesn't solve the _real_ problems. It's
focused on the economics of storage of mail, which, yeah, are a thing for a
few mail administrators, but generally is considered less important than the
spam-filtering problem in general.

I mean, I'm not willing to pay for infinite storage, but I certainly would be
willing to pay the cost of receiving 100x my current monthly spam load if I
didn't have to read any of it. Maybe even 1000x.

The major benefit of the IM2000 standard is that if you can squash the
spammer's mail servers, you squash the mail in question, which actually could
change things, because it would mean that spammers would need to control the
servers they use to send for longer than they do now.

~~~
thoughtexpt
That his software does not "run the internet" does not make it any less good.
It's there for whoever chooses to use it.

That he focuses on the software instead of pandering probably makes his
software better than the alternatives that aim to please even the most foolish
of users. At least I think so.

Maybe I interpreted the proposal incorrectly, but I always saw IM2000 (minus
the "notifications") as a "pull" solution.

By contrast, conventional email relies on "pushing" spam to the recipient (in
practice, a middleman called an "email provider").

A smart IM2000 recipient perhaps would not pull spam from the sender's server.

As such, the spam would never enter the network. It would just sit on the
sender's server.

Therefore, IM2000 not only conserves storage but also conserves bandwidth.

~~~
mike-cardwell
"A smart IM2000 recipient perhaps would not pull spam from the sender's
server."

Based on what criteria? You wouldn't have the message headers/content, so you
couldn't filter based on that. All you can filter on is the IP address. But
you can already do that with SMTP by just responding with a 5xx code as soon
as the sending server connects anyway... Or you could actually fetch the
message and then spam filter it. But then, you may as well use SMTP...

What difference does it make which side initiates the TCP connection? If I was
a spammer, IM2000 wouldn't concern me one little bit.

------
cm2187
The problem is not so much the attitude of the big guys. It is that smtp is
fundamentally broken. we need a better mail protocol that ensures: 1\. Traffic
always encrypted and content always signed 2\. Guarantee that the sender is
who it claims he is 3\. Decorrelating the email from the domain, a lot of
users are prisoners of their current provider just because the address they
gave everyone ends with the provider's domain name, very much like it is very
hard to switch bank accounts 4\. Ability to provide disposable adresses which
can be deleted when spammed

3 and 4 would require a sort of token system

~~~
JoshTriplett
> Decorrelating the email from the domain, a lot of users are prisoners of
> their current provider just because the address they gave everyone ends with
> the provider's domain name, very much like it is very hard to switch bank
> accounts

This one seems completely uninteresting in a world in which $15/year can get
you your own domain name, complete with reliable email servers, IMAP, and as
many email aliases as you like at that domain. Given that, why create an
entire infrastructure to "decouple" email addresses from domains?

~~~
cwyers
That's $15 a year that some people can't, won't or haven't paid before. If
someone decides later on that they wish to do so, they have to set up
forwarding and try and convince contacts to use the new e-mail address
instead.

------
dredmorbius
Many of the issues we're running into with online systems, particularly those
relating to quality and reputation (spam, collaborative filtering or content
rating as on HN or Reddit, etc.) have strong analogs in real-world social
spaces. And there were real-world mechanisms for dealing with these.

For a new businessman or professional setting out in the world, pre-Internet,
"establishing your name" was a requirement. These days the concept's often
referred to as "creating a personal brand", but the reality was pretty
straightforward: how does an unknown quantity become a known quantity?

A common method was the professional or social introduction. This is still
practiced, where a third-party _matchmaker_ will introduce two parties. The
matchmaker usually knows both, and can vouch for the newcomer and smooth the
path for introductions with the established party. Essentially, the matchmaker
stakes _their_ reputation by speaking for another.

Lawrence Lessig describes a similar concept in his book _Code and Other Laws
of Cypberspace_ , in a passage describing a physical messaging system, the
Yale Wall. This was a board onto which messages could be posted, with the
proviso that that they be signed. Unsigned messages would be posted from time
to time, effectively presenting an anonymous viewpoint. Removal wasn't
instantaneous or automatic, but rather, at some point _prior_ to garbage
collection, another individual could review the piece, and, if they felt it
warranted posting, sign for it. They weren't registering themselves as
_author_ , but as _vouching for the merits of the viewpoint_ \-- not
necessarily agreement.

A messaging system in which a new peer might be able to indicate that, hey,
peers X, Y, and Z, with established reputations can vouch for me, and for
which those peers could confirm their endorsement, might address the "how to
build reputation" issues of new mailservers.

I've also found that even large mail systems will frequently have _some_
procedure for getting at least provisionally vetted, effectively a workfactor
cost to coming online. Though the process absolutely could be improved.

------
skrebbel
It bothers me that this entire thread is about technology. This is
anticompetitive behavior by a small group of oligopolists, plain and simple.

The big providers need to build an accessible system for entering the email
market or face tremendous fines from worldwide governments. There are plenty
ways imaginable to do this, especially if you allow the process to involve
real people and paperwork.

It shouldn't matter whether this is simple negligence or conscious work to
keep out competition. The end result is that it's made nigh on impossible by a
few market leaders to enter said market.

------
eridius
I feel like this is a solvable problem without making any changes to email
whatsoever. The problem is the email recipient hosts are suspicious of the
sender (as opposed to the message itself being suspicious). So the solution is
to have a standardized way for senders to acquire an instantaneous reputation
by tying their real-world identity to it (which lets them be held accountable
if they do spam), and perhaps by throwing some money at it too. If there was
some company that did identity checks, similar to how EV certificates are
given out, then that ties your real-world identity to it (this could in fact
be done by literally requiring an EV certificate for the hostname of the
sender). This company could also take a decent-sized deposit (so you're
staking money on not being a spammer) and hold it in trust for a set amount of
time. Once the time has passed, and you've sent enough emails for recipients
to draw meaningful conclusions, if you have in fact not spammed, then you get
your deposit back (minus a service fee). Then all the big email hosts would
pay this company to query it about senders the host doesn't already trust, and
similarly they'd report any spam from these hosts back to the service.

Heck, this doesn't even have to be a new company. A big host like Google could
just start offering this service anyway, as a way to simplify their own
handling of unknown senders, although I'd feel more comfortable if this was
done by someone else.

~~~
notatoad
The core problem to this is that reputation is valuable. Google or Twitter or
Facebook or any of the other big web presences _could_ offer a method to
determine "reputation" of any of their users, and a way to canonically tie
your identity to that reputation, but they'd be undermining their own business
model. They want you to stay in their network, because the safety of
interacting with only reputable users is the big selling point of their
walled-garden social networks over open networks like email

~~~
eridius
What do you mean by "stay in their network", when it comes to sending emails?
Google is not in the business of being a paid mass email sender for
businesses. What business email they handle is specifically for being the
email host for a business's internal email, not sending their newsletter /
marketing / account notifications / whatever else.

------
codecamper
Here is an almost related story: I am in italy and was waiting for an email
from the local Apple store to tell me my macbook's repair was complete. After
6 days waiting, I called the store. They said the computer had been ready for
a few days.

I checked the spam folder. Gmail had plopped the apple email into the spam
folder. Gmail's reason was that the email was in a foreign language, Italian.
Didn't matter that the previous 2 weeks all my google.com searches were done
from Italy.

~~~
dredmorbius
Google's killed access to various accounts of mine at different times for:

1\. Accessing an account _one time_ over Tor. Subsequent reconnect attempts
from the same device I'd previously used, same IP, etc., failed. Took a couple
of weeks to get back.

2\. Accessing accounts from different IPs while travelling. Ultimately ending
up at the same IP as one of the email accounts I'd previously corresponded
heavily with.

3\. Accessing accounts from new device(s). Old (Android) devices could access
email but not Web-based account recovery tools.

"Who are you" is the most expensive question in technology. No matter how you
get it wrong, you're fucked.

[https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_go...](https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_google_identity_thing_again_who_are_you_is/)

~~~
georgenishimura
I was once disappointed not to hear back from a Google recruiter after an
interview. Turns out it was in my Gmail spam folder. Along with any other
@google.com address.

------
jasode
_> This isn't how the internet is supposed to work. _

The email architecture was started back when it was a smaller network of
researchers at universities, governments, etc. Everybody basically _trusted_
each other.

Once the "internet" is available to the general public and commercial
interests, it becomes vulnerable to the "bad actors" problem (e.g. spam
abuse). That's why we have the inevitable situation today of a few entities
(e.g. gmail, hotmail) being "trusted", and random residential SMTP servers run
by homeowners being "untrusted".

I haven't seen a realistic de-centralized trust proposal for email. Even if a
proposal is theoretically sound, what incentive is there for other big players
to adopt it?

~~~
chmike
How do you solve the problem of "cold call" with a trusting system ? How does
one build up trust with a new account ? Spammers could create million accounts
at once and fake trust build up between them. That's why I'm not convinced
that trust will solve the abuse of messaging. A messaging application must
allow "cold calls". Relying on trust built up by others is not a reliable
system because it can be abused.

~~~
jasode
_> How do you solve the problem of "cold call" with a trusting system ? [...]
That's why I'm not convinced that trust will solve the abuse of messaging._

You can't remove "trust" or "reputation" from a viable messaging system. The
concept of trust _always exists_ whether informally or formally.

When the internet was a small private network between universities and
government, how did a new unknown scientist "cold-call" an email? It was not
an issue. The scientist just sent the cold email. There was already _implicit_
trust within the system to allow that scenario. Everybody trusted that
researcher@someuniversity.edu didn't send a million emails about Nigerian
money transfers to researcher@army.mil.

Back then, email users didn't need "spam filters". The trust was informally
created because the institutional "system" _outside_ of the email system
vetted email users. (The "system" being the hiring procedures of
SomeUniversity, DepartmentOfDefense, etc.)

All those gates for reputation are gone when we expose that simplistic email
architecture to the general public. Trust doesn't go away. You now _must
recreate trust_ at a massive scale.

 _> Spammers could create million accounts at once and fake trust build up
between them. _

That's not the type of "trust" people are talking about in this thread. That
type of incestuous "trust" between bad actors won't work because it doesn't
have an initial authority (good actor) to "bless" them which starts an
acceptable "chain of trust". The concept is similar to Certificate
Authorities. We trust Chrome because we trust Google Inc; and in turn, Google
Inc trusts Verisign; and in turn, Verisign EV-certificate processing trusts
government offices that register businesses.

~~~
chmike
Ok my assumption of trust credit working model was inaccurate. Could you give
an example how trust could be used to control abuse ? If a user is given
initial trust credit, how can one prevent spammers to create million of such
"trusted" accounts and let them be banned after the hit and run spamming ?
That model us used in messaging systems like Twitter but has its limitation.
But is known to have its limit. Or did I still got it wrong ?

~~~
jasode
_> , how can one prevent spammers to create million of such "trusted" accounts
_

Notice what you did there? You characterized spam accounts as "trusted" even
though you didn't specify why or how they'd get trusted in the first place.
There's still a gap in your thinking of how the origination of _real_ (not
fake) trust happens.

It's like a 1970s email user asking, _" how does the system prevent a
'trusted' Nigerian money scammer from spamming everybody?"_ Well, why would
Department of Defense hire a Nigerian scammer? Would a UniversityOfWhatever
hire a Nigerian scammer? No. In that scenario, the trust _begins_ with the
institution. For the Nigerian scammer to be trusted, _somebody_ authoritative
has to "bless" him.

 _> Could you give an example how trust could be used to control abuse ? _

Well, we're still using _trust_ right now to help control abuse. Today,
reputable corporations and also millions of honest users (that's us) "trust"
Google Inc, Microsoft, Yahoo, FastMail, etc. Spam on a massive scale does not
originate from gmail/hotmail/etc because those corporations shut them down.

The issue is that it's _centralized_ trust concentrated in a few big players
instead of _de-centralized_ trust. That's the thorny problem that's very hard
to solve. All the clever decentralized proposals (sender pays for storage,
outbound payment bonds, cpu-proof-of-work, etc) are basically roundabout
proxies for recreating "trust" on a bigger scale. I don't know what the final
answer will be. I just know that trust/reputation is an unavoidable part of a
new solution.

------
lisa_henderson
This is a true story:

I rent some servers from the Rackspace cloud for personal use. I have my own
sites on these machines, and my own email servers.

Meanwhile, I have a day job, and lately it has been consuming 12 hours a day.
We missed a deadline and we have all been working like crazy to catch up. I
have fallen behind reading my personal email.

Roughly a month ago, my friends who use Gmail stopped getting my email. Or
rather, they did not know I was sending them email, because all of my email to
them was going to spam.

After a few weeks, I finally had a free weekend to catch up on my personal
life, so I did some investigations. Turns Rackspace had switched over to IP6
in a way that impacted my email. I did not have a Sender Policy Framework for
IP6, only IP4.

It's likely that Rackspace sent me an email about this, though I never read it
because I was busy.

This was easy to fix: I added a SPF for IP6.

However, these kinds of issues do make it harder to maintain a personal email
server. Its tough for us to keep up with the changes.

------
al2o3cr
Drinking game for this thread. Start at the top of the comments:

* every time someone implies spam filtering is a massive conspiracy against "the little guy", drink

* every time someone suggests Bitcoin be used for something, drink

* every time a solution utterly fails to account for compromised end-user machines sending spam, drink

* every time a comment can be summed up as "assuming a PKI exists, this problem is trivially solved", CHUG THE REST OF THE BOTTLE

~~~
dcposch
LOL

That reminds me of an old classic:
[http://craphound.com/spamsolutions.txt](http://craphound.com/spamsolutions.txt)

There are a lot of obvious problems with email.

* It's hard to run your own mail server, as described in the original post

* Your address is tied to your provider

* The standards involved in delivering mail (SMTP, POP, IMAP, etc) are complex, layered with hacks (SPF, STARTTLS, etc), hard to implement correctly, hard to secure.

* The standards that govern the actual body of email (multipart MIME, a 90s-vintage subset of "HTML" and "CSS") are even worse

But despite all those flaws, email is built on ubiquitous open standards
running on a federated infrastructure.

It's the best thing we have.

Proprietary standards and centralized designs like AIM, ICQ, MSN, FB
Messenger, Slack, and Twitter DMs come and go.

Email is forever, and is more useful than all of those combined. The flipside
is that it's very hard to "fix".

------
9248
> This isn't how the internet is supposed to work. As we continue to
> consolidate on a few big mail services, it's only going to become more
> difficult to start new servers.

And this is _exactly_ the reason I setup my own mail server. I'm only 1 man,
but I hope more people will do so with time, thus requiring the "big ones" to
work on better algorithms for filtering and not base it on reputation.

------
waynecochran
Create an email network where is would cost a penny to send email. It would be
payed into bitcoin wallet of folks maintaining infrastructure.

Every email would be digitally signed and encrypted. Certificate with keys
would connected to email address (and bitcoin wallet).

Spam would die.

Go build it please.

~~~
rokhayakebe
The minute you say bitcoin is the moment you cut out 99% of the population;
unless it is in the background and one does not have to interact with it
directly.

Other than that, I agree paying to send will reduce spam.

But also look at your postal mail box. There is arguably more spam there than
in your digital inbox, and that one costs (stamps).

~~~
waynecochran
Yes, but if the email is digitally signed I can be certain who sent it (at
least the address) and, unlike postal mail, I can electronically sort it.
Email from folks I know would receive the highest priority.

Are folks afraid of bitcoin? What can be worse than just having a 16 digit
number printed on a card -- that is a 1000x more vulnerable.

~~~
rokhayakebe
_Are folks afraid of bitcoin?_

You are part of the tiny percentage of the tiny percentage of the population
that understands bitcoin.

------
hannob
I have a bit of experience with running email servers. I can't really say that
I had similar encounters.

In my experience if you get blocked by big mail providers it's almost always
due to some reason. What's tricky is that it may be hard to tell what exactly
is wrong, because they won't necessarily tell you (or not in an easy way).

Some advice what I'd do to try to find out what's going on:

1\. Take a sent example mail that is like the blocked one (but obviously one
that reached its target destination) with all headers and run it through
spamassassin. Don't just look if it hit the spam score (then you did something
terribly wrong), look at each individual rule that spamassassin hit. They
might give you a clue. A proper mail usually shouldn't hit any or very few
positive spamassassin rules.

2\. Check your IP at a service like valli where you can query multiple DNS
black lists. If it is on any blacklist try to find out how you can be
delisted. There are some rogue blacklists that make it impossible to be
delisted at all, you may ignore them (google for them, their behavior is well
documented), but these shouldn't be more than 1 or 2. As already said by other
commenters, don't forget IPv6.

3\. Read whatever error message you can get your hands on. If you're blocked
on the SMTP level read the error message. If your message got sorted into a
spam folder look at all the headers. If the provider blocking you has some
online docs about their spam filtering read that. If they have some sort of
service for mail ISPs where you can sign up to get warnings sign up there.

Of course also the obvious stuff. If you do anything that is mass mailing you
are in extra danger. Make sure that you allow people to unsubscribe easily,
don't ignore manual attempts by them to unsubscribe ("I want to get off this
mailing list") and delete invalid mail addresses.

------
zrm
I was thinking about this a while ago and have been meaning to write it up and
post it somewhere, so I guess this is as good a time as any.

Hashcash (also known as the precursor to Bitcoin) was proposed to solve this
problem in 1997:

[https://en.wikipedia.org/wiki/Hashcash](https://en.wikipedia.org/wiki/Hashcash)

The trouble with it is that it requires computation for each sent message,
which is bad for senders with low resource devices or legitimate mailing
lists. I want to propose a variant.

Instead of creating an expensive hash against the message, create an even more
expensive hash against the (sender TLS certificate, receiver domain name)
pair. This implies using TLS but it works just as well even if the certificate
is self-signed. Then each mail server only has to generate a hash once per
recipient domain, ever. Every message that mail server sends to that domain is
tagged with that hash. A legitimate mail server will have already computed
hashes for all the domains its users regularly correspond with and rarely if
ever need to do any more expensive computations.

If spammers do the same thing then the receiving server can mark all messages
sent with that hash as spam. So there is a highly disproportionate cost to
spammers (even if they have more computing power) because to avoid that they
have to continuously generate expensive new hashes. Which can be made
arbitrarily expensive because legitimate servers only need to do it once. And
a new hash is much less valuable to a spammer than a domain name or IP address
is today because each hash can only be used against one recipient domain. The
required amount of computation can be set by the receiving server so domains
with more users can require more computation.

If a legitimate mail server is compromised by a spammer then it will have to
generate all new hashes (because the spammer will presumably immediately ruin
the reputation of the compromised ones), but the reputation of the legitimate
sender's email domains is unharmed because the reputation is tied to the hash
computation, not the sender's domain name(s).

And adding support to mail servers would require no configuration whatsoever.
You install server version N+1 and it starts tagging outgoing messages with
hashes that receiving servers can verify.

So here's the traditional form:
[http://craphound.com/spamsolutions.txt](http://craphound.com/spamsolutions.txt)

How'd I do?

~~~
tromp
Sounds like a plan!

With a memory-bound proof-of-work system like my Cuckoo Cycle, computing the
hash could require the use of more than 4GB of memory for over 5 minutes on a
20-thread server, thus preventing the use of botnets for avoiding the expense.

~~~
chmike
Not sure Internet providers or google will be enthusiast about this proposal.
This is equivalent as asking to pay to register a new destination address. In
this case the mail relay owner has to pay. It would be more fair to ask users
who send mail to pay to register a new destination address. Google and mail
relay provider would be more happy with this variant.

~~~
zrm
It's very friendly to large providers. The amount of computation needed to
send to a domain is the same regardless of how many emails you send so large
providers can amortize it over more emails. And large providers generally
operate their own infrastructure, which implies they already have a lot of
idle computing capacity during off peak hours, and "off peak" is staggered
across regions.

But even if some large providers refuse to hash outgoing mail, they can still
verify incoming mail. The interesting benefit is that a _small_ email provider
can burn some dollars in computation once to avoid having their mail discarded
by a large provider's spam filter indefinitely.

------
teddyh
I sometimes see similar tales of woe, and I can only say that this does _not_
match my experience. I’ve done this many times, you set up the mail server,
configure DNS correctly (including reverse lookup), and that’s it. Never had
problems being blacklisted or mail getting classified as spam.

I _suspect_ that people having trouble are _sending a lot of mail_ , like
“newletters”, etc. But I can’t prove this hypothesis.

~~~
rtehfm
Mail reputations are very real and pose an issue with mail servers. I had a
gaming server for years and had many issues with Gmail, Microsoft and Yahoo,
to name a few, filtering or blocking emails. Just last week, I setup a mail
server using mail in a box on a new server I spun up on Digital Ocean using an
IP that was on no blacklists and still had issues with sending emails to
various Gmail subscribers.

Even when I used to work at HostGator and handled many of their abuse issues,
they had many issues with being blacklisted just because RBLs didn't recognize
new HostGator IPs or the rate of email being sent from their new gateways.

So, at least with my personal and professional experience, I can attest to the
issues with using self-hosted mail servers.

~~~
zhte415
New IPs are often old IPs that have been used for other purposes before, and
often blacklisted. Especially IPs on easily spun-up and spun-down hosts like
DO. On DO, after every VPS I've removed, and then started up another a week
later, it has a different IP. It would be easy for a spammer/phisher/whoever
to abuse this quick address re-allocation, but hurts regular users.

------
MortenK
OP, I don't know if you are reading the comments here but in case you do:
Don't get discouraged so quickly.

The reason this is happening is as the blurb from the MS postmaster help page:
Your IP doesn't have a reputation yet.

The reason these rules are in place aren't about email monopoly, it's about
spam. If anybody could setup a SMTP server and start firing off large amounts
of mail, spam would be even more endemic than today.

You can configure your server perfectly, but that doesn't mean much, since
it's your IP that's the problem.

If you have legit objectives, it's a pain in the ass for sure. But you are not
the only one having this problem, and there's a solution for it.

All the big email service providers (ESP's) like Neolane, Exact Target,
Mailchimp, Campaign monitor etc share this problem when they onboard a new
client, who requires their own IP.

Deliverability is a surprisingly deep, technical topic, and all major ESP's
have entire teams of specialists working on this.

If you want to make such a service as Fastmail, you need to get really into
deliverability. It's not a walk in the park, but it's not impossible either.

I'm not a specialist in this particular area myself, so I can't give you that
much specific advice. I've just worked elbow to elbow with a lot of these
guys, so I know what kind of challenges they work with.

One thing I know for sure is really important, is the "warming up" of IP's.
Basically the IP you are sending from needs to accumulate some reputation over
a period of time, typically a month or two.

If you send out reasonably small amounts of mail to email addresses that
exists and the recipients does not explicitly report you for junk mail, your
IP get whitelisted and you will get a much higher delivery rate.

There's no quick fix unfortunately, and email reputation is hard to gain and
fast to lose.

But it certainly can be done. You sound very competent on the server side of
things, so to get your fastmail-like service up, I think it's just a matter of
a bit more persistence and studying deliverability as a technical subject.

Hope this helps.

~~~
jodyribton
Thanks for the encouragement! I might take another shot at it sometime. I had
this server running for about 3 months with just my personal mail, sending
probably 1 email per day on average. It's possible slightly higher volume
would do a better job of "warming up" the IP.

I self-hosted from about 2007-2008 and 2011-2013, and had nowhere near as much
trouble with deliverability. It came as a bit of a surprise how much more
difficult it is these days.

~~~
MortenK
If you try again, try to see if you can get a client with a decent sized
mailing list. Warming up is usually done with progressively larger sendouts.
So you'd start with maybe 500 sendouts, then a thousand, then 3000 etc.

But anyway yeah the barrier to entry has risen really much. It's much harder
these days than just a couple of years ago, and it probably won't get easier
either!

------
jimktrains2
Email has been the my last hold out from switch away from gapps completely. I
don't want to have to deal with any of this, especially as I do business
communication with clients via it. Email wasn't suppose to be like this, and
there has to be a better way to enable non-giants to successfully deliver
email.

~~~
seiji
Joke answer: the blockchain!

Real answer: It comes down to trust (duh). But, how do we manage trust online?
How do we manage trust in real life? Real life trust is through association of
groups. But, groups online are meaningless. A "gmail user" doesn't belong to a
community, they belong to The Nation of Google.

How do we break down online identities into manageable, trustable communities?
How do we bootstrap new communities into the system so we don't end up with a
pre-selected list of blessed communities while others languish in obscurity?

Open questions, no answers, and if you say Facebook you have to run 20 laps
and give me 1000 pushups.

~~~
jimktrains2
You joke about the blockchain, but HashCash (a similar technique to Bitcoin's
Blockchain-difficulty (not the metric itself, but how it's used)) was
originally conceived as a means of making email computationally costly to
send. The main issue was that spammers would use botnets while normal people
would be stuck taking a while sending emails (iirc).

Personally, I would like everyone to have their own rsa or ecc key. While not
as expensive as HashCash, it does require more computational effort to send an
encrypted email. (This precludes web portals from sending mail, but user
convince seems to always trump security! :( ). Still not a great solution.

Web of Trusts are an interesting idea, but fails for the same reason not
everyone has a public encryption key: bad ui, complex idea, and no one cares.
I honestly think this is the way forward. If someone isn't a few hops from me,
they probably don't have any business contacting me anyway -- if they really
want to, a.k.a. read my blog and want to ask me a question, they can use
HashCash or something else expensive to make the initial contact and slowly
build up trust/reputation as we exchange emails.

In the end, everyone is always told they need to check their spam folder for
false positives anyway. Maybe we should all just say "screw it" and read all
our email. If some combination of the sender and subject doesn't make us think
it's legit, then we can delete it.

Maybe we keep spam filters, but stop calling it a "Spam Box" and treat it like
trash. Maybe we start calling it "Unknown Box" and treat it with a little
skepticism instead of a dumpster.

~~~
pflanze
With regards to HashCash, the explanations I've seen didn't persuade me (the
counter arguments most often cited have been made by some researchers in some
IMHO very weak papers). To me it seems that the real reason why it's not being
used is that ISPs would have to allocate resources, which would make them less
competitive (who wants to offer free email then pay for a lot of
infrastructure, or deal with the mess explaining that users have to pay for
better delivery). The CPU cost could be moved to the clients, but somehow no
IMAP based mail clients seem to ever have implemented it (chicken and egg
problem?), and JavaScript has not been an efficient way to do it.

Also, I guess the big ISPs have an easy enough time (in their view?) analyzing
trust, which is probably a better solution in general if applicable. The
problem is that it's easier to judge (newly created) internal users (accounts)
than external ones (random IPs).

I would think improving the mail system would be a research topic, but perhaps
not interesting enough from the point of view of researchers (would it be best
done in social sciences or computer science?, and there are many real-world
variables, taking time to collect and verify, and working out simulations to
predict possible remedies).

------
staunch
This largely true and hugely disappointing. Our startup
([https://portal.cloud](https://portal.cloud)) is making it possible for non-
hackers to self-host their own email servers. It works very well for the most
part, but we have had to explain to a number of them why their email sometimes
gets bounced by the big proprietary cloud services.

All of our users have their own domain names, IP addresses, SPF records, and
correctly configured (and up to date) Postfix SMTP servers. There is
absolutely no excuse for not always delivering their email, and yet this is
what the big companies do.

~~~
ams6110
There is no obligation to accept email, from anybody.

~~~
staunch
There's no good reason a company should prevent their users from receiving
email from their self-hosted friends.

------
larrys
Edit: They said they checked the spam lists but the ones that I've listed
below may be helpful to others.

OP might have had an IP previously used by spammers or in a spam block. (See
edit).

Important to check the IP before using it if it's been given to you by, say,
the VPS Provider or the upstream. To see what, if any, reputation it has.

Here are two tools to use:

[http://multirbl.valli.org/](http://multirbl.valli.org/)

[http://mxtoolbox.com/SuperTool.aspx](http://mxtoolbox.com/SuperTool.aspx)

And there are a bunch of similar tools.

------
jorangreef
Outlook.com is possibly the worst offender at present:

1\. They set SPF policies of "-all" on customer domains. These customers have
no idea about SPF and often use 3rd party software to send account statements
that never get delivered. This breaks email for their customers and for
everyone else. They should set "~all" by default as Google Apps does.

2\. They claim not to, but they definitely block entire IP ranges rather than
a single IP address. A noisy neighbour in the same datacenter as you can get
you blacklisted. Use
[http://www.senderbase.org/lookup/ip](http://www.senderbase.org/lookup/ip) to
monitor your neighbours and report them to your hosting company. If you spot a
bad neighbour, you will no doubt see your own IP blacklisted at Outlook.com's
[https://postmaster.live.com/snds/ipStatus.aspx](https://postmaster.live.com/snds/ipStatus.aspx)
within a few hours.

3\. They are well known for accepting mail and then silently dropping it,
without bouncing sender or receiver and without putting it into a spam folder,
even when the sender is a longtime sender to that receiver and when the sender
is sending personal anticipated email. They should either reject at the SMTP
transaction (sending a bounce later is bad practice and leads to backscatter)
or else deliver to a spam folder. Silently dropping email breaks email for
everyone.

The sad thing is that many businesses are switching to Office 365 and
Outlook.com.

------
jwr
This is indeed worrying. I've been running my own E-mail servers for the last
20 years or so and even though my problems weren't as severe, I did run into a
few cases where the "big ones" were at least delaying my E-mail.

But this kind of problem invariably arises when we go from a fragmented
Internet with lots of small hosts/providers to an Internet of several walled
gardens, run by the big guys.

The real answer is to offer a reasonable self-hosting competitor to GMail.

------
calpaterson
I have self-hosted my mailserver for a long time and started to get problems a
couple of years ago. The main issue is corporate networks running McAfee's
"MxLogic" product that claim to bounce my mail and tell me so, but then go on
to deliver it almost all the time.

The difficulty in getting feedback is extremely frustrating, particularly
compared with getting feedback from, eg, google's webcrawlers.

------
technion
What has completely decimated mail management in my experience had been the
rise of cryptolocker related emails.

You could always deal with traditional viruses by blocking certain types of
attachments. A false negative on spam just annoyed someone with some Viagra
sale they probably didn't want.

Nowadays, you fail to block an email and suddenly a user loses his entire
department's file share for the day.

------
jmount
It is simple: the more fear uncertainty and doubt the "big email providers"
can cast on no using one of the big email providers the more they chase
everyone into their business (when they can read it). You can go on and on how
it is "technically hard to fix email" but that is a second order effect to not
even trying.

~~~
jmount
"no using" -> "not using". sorry.

------
karlshea
Having run my own email server about a decade ago I can kind of understand why
they are just rejecting things. Dealing with spam for even just a handful of
domains was a nightmare.

------
weihaw
Gmail has Postmaster tools that let domains with moderately large mail flow be
able to monitor their spam reputation scores, email authentication e.g DKIM /
SPF, DMARC rejection rates, etc. This tool can help address a number issues
folks here have mentioned wrt Gmail. In particular please look at the
"Delivery Errors Dashboard" description in the help center article.

help center post: [https://goo.gl/7QHoqc](https://goo.gl/7QHoqc) Postmaster
tool: [https://gmail.com/postmaster/](https://gmail.com/postmaster/)

(disclosure I work in Gmail)

~~~
antichrist
We are using Gmail Postmaster tools (I work at a small ESP), and while this
tool is certainly useful for diagnosing rejections, delivery to Gmail inbox /
promotions is still a black box.

In our case, a number of IP addresses we are using started showing up as "bad"
in Postmaster tools, despite having no increase in complaint rates (actually,
in Spam rate dashboard it shows 0.0% for the past 90 days). We have
implemented Gmail feedback loop headers into outgoing emails to diagnose the
issue, but the Feedback Loop dashboard also does not show any data.

All of these IPs have a Senderscore reputation in the high 90s and deliver
perfectly fine to all ISPs, except Gmail. Contacting Gmail team through their
form also yields no response.

My takeaway from this situation is that once Gmail starts not liking your
emails for some obscure reason, you have practically no way to fix that, apart
from getting a new IP / sending domain and starting to build sending
reputation from scratch.

------
ofir_geller
Did you try to acquire reputation by having real people with accounts in
"known" services send emails to your server?

~~~
josteink
If this sort of shit was required to get a basic www-server up and running, do
you think anyone would have cared about the internet at all?

The internet was supposed to be for everyone and anyone. Current internet
email is terribly flawed. There's just no way around it and the current status
cannot be defended.

------
gwu78
It may be infeasible to run a new SMTP-based mail service from "residential
IP's" that can interact with the existing email empire, dominated by store and
forward middlemen who expect to make money from the "free" email service they
provide.

That empire amounts to a junk email delivery service and later a way to gather
information about email users. The later purpose is probably why you want to
run a new email service?

However it is certainly feasible to run a new SMTP-based email service from
residential IP's that does NOT interact with the existing email empire. One
with no middlemen. The sender's SMTP server talks directly to the recipient's
SMTP server. You decide what port you want to use. There are thousands to
choose from.

There are multiple ways to do this, but I rarely if ever see this option
discussed. I suspect it's because like DNS most users are not comfortable
configuring mail servers nor with NAT traversal.

If indeed the motivation for running your own mail service is because you do
not want your mail stored on third part servers (whether in the sender's mail
folders or the recipient's), then the ability to interact with the existing
store and forward email providers seems a counterproductive requirement.

------
pilif
OP might still have some configuration issue (check the headers of the mail
that has been placed in the spam folder. Gmail usually tells you what the
problem is).

Last August, I have added IPv6 to our mail server, so its address for sure
didn't have any prior reputation. Once the PTR and SPF records were correct,
sending mail over v6 to gmail was no problem at all.

I have yet to see any delivery issue over v6 that's caused by a lack of
reputation

------
drumdance
Surprised he didn't mention third party reputation providers such at Return
Path.

~~~
vangale
I'm surprised as well, since these services reinforce his point even more. In
other words, good reputation = $$$.

Also, I currently use mailgun (and dabbling with mandrill on some new
projects) but I'm intrigued by postmarkapp.com take on dedicated IP addresses.
Apparently they won't sell dedicated IP addresses because of the time needed
to warm them up, so you get a shared IP and their TOS is stricter anti-spam
than other ESP's: [http://blog.postmarkapp.com/post/14127210172/the-false-
promi...](http://blog.postmarkapp.com/post/14127210172/the-false-promises-of-
dedicated-ips)

~~~
old-gregg
Mmm... not really. Dedicated IPs are hard to beat if you have a steady traffic
of high quality and low-latency delivery is a priority (think of PagerDuty).
Most legit businesses/customers have that. Most likely they're not offering
dedicated IPs because they're tough to get, especially if you are a small
company in email business.

The reason Mailgun and Mandrill can do this is because they belong to much
larger companies with better access to IPv4 stockpiles: Rackspace and
Mailchimp.

Source: I worked at Mailgun.

------
andrew-lucker
It's only a matter of time before we start seeing peering issues between major
email providers. This is not a system for sharing, it is competition.

------
andrewrothman
Email is such a broken system. It's unacceptably hard to get a message from
one machine to another and have it not marked as spam.

Furthermore I can't stand when I have to setup all of my accounts across the
multitude of devices that I own. Way too many settings to configure regarding
ports, domain names, encryption techniques, folder mappings, etc.

Don't get me wrong, I'm super impressed that email has managed to continue to
be an essential tool in many people's day to day workflow despite being old
enough to be found in a museum. But it's time for a change, and for us to put
all we've learned about encryption, data serialization, and user workflow out
on the table to design a better future for the universal method of digital
asynchronous threaded communication.

And if we've learned anything from Google Wave and friends, these changes have
to make sense, and not stray too far from what we're already familiar with and
know that works.

Anyone want to weigh in?

------
Animats
My experience has been that sending from my server works fine if it's in DNS
and RDNS. Sending in that server's name works fine if the SPF records are
present. But I don't bulk send; everything I send is something I typed, other
than some messages the server sends to me periodically.

------
jfaucett
You can talk about HTTPS and forcing encryption all over the web however much
you want but as long as you're stuck with SMTP and the few large coorps as the
only viable mail solutions you can forget individual citizen data privacy...
at least thats my 2cents.

------
grey-area
Perhaps the problem here is that there is no verified identity for email
servers?

~~~
jimktrains2
What's that even mean?

~~~
gruez
something like EV certificates for smtp servers?

~~~
jimktrains2
That proves I own the domain to some extent. Don't DKIM records do the same?
For the purposes of anti-spam, a cert doesn't do anything except prove that
you showed some CA that you own the domain (which can be done and is done in
other ways currently).

~~~
grey-area
At present SMTP is so broken that arbitrary servers are allowed to send mail
for any domain, with fake headers, there is no proper verification of sending
server or sending identity, the mail is accepted (along with legitimate mail),
and put in spam folders (sometimes, as is real mail sometimes). SPF/DKIM are
an attempt to solve that of course, but they are not enforced rigorously or
universally, and tactics like IP reputation and email content sniffing _still_
seem to be widely used too (the problem reported in the article). Compared to
the web, email seems pretty behind on verifying server identity.

~~~
jimktrains2
Yes, that's my point. It doesn't really solve any issues that SPF/DKIM don't
already solve -- pointing to certs isn't a new solution, it's a rehash of the
same solution.

And yes, the concept of relays should disappear along with much of the other
cruft SMTP brings. (And while we're at it, can we fix/replace IMAP, or at
least make the spec say that message ids are eternal and can't be
invalidated?)

~~~
grey-area
IMO Message ids should be defined as a sha512 of headers and content to ensure
they are unique. A new protocol is required.

~~~
jimktrains2
Currently headers get modified in-flight, so a content-based ID wouldn't work.

As for a new protocol, there are some out there, but all fail in some way or
another. I figured I'd throw my ideas out there and see how much they fail
[https://github.com/jimktrains/email_ng](https://github.com/jimktrains/email_ng)

------
phantom_oracle
The problem goes both ways.

Spam needs to stop and so too does the convergence of Internet services in
general (not just email).

Frankly, the solution already exists, and just like how billion-dollar
companies start with the tech community, techies need to embrace the idea of
decentralization by adopting some trust-based model.

Frankly, I do not see why encrypted emails cannot be accepted by users through
some peer-sharing agreement where the public-key is online.

There's tons of solutions that are easy for tech people and if this community
embraces it, it trickles down, like how a lot of other things do in tech.

~~~
redahs
I don't think we will see a trickle down effect unless there are substantial
benefits for the average user to outweigh the cost of switching and learning
something new.

If there is a single abstract advantages, like 'more security', that might not
be good enough.

We might as well throw in the kitchen sink and come up with a detailed list of
every possible advantage an ideal replacement protocol should offer:

\- no domain names required to create addresses

\- 1 terabyte+ attachment file sizes

\- ability to update and delete messages which recipients have not yet
downloaded

\- mail transfer and storage agent which don't have access to unencrypted
message contents

\- mail agents which can be distributed across multiple personal and home
devices

\- client apps which always and non-optionally use local encryption\decryption

\- ability to purchase mail serving and storage resources from a competitive
commercial market directly from client application.

\- ability to switch to a different commercial serving\storage resource
provider without loosing your previous identity\address.

------
grey-area
Everyone seems to agree that email is broken (and yet incredibly useful and
almost universal in reach).

So moving on from there, how do we fix it? Who is currently working on fixing
it? What would a new protocol look like?

~~~
bachmeier
Chat services such as Slack, and social media like Twitter, are excellent ways
to communicate. It's pretty simple. If you implement a request system so that
both parties have to agree to let the other send messages, all of the problems
are solved. There's no good reason that we need a system that allows anyone to
send an arbitrary number of messages to anyone else.

------
fensipens
> this server was configured perfectly: (...) SPF, DKIM and DMARC policies in
> place

SPF, DKIM and DMARC are no indicators of spamminess of a source. These systems
have a completely different purpose.

~~~
leni536
They should shift IP reputation to domain reputation though.

------
mgalka
I suspect many of my messages are getting flagged as spam as well. Is there an
easy way of checking whether emails are going through?

------
grinich
I've looked into this a bunch as we've been building Nylas. Our backend is
essentially a cloud mail user agent (MUA), but can have similar issues to a
MTA or mailbox provider.

It turns out that creating a successful product in the email space requires
you to build relationships and partnerships with the existing
vendors/providers. It takes a lot of work, and is part of what you pay for
when using Mailgun/Mandrill/etc.

These "greylists" and systems that drop mail from unknown IPs have been pretty
carefully designed and tuned to combat the insane amount of spam out there.
They work remarkably well, and for most of today's email users, spam is no
longer an issue. (The current state of email abuse innovation is
promotions/marketing/etc. which is a more subtle challenge.)

At the end of this article, the author writes, "This isn't how the internet is
supposed to work." Ironically this system of obscure reputation-based email is
a _direct result of how the email system was actually designed to work_ , with
a total lack of permissions or feedback loop. Many SMTP servers used to not
even require a password.

Stuff like DKIM/SPF and DMARC is a step in the right direction. But the RFCs
upon which our email system is based were written decades ago, and in many
cases have fundamental flaws, like SMTP leaking metadata no matter what. I
could go on and on about issues with email, and why I care about getting them
fixed, but let's just say it was designed in a different era of Internet with
different constraints and opportunities.

So how do you build a new email service and not get blocked? Well, you spend a
few weeks or months emailing, calling, Skyping, and meeting with folks in the
current space. You work your way up through marketing support and random
protocol discussion lists until you are talking with the folks who can
influence which IPs are blocked/unblocked. Then you convince them you are (1)
building a legit venture, (2) are worthy of their trust, and (3) don't
directly compete with them. Then you'll get a small number of clean IPs and
you _must not screw up!_

There are a few hacks, like sometimes a single partner/vendor will sell you a
block of clean IPs and help manage the spam reputation. But usually it's just
a lot of sweat and annoying phone calls. It takes way, way longer than setting
up SPF/DKIM. The challenge is more relationships than technical.

Once you have a new email sending provider, the burden shifts to _you_ for
doing abuse/spam prevention. And you find yourself implementing many of the
strategies and systems you cursed when getting started. But that's the circle
of rfc2822 life I guess.

Oh, and never use EC2 IPs for sending mail. Most of them have been burned by
spammers.

------
z3t4
I guess big providers has to deal with "newbies" all the time, that don't know
how to configure their server and run open relays. And don't know how to add
extra headers etc.

That said, silently dropping messages without a notification is probably
illegal! And pretty serious! So if you know what you are doing (and you are
not a spammer) you should send a cease and desist!

Just make sure you use double opt-in and that providing an e-mail address in
sign-ups/etc is optional!

Some will however put your mail in a spam-folder and it's not much you can do
about it, just hope your readers complain to their provider.

So basically: setup your smtp relay correctly! Make sure you are not on any
black-lists. Add extra headers like dkmi / precedence, add spf (don't froget
ipv6) and ptr. Add your relays to white-lists. Publicly publish a privacy and
e-mail policy (important! with opt in and optional clause); Link to them and
fill out some "email provider" forms at gmail/microsoft. Send out a bunch of
test mails. This will take a whole day, but if you do this, you will have no
problems unless your IP or domain is perma-banned.

~~~
cortesoft
There is absolutely no law that says you have to accept an email, nor that you
have to send a notification if you don't. A cease and desist?! Are you
kidding?

------
maerF0x0
Nobody else has mentioned it, so i will; This is a great situation for the
NSA. Get all 400M accounts in one fell swoop by using a gag ordered warrant on
Google (or microsoft) Ez peasy. Much harder than to contact a sysadmin about
their 1 account that isn't being fed into the behemoth..

~~~
patrickaljord
> Nobody else has mentioned it, so i will

Do we really need a comment against the NSA in every single thread mentioning
emails or hosting in general? Not that I'm a NSA supporter or anything.

------
TazeTSchnitzel
I use a private email provider (privateemail.com, via Namecheap), which
doesn't seem to stop others receiving my emails, but sites sometimes have
trouble emailing me, oddly enough.

Though that might just be because I have a 6-character domain (ajf.me).

------
chmike
SMTP Mail is broken. The best justification is that error messages can't even
be sent back to avoid abuse by spammers. People are constrained to use
services like gmail to avoid troubles. This is a serious privacy threat.

------
k2enemy
The anti-competitive consequences of this are really interesting. Does anyone
know if there are historical statistics on email provider market shares? It
would be interesting to see how things have changed over time.

------
atmosx
I did not realize that I am running my private SMTPd (+ imaps) for nearly 8
years. I never had issues. My score on test-mail is 9/10 because of lack of
DKIM. I will implement DKIM, see if I can get 10/10.

------
bad_user
The irony is that at least half the spam I get comes from @gmail.com
addresses.

------
jwatte
Charge the sender one cent per email through a combination of legislation and
technology. The problem will instantly go away. The transfer of mailing lists
to online forums will be a very small price to pay.

------
motoboi
McAfee Mail Gateway also uses a reputation, called GTI. Appliance hashes the
message, looks up it on this online service and get back a score.

There is a KB explaining that new senders get a high score.

------
mike_hearn
This guys story is a sad one but I'm sure we're missing some important
information here. New mail servers get set up all the time, it's not
impossible for such servers to be accepted by the big players.

If none of these major services ever learned that his server was OK, it's
likely that users weren't unmarking the mail he sent as spam. And that leads
to the question of why not.

~~~
angelbob
In general, everybody uses a service like MailChimp or SendGrid to do this.
That allows them to send email, but keeps them from building up a good
reputation for no spam on their IP addresses -- that reputation is built by
the email provider, not by you as the customer.

For awhile I wondered why so many SaaS apps bothered to pay external (not
cheap!) mail services when it's so easy to set up an email server.

This is why. My "easy to set up an email server" info was years out of date.

~~~
gingerlime
sadly, it's even hard to get some email across when using those ESPs (Email
Service Providers).

Our domain has all the correct SPF records, DKIM etc and we're using a
reputable email provider (Mandrill). All is sparking clean. Validated etc.
Still we see some activation emails to google get delivered, but severely
delayed on the Google side (after we see an OK from google for the email being
delivered). Our users can't reset their password or activate their account
until a certain time passes...

Emails from my mum (who I've emailed on the same account for nearly a decade)
suddenly get flagged as spam by gmail. Go figure.

------
gull
It's a lost cause.

The blacklisting ugliness is one more sign email was badly designed. One more
nail in the coffin.

------
leni536
So, why they look at the IP address at all when there is DKIM in place?

------
SHIT_TALKER
I see lots of threads shitting on the guy for doing it wrong vis-a-vis his
configuration whilst ignoring his actual problem: An IP address without a
reputation score. I've had the same problem and reached the same conclusion.
The address can't just be clean, as in not on a blacklist, but has to
essentially already be whitelisted via a "known good" reputation score or mail
automatically gets blackholed. How do I get my VPS provider of choice to give
me an IP address with a good reputation score?

------
chei0aiV
Another post lamenting this situation:

[https://sfconservancy.org/blog/2015/sep/15/email/](https://sfconservancy.org/blog/2015/sep/15/email/)

------
zer0defex
Just goes to show reputation isn't the end all, be all, solution to
everything. It's so often championed by the Linux kernel team as a reason why
the distributed model works, and it's evident that in that case it certainly
does. Here though, much outcry about how you can't setup a box and instantly
be as respected as established boxes that have earned rep over time. This post
just comes off way too butt hurt for my taste.

------
Sir_Cmpwn
Mail spammers are truly vile people. They have ruined it for everyone.

------
tete
I think it's exaggerated. I am happily running a private mail server on a tiny
vserver. My emails get through to whomever I might mail and I do so a lot.

I once heard that my email got flagged as spam.

Of course, if you don't set up your mail server correctly it might be that
it's flagged as spam, but it's not really harder than setting up various other
things correctly.

It sounds like it was set up correctly by the author. Most people use
SpamAssassin, also big companies do. So it should be good.

Maybe the network itself wasn't considered to be good by the mentioned big
companies.

