Fun fact: sns.amazonaws.com has no DMARC record. Basically all alerts from AWS can be forged or end up in your spam folder and you have no way to know they really came from Amazon. You might say: oh, there's probably a good reason they don't have DMARC, right? Apparently the reason is it's a feature request that they haven't gotten to. And SNS has been around for 12 years.
If I'm a hacker trying to phish someone into giving me their AWS creds, I'm faking CloudWatch alerts that say RDS is down and linking to my malware site. Because real CloudWatch alerts already end up in your spam folder (because no DKIM), no way to know if it's real or not unless you closely examine every link before clicking.
Many email clients don't preserve message integrity when mail is forwarded, which breaks DKIM as well as SPF.
Mailing lists still alter message fields which break DKIM and they forward mail, which breaks SPF. Mailing list software is being brought up to date to deal with this, by adjusting the "From" header and re-signing the message with updated DKIM information.
Many users are still not evaluating their email with an educated, critical eye. All the automated checking in the world still can't stop a user clicking on a message sent from "PayPa1" <no_really_this_is_paypal@hotmail.com>.
SPF, DKIM, DMARC, and ARC at Fastmail: inbound mail
Fastmail checks SPF, DKIM, DMARC, and ARC on all inbound mail. Passing or failing these checks only alters a message's spam score; we do not outright reject mail, only mark it as more or less suspicious. We add a standard Authentication Results header to all received mail explaining the results of the authentication checks.
SPF, DKIM, DMARC, and ARC at Fastmail: outbound mail
We publish a relaxed SPF policy and DKIM-sign all outbound mail from our domains.
If you have a custom domain, you can set your SPF, DKIM and DMARC policy on your DNS screens. We have instructions if you host your DNS with us. If you use another DNS provider, you need to publish the correct records at your DNS provider.
Fastmail domains have a DMARC policy of none, which means recipient mail servers should report whether the message passes or not, but not change deliverability. This allows users to send mail using our domains from anywhere, for legacy reasons.
In the future we will publish a p=reject policy for our domains. This means to send with a From address at one of our domains, youʼll have to send through our servers.
Any mail which passes through our system on the way to another provider will be ARC sealed with the authentication results when the mail was received.
DMARC is important if you want to create your own spam rules, such as "Send all e-mails from *.amazonaws.com that don't have valid DMARC to a Phishing folder".
Sure, an attacker could use no-reply@sns.amazonaws.com.attacker.domain. But most of our users have been trained to briefly look at the From: address before clicking things, so those would not succeed often. Without DMARC, we can't even catch phishes of the real domain.
This is going to sound funny, but is there a way to sign only certain emails (out of all emails a user sends) instead of signing all messages with DNS based verification?
I should mention specifically in a way that’s at least decently user accessible; ruling out GPG.
You can always sign based on specific rules via your MTA to sign some and not sign others. You can also send from many sender's (say allocated by spf allow rules) that aren't signed at all but matches the IP ranges allowed.
Why would you use SNS to send emails? That just seems really silly to me. It’s meant for pub-sub. I’d alert with PagerDuty or OpsGenie, so I’d not only get an email but a text or voice call.
SNS has two modes: A2A and A2P. A2A is pub-sub, A2P is for sending notifications to users (via e-mail, SMS, or mobile push)
CloudWatch Alerts are sent from SNS, and I would imagine others are as well, although some may use SES. I think SES has the same issue, as the AWS docs only mention DMARC/DKIM in the context of a custom domain.
As other commenters noted, this doesn't seem to be a "forged" message at all; it's DKIM-signed by youtube.com.
Probably whoever owns "alltimecaptaincool2019@gmail.com" has it forwarding to "robtoledoyour.com", who is then forwarding (with ARC!) to the author's Gmail mailbox.
I don't see why this shouldn't work--there's nothing in DKIM, DMARC, or ARC which is intended to prevent forwarding of legitimate emails (and such a feature would annoy many users of email!).
A well-functioning spam classifier knows that this email a) has a body that was signed by YouTube.com, b) was forwarded via robtoledoyour.com, and can make a spam classification decision based on that. Allowing the mail through seems reasonable given the contents are not spammy.
I'm unclear on what a protocol--SMTP or anything else!--could do better here other than just not support forwarding mail at all (or require bidirectional approval before allowing forwarding), which would be hugely disruptive to existing email users.
> I'm unclear on what a protocol--SMTP or anything else!--could do better
> here other than just not support forwarding mail at all (or require
> bidirectional approval before allowing forwarding)
I think requiring approval would be the correct behavior. For <robtoledoyour.com> to be allowed to send email on behalf of <youtube.com> should require explicit permission in <youtube.com> DNS records (or similarly canonical location).
As you note, this would break some of the remaining hold-outs who treat SMTP as an unauthenticated store-and-forward protocol. This seems both directionally correct, and aligned with how use of email has changed over the past ~20 years.
> I think requiring approval would be the correct behavior. For <robtoledoyour.com> to be allowed to send email on behalf of <youtube.com> should require explicit permission in <youtube.com> DNS records (or similarly canonical location).
Sorry, by "bidirectional approval" I meant that the author's (i.e. recipient's) mailbox would somehow confirm its desire to receive mail forwarded by robtoledoyour.com.
What you describe is, I think, impossible without eliminating certain extremely common use cases; it would break:
* Mailing lists (which are the whole reason ARC exists); I don't want to have to approve lists.kernel.org to send arbitrary mail on behalf of me (and certainly not on behalf of gmail.com, which I don't control!); I want it to be able to forward mail that it promises came from me and for recipients to trust it only if they trust lists.kernel.org (which is how ARC works).
* Mail forwarding between a user's different mailboxes; there's no way I can get Amazon.com to agree that myspambox@gmail.com should be allowed to forward mail on behalf of Amazon just so I can use a throwaway to forward mail to my main account.
I can imagine configurations where everything works dramatically differently, but it's hard for me to see what's wrong with the status quo (a signed message, with ARC, etc) except that To headers are allowed to not match RCPT TO, which as others have noted is both a point of confusion for users and how BCC works, and thus a hard to eliminate feature.
> Mailing lists (which are the whole reason ARC exists)
It would break mailing lists that want to have the original author's unmodified email address in the `From` header, which ... sounds fine? Contemporary mailing list software sends `From` headers like this:
'Jane User' via Some Mailing List <some-list@example.com>
which is verifiable.
> Mail forwarding between a user's different mailboxes
I would expect this to continue to work as before, as long as the mailboxes are in the same MTA -- for example admin@ and postmaster@ forwarding to jdoe@. If you mean `admin@example.com` forwarding to `jdoe@otherdomain.com`, then I think it would be appropriate to require explicit bidirectional opt-in (and Gmail IIRC does enforce just that).
edit: Also, this doesn't need to be enforced for all domains from the start. If domains want to continue to be open forwarders that let anyone use them to send spam, they don't need to set SPF/DKIM/DMARC records (and receivers like Gmail will continue to bit-bucket their mail).
I think you're basically describing the current ecosystem. :)
ARC allows what you describe for mailing lists, only, even better, it's machine-readable--the message metadata indicate that "Jane User" sent the mail, according to "Some Mailing List", and that Jane User's email was verified per DKIM!
And MTAs can (as you note) require bidirectional confirmation for forwarding and (as you note) can easily identify which authenticated senders actually sent the mail and, if they determine it's an open relay, bitbucket it!
So what you describe is pretty much the status quo, I think. The fundamental issue with the YouTube mail the author has encountered, of course, is that the forwarding domain is authenticated as having passed on an (unmodified) YouTube.com email, and Gmail--quite reasonably, I think--just doesn't know enough to know if the Gmail recipient of that forward wanted it. Since the message has no obvious signs of spam (like linking to a suspicious domain), it seems reasonable to me to treat this like any other authenticated, non-spammy email from a domain Gmail has not seen much volume from before (which I assume is the case here).
Ultimately, even with DKIM, DMARC, and ARC, email allows people to send mail to people they've never communicated with before, and this is an important function to preserve!
(As an aside, I totally acknowledge that a) the protocol complexity in email is a lot worse than it could be if all this functionality was built in up front, and b) there are alternative tradeoffs we could make that would remove some of this complexity. And the complexity is obviously a problem for user comprehension or else we wouldn't be having this conversation! But, fundamentally, if we want to make significant changes, we would have to revisit seemingly desirable features like non-matching envelope and header To, allowing unsolicited email from other people, authenticated forwarding, etc.)
I think the core disagreement is whether forwarding an unmodified signed email should be considered "from the original author" or "from the nearest authenticated hop".
You say that the email should be considered to be from YouTube, because it was originally created and signed by YouTube. In your model, the fact it was received from some unknown third party is non-notable.
I think it's the most recent hop that matters. Just because the original content of the email came from YouTube does not mean that it should be allowed to claim a @youtube.com From address. It should have a @robtoledoyour.com address, because that was the nearest hop in the forwarding chain that Gmail can verify (e.g. via TLS).
And then, since the email is "from" robtoledoyour.com but claims to be from YouTube, it should be discarded (or at least sent to spam).
As you note, this would break use cases that depend on DKIM to allow relaying email through unrelated third-party servers. I think that's fine, because I don't think third-party relays should be allowed to claim the identity of the original server.
I don't think that's the core disagreement at all. You're being too unspecific about what "from" you are talking about. :)
All the YouTube.com DKIM signature says is that YouTube.com did in fact author this content. Which is true!
ARC and SPF (both present on this message!) indicate that the last hop was not YouTube.com, and who the last hop was! So all the information you were asking for is in fact present here.
What you are suggesting is just a different phrasing of your prior suggestion: that people should not be allowed to forward email. And given this message has an ARC header that indicates it was forwarded, and from whom, it seems really strange to me to say that instead of messages that indicate they were forwarded, you think we should just ban forwarding entirely.
Maybe the problem is on Gmail's MUA: maybe this should be more prominently shown to the user as a forwarded email?
I'll try using different terms, in hopes that'll help:
* The author of the email is <no-reply@youtube.com>, verified by DKIM.
* The sender of the email is <postalerts@robtoledoyour.com>, verified by TLS.
If the author and the sender are the same, then there's no problem -- that's who the email is from.
If the author and the sender are different, you think the email is "from" the author, and I think the email is "from" the sender.
IMO whether the content was originally authored by YouTube might be interesting in some vague abstract chain-of-trust sense, but it's not useful to me as a consumer of email. I want to know (1) why some message came to arrive in my inbox and (2) who sent it to me. The desired UX is something like this:
[postalerts@robtoledoyour.com] Updates to YouTube's Terms of Service
(ideally also tagged as spam due to the mismatch between author and sender)
An analogy might be physical mail. If I receive a copy of Dante's Inferno in my mailbox, then I expect the return address to describe the person who put my address on the shipment. It would be beyond useless for the return address to say "Dante Alighieri, Italy" even though that is an accurate description of the work's author.
As a nitpick, TLS does not indicate the sender of the email.
You can try using different terms, but the standard terminology in the mail space is "envelope From" vs "header From", as I used above. These are different things, and there's enough information in this email for you to know which is which.
Some MUAs--like Outlook--would render this email as
"Sent by postalerts@robtoledoyour.com on behalf of no-reply@youtube.com"
As I said, it may be that you want your MUA to do that, in which case...use Outlook? But this has nothing to do with limitations of SMTP.
Here's the interesting thing: DKIM verifies the message contents. You can add many hops through many servers and still verify the DKIM - It's a design feature, actually, because of
Similarly to GPG, where an authenticated e-mail can be forwareded but you can still read and verify the original author's signature, DKIM hashes over the message contents and not the headers, so you can add remailer headers and still verify the original domain. It appears that the message contents are indeed from youtube.com, even if it was forwarded. This is the property that we want from mailing lists, where you can verify the original sender's domain authorized the letter even though you received it from a different domain.
Why someone is forwarding a legit google e-mail on and adding their own address, I cannot say. There's probably security implications, and ways to abuse it, but the message itself and the DKIM check are legit.
> I think requiring approval would be the correct behavior. For <robtoledoyour.com> to be allowed to send email on behalf of <youtube.com> should require explicit permission in <youtube.com> DNS records (or similarly canonical location).
What DMARC solves is that you can allow a service to originally send email on behalf of a domain. It does this by adding a 'alignment' requirement for SPF and DKIM, on top of the simple pass/fail mechanism.
- For SPF alignment the domain part of the rfc5321.mailfrom and rfc5322.from must match.
- For DKIM alignment, the public key must be published under the domain found in the rfc5322.from address.
If either SPF or DKIM is aligned the email is considered a DMARC pass.
Since SPF alignment breaks with forwarding (amongst other horrible flaws), it is common practice to focus on DKIM and not rely on SPF. Once an email is DKIM aligned (thus passes DMARC), it will remain aligned if the email is forwarded. This is by design, so that DMARC will not break forwarding.
The email in the article was originally DKIM aligned (thus passed DMARC inspection), then forwarded.
There are various strictness settings in SPF, DKIM and DMARC in both the DNS records and the email headers, but none will prevent forwarding of email.
... "In fact, an ARC chain can be counterfeited,[3] so ARC processing applies when receivers trust the good faith of ARC signers, but not so much their filtering practices."
Hmm, article references an e-mail post by good old John Levine, moderator of comp.compilers for over 30 years, who has some interesting things to say:
"[If] I were a certain kind of bad guy, I would take the two seal ARC chain
from a message from a virtuous sender, replace the message body and
>From and Subject line with my spam, add a fresh new i=3 seal and blast
it out. That ARC chain is 100% valid, even though the messsage is
spam."
You're implying that this message has been tampered with, but it has not. It is perfectly authentic. Whoever produced this message possesses the private signing key of youtube.com. This message has in no way been forged.
The only mystery here is why someone is mirroring google mail back to gmail with unexpected envelope recipients. It could be a weird error, or it could be they think they can game the IP reputation system by doing it.
In any case, the fine moderators ought to correct the title, because it is wrong and misleading.
The message has been forged, as can be observed from the From: header being a domain that the sending server was not authorized to send from.
Whether a subset of the message -- in this case the body -- is authentic doesn't matter. If I were to MITM google.com and send back an archived snapshot from a month ago, that would be forged traffic even if it matches responses that Google had once sent.
Remembering back to the uucp days and when certain OOB nodes came back up after outages, this is precisely why email as a standard can handle delayed delivery and MX records have preferences for relays that can queue for future delivery.
You are inventing imaginary requirements for email that do not exist. The "From:" header is not part of the envelope. The "From:" header is protected by the DKIM signature. This message is from youtube.com. You received it via some other relay. Nothing about that is weird. People love relaying email all over the place.
Do you have a hypothesis about why someone would be re-sending these YouTube messages? One possibility is that the original To: recipient of this message is an bad actor's address. Maybe some small percentage of users will contact that address, and thereby get harvested.
Interesting; I actually thought about that too! But there are ways it could backfire, like people reporting this junk to blacklist sites. Spamcop doesn't care that what you sent is a copy of a valid e-mail from YouTube with good DKIM.
> That email was sent by <robtoledoyour.com>, claiming to be <youtube.com>.
Not exactly! The "Received:" header lines track the identities of forwarding hosts. So we know that the mail was forwarded by a certain machine. Here is what I think the line exactly nmeans off the top of my head:
from 7n.robtoledoyour.com (7n.robtoledoyour.com. [2a01:7c8:bb01:51a::7])
^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
A B C
The A, I think, comes from the string that the host gave in the SMTP HELO command. C is the IP address pulled from the SMTP connection, and B is what that resolved to under reverse DNS.
Now, none of these is understood to be the sender. For the purposes of SMTP, the sender is the envelope sender: the argument given in the SMTP "MAIL FROM:" command.
So I think when we are talking about, say, the SPF system, the S refers to that sender. An SPF record published by a domain example.com says something like "if such and such hosts send an e-mail whose envelope sender is example.com, they are blessed to do that".
So yes, there is the concept of whether robtoledyour.com is allowed to forward mails which are from a certain sender.
WHAT WE DON'T KNOW HERE: is what that sender is! We know only the From: address in the header, which is no-reply@youtube.com. That is not necessarily the envelope sender!
The e-mail just has a forged From: line, which is easy to do.
Now, let's look at the full e-mail again. There is more information:
Received: from 7n.robtoledoyour.com (7n.robtoledoyour.com. [2a01:7c8:bb01:51a::7])
by mx.google.com with ESMTPS id es15-20020a056402380f00b0042de38ce571si326031edb.295.2022.05.31.10.35.25
for <jmillikin@gmail.com>
(version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128);
Tue, 31 May 2022 10:35:25 -0700 (PDT)
Received-SPF: pass (google.com: domain of postalerts@robtoledoyour.com designates 2a01:7c8:bb01:51a::7 as permitted sender) client-ip=2a01:7c8:bb01:51a::7;
The Received-SPF line reveals to us who the sender is! You see? The sender is postalerts@robtoledoyour.com, not Youtube!
Here is what happened: the spammer directly sent the mail from the their machine to google, using postalerts@robtoledoyour.com as the sender. Google verified the SPF: yes that spammer machine 2a01:7c8:bb01:51a::7 is allowed to send messages on behalf of "postalerts@robtoledoyour.com". Meaning that that machine can connect to a Google mail server and use the command:
MAIL from: postalerts@robtoledoyour.com
Google's delivery checks (whatever they are) didn't care that the From: header purports to be from Youtube, not matching the envelope sender at all.
It is legitimate for envelope and From senders not to match. It happens with mailing lists, for instance.
When you get a mailing list post, the From: will indicate the original person who wrote the post. (It had better!)
But the mailing list posting is actually from the mailing list, not from that user. When the mailing list robot connects to its forwarding server (or perhaps directly to your server) it will use a command like command "mail from: foo-list-bounces@example.com" and not the address in the From: line.
The mailing list cannot use the address in the From: line because that will not pass SPF checks! Most mailing list sites do not have SPF permission to send mail on behalf of the list participants, as the envelope senders. They have to send as themselves, and just forward the mail data with the original From:.
Now let's discuss the DKIM aspect of this. Google checked the DKIM signature and it passed. This is because what the spammer sent is in fact a Youtube message, verbatim.
Nothing in the body of the message points to the spammer's domain, for instance. All the links in the HTML are the original Youtube material.
Here is possibly what the spammer might be hoping for. It could be that the original target of the Youtube message, namely alltimec____ncool2019@gmail.com (the one in the To: header) is an address controlled by the spammer, who is hoping that some of the recipients of the e-mail will send a message to that address.
That will be unlikely because YouTube also set the Reply-to: header. Maybe in some mail clients, if "reply all" is used, it will include alltimec_____ncool2019@gmail.com.
> Correct, this message isn't forged, it's authentic
I personally receive a lot of emails that aren't addressed to me and aren't marked as spam and have always wondered how this works.
Can someone go into a bit more details about why this works? In what way is this message authentic?
The ARC wiki page says "Validating an ARC chain only makes sense if the receiver trusts the ARC signers."; in this case the receiver is Gmail. Why does gmail trust a random domain (robtoledoyour.com)?
The envelope recipient - the account to which the message is delivered - hasn't got anything at all to do with the contents of the "To:" header, which can be absolutely anything. That's how the "Bcc" feature of email works!
The "To:" header is not involved in any way with the delivery of emails.
I think it’s because YouTube.com is the original sender and it’s considered ok if the receiver trusts the signers. The problem IMO is that the receiver’s mail provider (ex: ms365) doesn’t know if the receiver trusts the signers when deciding to deliver messages.
I’ve seen bad actors flood mailboxes with forwarded mail to obfuscate malicious activity. For example, they send a forged message asking for banking info updates and flood the real address that’ll get the “ok done” reply which I assume is an attempt to delay discovery of the attack.
Yes I would love to know why this works as well. I've noticed over the past few months, I get emails that land in my gmail inbox that are obviously spam (various bs offers), and I've always wondered how they are able to bypass the spam filter with ease it seems. Also the To email line is to an email address that is a variation of my email address @aol.com that doesn't actually exist
The latest thing that started happening a week or two ago was an endless stream of unsolicited mailing list messages addressed to "Prunella" trying to get me to click on links. Ended up just setting up an autodelete when that name is in the Send field (because I discovered Gmail doesn't let you auto-spam folder things).
The reason that message has that addressee is because the To: is protected by the DKIM signature, so whoever relayed the message cannot have changed it.
This is because John was the envelope recipient: the one named in the SMTP protocol "RCPT to:" command.
There are ways that a perfectly legitimate e-mail you receive night not mention any of your addresses in To: or Cc:.
E.g. you are subscribed to a mailing list and some new posting (not part of some discussion you are in) arrives. It might just be addressed as "To: whatever-mailing-list@example.com". You received it because the mailing list robot used your e-mail as the RCPT to: envelope address at the SMTP protocol level.
Any ideas for why that email appeared in your inbox? It doesn't look like it's a phishing attack - "Updates to YouTube's Terms of Service" is a subject line that guarantees that precisely zero people will even bother opening it.
My rule: If there isn't a prominent, 1-click unsubscribe link in an email, it gets marked as spam.
If I click unsubscribe, and the form asks me to fill in information, it gets marked as spam. No, I'm not filling in my email, let alone completing a CAPTCHA, to ask you to stop sending me emails.
However, I don't click an unsubscribe link in an email where I suspect the unsubscribe link is itself a scam to collect information.
Perhaps to check if the inbox is valid? Sending from some random domain you risk having your mail silently dropped. If the server thinks you're youtube, then you're going to get through.
Yes, but those likely to read ToS updates are not going to be the demographic that likely falls for scams; proper scammers target the population that is less adept at technology so they don't notice they're being duped.
I have had the opposite problem for years. (Well, if it's a problem.) I have many Google accounts opened with disposable addresses, one for each service. (They are pretty old (10+ years) and to many of them I have lost access. I have the password, but Google does not let me in without providing a phone number first. Because I don't want to share my phone number every time I cannot register to any new Google services. But I guess that's only a good thing, the company has become too evil a long time ago.)
All addresses get forwarded to my "main" gmail. And since the beginning of times all updates of terms and whatnot go to spam. I believe for phishing, not even for similar messages have been recognized as spam before. I mark them as not spam every time, but their filter just doesn't learn.
Because I use disposable addresses everywhere I get very little spam. I close addresses that end up at a spammer. For me Gmail's filters produce at least 80% false positives. And marking messages as not spam seems to have little to no persistent impact.
Gmail / Google Workspaces don't care about their paying customer's privacy and are too lazy to improve their deliverability. Issues with inbound header handling wouldn't surprise me either.
If you use multiple domains with Google Workspaces and noticed your emails not getting through when sent from your non-primary domain; you're not imagining it. Any mail receivers that are properly checking SPF are not going to give you a good deliverability score, no matter what you do. Gmail leaks your primary domain in the Reply-Path rather than specifying the domain you sent the email from. I've reported this to them, they don't give a shit. Fun.
Yeah, there are reports of Gmail setting the Return-Path to the primary domain going back a decade ago. That makes the alternative domain pretty useless if you want to get SPF passing. Fastmail does the right thing FWIW, they switch the Return-Path to the alternative domain when you send from there.
Could there be a Sender header that was being verified instead of the From? Is that how it works? What about envelope From? Which of those are even protected by DKIM/SPF?
Meanwhile, Gmail is obsessed with tagging messages I get from nearly every mailing list I'm on, including ones hosted by google groups, as spam. Even messages sent by a gmail user, to a google group.
Outdoor groups? Spam.
Technical mailing lists? Spam.
Social group mailing lists? Spam.
I have filters assigning these messages to labels.
I have been on some of these lists for ten fucking years.
I routinely go into my spam folder and find a dozen messages from these lists tagged as spam, dutifully click "not spam", and rinse/wash/repeat.
Rarely does spam make it into my inbox, but it's so much worse that messages I give a shit about are routinely dumped into my spam box. But not so routinely that I think to check it often enough...
That's the consequence of machine learning and allowing people to mark stuff as spam. 80% of all spam reports I get are for legitimate mails. A surprisingly large amount of them are people who mark their own mails as spam.
While these factors can cause a quarantine or even silent disposal, empirical evidence suggests that there are custom mechanisms (ie spam filters) downstream that do the larger part of the filtering.
I have been running private Mail servers since the early 2000s and this last decade has certainly not been nice to folks that want to self host. I cannot even send my mother in law email because Google just plainly denies delivery with no way to circumvent. Funnily enough mass mailing from the same account 10000 other gmail addresses has delivery metrics just like doing it with mailchimp. I think it is safe to say that big email hosting has grown so complex internally that the companies themselves can’t really make sense of it any more.
Edit: to preempt possible “you are doing it wrong” comments, my Mail server setup checks out in every conceivable test suite and email content as well :-)
It's not just you, and it's not just people who are self-hosting. The agressive spam filtering at Gmail/Hotmail is an industry-wide issue and while most of us get by, there's no denying that there's a lot of false positives.
The Mailop mailing list has a nice note about it on their Best Practices page:
> If you want to send mail to recipients who have accounts at big email providers, be aware that all of the above cannot guarantee that these providers won’t reject your mail, put it straight into recipient’s spam folder or just silently discard it - they just impose their own rules on anyone and you virtually can’t do anything about it.
Failed SPF/DKIM/DMARC doesn't trigger abuse reports through feedback loops. Marking messages as spam does.
These are cases where one party has decided to mark one or more emails as spam. None of the emails in question are bulk messages -- it's all sent from properly authenticated domains and most of the mails are from ongoing conversations. A lot of it's correspondence between private individuals, others between businesses.
Giving people a Junk button is a bad idea at scale. There's been a lot of discussion about this being a linguistical issue (that people don't realise Trash and Junk are different things), but I regularly see people marking entire conversations as Junk when they're angry about not getting an invoice's due date moved, or when arguing with a family member. The worst part is that people don't realise the Junk button often triggers a report to the sender's ESP. The contents of the reported emails are usually included, and in most cases the reports are read by someone else.
All this is also processed by spam filters, regardless if it's simple SpamAssassin/Bayes setups, or more advanced systems at Hotmail/Gmail.
I have the _exact_ same problem, and I've never had it with any other mail provider. They say Gmail is 'the most advanced'. I may be wrong but I don't remember them being as bad going back, so I guess their advancements aren't working.
I mean, Paul Graham seems to be doing fine with a simple[1] algorithm. If I remember correctly, most mail providers used to do the same thing as he described.
"Good security means constantly staying ahead of threats, and our existing ML models are highly effective at doing this—in conjunction with our other protections, they help block more than 99.9 percent of spam, phishing, and malware from reaching Gmail inboxes" [2]
That may be true, if you don't consider false positives.
This is so true. This false spam tagging has caused me so much pain with paying customers, I ended up white listing all email to a particular to address using a filter.
It's unbelievable that email is so fragile and broken.
Big email providers need to set a flag day after which they will drop all email from domains that don't have DKIM+SPF. Their servers also have to support TLS for sending and receiving mail.
That's the easy half. Next, we need a stronger system of domain reputation. For this, we grandfather in all existing domain names, but any newly registered domains need to put up a bond for good behaviour. If one of the big email providers detects high levels of spam from the domain, they can slash the bond, but after a few years of good behaviour the bond is returned.
Of course this gives a lot of unelected power to the big providers, but guess what, they already have that power, except they're using it for making their moat bigger, not for actually solving the problem. If this bond system used a public ledger, then smaller competitors would be able to see the reputation decisions that these big providers made, and crypto-currency would also be suitably international for a global system like email.
This is fully misguided. Did you ever maintain a mail server and/or check your spam? SPAM is for the vast majority SPF/DKIM/whatever compliant (in fact, my regular email has less DKIM signatures). The joke was always that SPAM was passing all these checks _earlier_ than real email systems.
Why?
SPAM is mostly sent through illegitimately acquired accounts on big-brand servers now. Add a new check? Well, sorry to break it to you, your spam is going to have it too. In fact, on the systems I manage, 70%+ of all spam is coming from gmail itself.
People conflate SPF/DKIM/DMARC with spam checking.. they're not systems to prevent spam. They're intended to prevent forgery.
Realistically, to protect from the dumbest forms of forgery (which is what spam initially was leveraging on), SPF is really all you need.
If a spam email can leave a system configured with SPF, in practice it's already a problem with the organization that let that message out. No system is going to help you with such a problem, and so they're equally useless as spam filters.
> People conflate SPF/DKIM/DMARC with spam checking.. they're not systems to prevent spam. They're intended to prevent forgery.
That's why I said that mandating these things was the easy half.
Once spoofing isn't possible, we have to build a proper (decentralised) system of domain reputation, with financial penalties attached. (And in order to get all existing internet users to agree to this, we have to exempt them and only burden future domain registrants, since there are no lobbyists for corporations that don't exist yet).
My understanding is that a lot of spam comes from cheaply and temporarily registered domains which are discarded as soon as the emails have been sent, because the domain's reputation is trashed within hours. If doing that caused the spammers to lose even $100, it would put their costs up considerably, and hopefully destroy their profit margin.
It is not. There's still a significant amount of spam that doesn't have SPF, there's a lot of forgery and it's easier to filter if everyone used the holy trinity.
> SPAM is mostly sent through illegitimately acquired accounts on big-brand servers now.
That's what you see on and from gmail, but it's not the majority of spam.
> It is not. There's still a significant amount of spam that doesn't have SPF, there's a lot of forgery and it's easier to filter if everyone used the holy trinity.
Absolutely. But but again, this will _not_ solve SPAM as the parent implied.
> That's what you see on and from gmail, but it's not the majority of spam.
There's a huge variability on sources, depending also on the class of users you have on your server. However I do keep all copies of spam that could pass greylisting for classification purposes since the early 2010' from various sources and honeypot addresses -- correctly signed spam _is_ the vast majority.
Low-effort spammers are pretty easy to weed out. And while rejecting messages without SPF is still not feasible, it wouldn't _improve_ the ham/spam filtering ratio in my case.
It's a very subjective view, but while I do absolutely recommend for every single system to have SPF at the very minimum, I didn't see any practical advantage for spam filtering for newer proposals such as DMARC/DKIM. Any spammer that has SPF enabled, will have the rest.
I liked Bill Gates' idea of making senders pay a fee (perhaps a minimum amount specified by the recipient) and giving recipients the option of refunding it for personal contacts.
Ok. So. I'm not going to say that is a bad idea, but I will share a story.
I used to work for a company that was the USPS's #1 or #2 customer (depending on the year). We spent a TON of money on spam. The biggest cost was the stamp.
If people have to start paying for emails- it will eliminate a lot of the spam -but it won't eliminate all of the spam- and companies like the one I used to work for will start to shine.
What if you are able to set your own fee, so before sending mail you lookup the difficulty level for the receiver. People can just keep raising the dificulty until they stop seeing spam.
This was the concept behind HashCash, the concept behind BitCoin/Cryptocurrencies in general.
The idea being that each email would do a non-trivial amount of work, such compute a valid hash - encode it in the message header.
This message header can be checked quickly on receivers side, but takes time for the sender.
This makes sending a large amount of email quickly infeasible without a significant amount of compute power.
Honestly I think email needs to be rethought from the ground up, just a whole new protocol with a killer app to get it going.
Some of the most salient uses/qualities of email:
- Asynchronous communication
- Decentralized / Federated
- Identity
Issues:
- Spam
- Lack of room for the format to expand
- Identification verification is hard
My ideal replacement for email would look something like a decentralized/federated identity protocol with default whitelist for contact upon which you build out data formats for sharing.
One key spam reduction technique in a federated system is to realize that most legitimate interactions are with people and services you are already in contact with so it’s better to make initial connections harder to improve the quality of daily use.
Another tool that works decently in a decentralized system is introductions/vouching. Eg. you can ask mutual contacts to connect you or at larger scale you can entrust third party services to “vouch” for a user, similar to how certs work. If a vouching service does a poor job of vetting who it lets through than users will stop trusting it.
> Honestly I think email needs to be rethought from the ground up, just a whole new protocol with a killer app to get it going.
That's why it hasn't and won't happen any time soon.
> My ideal replacement for email would look something like a decentralized/federated identity protocol with default whitelist for contact upon which you build out data formats for sharing.
Nobody stops you from implementing sender whitelists with email.
> Another tool that works decently in a decentralized system is introductions/vouching.
This idea boils down to PKI but that's far from perfect due to humans, not tech.
We could already absolutely significantly increase mail trustworthiness if we'd use S/MIME en-masse. If postmasters aren't improving mail integrity, end-users could. A few EU countries have deployed certificates to people, but it hasn't reached critical mass to act as an indicator yet.
Ban Microsoft and Google from email to start (they both control an awful lot of email but actively undermine its usefulness). Death penalty for spammers (maybe three strikes, or 300% of revenue from spam + audit costs if you want to be really nice). Corporal punishment for inappropriate reply-all. Refuse HTML email.
Maybe some sort of nice and easy to use mailing list setups, so I can invite you to my family / friend / crazy internet people list, and I'll send a mail every once in a while and you can read about what's going on with your family / friends / crazy internet people in a nice text optimized way.
I'd also love someone to make server side filters that operate on read email or email left in inbox for 30 days. I want my mail to sit in a big mixed pile of garbage when it's newish, but after a while, settle into a decent taxonomy.
Well, you shouldn't use GMail. They're mining your emails for information and occasionally passing them on to the US government, as Edward Snowden has revealed. There are other email providers, both free and for-pay, which should suit your needs well enough.
On one hand, they aren't special casing their own products, so good on them I guess? On the other hand, if they can't detect this, what point is their spam filtering even?
Public mail services can be surprisingly bad at what they decide not to catch. I love Fastmail but because some existing customers rely on it, the default behavior is not to block inbound emails with the same domain as you've delegated entirely to Fastmail.
One would expect a given provider should, unless instructed explicitly otherwise, block outside emails from domains it controls mail for.
> One would expect a given provider should, unless instructed explicitly otherwise, block outside emails from domains it controls mail for.
It's not even something you need to specifically craft much infrastructure for. There are public standards for "don't accept email from this domain unless the sender has a cert encrypted by one of these CAs" (Google runs their own CA, yes?) and "don't accept email from this domain except from these servers" and so on.
If I'm a hacker trying to phish someone into giving me their AWS creds, I'm faking CloudWatch alerts that say RDS is down and linking to my malware site. Because real CloudWatch alerts already end up in your spam folder (because no DKIM), no way to know if it's real or not unless you closely examine every link before clicking.