
DMARC Secured Email Identities but Broke Mailing Lists - samtoday
https://learntemail.sam.today/blog/dmarc-secured-your-email-identity-but-see-how-it-ruined-mailing-lists/
======
zAy0LfpBZLC8mAC
This is a terrible article that's almost completely wrong as it doesn't even
mention the distinction between envelope (SMTP) from and header (RFC822) from.

The envelope from is what is transmitted in SMTP commands and specifies where
bounces due to delivery delays or failures are supposed to be sent, the header
from is what is displayed to the recipient as the sender, but is completely
ignored by SMTP.

SPF only deals with the envelope from, DKIM only deals with the header from
(and other parts of the email headers/content).

Really, there is nothing there that necessarily prevents mailing lists from
working just fine:

A mailing list can (and should) replace the envelope from with its own
address, so that bounces from subscribers aren't sent to the author of the
message, but to the mailing list software, which then can do bounce
management, such as automatically unsubscribing addresses that consistently
bounce because they don't exist anymore. As the mailing list software should
use its own domain for the bounce addresses, the mailing list operator can set
up SPF to authorize the mailing list server as an outbound server for that
domain just fine, and that has no effect whatsoever on the header from that's
shown to the recipient, and that replies would go to by default.

As for DKIM, you simply should not modify the message, and it will deliver
just fine, whether through a mailing list or directly. Modifying the message
mostly really shouldn't be necessary. Reply-to mangling is a bad idea anyhow,
and the mailing list should be recognized by the client software either
because it's in the destination headers, or by using mailing-list headers
added by the mailing list software, instead of mangling the body or the
subject.

~~~
temprature
_> As the mailing list software should use its own domain for the bounce
addresses, the mailing list operator can set up SPF to authorize the mailing
list server as an outbound server for that domain just fine_

This will make SPF pass but it has no effect on the DMARC result. For DMARC to
use the SPF result, the envelope FROM needs to be aligned with the header
From. Pretty much every mailing list already does what you've described but it
doesn't help with DMARC.

 _> As for DKIM, you simply should not modify the message_

Many mailing lists modify every message to add a footer with unsubscribe
information. Others only modify some messages when necessary such as to
convert HTML messages to plain-text, to strip attachments, to wrap lines to 72
characters etc.

~~~
CodeWriter23
Yes, modify the message and submit to the SMTP server, which calculates the
DKIM signature.

~~~
temprature
I don't understand what you mean.

If you're saying the mailing list SMTP server could sign the message with
their own DKIM key, that doesn't work because the DKIM result only gets used
if the signing domain is aligned with the domain in the From header.

~~~
CodeWriter23
Ooops, I probably should have quoted you

> Many mailing lists modify every message to add a footer with unsubscribe
> information.

That will not affect DKIM. The DKIM signing occurs after the listserv creates
the email message and hands it off to the SMTP server.

~~~
temprature
_> That will not affect DKIM._

It will affect DKIM and it does.

 _> The DKIM signing occurs after the listserv creates the email message and
hands it off to the SMTP server._

The DKIM signing occurs before the listserv even receives the message. It's
done by the sender's SMTP server before it gets relayed. The listserv receives
the signed message and adds a footer which breaks the signature.

~~~
CodeWriter23
The listserv has to copy any incoming email message and send one copy of the
inbound message to each recipient on the listserv list. To assure
deliverability, it has to send it from it's own address @ it's own domain, to
avoid SPF failure. Each individual copy is passed by the listserv to the SMTP
server to go to each recipient on the listserv list. The SMTP server
calculates the DKIM on each message as it is sent.

Again, mail servers adding links is completely irrelevant to the DKIM signing
process.

~~~
temprature
_> it has to send it from it's own address @ it's own domain, to avoid SPF
failure._

To avoid SPF failure it uses its own domain in the MAIL FROM envelope address.
There are very few lists that change the From header, I can't name a single
one although they probably do exist.

 _> The SMTP server calculates the DKIM on each message as it is sent._

Like I said, even if they did, that only works if they were also changing the
From header to their own domain. Otherwise, even though there is a valid DKIM
signature there is no alignment with the From header so it is ignored when
evaluating DMARC. DKIM does not use the MAIL FROM address.

 _> Again, mail servers adding links is completely irrelevant to the DKIM
signing process._

I send a mail to a mailing list, LKML is a good example. My SMTP server signs
the message with my DKIM private key and relays it to the LKML SMTP server.
LKML receives it, appends a footer and relays it to all subscribers, with my
name and email address in the From header but using its own address in the
MAIL FROM address. The footer breaks the DKIM signature because when my
message was signed that footer wasn't there. LKML does not re-sign the
outgoing messages, it relays the DKIM signature provided in the original
message. This is standard of many, possibly even most, mailing lists.

------
mileswu
Forwarding email with outlook.com/Office365/Microsoft Exchange also breaks
DMARC. Exchange Server sometimes modifies the headers of emails when
forwarding them, invalidating the DKIM signature, so then a DMARC policy
rejects the forwarded email. Apparently Microsoft have a fix in the pipeline
for this, but it's been taking ages.

More info at: [https://blogs.msdn.microsoft.com/tzink/2016/05/19/why-
does-m...](https://blogs.msdn.microsoft.com/tzink/2016/05/19/why-does-my-
email-from-facebook-that-i-forward-from-my-outlook-com-account-get-rejected/)

------
mpa000
I stopped reading when it referred to SPF as "Sender Protection Framework." As
usual, the discussion on HN is far better than R'ing TFA.

We recently had a collision between the practices of one of our vendors, the
silly way that .edu's tend to forward mail, and Gmail's DMARC policy. The
vendor is doing everything "right" in terms of their own (completely
transactional) mailings on our behalf but some still want to blame them for
poorly forwarded messages sent to .edu addresses that subsequently get
swallowed up by Gmail and are never seen by the authors or referees.

~~~
discreditable
I've seen the same issue with an edu address using office 365. The user set
his mail for our org to forward to his .edu address. The .edu address was set
to forward to his personal Gmail. The Gmail at the end of the chain was
rejecting the mail because the edu didn't rewrite the from address.

------
roketridah
Authenticated Received Chain (ARC) is being developed to work simultaneously
with DMARC and help provide relief to mail agents that break DKIM by allowing
ARC aware receivers to sign authentication results that allow downstream
processors to review what the initial results were and make their decisions.

Details at [http://arc-spec.org/](http://arc-spec.org/)

------
gcp
So what did LKML end up doing?

Edit: given the Ts'o response above I guess they enforce the sender to not
have DMARC.

"There are people with google.com addresses that need to use non-Google
addresses in order to participate on the Linux Kernel Mailing List."

See also: [https://www.ietf.org/mail-
archive/web/dmarc/current/msg03236...](https://www.ietf.org/mail-
archive/web/dmarc/current/msg03236.html)

------
mikegerwitz
> For many years, there was no real way to verify that you really got the
> email the person that the From header states.

DMARC doesn't really assert your identity---it asserts the identity of the
server.

Remember that you can also use PGP to assert _yourself_, and this works well
with mailing lists (your mail client will hopefully distinguish between the
signed message and the mailing list footer, which is unsigned). It also
persists---if a mailing list is stripping DMARC headers, then that doesn't
help you any.

~~~
jedbrown
PGP signatures can't be used to prevent delivery of scam/spam email.

~~~
mikegerwitz
Yes, you're right.

I modified my message to make clear what I was commenting on.

------
shutton
I work on Gaggle Mail which is a group mailing list provider that avoids all
this by only using a From address that we control. I do think stricter
addressing policies will become the norm in years to come which is why we took
this approach.

More details here: [https://gaggle.email/how-emails-are-addressed-and-
sent](https://gaggle.email/how-emails-are-addressed-and-sent)

------
proaralyst
As someone who thought it'd be a good idea to run their own mail server, I
found that without SPF/DKIM/DMARC your mail gets identified as spam a lot.
Having DMARC just means you get told about it.

~~~
hannob
That is nonsense. I run a mail server. I have SPF, but no DKIM/DMARC. I have
no problems with mail deliverability.

In my experience there are a lot of urban legends around this topic.

~~~
proaralyst
I have had (and continue to have) perfectly innocuous emails land in the spam
folders of others with gmail.

~~~
icebraining
So have I - including when sending from another Gmail account. Their spam
filters can be fickle.

------
peterwaller
SPF has another problem. I sent an email to someone recently @theirdomain.com.
I subsequently saw by chance that the email was rejected because they were
hosting @theirdomain.com with a random ISP but they had configured the mail to
be forwarded to a mailbox in @gmail.com.

Gmail sees the email coming from @theirdomain.com's servers, rather than my
server. Gmail checks the SPF record which doesn't match, and it rejects it.

I understand that this style of forwarding is anyway bad because gmail see's
all email the user receives @theirdomain.com as coming from those servers, not
their true origin. If @theirdomain.com receives (and forwards) any spam, it
looks like a spammer to gmail.

~~~
mwpmaybe
The solution to this is SRS[0], but yes, not every mail-forwarder implements
this (correctly), and it seems to break DKIM. The good news is that an SPF
soft-fail and a DKIM pass usually means a DMARC pass. More information
here[1].

0\. [http://www.openspf.org/SRS](http://www.openspf.org/SRS)

1\. [https://blog.fastmail.com/2016/12/24/spf-dkim-
dmarc/](https://blog.fastmail.com/2016/12/24/spf-dkim-dmarc/)

~~~
jlgaddis
Even with SRS, the "reputation" of the "middleman" (from Google's perspective)
still suffers.

------
__david__
If the mailing list changes the From address, then the message won't show the
correct author in the message list in my email client. Replying may work
(because of Reply-To:), but not having the correct "From" in the client is
really a deal breaker.

Besides, DKIM already solved this problem by not caring _where_ the mail comes
from, just that the message was signed. It seems the solution to me is to stop
using SPF if you want to use mailing lists.

~~~
garaetjjte
DMARC policy rejects messages only when they failed both SPF and DKIM checks.
It seems problem is that mailing lists like to mangle message content.

~~~
Nazzareno
Not exactly, the DMARC policy is very flexible. Each server, after a period of
monitoring when he receives a detailed report of what would happen if a policy
is enforced, can define his own policies.

------
joecot
Ran into this problem setting up a new mailing list system. I'd always
wondered by mailing list mail sent from yahoo went directly to spam, while
mail from gmail worked just fine.

Mailing list mail will always fail DMARC if you keep the original sender's
address, but when setting up DMARC for a domain, you specify what should
happen to mail from your domain if it fails the DMARC check.

For gmail and other sane providers, their DMARC failure is set to ignore,
which means DMARC failing for those addresses gets calculated into spam
probability, but isn't an auto-fail. For yahoo and AOL, they have it set to
reject, which means mailing list mail from those domains automatically goes to
spam for gmail.

Groupserver mailing list software resolves this by checking the DMARC settings
for the sender. If DMARC is set to ignore, the sender address is kept, because
it won't actually affect delivery. If it's set to reject, it mangles the
sender address so that the mail at least gets delivered.

------
INTPenis
I've had some contact with a big client e-mail filter, (30k+ users) and we
chose to expose a soft SPF in the external DNS and a hard SPF in an internal
DNS used as a cache by the mailfilter.

This was a workaround because the big org had countless 3rd party suppliers of
services who many times wanted to mail as @big.corp. And this was fine when
mailing to big corp as we could whitelist their senders in the SPF service
config, but when mailing to the rest of the internet like gmail, yahoo and
hotmail we needed a soft SPF fail instead of having to add those countless
servers in our DNS record.

------
jedbrown
Unfortunately, having the list software create a new message breaks threading.
It is common practice to Cc people (subscribed to the list or not) that may
need to interact with the thread, but they get the message directly from the
sender while list recipients get a different message (different From,
different Message-ID). Similarly, if the original sender replies to their own
outgoing message, it will not thread for list recipients.

~~~
shutton
Threading can be preserved when creating a new message as long as the mailing
software re-uses the same: 'message-id', 'references', 'in-reply-to' and
'thread-index' headers.

------
tytso
The primary problem is that DMARC is fundamentally flawed, and was not enacted
using a standards process that respected all of the stakeholders. As a result,
it fundamentally becomes a matter of power politics.

If there are a bunch of people who need to participate in a particular mailing
list --- say, IETF mailing list or the Linux Kernel development lists --- more
than they need to stick with a particular mail provider, it becomes possible
to say to them, "you want to participate in our community"? Change mail
providers.

In the cases where a mailing list community badly needs the Yahoo users, Yahoo
can dictate to the mailing list --- change your mailing list software and
inflict pain all on your mailing list users, or you don't get access to our
e-mail user community. (And rewriting the from field has all sorts of bad
effects, including corrupting contact databases that auto populate based on
names and addresses in the from field, making the mail summary index useless,
breaking reply-to sender, etc. So there is real pain involved here.)

Part of the issue was that DMARC was originally intended for domains that only
sent official announcements (e.g. for credit card companies or banks) and
where employee e-mails came from a different domain. For that original use
case, DMARC worked perfectly well. Apparently Yahoo then decided it had a
terrible SPAM problem, and decided DMARC was a blunt instrument to use, and
inflicted on consumer e-mails. Other companies didn't think ahead, and used
the same domain for official e-mails as their employee e-mails, and decided
that protecting their users was more important than inconveniencing their
employees.

This is why it's ultimately all about power politics. If you want to
participate in Linux Kernel development, you need an e-mail address that won't
arbitrarily cause your e-mails to be dropped (for example, so Linus Torvalds
doesn't get your pull requests). Despite this, growth in Linux Kernel
Development continues to be growing, which means that when choosing between
the inconvenience of changing or using an alternate e-mail provider, versus
participating in Linux development, developers choose the former. But that
only work because Linux has the economic power and clout to get away with it.

Heck, over ten years ago, refusing to buckle in to crap e-mail systems forced
IBM to allow its Linux Technology Center folks install a standards-complaint
IMAP server instead of using that abomination caused Lotus Notes which the
rest of the company was forced to use. (And this was probably a better
demonstration of the power of Linux, given how much IBM was stubbornly
attached to Blotus Goats.) But make no mistake. This is Trump style, power
politics. It doesn't hurt those with a lot of economic power, but if you're
some tiny, podunk church mailing list, or some other group lacking in economic
power, you're screwed.

DMARC: making email "great" again.

------
dekhn
Once, I needed to email something to djb. I tried, and his email system sent
back a challenge which was dumped into my spam box. I never did manage to get
an email to him.

------
thresh
DMARC sucks.

Source: I run two big mailman installations.

~~~
throwanem
Mailman sucks, too.

Source: I administered a variety of Mailman-driven lists for years. Never
again.

