B1 is displaying a verified relay that doesnt match the sender domain.
The problem seems to be that the author doesnt want to display "via way.hanami.com", in which case he should set up DKIM for the domain he actually wants (the redacted domain) instead of "way.hanami.com".
Really shouldn't be advertising yourself as a "mail forwarding service" if you dont understand that...
Gmail's UI has a lot of quirks (can we talk about how it's a productivity app with a fantastic search, that doesn't let you save searches?) but it's correctly calling out people who don't do DKIM correctly. OP's article doesn't even talk about anything other than DKIM; intended or not, the title is pure clickbait.
You can actually create filters in Gmail which would be like a saved search. Once you've searched for something, click on the arrow on the right, and then "Create filter".
Of course, and you can create labels that track that filter's application in the same way a materialized view would work. But we don't CREATE MATERIALIZED VIEW, or worse yet UPDATE a many-to-many table, every time we want to run an exploratory analysis on our SQL databases and save our ability to rerun that analysis for later, right? Because that would create a ton of bloat, and we might lose track of the labels that we actually want to be permanently meaningful, and if a label is ever added manually, there's no guarantee that the label tracks the initial intent of the search query.
And that doesn't even get into the fact that any such labeling is visible as a tag on every matched conversation, leading to a need to have a concise name for your analysis to avoid information overload. Naming things is hard, and when designing interfaces, reducing cognitive load isn't just a nice-to-have.
My biggest gripe is that you can't sort by sender. Sure, you can filter but sorting by sender and visually pattern matching is the easiest way to cull and unsub from repetitive promotional emails that might otherwise not stick out in a cluttered inbox. It's absolutely insane to me that in 2021 gmail doesn't have this feature that every other email client has had forever.
I understand DKIM and i have written this part in the article
> Somehow when we signed a message with DKIM, google decided to show the “Via way.hanami.run”, which is technically correct because the message lack of DKIM signature for original email(I made a typo and type gmail in original article) address
As in, I agree with gmail decision. My point is that, why when a message has nothing, gmail didn't say "warning: no DKIM signing".
I just hope gmail makes that fact clear on their UI when a message has no DKIM signing.
And sometime, to reduce minimal work for users, DKIM is optional for them. If they add our public key to `hanami._domain` we will sign it. But it's really tricky to configure DKIM for a normal user. Even gmail makes it optionally. In that case, gmail signed it with gappssmtp.com and yet, they don't say "Via gappssmtp.com"...
> My point is that, why when a message has nothing, gmail didn't say "warning: no DKIM signing".
Because 99.9% of users would say "What is a DKIM?"
Google has decided that regular uses don't need to know if DKIM and SPF passed or not, because they don't know what that is anyway. They use those things in their spam filtering. If a message got past their filters, regular users assume they are safe, and even if it had DKIM status on it, that wouldn't change.
Power users can bring up the second screen you showed if they want to be sure.
If Chrome operated that way, we would still have broken locks on mixed-content SSL pages and no warning at all on non-SSL pages. The fact is, an attempt at security is better than no attempt at all, and visibly marking the insecure option as less secure is better than pretending security doesn’t exist unless it’s implemented 100% as expected (lock icon appears, etc.)
The world has far better uptake of HTTPS than of DKIM. If DKIM was as widely used as HTTPS, then Google likely would indicate when a message was not signed.
> 86.8% of the emails we received are signed according to the (DKIM) standard (up from 76.9% in 2013). Over two million domains (weekly active) have adopted this standard (up from 0.5 millions 2013).
> 95.3% of incoming emails we receive come from SMTP servers that are authenticated using the SPF standard (up from 89.1% in 2013). Over 7.8 million domains (weekly active) have adopted the SPF standard (up from 3.5 million domains in 2013).
> 85% of incoming emails we receive are protected by both the DKIM and SPF standards (up from 74.7% in 2013).
> Over 162,000 domains have deployed domain-wide policies that allow us to reject hundreds of millions of unauthenticated emails every week via the DMARC standard (up from 80,000 in 2013).
In conclusion, the evidence -- from Google itself -- shows that only as of 2017-18 has HTTPS adoption even remotely compared to that of DMARC adoption's statistics from 2013!
Gmail is terribly behind Chrome in this regard. Google Search began down-ranking sites not SSL protected in 2015, Chrome warned about submitting passwords on insecure forms in 2016, and by 2017 was rolling out or planning to roll out a series of changes to visibly mark insecure pages as insecure, one address bar change at a time.
In that same time, Google launched Inbox in 2015 ... and shuttered it by 2018, redesigning Gmail. They had plenty of opportunities to highlight insecure email transmission, they chose not to.
I agree with you in theory, but in practice they are very different situations. For websites, the website owner loses traffic if they don't offer a secure experience -- they are in complete control of their destiny.
For email, the user loses deliverability and may not even be aware of it. Also it's often not within their control to fix. Marking non-signed emails as such would put a huge burden on email users and they might not even be aware of it.
Th lock means "this is secure", but we're not talking about that with DKIM/SPF. We're just talking about if the sender can be verified. It doesn't really mean "secure" or not. It just means "we're pretty sure the sender isn't lying".
Also, for websites, the website owner loses traffic if they don't offer a secure experience -- they are in complete control of their destiny.
For email, the user loses deliverability and may not even be aware of it. Also it's often not within their control to fix. Marking non-signed emails as such would put a huge burden on email users and they might not even be aware of it.
For anyone looking for a forwarder that has done their homework, I can recommend forwardememail.net [0]. The developer has commented on the issues here on HN a few times.
> while the message that failed both SPF and DKIM looks correct on their UI.
While the behavior can be annoying, the first picture in the article conveniently cuts of the profile image near the sender details. I believe when both fail gmail adds a question mark icon of a red unlock. So no, it doesn't look more correct.
If you add your own DKIM then you have to mangle the from address to make it appear that the email is from your email domain. Which usually means some sort of "via" usage. So that would not fix the problem.
This is a generic problem with DKIM, it breaks email forwarding.
DKIM doesnt break anything because it applies to the original message. You simply forward the signature header (well, you forward everything - thats what makes it forwarding!).
As I mentioned elsewhere, its not clear what the authors intent is - is it an originating server, or is it a relay. In both cases he shouldnt be signing with the relays key, but should either be signing with the domains key (if available) or forwarding the existing signature. Signing with the relays key is all but useless.
>DKIM doesnt break anything because it applies to the message body.
DKIM applies to what you set it to apply to in the DKIM entry. Here is what Gmail signs from the header (the body is signed by default) for their DKIM (taken from a recent Gmail originated message):
Why would you change the from address? The whole point here is to forward, not re-mail. Everything should be forwarded.
I kinda agree with what you said, but the point is that you shouldnt be mangling the from address when forwarding. That will definitely break DKIM, as its actually a required header fields in the sig - but theres NO reason to do that.
p.s. Ive clarified that i meant "original message" not "message body" in my post.
p.p.s Actually, I do think we are in agreement. I mentioned elsewhere that we dont know what the intent was - relay or origin, but in no case should they sign with the relays domain key.
That's a From inside the message. MAIL FROM in the SMTP protocol (envelope from, which you change during forwarding) can be anything else, and DKIM will still validate the message.
It is the opposite. Forwarding (can sometimes) break DKIM. If you are reliant on DKIM (such as blocking messages with DMARC) then you need to account for the impact of forwarding. This has been a big problem with mailing lists, since they get caught in the middle. Mail might get blocked because of the signature failed and the mailing list gets blamed.
I think it's because the author thinks "via" text is a negative thing?
They run an email forwarding service, so their SPF records will fail, but an untampered email should still pass DKIM. in fact, that's how most email forwarding setups work. If the message is tampared, and re-signed with the forwarding service's DKIM key, it's correct for Gmail/email-client to show the via-text.
I can't help pointing out that the root cause here is that the email server is signing various things in the email that the email server did not originate. There would be no problems with forwarding if the originating user signed the email instead. The identity of the email originator is what the email recipient is actually interested in. The UI could then show if you actually know the entity sending the email rather than if your email server knows the originating email server.
I've created two email startups (https://ohmysmtp.com and https://idbloc.co) - email is hard, thanks in no small part to these issues which are basically undocumented or at least not specified.
To avoid this, the Hanami team could add a reply-to header that routes back to the original sender, update the from header to come from them (to pass DKIM and SPF), and instead of forwarding on directly, re-build the email and resend it as a fresh, DKIM signed-one. This is how we do it at IdBloc (we add the via bit):
from: Original Sender via IdBloc.co <sender-at-domain.com@users.idbloc.co>
reply-to: "Original Sender" <sender@domain.com>
If they want to act as a complete firewall in between, they can change the reply-to address as well and completely rebuild the email each time.
These "issues" are EXTREMELY well documented AND specified.
To be fair, we don't actually know the authors intent (and I would argue that they dont either).
They should be either forwarding the signature (in the case of relaying) or signing with the senders domain key. They are instead signing with the relay domain key, which is all but useless.
@supermatt you really get this stuff. Is there an OReilly (or other) book you'd recommend, or is all this knowledge part of the vast, distributed online corpus of IETF RFCs and StackOverflow questions?
Agreed, any links (or references to books) about the subject matter would be helpful!
At this point i feel like a lot of e-mail related topics are full of "unknown unknowns" to me and many other developers. It's hard to say whether we're missing bits of information that are considered common knowledge or maybe e-mail related development doesn't have enough discoverability (like blog posts, tutorials and so on) to make it easier, like configuring HTTPS in Apache2/httpd would have.
Or maybe the whole domain is just full of inherent complexity due to its history (SPF, DKIM, DMARC etc.).
> SPF
Is the relay authorised for this domain
> DKIM
Which domain(s) authored/approved this email
> DMARC
Formalises some behaviours for the above + reporting
Much of this is to prevent a bad actor sending out emails from a domain they do not control (i.e. prevent spoofing).
The REAL problems with email come with deliverability. Basically an MTA not forwarding or accepting mail from you. This can be for a variety of reasons: you are misconfigured, they are misconfigured, one of your relays is on a blocklist, your relay exceeds a rate limit, your emails exceed some spam score, etc. This is the stuff that makes service management painful.
For example, gmail may give you a few "spam points" for sending them more email than they are used to from your relay - this means your emails end up in spam, get bounced and requeued for delivery (if you are configured correctly!), outright refused, or even mysteriously blackholed. It takes time to build reputation in that respect. Emails may be marked as spam by recipients, which can further tarnish your relays reputation with that provider (potentially multiple if they share reports with other providers).
Using an IP with a history of abuse, or having bad neighbours on your subnet, or having emails flagged with a spam reporting service can result in you being added to blocklists - requiring you to be vigilant in log monitoring, postmaster duties, etc.
Email isnt difficult, its just that bad actors have meant that you need to establish a positive reputation, rather than simply avoiding a bad one if you want to send any realistic volume of email to a large provider. DMARC/SPF/DKIM are intended to give accoutability in order to establish domain-based, rather than relay-based reputation - but ultimately its up to the receiving provider (and occasionally 3rd party relays, depending on your setup) how they handle it.
Other RFCs are either obsolete, or talk about some crypto specifics.
All you really need to know is that DKIM signs the message to prevent tampering. This includes the content and "from" header (optionally other stuff). Anyone can sign it, but its up to the verifier to decide which of the signatories to "trust". For example, you dont NEED to sign with the senders domain key, which is why google is flagging this 3rd party signature with some custom UI.
DMARC formalises this with identifer alignment, to ensure that a signatories domain key matches the "from" address domain.
DMARC RFC says something along those lines (Identifier Alignment).
With DKIM anyone can sign the email, its up to the verifier to decide who to trust and how to represent that. DMARC formalises who you should trust (i.e. the domain key for the sender).
p.s. It seems you can click on the "via" in gmail and it gives you some (rubbish and borderline misleading) information.
Thank you. I'm thinking about it as well but many other email forwarding service forward it as it's. so when I do different, they kind of confused. Plus, many mail client show the from as well. So it confused user quite a bit.
Running an email forwarding service has to be a pretty thankless task, so hats off to the author if they're doing it right, even if the UI isn't quite on their side.
I send a lot of (wanted) mail and some subscribers have various forwarders set up which result in the eventual recipient server rejecting the mail due to the forwarded mail failing SPF and DMARC. It's a nightmare as we get the bounce backs telling us off but it's because the forwarders aren't, of course, sending from IPs included in our SPF record. (There are clearly ways to do it properly, but a lot of mail forwarding systems appear not to?)
The overall UX of Gmail as of 2021 is a mixed bag, and this is somewhat generously speaking. I mostly rely on search to find emails, as tags get hidden, the "Important" mailbox overwhelms messages the AI didn't deem "important" and the spam filter has never been worse. I was expecting a review of... the quirks of the Gmail UI.
My favorite Gmail quirk is how your account gets locked out for a few hours if you bulk delete thousands of emails, and how trying to archive every message from a very large inbox fails silently.
It'd be nice if Gmail could get with the times and add proper custom domain support to the consumer grade service. These email forwarding services aren't really ideal, and people shouldn't need to use them. They're never really going to have the tier of security as, say, Gmail.
More generally, the problem of email address exhaustion on consumer services is getting worse. Imagine being a kid making your first email account on Gmail or Outlook.com that doesn't look totally unprofessional. I got lucky to get full name @ gmail. (But, it's to Google's credit that they don't do the inane Yahoo approach of recycling addresses.)
Not defending gmail, but for them offering a gmail.com address is working out for many. Outside HN and tech, I doubt people care if their email suffix is `@gmail.com`.
Google Apps (or whatever they call it these days) is very affordable and gets many aspects of emails right. There are other well established alternatives too, like protonmail
> if Gmail could get with the times and add proper custom domain support to the consumer grade service
Yes please. Especially since Microsoft offers it to a limited extent on Outlook.com for Microsoft 365 Home subscribers ('limited' in the sense, the domain must be registered with GoDaddy).
In case any Google employees are reading this: this would be a very useful feature for one.google.com.
> More generally, the problem of email address exhaustion on consumer services is getting worse.
And there's another pernicious problem: if you have a firstname@gmail.com or first.lastname@gmail.com, you're likely getting a ton of misdirected email even if your name isn't that common.
The most infuriating thing about gmail is how icons with destructive actions appear under my mouse cursor when I move my mouse over it and I'm just trying to click on the email to drill down into the conversation from the inbox. Every, f*cking time i'll just lazy throw my cursor over and just happen to end up in the zone where Archive or Send To Trash is and click it without a chance to react. I despise gmail with a burning passion. Google UX design is a complete joke overall.
And don't forget the image attachment that Google ~~insists~~ forcibly put the image inline the message instead of as attachment. Why would they make it complicated and made a weird drag-n-drop rules for image only. It would be nice if Gmail team simply add the option to disable adding the image inline the message. Since it is Google, more likely Gmail will sunset their service before adding that option.
I have a similar keyboard problem. With relative frequency the focus is stolen from a text input box to the top level. Usually this is just an annoyance, but sometimes it happens right as I'm typing an "m" (which results in me accidentally muting a conversation) or an "e" (which results in me accidentally archiving a conversation).
I've noticed this too. It feels a lot like hitting the touchpad by mistake, except I don't use a touchpad. The focus is just lost mid-typing and a bunch of shortcuts get triggered.
I recently had to just disable the Gmail spam filter using a custom rule [1] because the false positive rate was awful – 90% of the stuff in my spam folder was legitimate, including Google Calendar invites and normal GitHub communication.
Unfortunately I'm starting to get the feeling that Gmail is no longer a high priority for Google, and so it's slowly bitrotting like so many other Google projects.
[1] Matches: {(to:me) (deliveredto:myaddress@gmail.com)}
Do this: Never send it to Spam
Google Docs[0] and Calendar invites[1] themselves are often spam so i'm at least glad they don't outright whitelist domains. What is spam to some might not be spam to others.
I didn't realize how many false positives Gmail had with spam for the longest time. An astonishing amount of legit messages get wrongly flagged for my account. Given that the UI hides the spam folder, I would bet that most people have no idea how many messages they miss from this.
Hi, I'm the OP of this article. I'm not intending to click bait this article and my mistake for not compeltely explanation DKIM.
My point is that as an average user, when seeing this, what is their expectation. This is what happen when a user asked me about that today.
I understand DKIM. However, when one send a plain email without any DKIM, why google say nothing on their UI. If they say: "This email has no DKIM", it would be better. Now, the one who signed that email(original email didn't) get this "Via way.hanami.run" and an normal user will confuse when seeing it.
BTW, We do support DKIM signing in our dashboard, it will ask user to setup MX, SPF, and DKIM. However, many mail services will let DKIM an optional thing. Even google/zoho.
You can add domain and send email just fine without configuring DKIM for your domain. Google will sign your message with gappssmtp.com domain and won't say anything about "Via"
As a typical end user you probably won't understand what that "via" means. All you see is that your e-mail was sent "via" some unknown thing you don't recognize.
Even if you do understand - isn't the "via" specifically designed to reduce trust? "Hey, head's up, this message actually came in through this third-party"
Noob question from someone with little knowledge of email: Doesn't Gmail just move emails which fail DMARC to spam outright? Even though A1 in the op does look more trustworthy, I would have thought Gmail would just disregard it, since a DMARC fail is a big red flag? In that case, I would think B1 looks a lot more trustworthy, since it's not marked as spam. Or is the picture a bit more complex than that?
No, Dmarc is not a red flag if it’s not setup, most email providers don’t require dmarc to be setup. Many, many emails are sent that do not pass dmarc.
The reason is that if you maintain dmarc records you have to handle spam complaints and bounce reports. So if you’re using a provider like mailgun or ohmysmtp, you still have to set up a receiving smtp server even if you’re only sending outbound mail.
It does _fail_ DMARC.. but the DMARC record for getopty.com tells it to do nothing if it fails.
It's policy is set to "none". You normally do this when you are starting to investigate your DMARC setup and want the stats. This way you can fix things before you up the policy through quarantine and then to reject.
Folks discussing this are knowledgeable about the SMTP protocol. So let me ask something I have always wondered about -- what was the idea behind HELO, what purpose does it serve in the protocol.
Thanks to all who take out their time to answer this 'ought to be obvious' question.
SMTP is an interesting protocol because the server opens the session with the banner, unlike HTTP where the client speaks first. I think a better question is why does the banner exist, and what purpose does it serve?
HELO/EHLO gives us a way to negotiate ESMTP, which is what practically everyone uses today.
Indeed but what's the purpose of that. The SMTP server knows the IP address anyway and can DNS lookup if needed. The one chatting with server can say hello followed by any arbitrary string
Reverse DNS was added to SMTP servers as an anti-spam feature about 15 years after HELO was established.
But to be honest, I'm not sure HELO was ever really useful. In the very early years, it might have been appropriate to reject messages where the MAIL-FROM domain did not match the HELO domain. Not for long though.
HELO is not really used that much anymore. It's EHLO now. And response to it includes capabilities of the server. So it serves negotiation of those at the beginning of the session, too.
It's better to be prompted for those, instead of just sending them unannounced, in case client just want to switch to TLS first via starttls.
Yes - the first email should clearly say that DKIM/SPF is not correct. In other words, if I mistakenly un-spam an email which has incorrect DKIM/SPF, then the gmail interface should still say that headers are not ok.
This is kind of like how http sites look totally fine in most browsers, but an https site with a self-signed certificate will cause a "DANGER! ENTER AT YOUR OWN PERIL" screen to be shown.
The most annoying thing about the current Gmail UI is that I have to have the tab open for another minute after deleting or achiving mail, otherwise it gets restored automatically because of the Undo countdown feature. Insane!
B1 is displaying a verified relay that doesnt match the sender domain.
The problem seems to be that the author doesnt want to display "via way.hanami.com", in which case he should set up DKIM for the domain he actually wants (the redacted domain) instead of "way.hanami.com".
Really shouldn't be advertising yourself as a "mail forwarding service" if you dont understand that...