We'd contact them, they would fix things, and it would work for a while...but invariably it would go back to classifying it all as spam, and we'd get customers calling to complain about not getting instructions or support responses.
Finally, our guy who managed our ad spending called up our rep in ad sales at the email company, and asked in rather forceful terms just why we should continue spending $large_number/month on ads there to acquire new customers, when we were then going to be blocked from emailing those customers?
The ad rep conference called in their head of IT, who conference called in their engineer in charge of spam filtering, and had him right there add us to a whitelist that would prevent anything from us being classified as spam no matter what the rest of their spam system decided. As far as I know we never had another email go into the spam filter.
If we hadn't been spending a large amount on ads with them, I doubt the problem would ever have been resolved.
If you sell a software product to end users, and have non-marketing messages that you need users to see such as responses to support requests or reminders that subscriptions are going to automatically re-bill, I'd recommend including some kind of messaging system specifically for this in the product itself. Still send them as email, but also send them via this in-product channel, or at least send notices that there is a message with links to a way to view it on the web. Email only is simply not reliable enough.
That is because implementations, especially gmail, ignore the specifications: (EDIT) there are specific error codes and messages should be sent back to the originator. The reasons to ignore it is, in theory, not to give spammers ideas how to avoid it.
Email is incredibly reliable, it can tolerate outages, configuration errors, delays, etc, by specification, but ever since "modern" spam filtering, gmail (and those who follow their practices) it became a nightmare, because everyone ignores how it _should_ work. It was done in a time when connections were slow and prone to errors; it had to be flexible.
Yes, I know spam was bad I was there. But I'd rather have spam, than actually important mails getting dropped because of false positive, paranoid spamfilters.
It doesn't take very long to manually delete spam from the inbox. And if your spam filter is putting legitimate emails in your spam folder, you need to manually filter them anyway.
I haven’t checked my spam folder on a routine basis in many years.
Now spam means "anything the user doesn't want to receive". Even if the user previously deliberately created a relationship with the sender. Gmail successfully trained users into clicking the spam button for anything such. People just click Gmail's magic make-it-go-away button for things as legitimate as a mailing list they deliberately signed up for and are simply too lazy to properly unsubscribe.
Legitimacy isn't a binary either-or, it's a continuum. You bought from a merchant ten years ago and they email you again, is that spam? How about the company that acquired that merchant which is now trying to sell you a completely different line of business? You donated to a politician and now their successor in the same party emails you, is that spam? That's what modern spam filtering deals with.
Spam means anything from someone who is indifferent to whether the recipient wants to receive it and doesn't get "sufficient" permission. I think this is consistent with the original type of spam, even if there is no bright line delineating it.
If you're implying that anything with a working unsubscribe link shouldn't be considered spam, then I think this definitional issue relates to the same controversies about what constitutes consent that have been discussed a lot recently.
I mean, I agree with your last paragraph, so far as that goes, but I would suggest rather than the definition of spam changing, ordinary businesses got spammier/sleazier.
It wouldn't be efficient today with spam being more more diverse and often hardly distinguishable from legitimate emails. A lot of spam nowadays is like forced advertising, by opposition to plain scam and dodgy stuff.
The only things I can think of that I do differently than many people that might make a difference is that I have my mail client set up to not show remote images ever and I never dirctly click any link from email no matter the source. Possibly things might be much worse otherwise, but from my perspective it doesn't seem like spam is a huge issue these days with basic filtering.
An alternative narrative is that previously email servers were supposed to accept almost anything from everyone and spammers took advantage of this. From this perspective it is the relatively recent requirement of sender verification that made the key difference and allows Spamhaus to work (obviously they need to do something to determine who to block). This perspective seems to match what I am seeing.
I have tuned the spam filters for a fairly major email provider for a while and while you are right for a lot of people, there's a significant fraction of them that receives hundreds or even thousands of spam emails per day. For them, it is not practical - and everyone needs to be able to use it.
So while gmail has made life harder for people who want to email me, it have made life much easier for me as an email user - which makes me more willing to give out my email.
They all go through some provider who can block you.
It’s a rare product that people check on their own without notifications!
Any ideas how to reach people without gatekeepers? I have just one way: a desktop app or mobile app that is allowed to periodically wake up and poll a server.
If you're asking "anybody today" then yes, that's probably one of the only ways.
But maybe tomorrow might be better.
On the horizon (where it may stay - it's been slow in coming) is the idea of "distributed and serverless" internet. Also known as "peer-to-peer".
You can think of it as everyone (well, their device) being their own server - kinda like the internet was originally meant to be (before vast sums of money got involved).
Through "magic" - each person that is on the network can browse, send, receive, post, etc - to others without needing much of anything else - just the pipe (not really any way around that kind of "gatekeeper" unless you have a mesh network of some sort - and even there, long-haul can be tough or impossible). A well known example:
...there are others out there as well; just google around for "peer-to-peer web" and similar terms.
I think that is super important given what’s going on:
We have started to work on captive portals so your device can surf from wifi to wifi just like a cellphone does with towers, and check in wherever you visit without you doing anything (with your prior permission, of course)
IPFS is great, as is BitTorrent. I have met a lot of the people running these projects, like Tom Berners-Lee and the guys from Solid (back then), David Irvine and the guys from MaidSAFE, and Petar Maymounkov from NYU who invented Kademlia DHT.
I think there are several things that are needed:
NAT hole punching
End to end encryption
Kademlia or other DHT
Small group consensus
Merkle DAG with validators
For me it’s not even so much about privacy as being permissionless. No gatekeepers.
Whether it’s your cellphone company or your cable internet ISP or the Google Maps API or Amazon Web Services, they require you to pay them for their closed-source infrastructure and data.
The infrastructure should be a mesh and the data should be open source. The challenge is simply in the software to coordinate the mesh participants, and for the data it’s software to enable the right rules to maximize the chances of data being correct.
There is tons, tons to do and frankly most VCs and investors don’t get it. The payoff is huge for humanity as a whole, and can unleash even more innovation than the Web did after it disrupted AOL etc. But they don’t capture even 10% of the whole value. It’s free to the world.
So we’re chronically underfunded but we have already put together so much. And now we are making money the old fashioned way: earning it from users and clients hahaha. The Basecamp guys would be proud.
If it is your first time, you will be asked to create an identity on Blockstack: https://docs.blockstack.org/core/naming/introduction.html
Each instance of the "browser" whether on desktop or phone, requires a unique password, associated with the id. The id is stored on the Blockchain, the data is stored in your own gaia hub which you can host wherever you like: https://docs.blockstack.org/storage/overview.html
In regards to your comment about "Kademlia or other DHT": Blockstack previously used Kademlia, but iterated on an improvement to Kademlia by creating Atlas which is what we use at Blockstack. More here: https://github.com/blockstack/atlas
and the specific reasons about why Atlas was built in lieu of Kademlia here: https://blog.blockstack.org/blockstack-core-v0-14-release/
Together, Atlas, BNS, and gaia create the foundation upon which the Blockstack API is built on.
You can build applications using the Blockstack API here: https://docs.blockstack.org/develop/zero_to_dapp_1.html
If you want to explore the ecosystem of existing decentralised applications on Blockstack and other decentralised solutions, you can see that here: https://app.co/
You can read more about how identity, Atlas, and user owned storage works here: https://blockstack.org/whitepaper.pdf and maybe poke around here a bit: https://core.blockstack.org/
Oh yes, also you can download the browser on your desktop if you like.
Can you drop me a line at username greg with the domain qbix.com ?
Tox, a P2P Skype alternative.
Unfortunately it is not well known outside of tinfoil hat community.
Here the AppStore is the gatekeeper. OP is probably right, your website is the most reliable place to communicate with your customers.
The company I work for was sending confirmation emails doing everything properly: separate IP range mx boxes, warmed up, DMARC, SPF, DKIM, all the bells and whistles. Nothing else was allowed to go through them.
Suddenly people weren't getting those confirmation letters. It turned out that soon after Gmail introduced the "Promotions" tab, a silent, new feature was added: anything from noreply@ was put in there.
We tried going through the official channels - once we got to the end of the tunnel, it told us "we'll get back to you in 2 weeks". That's when we started digging into internal google connections across the company, and thank god, we found someone who know someone, and in was sorted within a couple of hours. (It might have helped that the company spends a lot of money on google ads.)
Gmail is lovely.
There are valid scenarios for one way communication; those emails are only verifications of transaction that can't be simply altered via responding to a mail.
Email has the notion of Reply-to header, although this case, that was not populated.
Making a decision solely noreply@ is why the life of Mr. Null is quite hard.[^1]
No, gmail is not right about this.
I understand the frustration of seeing our simple and low effort mechanisms decay over the past 30 years. But I also appreciate that there might just be a need for a more symmetric social contract for engaging in the use of communications systems and other common infrastructure.
But, the task recipients face for sifting and sorting their inbox into different buckets is essentially the same faced by a notification sender who accepts feedback instead of hiding behind a no-reply address. Lacking real message authentication and authorization tools, we have to look at a mixture of content and metadata patterns to try to sort it out.
In this every gray world, I can imagine reciprocity being a useful measure...
The reason I ask is because I'm pretty sure lot's of sites (invoices/receipts/newsletters) send out stuff via a noreply intentionally.
This is my frustration with Google and their suddenly deciding this is how things should be done. For instance I use my camera to capture receipts, QR codes I may want later and a dozen other things. Google is of the opinion this wastes space so I get regular "Clear up the clutter!" cards. One man's clutter is another man's system of record for important stuff. Stop imposing your opinion on how I should use your product, Google.
Essentially Gmail (and other mail services) put messages from the same sender with the same subject into a Conversation View. Users are confused by it, and clicking the oldest password reset link (expired) rather than the newest.
Claims that users being unable to reset their passwords costs Expedia $187M.
Well, it turns out that Google randomly groups some of these messages in a single thread. And yet this thread looks identical to a single message. So I click on the thread, read the first message, see the email footer and go back to Inbox to read the next one. This marks the entire thread read even though I didn't actually read the subsequent messages. So I never come around to read them.
I completely believe that most users will not read any email in the thread beyond the first one, because there's absolutely no indication there's more stuff to read.
There's still a lot of improvements for mail UX. I don't expect google will be the one to make them, but maybe they'll copy the other people who do.
IMO, gmail is in the wrong here. If the message id is not in the References or the In-Reply-To header, it's a new conversation.
EDIT: Hmmm. It appears that I CAN change that.
Recurring messages having the same title is an absurdly common pattern. Almost every service that has ever sent me a password reset email has used the same exact subject. Same goes for things like shipment notifications, order confirmations, purchase confirmations from PayPal, etc. So given this, it's kind of absurd that a MUA would be designed in a way that buries all those emails, isn't it?
Let's assume that it really is intentional that gmail buries all those messages, on some presumption that they aren't important or on a 'well fuck you, change it then' basis. There definitely are services out there that append random numbers to the end of automated emails. I always assumed this was to make it easy to sort threads by unique id, i.e. customer service systems - so maybe they've been applying an undocumented Gmail Best Practice this whole time. Assuming that this is a good feature implemented correctly, why does unread/read status not work right? If I click into a 50 email long unread thread and then click back why is it INSTANTLY marked read? How is that a useful behavior that would seem intended to anyone? Non-threaded views like in Outlook do not work this way. The common 'reply up top, history at bottom' email formatting also avoids this problem, which you'd think the gmail frontend designers would be aware of.
I would argue that both behaviors are either a bug or user-hostile design. Calling it a bug is generous to the designers because it assumes goodwill and just views it as an oversight or error in a very complex system. I'd personally be inclined to call it bad design, because Gmail is full of bad design, but there's nothing weird or bad about a user calling it a bug!
A $187M figure is super realistic to me given how often I see this particular problem affect me. It literally happens daily. Naturally, I learned years ago that gmail does this and got used to having to dig through my email history to find out where a notification went, but it's still a bad behavior and it still catches me unaware sometimes. I've missed important emails this way.
The difficulty of maintaining Inbox Zero in 2019 also combines poorly with these behaviors - when a notification for password reset or whatever gets threaded in to an email with an old date on it, it can make it harder for your brain to process what just happened.
I wonder at which point people will start realizing that Gmail is not all it's cracked up to be. Apart from the privacy issues (you basically have to assume that Google is reading all your mail and mining data from it), Gmail treats your mail as their mail: they will do anything they like with it, including hiding it from you.
If that sounds like a rant, it is — I am worried about the increasingly centralized nature of E-mail.
Marked as "not spam". Further correspondence from them, via the same address, ends up in spam list. Filter seems to be ignoring me reporting the misclassifications.
I can't tell whether the spam filter is personalized per user, or whether my reports are drops in the ocean. Either way, I get that feeling that I don't own my own email account either.
(No affiliation with fastmail, just a customer.)
There are a few settings to configure to be able to receive and send emails and many companies can't bother to do it. This can be as stupid as not setting the MX records or sending email from yorcompany.com (admire the subtle typo). Spam folder guaranteed.
In this article for instance, the issue is incredibly stupid and only surpassed by the triviality of the workaround. It's been known for years, yet nothing is done about it.
I wanted to clarify that because the above comment makes the issue seem less important. It is important.
For gmail, you can register your domain in the google webmaster tools and it will show how many emails were filtered or not, by gmail users.
That's opened up to infinite loop of rejection emails.
I know my ISP's mail server actually does reject at sending time since I am on the gcc mailing lists, which unfortunately pass along a fair amount of spam, and when it is rejected the list software then sends me the bounce message (rejected based on From header checking, I'm not sure of the full details).
Possibly Google does this too, I don't have direct knowledge, just going off what others said. But it seems like systems wouldn't stay misconfigured long if they were.
My main point is that IMO, mail that fails sender verification should not be delivered at all, not even to spam folders. Anything else is just making a bigger mess and helping spammers. Ideally the sender should be notified that it wasn't received. But I'm sure Google has good reasons for whatever they are actually doing and my understanding is that the main reason that sender verification has become mandatory as quickly as it has is due to Google pushing it.
I have the exact opposite problem. My clients with ISP email, free email, low quality shared hosting email have the problems. My clients with G Suite, and Office365 have no deliverability, spam or reliability problems.
The Gmail spam filtering is totally hit and miss - I have had a number of Gmail/Gsuite accounts and the level of deliverability is all over the place. When it's good, it's great. When it isn't, no action you take will prevent it from sending important messages to the spam folder.
Issue before that was related to Earthlink (yes that Earthlink) doing the same thing. my solution in that case was to put my G Suite email on the BCC list - guess who got 100% of the form submissions? > This guy
Gmail spam filtering is not hit or miss, it's very reliable both from my own experience and my clients, whom typically "upgrade" to solve spam and deliverability issues with other providers.
Yes, it is hit or miss, as our vastly different experiences show. Be happy you've gotten lucky. I have not. And I know I'm not alone as I regularly come across others who get confused when they have their first experience with this, because they used to have the same belief about Gmail as you do.
The only reason I noticed is because I really cared about talking to customers and because I run my own server and can look in the logs to see if the E-mail was accepted by Google.
It might be scummy, and certainly risky, but you can hardly say it would be boneheaded to risk damaging a $1B segment to improve an $80B segment.
Never forget: Google is an advertising company. They're not a hosting company, or a SAAS company, or a phone/laptop company, and certainly not a browser company or a mobile OS company. They may dip their toes in lots of areas, but the core is solidly about advertising.
Heck, with how common G-Suite is used in schools, getting kids used to Gmail early would be a powerful argument on its own. Though writing it out like that kind of feels a bit dirty.
Speaking as a long-time G-Suite user the offering currently feels a little lost. It's a bit disappointing considering the initial promise of the product. Aside from personal inconvenience, letting it die would be damaging to Google's ability to sell into enterprise markets.
Researched briefly and this was the best I found but I have memories of something even more juicy: https://www.theguardian.com/technology/2015/dec/02/google-ef...
Not to me, and probably not to anyone else that has to administrate GSuite. I am continually astounded at how lacking in features and management capabilities corporate gmail accounts are.
Its a discount office 365 in every way - lower price, lower features, lower usability.
It sure seems like google is itching to discontinue.
In the real world, the buck stops with you. Even if it's not fair, if there's an issue your users are having, you need to fix it.
At their scale, there is no excuse for not polishing every minute aspect of the UX, and that it includes how it interacts with every email service.
Expedia stole 187m from themselves by not having their shit together.
e.g instead of:
Subject:"Password Reset Notification" (or)
Subject:"Website Support Request"
Subject:"Your Password Reset Request - March 14th 10:19am" (and)
Subject:"Website Support Request - Jimmy Davis, Failed Login"
The emails should have specified the expiration date in the text of the email document from day one.
That doesn't avoid the issue. Similar emails are collapsed, it's not about being identical.
For example, two orders listing the (different) products you bought might get collapsed, despite being fairly different.
It still threads the email, but the last one will show up when clicking on it.
When you send an email that you don't want to be collapsed into any previous thread on Gmail, include an `X-Entity-Ref-ID` header with a random value.
I don't remember where I learned this. I can't seem to find any official documentation mentioning it. But it works.
It’s gmails bug, but I’m amazed they couldn’t figure out hiw to implement the fix. Amazed but not too surprised given how not nimble big companies are.
Incredible how Hacker News figured out the workaround in 6 hours, when they couldn't in 6 years.
Otherwise a unique subject is always easy for some people.
So the "bug" is collapsed message/sort/thread view?
Feels like this is about as un-bug-like as you can get. Adding the unique Subject: line feels like the right fix (in you) rather than shout at google "you have a bug"
> So the "bug" is collapsed message/sort/thread view?
No - the bug is that new messages were being hidden because they were mischaracterized as redundant.
I quickly disabled Google's 'conversations' feature (or whatever they call it), when it was first introduced, because it did not seem to be doing a good job, but did not think about its wider implications.
I probably should not be surprised about these reactions, as I have had experience with fellow developers who insist that what is manifestly a bug is not one because it is "working as designed", or because there is an undocumented workaround to the undocumented problem.
The current way yeah you might have to see where the conversation left off- but it's the safest to not hide something from the user.
The problem is that the first message is also uncollapsed, so the user has to know to scroll to the bottom.
I also find their numbers very hard to believe. Expedia's net income last year was ~$400M. They could apparently make a 60-character fix and increase that by $187M but they don't, because...reasons?
Changes can be impossibly difficult in large companies for no particular reason.
A less advanced non-threaded show-most-recent email view would not suffer from this issue. When you add a more advanced feature, make it the default, and inadvertently reduce usability, you are at fault.
Their add contact screen, with the form to fill in contact info, had two same-colored buttons, not that far apart, same copy. One added the contact whose info you'd just been putting in. The other helpfully erased it (add a new contact, was what that one meant).
That's probably been fixed, but the whole settings area was kinda that way back then—like someone slapped it together without once thinking about how it'd be used. IDK what it's like now now, I use basic HTML gmail.
[EDIT] who's to whose. I cannot believe I wrote that.
What, you can't remember literally dozens of different passwords from different sites, that all have differing and incompatible password requirements?
This statement is bizarre, I don't really see anyone defending Gmail here - just commenters offering further explanation on what's happening and other people mentioning specific examples of being caught out by this behaviour.
The Gmail UI is horrible, the amount of confusion it creates and how illogical it does collapse and order things amazes me. Why not just show me the content as it is and let me figure out how to handle it?
But it will not solve other UX issues, like the fact that gmail UI encourages top-posting, even if the sender replied inside the body of the original e-mail. (it skillfully hides the option to reply inline inside the quoted body, even in this case)
It also makes a complete mess when you write responses inside the quoted text, if you're not extremely careful and aware of this fact.
For example these are identical messages:
As gmail user sent it: https://megous.com/dl/tmp/gmail-garbage2.png
As I received it (text e-mail message gmail actually generated): https://megous.com/dl/tmp/gmail-garbage4.png
Now someone tell me this is not a complete garbage!
Honestly, conversing with gmail users is my least favorite thing, as long as they use the web client. Gmail webmail is not a serious e-mail client. It does not even implement threading.
That said, such overblown criticisms reminds me of the quote "There are only two kinds of programming languages: those people always bitch about and those nobody uses". It's true for everything.
Back when each new message in a thread had a different-colored header, it was vastly easier to use by skimming.
Now, with everything in various shades of monochrome, threads collapsed or expanded, quotes hidden or not hidden, signatures here and there, it's virtually impossible to tell at a glance where one message begin and the other one ends.
I've noticed this myself on the occasion I end up requesting a password reset multiple times. It's annoying at best to have to click open a closed message and then click again to show the quoted text. I'm sure it took me a minute to figure out what was going on the first time I encountered this.
So after vacation > turn collapse back on > clean up vacation built up email > turn collapse off again
Edit: oh, I also proactively search for "lost" emails with the following search string:
has:nouserlabels -label:inbox -label:drafts
With my workflow, I apply a label to everything I want to save when I move it out of the inbox, meaning I've finished dealing with whatever the email requires. So that search string finds whatever has fallen out of my workflow.
Basically, if I'm remembering right, there was a semi-standard header field in emails that used some sort of hash to identify which email was being replied to. This was how emails were threaded together in other clients, and worked well for the most part.
But there were some situations where it didn't work accurately, leaving some emails un-threaded, so Google created its new conversation view to both flatten the reply tree (so you'd see all emails so far before responding, instead of making the same reply as someone else) and group up emails sent from clients that didn't include the hash.
And now we're here.
The average person doesn't get so many emails that they need to have them collapsed and sorted based on the sender. Most people don't have a problem reading every email in their inbox each day - In fact, that's what they want to do.
"The direct impact of this bug"
"Actually, the bug is still active"
So I thought the author realized it was a feature, not a bug. But then the author goes on to claim it's a bug, several times. It's clearly not, it may be a shitty feature but it's not a bug.
It took me one instance of this to realize what was going on. Why only one time? Because I took the time to learn how my tools work.
(If this reference is too old: https://en.wikipedia.org/wiki/Pine_(email_client) )
So use it then. alpine (the successor to pine) is my daily email client. I currently have 161,210 messages in my inbox, it handles it just fine.
It's still supported, and has new features like UTF-8 support and viewing HTML emails (without the images).
It's better than any other email program I've used with one exception: No images in HTML email (although you can view an HTML email in your browser, the attachments don't come along).
I use this - it's way more responsive and faster than the current gmail UI.
My only gripe with the basic html version is that the back button is broken when you're trying to go back to search results after clicking on an email (you need to click on "Go back to search results").
are you sure your recipients
seeing emails that look like
One major problem with plain text emails is the standard was created when nearly everyone had a screen wide enough to display at least 78 characters per line (or whatever it was). Today people spend most of their time on screens much narrower than that. FF is a half assed solution to that problem
html was designed to work on a much wider variety of screen sizes so it works much better for email.
I say this as an offlineimap/mu4e user who knows some of my recipients are seeing bad formatting and doesn't really care.
This basically describes me (though substitute "bower" for "mu4e").
It occurs to me that it would be relatively simple to write a filter that converts text e-mails to a multipart with an html e-mail having the same text.
Still no solution to replying to html e-mails inline, but the intersection of (Doesn't topost) and (uses html e-mail preferrentially to text e-mail) is relatively small.
In the case I described, the email is not opened with only 3 dots showing, it is instead completely collapsed. Like if you clicked the header to collapse it manually.
If true, I wonder how similar these emails were to have that happen.
Enter Outlook mobile, absolutely love it, no wonder its the most rated mail app in app stores!
At Blekko we ran into a variant of this caused all of our regular logging reports to get thrown into a single gmail thread and collapsed (making it hard to find the results) so we changed our logging script to include the date in the subject line.
The fix for the author would have been to change the subject to 'change password request received on 14-mar-2019' or something similar (you can include the time too for more uniqueness). Then it always starts a new thread and the messages are always visible.
Sarcasm aside, it stinks to have to workaround email clients that deliberately don't pay attention to convention. Putting the date in the subject should be a clear indicator that Gmail is in the wrong here—there's already a date field in the headers.
On the other hand, I chose to use the Outlook client instead of the built in mail client for iOS, specifically because there is a grey area between “spam” and automated email that isn’t urgent but I still want to see.
Most of the time Outlook gets it right.
Wow, I just saved you $100M! /s
Edit: Well, actually it didn't work and it is still an ongoing bug.
"Actually, the bug is still active. I never had the opportunity to fix it, didn’t stay long."