Google is forwarding mail for dnalounge.com but the SPF rule doesn't allow Google's SMTP servers to do that: "v=spf1 a mx ptr ~all". That could explain why the email gets in the spam box; failing SFP should increase the "spam score".
Besides I think SOFTFAIL shouldn't be used in production; and I also doubt that Google should be taking seriously a SOFTFAIL anyway; so please take this comment with a pinch of salt.
SPF is easy to get wrong, and it always backfires at you ;)
EDIT: seems that I may be right, according to this comment http://www.jwz.org/blog/2015/03/google-seems-to-have-broken-...
"A@dnalounge.com is logged in to GMail Web Client as A@gmail.com and sends a message to B@dnalounge.com. Google's SMTP servers deliver that to cerebrum.dnalounge.com with an envelope sender of A@gmail.com. (THIS IS WRONG ON SO MANY LEVELS.) cerebrum turns around and bent-pipe forwards back to Google's SMTP servers, who determine that Google's SPF record doesn't list cerebrum as a designated sender for gmail.com (given the preserved envelope sender of A@gmail.com)."
The goog then sends mail to the receiver at 'dnalounge' MX, where it's then shuffled back to the receiver's gmail mailbox. I don't read his setup as having the sender involved with the dnalounge SMTP server at all.
Shame this comment isn't further up.
I have a similar setup, and upon reading the RFC, I though ~all was the flag for the desired behavior. Turns out, it's not.
Many SPF docs recommend using SOFTFAILs, including Gmail's docs . What's your recommended SPF setup for his use case?
Using "~all" means that you can only tell which hosts are definitely allowed to send mail for the domain, and you're unsure of anything else.
IMHO that reduces the effectivity of SPF. SOFTFAIL is useful as a debug method when you're testing rules and you don't want mail to be rejected by mistake; but I think it should be transitory and finally replaced by FAIL ("-all").
If you configure SPF to allow mail being delivered by Google's SMTP servers for that domain, you're again reducing its effectivity (Google's SMTP servers are used to send spam); but still better than a "SOFTFAIL all" I think :)
Spamassassin's rationale was that many of the tutorials online never explained the difference, so the majority of mailservers were just using SOFTFAIL everywhere. More paradoxically, messages that had a hard FAIL result were statistically more likely to be due to a misconfiguration, based on an empirical analysis.
Like I said though, it's been quite a while since I've had to deal with this. I'm not sure if it's still true or not.
I figure Google takes all email in a 2 step process (either intentional or accidental)
1: get a good fraction of the world on GMail
2: intermittently declare any non GMail mail spam (
greatly lowering the utility of non GMail).
And for the "I don't like conspiracy theories crowds." From the first this said "either intentional or accidental". It would be enough for this to be a fortuitous bug for the effect to hurt non GMail users disproportionately. Google could even fail a fraction of GMail and the overall image would still be "email is flakey in general, we'd better all switch to the service provider for safety."
I known spam filtering is hard- but GMail has some really strong signals that I would think could dominate here (like incoming email contains text unique to a recent outgoing email). Or email is from a sender you clearly have a relation with.
(note: "Google takes all email" is a possible outcome, not a plot/conspiracy. A bug like this on a system of this size can have a big impact. Of course that does mean somebody running a system of this size might have an extra responsibility to look out for such things.)
It is also apparently unfixable. Regular people cannot inquire as to why mail is not getting through. And amazingly, even having friends at Google does me no good here. I'm told that the anti-spam team is so mysterious and opaque that they won't even talk to co-workers about this.
It's maddening. I get why Google grew up with absolutely no conception of customer service. (It's hard and expensive, and not necessary for their business model. Chicken farms don't have customer service windows for the chickens to lodge complaints.) But their apparent utter indifference to the problems they are creating for other people is maddening.
(Full disclosure: I'm an employee of Mailgun)
It is possible to send email from your own server; you need to follow onerous industry-standard practices, like using a dedicated, known IP that has a certain amount of increasing email volume, known as "IP warming". After a few months of constant interaction with an ESP, your sender score improves. Unless you're sending massive amounts of email, the cost-benefit ratio isn't in your favor to use this method.
However Yahoo are an order of magnitude worse when it comes to delivering email. If they decide you aren't their friend, they send back an error code with a URL in it to go and visit and identify yourself via a web form that is reviewed rarely by some humans. Then if you fuck that form up or if they don't like you, blackholed for 6 months and they stick their fingers in their ears. Try telling a yahoo user about this as well and it's "meh".
Email is broken unless you have a small provider or DIY and it's fucking frustrating.
I understand spam is a problem but email is perhaps the last communication identity you can own, and have interoperate with others. I have had my own domain and own email for 16 years now, and I am not going to give that up. Free email services come and go, but I own my identity.
I think the problem with DIY/small business email is that it is surprisingly difficult to set up a proper mail server. Sendmail book is about 4 inches thick, and my current setup is using 3 projects to achieve a simple mail server with SSL auth/IMAP. (Dovecot, postfix, SASL). I am a software dev, but even very good sysadmins I know do not want to have anything to do with email anymore and will often farm it out to Google.
Might want to set up greylisting, though, to curb incoming spam.
What succeeded for one client was my posting pflogsumm stats showing delivery rates and times for Yahoo vs. numerous other large mail providers, along with proof of SPF and DKIM support, to all the Yahoo C-level (and several lower) execs, indicating that they had a problem, and I'd been attempting for months to resolve it.
Apparently there's a Yahoo "concierge", and we managed to get onto the company's approved senders list shortly after.
That said: email's a pain in the ass, and I do know for a fact that that particular client was failing to scrub known dead recipients (as in: the domains no longer existed). Some fights you just concede....
Turned out all the emails from @google.com were going into my Gmail spam. I think they still do (no matter how many I've marked as not spam).
Yup same exact thing. Just the other day I sent my wife a few links from my business email account @company to her @gmail and it ended up in her spam folder. And I have been mailing her for longer than a few months as well.
In other words google does not differentiate it appears on whether you have an ongoing relationship (email wise) with the person that you are sending an email to as if they are just looking at the mail message and nothing else.
Spam classification is a hard problem, but I don't think they've put the false-positive/false-negative tradeoff in the right place, and they really need some safety valves like offering to whitelist an address when you un-mark it as spam.
Apparently forwarding can sometimes invalidate the SPF or DKIM  depending on the settings involved.
But there is a white list, if you add the sender to your contact list it is white listed.
It's horribly convoluted but it /is/ possible to do so:
I've been running it this way for about 18 months now after having a few important emails get gobbled up by their spam filter.
Never because my grocery store isn't a website.
Amazon can't give you a printed receipt because you are not there, physically. So they send you an email instead.
This is not that hard to understand.
(And keep this discussion in mind next time you read a HN story teaching people how cool e-mail marketing is for your startup - what it's cool at is often pissing off your users.)
Also Google took a good approach to that issue - when an e-mail has an unsubscribe link and you try to mark it as spam, you get asked if you'd like to unsubscribe instead. And yes, I sometimes do exactly that.
1) Unsubscribe should be one click, no confirmation. In the case of accidentally unsubscribing, if it's really that valuable, the person will re-subscribe, and I guarantee it's easy to subscribe because that's already optimized, for obvious reasons.
2) GMail shouldn't give you the option to choose; if you mark something as spam, it should silently attempt to unsubscribe in the background, then keep track of which services don't actually support unsubscribe, and mark those as spam. Far too many people are vindictive and lazy and will intentionally mark things as spam either because they don't want to go through the process of unsubscribing, or they forgot they subscribed, or think they've been "wronged" by getting an email they opted in to.
For the record, I too despise the oblivious email marketing stories on HN, eg the "cold emailing" C level execs whose addresses aren't publicly available. Dirty, dirty, dirty.
so... sorry? :)
If you signed up, then you opted in. Don't be lazy and mark things you signed up for as spam. I'm not understanding how people get email from Amazon that aren't customers, and how customers can't turn off everything but confirmations, which I know is possible, because I have.
I'm a customer, I consider that opt out. You don't? I signed up for amazon, not their bullshit emails. That you consider the situation opt in shows they have you trained well.
That said, this is no excuse for Google to mark their transactional emails as spam for people that clearly want to receive them. Google is just being lazy and doing a bad job. Even Yahoo gets that right, Google can do it too.
of the 64 emails I have from them 50 are spam of this nature.
just because i signed up for amazon.com, doesn't mean they get to spam me crap because they've buried a 2pt font unsubscribe feature that may or may not work over the course of a few weeks.
You can intentionally not fix a bug.
I don't disagree. E-mail is a mess. Still, on the whole, for any person or organization with modest technical chops, I feel the freedom and decentralization (which is a positive for the entire ecosystem) is reason alone for retaining control by self-hosting.
Decentralization is cool and all, but until we have some kind of self-contained, easy to deploy and maintain solution for e-mail servers, it's hard to me to recommend anything other than GMail to people - especially that typical small company or organization has a lifespan that's shorter than Google, so they don't get any important benefit from decentralization.
There are other companies that offer e-mail services, without the embrace, expand, and extinguish strategy that Google seems to follow these days.
I started with their new theme called "Larry" and then corrected the fonts, colors, graphics, and certain element sizing decisions.
Another large portion of my spam is from Chinese product spammers using Picasa albums and send to a friend feature. There are limited ways you can report this and despite marking almost every Picasa email I've received as spam, every single Picasa email still makes it through.
That is until you get a court summons after an invoice was sent several times and never got there because some fuckwit mail policy just blackholed it. This stuff does happen and when it does its costly.
I nearly always agree or can be convinced by the higher karma users, but the fresh users seem to be defaulting to reddit-level arguments (uneducated, immutable beliefs, hyperbole, taking things personally, no objectivity). I.e.
> just that everyone who doesn't run a mail server hasn't had to deal with this shit
"Everyone" in that sentence should not have the right to be downvoting in the first place.
Also, Reddit is cool, and there's ton of quality discussions - usually under particular subreddits. For example, I love /r/KerbalSpaceProgram for a perfect combination of being large, friendly, fun, and full of people willing to teach each other some more advanced science.
To address your point: I think current downvote gate is fine and adjusting it up won't really help much. What would help (though I don't know how to achieve that) is people realizing that when saying anything controversial, you might end up getting a bunch of downvotes immediately, but the score usually settles to a reasonable value within an hour or two. It takes time, but one just has to get used to it.
TL;DR: karma comes and goes; look at it through a low-pass filter, to filter small variations.
Huh. Maybe we should take that metaphor literally and implement it!
I find hn a really to be a really good combination of intelligent, grown up discussion with little name-calling etc.
That said, I think on hn one must be prepared to be corrected, -that's just part of being scientifically minded. As the old saying goes: "Iron sharpens iron, and one man sharpens another."
Who cares about my stupid comments, sure, I'm an idiot- I'll agree to that. Unfortunately, I see it happen all the time. The parent of this, for example, was downvoted immediately.
Everyone in this community is prepared to be corrected- the problem is when there's no correction offered. It's a community of suppression, and it's not hard to see.
I don't think it is a big problem though only that it has been slightly increasing.
(one comment you'd see back in the days when comment scores where displayed was: "sorry for the downvote, -reading hn from a mobile device ". You don't see this comment anymore but I'm not sure if that is because it was easier to detect accidental downvotes back then or because people where more polite.)
Perhaps HN users need to be reminded to upvote things that have been unfairly downvoted? (Although I've seen plenty of examples of corrective upvotes supplied pretty quickly).
Another issue I have with your argument - I don't believe Gmail has the market share to make such a fiendish strategy viable.
And try doing business with small businesses and then tell me what market share you perceive GMail as having.
I think, right or wrong, you'd probably get a better result if the rage in your text was clearly directed at the consequences rather than the motives - if nothing else because, as you say, the motives don't actually change the consequences and it's the consequences you, I, and everybody else who likes having their own email setup has to suffer.
(to be clear, I'm not commenting on whether or not you should have been downvoted, only on my best guess as to how to get across the same information without it happening)
Another educational moment for you would be to look up the word "conspiracy" to learn what it means. I understand that you were hoping to use it as a dismissive term, but you would have been better off using a universally dismissive term like "crazy", as there are a lot of rational people that understand the factual meaning of "conspiracy" and don't interpret it in the same way you do.
It doesn't matter. The conversation at HN is toxic. I've bitched enough. I was just reaching out for commiseration.
His employees want to receive mail sent to @dnalounge.com in their Gmail inboxes. It's a non-standard configuration and brittle. You can't really expect them to support it for free.
Google offers a couple of supported ways to do it. Most prominently: pay for Google Apps. But you can also use Gmail like a regular mail client and retrieve mail using POP3.
No, he is using a normal forward for those who want to receive their mail at gmail. He is not using gmail as his backend. You should read the original article.
I run my own mail server, but most of my employees use Gmail. So I have forwarding set up: firstname.lastname@example.org simply forwards to email@example.com. When sending mail using their Gmail account, they set their From line to firstname.lastname@example.org. (Google lets you do this if you jump through some hoops to verify that you can actually receive mail at that account.)
There are basically two cases here:
1. The sender is sending one email to one receiver. The sender computes the proof of work and sends it along with the message. This takes some time, but typically it can be done in the background. Waiting few seconds between sending and receiving an email is typically not an issue, and in fact many email clients wait for some time before sending anyway to allow "undo send" functionality. Rather that doing this computation on the mail server, it should be done on the sender's client, so that the server doesn't get overloaded with proof-of-work computations.
2. The sender is sending many emails to many receivers (a mailing list). In this case when the receiver signs up for the mailing list, the sender sends a request for a whitelist token to the receiver's mail server. IF the user accepts the request, the receiver's mail server returns the token and then the sender sends all further communication with the token instead of with a proof-of-work. This solves a few problems: a) Receivers opt in rather than opting out of receiving communications. The ubiquitous pre-checked "send me spam" checkbox loses its effectiveness. b) Receivers can revoke tokens at any time. c) Senders who are sending large amounts of legitimate mail don't have to compute a proof-of-work for every email they send.
Receivers simply drop email which doesn't come with either a proof-of-work or a whitelist token. This drives up the cost of sending large numbers of spam emails because each spam email requires a large amount of computation. And even in cases where a spammer has a large amount of computation at their disposal (botnets are a common case) it makes it easier for servers to distinguish between mailing lists and spam: a large number of identical emails could be either, but mailing list mailings should come with a whitelist token.
Doing things this way would mean we can drop these terrible DKIM and SPF systems that both fail to prevent spam and make it difficult to send legitimate mail.
I can understand your feelings about SPF, but why is DKIM a terrible system in your opinion?
And so, I have now moved my e-mail to Rackspace. I had to use their chat-based support several times while migrating my e-mail to the new server, and it was truly fantastic. On that point alone, I feel fortunate.
The $20/month I'm paying Rackspace is a pittance compared to the time and effort I was spending trying to keep my old SMTP/IMAP servers secure, as well as the false-spam tagging that happened all-too-often.
It sounds like jwz wants to have his cake and eat it too, and I sympathize. But I'm not sure that it's possible any more to spend a non-trivial amount of time configuring e-mail servers, tinkering with them such that they'll work with big companies (and especially Google). The Internet is no longer the simple, fun playground that we old-timers remember, and that effectively means giving control over some services to people who are paid full-time salaries to take care of these inter-connectivity issues.
Gmail accounts created on or about June 2014 (exact date unknown, Google only mentions "second half of 2014" in their new authentication blog post) won't work with Thunderbird until the Thunderbird team implements Google's non-email-standard authentication. This is currently scheduled for Thunderbird 38 which will be released on April 7, 2015. See bug 849540 for the full technical details
It sounds like he doesn't want to force a work email account on his employees. I don't think many people would have a problem with a work email, it's the norm after all, but fine.
But he also doesn't want to add their personal email addresses to his address book (is this why he wants the @dnalounge.com addresses?)
I'm not sure what this setup is attempting to accomplish.
Actually, makes sense, because he can then save those emails on the server, back them up, and they'll be in his control if an employee ever leaves. It's good corporate policy. I believe the White House is having a similar conversation lately.
So again, I'm not sure what problem he's trying to solve here. To be clear I'm not saying he shouldn't be able to set up his email this way and have it work, just that I don't get why he WANTS to do it this way. What is it accomplishing?
Especially since you can configure gmail to serve as an interface to arbitrary third party POP3/IMAP services.
"This would work but I'd rather find a more elegant solution" seems like a perfectly reasonable desire to me.
With office 365 and hotmail email just goes into the abyss. We have SPF and DKIM yet some mail just never arrives when sent to hotmail. The server responds with "250 queued for delivery" but the mail never arrives and doesn't go into spam either.
Very annoying when order confirmations don't arrive.
I did consider setting up a route in my mailserver to GMail using my Google account credentials for their SMTP server, but then decided they should better get their own shit straight. It wouldn't scale to do this for everyone.
It doesn't seem to matter how many times I mark an email as "Not Spam". The next day an email from the same sender from the same server with the same Received: path will land right back in Spam again.
I realize that gmail can't immediately globally unblock based on "Not Spam" reports (otherwise spammers would ruin things by marking their own bulk mail) but could they at least apply them to my mailbox?
I guess adding the senders to my address book would help, but it seems like a silly thing to have to do. I already told it that a message is "Not spam", can't it take it from there?
So for this reason people give up and either send their emails through well known mail senders or switch their users to services like GMail. This is sad as perhaps the last communications medium with open standards is being forced proprietary due to the difficulty in dealing with the large email providers. My company switched to GMail for our email, and Mandrill for sending out our mailing list although we have competent sysadmins.
This is a pretty strong understatement. The spam filter in gmail is incredibly aggressive. My secondary email is a gmail account and anytime I have a registration email from a service or anything with a couple URLs in it, it goes straight to spam. Meanwhile, the product we use at work is tons smarter, but gives the occasional spam into the inbox.
I guess someone at google decided that people seem happier when there's no spam in their inbox. People seem to drop 20 IQ points when they see spam. They either instantly rage about it or send the email to the helpdesk demanding an explanation on how someone dared email them something they weren't interested in. Its a bizarre behavior. Yet, when they're forced to fish through a spam folder or an anti-spam web interface, they seem to mind that less. Which is bizarre as its tons more work.
I think the guys at gmail realized this and just erred on the side of caution by giving people as much as a clean inbox as they can handle. Even if they don't like it, who are they going to complain to? Google? Who at google gives two shits and even if someone did, how could your reach them, and if you did who has the power to change policy? Meanwhile if you run mail servers, you get complaints about how your email goes into spam, even when you have all your DNS ducks in a row and have a good reputation. When you tell gmail users to talk to google about it, they scoff and tell you to piss off.
What is difficult in spam classification is having no false negatives and no false positives. Any decent CS graduate can create a spam classification system with no false negatives given the leeway to introduce false positives. That's what Gmail did.
I think this is the case for the user. For the sender it's a major issue and a huge annoyance, but you don't measure user satisfaction by asking senders.
Maybe I'm just an outlier, but I'm perfectly content with my gmail.
Example of a message from Google, to a custom domain gmail account, forwarded to another custom domain gmail account — found in my spam folder:
They're still no where to be found, might be in limbo, but email forwarding definitely took a hit yesterday
Turns out, freaking difficult. For one of the simplest protocols in the world, email is now a byzantine disaster. Different clients render things differently, different providers block or don't block things -- and that's not even getting into trusted domains and spam issues, which you end up depending on third parties to assist.
Add into this mix Google's notorious black-box/impenetrability/lost-in-space-customer-support? I don't see an optimistic future for you.
I'd rather go back to hand-coding COM in C++ that become a network email engineer. That's got to be a brutal job. You're dependent on so many moving pieces and you control so very little.
The way it's set up now, with hundreds of vendors implementing different control protocols and tens of thousands of spammers trying to break in? I don't see anybody being happy -- some spammers get in anyway, and email users have different experiences for the same dang net service. Admins want to pull their hair out. It really is a miracle things work as well as they do.
If we can't get email working right? It doesn't portend for a bright future for the rest of internet traffic.
We simple prohibit @yahoo emails to be added as contact and use Mailgun.
Maybe Google gives extra trust to 1&1; or maybe there is some other thing that they check for.
I always wanted to do the opposite: Run a filter on the spam folder to delete emails that I know for sure that really are spam, so I wouldn't need to delete them manually.
Stop using it. You already have your own email server.
And, well, I am sympathetic to the argument that Google shouldn't do things the wrong way. But you really have no leverage to make them. Google offers the ability to use Gmail for e-mail sent to your domain, that's Google Apps For Business (or I guess Google Apps For Work). If that's unacceptable, you either have to deal with Google not caring about your needs (unless you're paying them, you're not the customer anyway) or do without Google.
If he wants to use Gmail for his business he needs to get Google Apps. Having your employees use their personal e-mail accounts for business is completely unprofessional.
Why even bother running an in-house mail server if all it does is forward e-mails? Cut the costs and pay for the services Google offers for business and get your employees more focused by keeping them off their personal e-mail on the job.
I've recently been through this for my own personal email. I have a domain with some interesting rewrites to the address that I forward to GMail. I put together my set up into a Docker container:
It is worth breaking this down into each direction -- sending and receiving.
This is the easy side. You simply run an SMTP server and have GMail use this to send. If you want others to not mark you as spam you'll want to look at setting up SPF (publishing which IPs are allowed to send from your domain) and perhaps DKIM (digitally sign your email).
GMail used to allow sending from aliases directly without an external SMTP server but that is disabled. Old accounts are grandfathered in.
This is where things get complicated. Essentially, you are having folks send mail to your server and you are turning around and relaying that email to GMail. GMail has a hard job here. It doesn't know if it should trust you and that you are acting on behalf of the user or if you are an open relay sending spam. I've found after I change my forwarding set up I have to police my spam folder for a couple of days to retrain GMail.
The real complication here is SPF. Sending domains will publish their own SPF records. When doing this they can either specify a soft or hard fail. If they specify a soft fail (`~all`) then there is a chance that GMail won't mark it as spam. But if they use soft fail (`-all`) then it will go to spam all the time. The problem is that the sending domain (say evite.com) doesn't list your relay IP as having permission to send that mail and so GMail respects that.
Having GMail pull the mail via POP is one solution here but that introduces latency.
Or you can rewrite the "envelope sender" so that you are honest about the mail coming through your server. The accepted scheme to do this is SRS. This is not a silver bullet though. If you are forwarding a lot of spam, GMail may decide that your SRS domain is spammy and penalize all incoming mail.
Also, if you are forwarding a lot of spam, GMail will throttle you. It'll have you back off and wait to forward more mail. Your best bet is to find ways to eliminate obvious spam before you forward to GMail.
My solution seems to be working okay for now, but it is a pain the ass and I'm honestly not sure it is worth it.
EDIT: Here is a tutorial that I used to inform my approach. It is worth digging in to. http://seasonofcode.com/posts/setting-up-dkim-and-srs-in-pos...
2: when you are reliant on an external source to solve a problem within your own house you are not prepared.
3: solution: reduce reliance or point of failure by either bringing it all in house and forwarding the remains or bring root to the source and source from root and use mask and forwarding.
he could bounce off another provider/server so he can still maintain what he is doing now - he just has not explored any other solution except blaming. the arguement "it should just work" is not vaild if you are reliant on another service.
Whelp, there's your problem.
Programmer misconfigures SPF record and blames google instead, news at 11.