The solution we have used, it was bring Gmail to understand that your emails have been sent from a real user. You can simulate a conversation between your domain and other Gmail accounts. Send a first email, with a realistic text, to a Gmail account, if it goes in spam, mark as safe (remove it from spam), and reply to it, always with a realistic text. Continue to simulate a normal conversation, sending 4/5 emails between the 2 account. Re-do the operation from another Gmail account, but this time do the opposite:
start from the Gmail one.
It not bulletproof, but it has worked in the past for us.
There is no way that this method would work id the domain it does actually send spam mails.
Basically even if user expressly add you to their contact list and pull you out of spam, etc, if you sending them just your email ignoring what they reply back to you, eventually you get back to spam. I think this is very clever and is based on simple human rule that most follow when conversating over email: when people reply to email the do not start a new email, they hit reply and previous email from their convo lands on the bottom of this new email.
Once we started emailing new emails with adding content from addessee previous email to the bottom, our emails start getting back to addressee major mailbox.
And this make perfect sense on how to detect a real conversation (gmail has your sent items so if incoming email contains something you wrote before, its an indication you have a real live conversation with someone)
If your numbers are low, so even small change can make a big difference
In this case, the mail was bounced at the SMTP level, and the bounce is (strangely) identified as coming from Google Groups.
"Mark as safe" will not SMTP non-delivery problems, I think, just the mis-classification of e-mail as spam at the folder delivery level.
The reason given is "parked domain".
Edit: I am referring to the bitbin.de domain
If you are curious, try to go on our italian website: qboxmail.it
"I had the exact same problems in the past. For me it helped just to register and immediately cancel a trial of Google Apps for the domain. It’s annoying to have this as a necessary step but at least it was like that done in just a few minutes (after I tried to find a contact email of Google for a much longer time). I tried it because I had other domains on the same server which didn’t had this issue. All domains which weren’t filtered had a Google Apps account in the past. So I thought it’s worth a try, and yeah it was solved."
After 2+ years of hosting my domain myself, it doesn't seem possible to build up enough reputation with them personally.
What this looks like to me is a reputation problem with one or more domains, either in your headers or your message content.
From a domain reputation perspective, you don't generally need to 'build' reputation in the same way you often do with IP reputation; rather, Gmail tends to tag domains with negative reputation only when they've observed unusually bad behavior associated with the domain.
Sometimes this can happen due to domain forgery or spoofing. Sometimes it's because a sender is doing overly aggressive email marketing. Sometimes it's just because your domain shows up in someone else's bulk email.
With full authentication, spoofing is unlikely; I also don't see any indication you're doing email marketing here, and your IP 18.104.22.168 doesn't appear to generate much more than a trickle of volume. So - perhaps the latter? Has anyone included a link to one of your blog posts in an email newsletter, for example?
It would also be helpful to see full headers and content; it's great you're doing SPF/DKIM/DMARC, but they're not going to prevent all issues - there might be something elsewhere in your message that gives us a better idea where the problem is.
Finally, keep in mind that anti-spam systems are highly dynamic, and results can change - sometimes in as little as a few minutes - based on the latest data feeding into the system. If you're lucky, you might see this resolve on its own fairly quickly.
>anti-spam systems are highly dynamic, and results can change - sometimes in as little as a few minutes - based on the latest data feeding into the system. If you're lucky, you might see this resolve on its own fairly quickly.
>this will most likely be fixed tomorrow morning as a result of this thread.
How was she 'absolutely right'?
I think he understands that it's different. He was humorously implying that the issue will be resolved by a Gmail person seeing the thread, as opposed to some AI spam filtering.
I doubt you'd get away with doing it for a domain sending legitimate mail, or at least not for very long.
Running the email through something like Amazon SES or Mandrill would probably be helpful. Both have generous free tiers.
So I don't know if that theory explains the issue either, though it could be a component.
That volume (assuming non-spammy content, of course) would over time build up a positive reputation for your IPs, outbound addresses, and content.
A new or low-volume IP on Hetzner is going to have a harder time overcoming the initial suspicion Gmail saddles it with.
Gmail's approach with a low-reputation IP is generally to throttle it aggressively; mail still gets through, just not very quickly if you're sending in bulk.
Domain reputation is not initially distrusted unless it's a brand-new domain (registered in the last 30 days or so), and generally doesn't come into play unless there's very unusually poor metrics (think sky-high spam complaints) associated with the domain. Content reputation is applied in a similar manner.
The headers I'm looking at should most likely give Gmail enough distinct identifiers to work with to isolate it from broader reputation issues at Hetzner, but the post only included partial headers, so I can't quite say that conclusively.
It's oh so much fun trying to get them removed.
But you are still sending all your outgoing email through an external 3rd party. Not ideal. Also, these days every email is a top-reply orgy of all correspondence ever sent, so your SMTP relay essentially gets a copy of your entire conversational inbox (via "QUOTE ALL THE THINGS" reply garbage) as well as your outbox.
smtp delivery is something of a black art, but at the very least those things need to be set correctly. Its also worth mentioning that if you have 'spammy' applications that opt people in to mailings then a lot of people just get pissed off and click the 'junk' button over and over instead of unsubscribing properly. So think twice before auto-subing customers or enabling 'mail me replies' by default on forums and such.
I also learned the hard way that once you're on Cisco Ironport's spam list, then you really can't get off. There's no one to talk to. Its supposed to be automated. In practice, that means months before your domain is unblocked. I find most Ironport admins just use that sole list even while Cisco recommends they use many and weigh the average. After a stupid config change that opened our smtp to relaying, we got off everyone's blacklist but IronPort. IronPort is pretty much the defacto standard in several types of enterprise. My fix? Get a different IP address for the mail server.
Mandrill is top notch, just like other professional email providers such as Mailgun and Sendgrid.
SES (as with Mandrill, Sendgrid, etc.) runs on a separate block of IPs and is very strict about spam/bounce reports. Very easy to get yourself throttled or cut off entirely.
You might have to spin up a bunch of servers to find one which doesn't have a blacklisted IP. Then you have to build the IP's "reputation" with the various providers before you can send a significant amount of email through it.
Just a few years ago, I personally experienced this sending email between two GMail accounts, and then I noticed that my (workplace!!) IP address ended up on a public blacklist within an hour. Fortunately I noticed fast enough to get it removed before any mail delivery was affected.
Good to know! I bet they have their own internal ranking system which is far more accurate than the public blacklists.
I've never had issues inboxing on gmail accounts using SPF/DKIM/DMARC + Sendgrid, even when sending 125,000+ emails (legit!) per day.
The primary limiting factor for most blacklists is not scale, but simply the fact that most of them have no more than a few data sources - most commonly spamtrap data. It's useful, but it's not a comprehensive enough data point to accurately evaluate mail.
Having dozens or hundreds of data points available - things like how many recipients open a message and spend time reading it, how quickly they seek out a message when initially opening their inbox, or a sending domain's pagerank - gives Google a considerable edge in assessing overall mail quality.
(Caveat: outbound filtering is often more difficult than inbound filtering - perhaps in part because there are fewer data points available when assessing outbound mail.)
I observed a former boss, who was very technically competent otherwise, using "mark as spam" instead of delete.
I guess if he could underestimate that button so can a million other people.
The fact that Linode's lowest tier has been $20/mo until recently ($10/mom nowadays) also helps. That makes Linode unnecessarily expensive for people who just want to burn IPs, since they can easily get an IP for $5/mo with DO or even less with low-end VPS providers.
My newsletters are aimed at developers, and one issue went out and was considered by Gmail to be a 'phishing' attempt. I couldn't figure it out. Several issues later, another one was picked up the same way and I figured it out.. In both issues, one of the items was linking to domains that looked a bit like this "www.0x10abcdef.com" (this is NOT the actual domain) - basically a domain that looked like a hexadecimal number. I ran numerous tests and Gmail always considered mails with links to domains like this to be phishing attempts.
I reported this as a bug (since nothing was wrong or reported with the domains in question, it was basically Gmail's filter being in error) but no idea if it was ever resolved.
That's my guess, at any rate.
Edit: You can usually find out what they mean by going to https://www.google.com/doodles
tl;dr: I would blackhole my own mailserver.
Have you any advice for me?
Im planning to use confirmation as is done by free software mailing lists but my concern is how to flog my website without offending anyone by flogging my website.
it'll usually serve remote http pixels for "open tracking" either way though
given how most clients/web-mails filter these by default, is this of any use? Only users which explicitly click on "show images" will get tracked, and the rest won't even show see your email properly.
I regard such tracking pixels as morally reprehensible. While I know most of my subscribers will disable remote images anyway, quite likely they would think poorly of me for serving them.
All the stuff I read about email marketing is all about all the kewel things one can do with email bugs.
I might tolerate your initiation of contact, but I will not tolerate your observation of my reaction, without consent.
For this reason, I will not click on links with obvious tracking parameters. I strip them out first, or come to get the information some other way.
So if you use them, you're taking advantage of peoples ignorance. Seems morally reprehensible to me...
I am not cool with them discovering I read their eMail while receiving pleasure in a hoyse of ill repute.
Mandrill sets open tracking to 'off' by default and no doubt Sendgrid, et. al make it optional as well. Same goes for link tracking.
(I use Mandrill for emailing dev-related stuff and was also sensitive about this given the demographic/privacy)
How about asking a few of your friends to select your emails in their gmail spam folders and click "Not Spam". Hopefully that gets the ball rolling and the situation improves...
My best guesses as to why my domain has been dinged:
1) It's on a VPS, that IP may be flagged already
2) I often use a VPN, and sometimes send emails out through my server using it. This probably raises the red flags.
3) It's a non-standard TLD, .co
4) I don't use any Google services with that email address/domain (I assume doing so adds some level of measurable trust)
5) I'm not in the address book of a lot of gmail users, because this is my private server, that I use only for job seeking, and personal communication.
This has really done damage in the past. I've applied for jobs, and heard back weeks after they hired someone, letting me know, woops, I ended up in their spam folder.
Trying to meet with a friend? I can't email them, I need to use Facebook or Gmail... welp.
This is super annoying. I kind of understand why it happens, but it's just a little sad that building your own fort, so to speak, is so impossible.
Things I tried to do to mitigate this:
1) Made my web domain https only (why not?)
2) set up DKIM and SPF (didn't seem to have any effect)
3) proper SMTP authentication, secure port only
4) Reached out to Google via the typical forms they offer, and heard nothing, obviously.
Google stands to lose a lot more from a potential PR disaster for burning former customers who move away from hosting than they do from trying to convert a tiny portion of users to a free mail hosting service.
If you dont want to receive spam use lafn.org. I dont know how one gets an account there but I expect it doesnt cost anything.
If you want a full VPS use http://prgmr.com/ - "we dont assume you are stupid.". Its a hosting service operated by neckbeards for the benefit of their fellow neckbeards.
Quite cheap, I get mine for free because I help them with their marketing.
I have devoted years to beating a clue about marketing through his pointy skull but then he complains that he has to take contract comouter janitor work to pay his data center.
I expect he shops at that dame Safeway.
The way he needs to market was established in the 1960s by the stanford alumni association but Luke refuses to Read The Fine Manual.
He knows all about Open Source though.
Whenever they solicit me I respond immediately with links to my own SEO articles. Of course the very best SEO is for ones own website to go viral at an SEO board.
One enterprising young South Asian was obviously a clueless newbie so I gave my reply a great deal of care. A few hours later he responded with:
"You are my SEO master."
I dontbreally offer SEO consulting but to claim that I do results in lots of Google Juice.
Doing a quick search on my inbox, there are some relevant emails containing the word 'unsubscribe'
I run my own mail server and manage a number of small business clients who have mail servers. Email trust is getting more and more tedious.
Recently I was resolving a domain registrar issue with Network Solutions. They required forms filled out and signed, a copy of a utility bill from my client, a copy of my ID...
I bundled up everything and emailed them the scans. I contacted them 5 business days later, they claim to never have received it.
I sent it again while I had a rep on the phone, it went into their spam hole, probably due to size of attachment.
They helpfully suggested I get a GMail account to send the same message.
They are my registrar, they host my DNS, including my MX record. I have an spf record...
I thought it was pretty farcical, and a sad statement of digital trust/authenticity.
Some of my clients are giving up and just going with the flow, I have had several conversions to Google/Microsoft cloud-hosted solutions for email...
I hope this story gets traction and someone on the Gmail team finds it and comments.
That said I've never seen a DSN like the one in the screenshot. It certainly is not generated by the gmail spam checking system, because gmail does not bounce spam. Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders.
This is a million times better. Meanwhile, microsoft accepts it, sends it at passed on DMARC reports, but discards the mail silently without it even reaching the spam mailbox.
Apologies if you took it as a conclusion, it was only hand waving speculation, but google does have an incentive to make it difficult for the little guy...
> Gmail either rejects spam at SMTP DATA time, or delivers it to spam folders.
They do if you have DMARC enabled, and a REJECT policy, although they still seem to ignore it sometimes and I have no clue my mail is sitting in a spam box.
https://github.com/cmail-mta/cmail if you're interested.
Also make sure a DMARC record is setup.
DMARC is enabled (and in one of the screenshots, Google shows it passing). I get reports from them, but it provides no insight into their decisions.
This part sums up as I see it as well:
I can only think this is intentional on Google’s part – they have a near monopoly; the vast majority of mail I send these days goes to Google – and if a small company is running their own mail server is too much of a hassle, then maybe they’d buy Google Apps. It’s bad, anti-competitive behavior on Google’s part. Shame on them if its true. I don’t know if it is, I can only guess, but they certainly have an incentive to make it difficult for the little guy.
I’m just a geek that likes running my own servers. My pleas to Google’s impersonal forms fall on deaf ears, and I’m getting tired of telling everyone I e-mail to check their spam folders.
Yahoo apparently has procedures to deal with this but they are difficult to find. The annoying thing is that now I'm not even receiving any notification of the blocking anymore. The mail just disappears.
Btw, mail from my Drupal system also always ended up in spam. But I kept removing the spam tag and now it is just goes into regular mail.
Our 2 person small time company sends a few hundred mails a month at most. We replay to someone on Gmail and get spam-binned. They whitelist us, we reply again and get spam binned. We send the mail via a major provider and it gets through.
On Outlook a website mail form sending emails to a hotmail.co.uk address was getting blocked - the server has the same IP it's had for years, the form has been used for years, the recipient has whitelisted the email address. I forget what eventually fixed it, think it was addition of a reply-to address; quite ridiculous.
In both cases they are long term domains with real ID info that hasn't changed, the domains have been on the same IP held by the same ISP for at least 3 years and owned by the same owner used for the same businesses for at least 10 years. Both domains are long-term registered in (Google|Microsoft) analytics.
Yes I can see that such domains could be purchased by spammers and the prior owners may not change their ID info and the new holders may be able to purchase space on the old server and so keep the IP address (despite the established ISP having strong anti-spam policies) and may be able to then send out spam emails, but who would whitelist those emails???
IMO on either Outlook or Gmail if you whitelist something, even if it were spam from a known spammer, then they should let it through (sanitised if needs be). If they wanted to they could add a "99.999996% of others blocked this but you have whitelisted it, do you want to block emails from YourBestFriendWhoSendsSpam@theirISP.com in the future?".
Remember, I am paid user and I went through rounds of bs with their support.
I feel sorry for those, who have to send email to Google Apps/Gmail as part of their business.
(I've posted this as a comment on original story, but decided to duplicate here, if someone will find this info helpful)
To answer your question about what you can do, you can send mail to Google legal asking them to accept your mail gmail users - just copy your whole blog post, I'm sure they'll action it right away:
Antitrust is serious business. (They already have trouble in the EU for it - https://www.google.com/search?q=antitrust+eu+google - by the way I think it's completely unfounded.) It takes them seconds to whitelist you, and this really is a "15 seconds could save you $15 million on your next antitrust case" and anyone in Google legal can probably see that.
EDIT: I don't see how this got downvoted. This literally answers OP's question about what else he can do.
But Google does accept almost everybody else's email, including many actual commercial competitors, and there is no reason to believe anyone at Google is deliberating singling OP out or even knows who OP is, so I can't imagine where you're seeing anticompetitive behavior.
> Since email is an open protocol, and since his mail server is easy to verify and sends no spam (as he outlines at long length in his blog post), this really is quite blatant. Imagine if, early in the millennium, Internet Explorer refused to connect over http with servers that weren't IIS.
This is more like if Internet Explorer failed to connect to some random Joe's homebrew personal server but could connect to most other people's just fine whether or not they were using IIS. That's not anticompetitive behavior, it's just a bug, and in fact there were many things that Internet Explorer did not handle correctly.
Again, Google accepts email from a huge number of sources, including major competitors, of which the OP is not one. Many people here even report running their own mail servers and experiencing success sending to Gmail. What kind of anticompetitive behavior is that? There is no indication that this is anything more than a bug.
If all small scale email providers get blocked then that's surely anti-competitive. It doesn't require them to have specifically blocked his server for them to be acting anti-competitively.
If the only practical way to get your [non-spam] email passed on to a person who uses gmail is to move away from your [non-spamming] email provider that's also anti-competitive IMO.
"but sometimes these wanted messages are mistakenly classified as spam. When this happens, you might have to wade through your spam folder to find that one important email (yuck!). We can help senders to do better, so today we’re launching the Gmail Postmaster Tools."
Check-list before you attempt to get white-listed/ban-lifted on Gmail:
Send bulk with "precedence: bulk" header. Use spf and dkim (optional). Always send from the same domain. Have your users "opt-in" to receive mail. Have a public e-mail policy. Basically, don't require users to enter their e-mail address.
I have a lot of mail that gets forwarded to gmail, including from cron that goes to my own mailbox at gmail. I sometimes have to unmark it as spam, but not too often. My IPv4 doesn't even have a custom rDNS -- only provider-specific one -- nor have I bothered to implement DKIM, although I do have SPF and am also registered for Webmaster Tools (although I somehow doubt that really matters).
I would maybe change the SPF record to fail instead of soft-fail (~all to -all).
I have a @gmail.com account that I use for testing whenever I change something.
It is the blacklist removal center. I removed my email address from this list and since then, everything seems to be working perfect.
What? How does e-mailing a company end up with a bounce from ... Google Groups? (See attached screenshot of the bounce.)
Maybe the original outbound message had something funny in the To: or Cc: recipient lists.
People tell you not to run your own mail server as it's a nightmare to keep on top of all the security aspects, and yes that's a thing. The greater problem is getting your mail delivered to the vast majority of people with gmail/hotmail/yahooo mail.
I'd rather have a few geeky people like me not have the fun of self hosting an e-mail server, than waste millions of man hours round the world because of people dealing with spam.
If you must hack, however, use something like Mailgun. It is more hacky in a way that you can program your incoming mail the way you want, not just install a mail server with a few commands.
Second, when a technology becomes extremely mainstream, it begins to generate federation and structure around it. It's called evolution. This is higher in services where stakes are higher, and where giving an individual excess power can screw up other individual's lives. It is the same reason you cannot install your own cell tower. You can start a tiny in-house telephony service, but you cannot complain if AT&T declines to integrate with it.
Third, the openness is a wild, wild west concept. If openness allows him to install mail server, it also allows Google to block it.
Fourth, you underestimate the amount of turtles the spam cave has. A lot of very smart people, leave mail servers on for years sending only a few emails before using it as a spam weapon. These servers can be rented for high price on the black market. If you were a measly C++ program, it is impossible for you to distinguish his good motives from a bad person. From what Google knows from 99% cases, this server will start sending spam any moment now.
Fifth, you underestimate the effect spam has. This guy suddenly starts sending out phishy Apple looking emails, asking people to change password. Next thing you know, Jennifer's pics (or your wife's, or your daughter's) are online.
We, as hackers, drink way too much Kool-Aid.
I don't think that's a good analogy. It misses the crucial point that the emails are being whitelisted by the receiver but still get blocked. A better analogy IMO would be that you invite me in to your living room for a chat, I say I'll bring the gun you wanted to see, you say that's fine. Then Google come along and muscle me away at your door because I have a gun. Then another day I ring and say can I come over for a chat, I say I have the gun you wanted to see, you say that's fine and that you've let Google know you want me to come over. Then Google meet me at the door and say I can't come in because I have a gun - I don't even get to knock, they don't ask you if you want to see me despite me having what you requested, they pay no attention to your request to them to let me in.
In case it's not clear what is happening outside the analogy is that I'm sending a message with content (the gun) that Google thinks is harmful. You want that content (to see the gun). You tell Google you want that content (whitelisting my email address, marking my email as not-spam) but they continue to block the email. What's worst is they block the email now without notifying and ignore the whitelist (they meet me at the door). You never get my emails that you want (never get to see the gun), because Google have made an erroneous assumption based on generalities and ignored the interventions of their customer.
Meanwhile - inside the analogy - a third party can bring a gun to your friends house but they happen to hire Google doorstaff themselves ...
It has the complexion of a protection racket: "Wouldn't it be a shame if you used some other supplier and your emails didn't get through, oh no. /s". Then as soon as the "you paid Google" flag is raised [eg by giving them access to all your emails] suddenly the exact same messages get through to the exact same people.
Google may be completely innocent but it stinks real bad.
It's like eliminating poverty by killing all poor people. Yes, it would work, but the side effects are worse than the outcome.
Your 'majority rule' analysis is flawed as the ability for individuals to access the internet without relying on [specific] mega corps or other major organisations serves the needs of the majority. The ultimate end of requiring people to satisfy a corporations demands before being allowed to communicate using the common means (email here) is anti-democratic, it gives too much power to those companies.