
Google backslides on federated instant messaging, on purpose? - cs702
http://www.fsf.org/blogs/sysadmin/google-backslides-on-federated-instant-messaging-on-purpose
======
agwa
> This change is akin to Google no longer accepting incoming e-mail for
> @gmail.com addresses from non-Google domains. That would be unthinkable.

The sad thing is that I no longer consider this unthinkable. There's an
extremely disconcerting email monoculture emerging around gmail. Practically
everyone I exchange email with uses gmail. Companies and universities are
switching to gmail. It can be difficult to get your mail accepted by gmail if
you run your own servers: I help administer the servers at one organization
whose mail (personal correspondence, _not_ mailing list posts or the like)
often gets sent straight to gmail's spam folder and we're doing _everything_
right in terms of DNS/SPF/DKIM/etc (in fact, the same config works great
elsewhere). Try to look for help on gmail's website and all you can find for
this problem are the "Bulk Sender Guidelines" - as if the only people who
aren't using gmail already are bulk senders.

Now consider Google's actions as of late. I could totally see them one day
saying, "we're not going to accept email from you unless we've emailed you
first or you've contacted us to ask for permission." There would be outrage,
but I also wonder how many people would _actually_ stop using gmail if they
did this.

Edit: I'm not saying this will happen, I'm just saying it's not unthinkable,
which is sad.

~~~
__david__
> It can be difficult to get your mail accepted by gmail if you run your own
> servers: I help administer the servers at one organization whose mail [...]
> often gets sent straight to gmail's spam folder...

As a counterpoint, I've been running my own mail server for 15 years (from
various hosted and dedicated servers) and I haven't ever had a problem
delivering to gmail...

~~~
agwa
Yeah, I run other mail servers as well, and none of them have problems
delivering to gmail. It's just this one place which has a problem, and there
is no way to make Google care about it. I fear it will only get worse as the
gmail monoculture spreads.

------
cs702
The "unintended side effect of filtering spam" explanation makes no sense.

How likely is it that the same company which builds bleeding-edge machine-
learning systems to track and predict our behavior online, and which uses
these AI predictions constantly to maximize their ad revenue, somehow cannot
find a better way to filter out spam invites?

How likely is it that the same company that houses the likes of Hinton, Norvig
and Kurzweil under the same roof can't find a better way?

Google is _packed_ with experts at solving the "spam filtering" (i.e., pattern
recognition) problem.

It appears this was done on purpose[1], driven by a corporate culture that no
longer cares as much about openness. _[Please read jholman's responses below.
He's right, I went too far with this last sentence.]_

\--

[1]
[http://mail.jabber.org/pipermail/operators/2013-February/001...](http://mail.jabber.org/pipermail/operators/2013-February/001571.html)

\--

Edits: added "it appears" at the end, to tone down the language. Also,
reworded and added sentences to make my point clearer, and corrected text to
refer to invites, not messages (thanks for pointing that out, mdc!) and point
out that this was indeed done on purpose.

~~~
onemorepassword
I have spam-filtered email addresses through 3 different email providers, one
of them being Google.

Only Google manages to constantly produce falls positives (including mail
from, sweet irony, Google services like Analytics) _and_ regularly allow spam
and phishing mails through.

The other two, ran by relatively small providers, are nearly perfect.

Don't overestimate Google.

(The same "intelligent" Google also seems to be unable to figure out which
language I use, despite me telling them on a regular basis.)

~~~
eridius
How many spammy emails do you get at each address? You seem to be measuring
the absolute rate of false negatives/positives, but not measuring the relative
rate.

~~~
onemorepassword
I have no numbers, but the non-Google addresses are much older (one dating
back to the mid-90's) and have been liberally strewn around the internet for
well over a decade.

They both get several times more spam than the much more recent business-only
Google address, yet if their filtering lets through one per month it's a lot.
I can't even remember the last false positive.

The majority of the mail that ends up in my Google spambox consists of
legitimate email from reputable sources (Amazon, Facebook, Google itself), and
barely any actually spam. Extra annoying: perfectly fine email from our own
services regularly gets flagged as spam by Google, and we often have no f-ing
clue why.

And don't get me started on Google Groups spam filter, which for some reason
is even worse. I have to turn it off for any group-address I want to make
accessible to non-members.

~~~
eridius
They may be much older, but if your Google address is @gmail then I guarantee
you it gets _far_ more spam, just from scattershot spammers.

As an anecdotal example, I have an email address that's been strewn about the
internet for almost 2 decades. It's currently hosted on Google Apps, but it
has a non-Google domain. I get a spam message in my inbox maybe once every
couple of months.

I also have a gmail address. I never use the thing. But it gets inundated with
spam, and every month or so when I go look at it I have to clean lots of junk
out of the inbox.

Since they're both hosted by Google, I'm forced to conclude that the gmail one
gets many orders of magnitude more spam merely by virtue of ending in
@gmail.com.

~~~
onemorepassword
It's not an @gmail.com address.

And why do people keep making excuses for Google?

Google simply isn't very good at filtering spam, something most regular ISP's
can handle perfectly well, and the lack of options in Gmail and Groups clearly
show that they don't care very much about it either.

~~~
polyfractal
Fun fact, Google doesn't play nicely with ESPs either. Most mail providers
(Yahoo, Hotmail, etc) use a feedback loop when you report spam. After you mash
the spam button, the ESP that sent the mail is notified that a particular
email was flagged as spam.

This allows the ESP to curtail spam problems on their end (for example,
Mailchimp heavily throttles your emails or outright bans you if your spam rate
creeps past a very low percentage). It's an all-around good thing for the
ecosystem, with the exception perhaps of publishers that get the unlucky "spam
instead of unsubscribe" user action.

But Gmail does not participate in this loop. They don't tell any ESP that a
user has marked an email as spam...that data all stays in house. Why? Hell if
I know - perhaps they don't want to tip off spammers to being detected. On the
flipside, reputable ESPs get less leverage on spammers in their network.

~~~
Thing_Two
because it would allow spam houses to train their software to avoid the Google
spam filter. No feedback loop makes it harder to train (not impossible, just
harder)

------
zeen
I'm one of the authors of the Prosody XMPP server, and a member of the XMPP
software foundation. Prosody operators have been reporting this for more than
a week now.

Google users have apparently been flooded with subscription requests from
spammers, and the flooding suddenly became massive. The problem is, there are
a large number of jabber servers out there which have open account
registration without captchas. Most jabber server software doesn't come with a
captcha module included by default, and of course, most admins don't bother
changing defaults, even while running a public server with open registration.

Unlike some other comments here, I don't think Google has any malicious intent
in this. This seems like a stop-gap measure, while they figure out and
implement a proper solution.

As to the proper solution, the XMPP community is largely moving towards having
captchas, or other forms of verification, and there are a number of proposed
standards.

The thing to understand here is that the XMPP community has historically not
had a spam problem. Due to the nature of the protocol, spoofing wasn't
possible from the start, and there were no large lists of JIDs for spammers to
abuse, so things worked out fine for a decade despite a lack of captchas. The
good news is that the XSF was already preemptively working on the spam
problem, and the speed with which XMPP specs (XEPs) get defined, implemented
and deployed in servers and clients is far faster than any other large scale
open protocol that I'm aware of.

------
mmanfrin
I think I might be a minority here, but I think this is an okay move. I use
gchat exclusively with people within my workplace (also on gchat) and friends
who also work at places with Google apps.

However, I've been getting a consistent barrage of requests from spam email
accounts to chat; accounts with obscene names like
'sweety+69+for+free@freemail.ru' or something to the sort. The names/domains
of each are different every time, so I can't simply block a domains.

I ask: if Google does not block outside requests, what could they do to stop
this sort of thing?

~~~
mindcrime
Great, do you want them to block incoming emails from any domain other than
your own, as well?

~~~
zdw
Obviously not. But anti-spam measures are decades old at this point and well
known.

The argument is a "the technology isn't there yet and spammers are running
rampant, so the obvious solution is block it, which isn't ideal", not "Google
is doing this because they want a monopoly on chat, and broke the protocol to
do it".

~~~
mindcrime
Personally, I'm not (necessarily) suggesting that "Google is doing this
because they want a monopoly on chat, and broke the protocol to do it". But it
seems to me that as much experience as Google engineers have dealing with spam
in the email world, they could come up with something feasible on XMPP as
well.

Doing this strikes me as throwing the baby out with the bathwater.

~~~
cjbprime
> it seems to me that as much experience as Google engineers have dealing with
> spam in the email world, they could come up with something feasible on XMPP
> as well.

But before they start on this research project, they might want to do
something about the spam their users are getting now, right?

~~~
mindcrime
By breaking something fundamental about how XMPP chat works? Personally, I
vote "no".

~~~
enjo
I for one vote "yes". I was starting to receive lots of spam requests from
within google talk. It was really annoying. Whatever they did it stopped and
I'm glad for it.

~~~
mindcrime
Great, you lose one minor annoyance, and the rest of us deal with the fallout
of a broken Internet. Sounds like a great tradeoff to me.

What is it with modern times, where people are willing to sacrifice
fundamental things, like applications that adhere to standard protocols, to
gain some minute level of relief from something that's just "annoying"?

~~~
jamespo
That is the whole reason people use google, it's convenient.

They're not stopping anyone running a personal ejabberd server.

~~~
slashclee
They _are_ effectively stopping people from running personal ejabberd servers.
What the hell is the point of running your own jabber server if your users
can't talk to users on the largest jabber network on the planet?

------
RyanZAG
Obvious fix here: only allow chat invites from people you have sent an email
to. Google could return an error to blocked invites to inform them that they
must get the recipient to send them an email first.

I can't see any downsides and it definitely solves the spam problem. If they
could make it apply to other gmail accounts as well, even better. I've gotten
annoying gchat requests from sweety69@gmail.com as well.

~~~
guard-of-terra
Not all Jabber services provide e-mail. How would you write an email from one
that doesn't?

~~~
jrockway
"Great is the enemy of good."

~~~
codemac
Most people on federated servers are specifically on servers like xmpp.us,
jabber.org, etc that _don't_ have associated email accounts.

This isn't perfect being the enemy of the good, it's a description of why it's
not even good.

------
rachelbythebay
I was very disappointed when I found out they don't use TLS when federating
XMPP. Basically, if you're chatting with someone who uses them for XMPP
(Jabber) service, your stuff is riding in the clear between the two servers.

It's been like this for years.

~~~
mike-cardwell
People should use OTR anyway. That way not even Google can see your messages.

~~~
pseut
How many gmail chat users use an external IM client?

~~~
mike-cardwell
I don't know. Most of them?

~~~
pseut
I was thinking almost none of them.

~~~
mike-cardwell
I reckon nearly all of them. Pointless this sort of speculation isn't it?

------
speeder
Every time someone break stuff to "fight spam" I can only conclude it is one
of two things: on purpose unethical behavior, or incompetence.

~~~
jxf
Kind of like how we break a lot of our legal system to "fight child
pornography". Yes, they're both bad things, but fighting them cannot come at
the cost of breaking otherwise perfectly legitimate and functional endeavors.

~~~
speeder
This might be also lots of overreaction.

I've met two very intelligent women (That don't know each other) that for most
things have very reasonable opinions.

But throw "child abuse" and it is like telling a robot to hide in a corner in
a round room, they just break and if they could they would pass laws about
instantly killing suspects of child abuse.

When I try to argue with them of how "x" or "y" is bad idea because of its
side effects, they always reply: "I don't care, EVERYTHING is worth doing to
protect children."

And then I understand why so much politicians use "for the children" rhetoric
when they want something.

~~~
guard-of-terra
Maybe they're compensating for not having children they want to have or not
spending enough time with their own children.

They redirect their frustration by wanting to punish "child abusers" when the
source of this fictional abuse is in their head.

I guess that's what happen. The room is round, but their upbringing tells them
to stand in a corner.

~~~
ceejayoz
Maybe they're space aliens, while we're wildly speculating.

~~~
guard-of-terra
Do you prefer "they're stupid and dangerous" answer? Because it's either this
or that.

~~~
ceejayoz
Well, that's a false dichotomy if I ever saw one.

I prefer "well intentioned but misguided", "worked up by a sensationalist
media that obscures the fact that kids are safer than in the 'idyllic' 1950s",
etc. theories.

~~~
guard-of-terra
"well intentioned but misguided" equals "stupid and dangerous" once you shave
off the bullshit.

At least in this case. It's a textbook case, exactly like the one in South
Park Uncut.

------
fluidcruft
Ha! I think Google must have realized that over the last two days I have
deleted my Google+ and youtube accounts, shut off/deleted blogger, purged
Chrome from all my desktops, laptops and phones and registered for an
@jabber.org XMPP address. And I'm not the only one, my friends are all on
board with excising the hydra. Still hesitating about the effort needed to
migrated from gmail, though. Time to organize exodus parties.

~~~
icebraining
It's not really that hard. Backing up is relatively easy (offlineimap or
similar) and you just need to forward the Gmail account to your new one for a
transitional period, while you gradually change your address.

------
mindcrime
Wow, not a great week for Google here. Their reputation has certainly taken a
big "ding" in my eyes, between this, Reader and CalDAV.

~~~
bentcorner
I'm not saying this is going to happen at all, but I wonder what would happen
if Google had to end up putting all its horses behind automated cars and Glass
(because of changes to the search marketplace, of which these changes we're
seeing are just the start of a response to). In 20 years we'll be driving
Google Cars and we'd tell our grandkids "You know, Google used to do email" to
their great surprise.

In real history, this is like telling kids that Nintendo used to make Japanese
playing cards.

~~~
patrickaljord
> In real history, this is like telling kids that Nintendo used to make
> Japanese playing cards.

Not really, Nintendo cards weren't used by most countries on earth minus China
and didn't revolutionized gaming world widely the way google did with
organizing knowledge.

~~~
mh-
perhaps their cards didn't, but NES certainly did [revolutionize gaming].

------
zdw
There are legitimate ways that follow the letter of the RFC to prevent spam
over SMTP (greylisting, etc.), or to overlay sender/server validation on top
of it (SPF, etc.), but most of these lessons aren't cross applicable to XMPP

Sure, Google could spend a lot of time trying to come up with a technological
solution that doesn't break federation (except for in spam cases), but it
would be difficult to do on a service that fundamentally doesn't make money.

This sounds like blog trackback all over again - useful, a nice idea, but
nearly worthless once spammers figure out how to pee in the communal pool.

------
dn7dt3qmpldi
“This change is akin to Google no longer accepting incoming e-mail for
@gmail.com addresses from non-Google domains”.

No, it’s not. It’s this kind of constant exaggerated claims that give a bad
reputation to people that speak for free software, and make them look like
out‐of‐touch conspiracy theorists (think Stallman).

Now, I’m not saying I disagree with the message as a whole; _it is_ bad that
google closes the door to this kind of interaction, but it’s nowhere close to
what they claim (and in bold). With their example, you’d effectively only be
able to speak to the person on the other side, but never receive their
messages. This case is very different, as only the _initial_ contact is
unilateral (the person with the gmail account has to invite the other one),
but after that, it works just as well.

It’s this kind of stupid exaggerated argument that drives people away from
your message.

------
morsch
So what's the best alternative for mobile XMPP messaging? I've tried a couple
of the popular ones, and none of seemed to work as well as GTalk (reliable
delivery with no effect on battery life). I assume that GTalk is only a slim
client, and the XMPP stuff happens server side, which aids in that. I suppose
there are other services that work the same way. Anything self hosted?

Also, not having OTR messaging (ie. the socialist millionaire protocol, not
Google's private mode) just seems stupid in this day and age. And so does not
using TLS for federated messages, if what I've just read in another comment is
true. It really is time to switch.

~~~
mh-
GTalk does not have "no effect on battery life".

I'm assuming you're on an android device (you didn't say) -- all¹ Android
devices are "always" connected to GTalk already. GCM (C2DM) sends all push
notifications/wakeup requests over this channel, and therefore the framework
ensures the socket remains open even when sleeping.

 _¹ - ones with Google Services_

~~~
morsch
Okay. TANSTAAFL. Regardless, in my experience, apps like Jabiru tended to
drain the battery noticeably faster while at the same time being less
resistant to outages when interrupted briefly (elevator, subway, etc.).

------
Zash
They have been doing this since at least the beginning of March. And the
blocking is also done in a way that causes the user to not be notified,
basically by replying in the wrong direction.

[http://mail.jabber.org/pipermail/operators/2013-March/001610...](http://mail.jabber.org/pipermail/operators/2013-March/001610.html)

------
cbhl
While this is unfortunate, I don't think it's particularly surprising.

For example, Facebook allows accessing their Chat servers using XMPP, but they
don't support federation _at all_.

------
bgruber
while this does sound really horrible, the pragmatic part of me is forced to
agree with google's decision here. Other gtalk users undoubtedly account for
the lion's share of incoming requests, while non-gtalk requests undoubtedly
account for the lion's share of spam. it's much, much easier to whitelist the
few 'good' jabber providers than to filter the spam.

------
diminoten
> We hope that Google will retract this change and find a solution that does
> not undermine the distributed nature of the Internet.

I'm sick of this kind of language, why does EVERYONE who does advocacy have to
speak like this? Can no one be reasonable?

~~~
icebraining
What exactly is your issue with it?

~~~
diminoten
It's too grandiose a statement, considering the act.

~~~
sergiosgc
A distributed Internet based on open protocols is grandiose now??? Are we that
far down the rabbit hole?

~~~
diminoten
No, the idea that Google making this specific action is "DESTROYING" a
distributed Internet is grandiose.

~~~
icebraining
They didn't say destroy, they said "undermine", which means "To weaken or work
against", which this action certainly does.

~~~
diminoten
Same problem, and no it doesn't.

------
ChrisCinelli
Frankly, after I received some spamming bot services to contact me, I do not
think that the Google solutions is that bad. You can request to be white
listed if you are a legit site. This increases the quality of gmail's user
service and still let other legit player to access gmail's users. As a gmail
user, I think what they did was good and if somebody has a better idea...
please... I am waiting =)

------
Nux
I can see a pattern here. :-)

Again: people, run your own shit. Stop playing the google roulette.

~~~
ebiester
I have 15 minutes a month to run my own shit. What solution do you have?

No, really. I would rather spend that five hours making sure shit works with
my boyfriend, or my family, or my friends, or learning something that will
make me money, or working on a pretty cool hack I thought of while in the
museum last week, or working on my unfinished novel that has been languishing,
or work on proofreading a paper for a friend, or just relax.

Yeah, system administration is under those priorities.

I am more than willing to pay. I just want to pay one person for everything. I
want that integration. I want webmail. I want to be able to access it from my
phone without issue. I _like_ integration.

I may be a geek, but I have better things to do than figure out why my email
isn't getting to a friend because someone's IP address ended up on spamhaus.

~~~
Nux
Ebiester: if you want a decent email platform with webmail, calendaring and
maybe xmpp, have a look at Zimbra. It's super easy to install, all you need is
a VPS with Centos 6 or Ubuntu 12.04 and a static IP address, then download and
execute the installer.

I understand it's not so easy to run your own stuff, but it's also not as hard
as some people believe, especially if all your need is a small setup for
personal use.

I'll try to wrap up some tutorials in the near future to help people run stuff
on their own. I'll remember to ping you when done.

~~~
ebiester
So... what do I do if an IP block gets on Spamhaus? How about two factor
authentication? (DnaBolt's Legitimi is minimum 1000 users)

That's not to mention that I'm seeing 1GB RAM minimum to run the thing. That's
a lot of wasted resources for one person.

I can run my own email server. I just have better things to do with my time.

~~~
Nux
Then why did you even comment if you are not willing to do something?

Whatever you do you must pay, with time, money, your privacy or control; I see
you've made your choice.

~~~
jamie_ca
Yes. ebiester even spelled it out:

> I am more than willing to pay. I just want to pay one person for everything.
> I want that integration. I want webmail. I want to be able to access it from
> my phone without issue.

I'm in the same boat - currently looking for somewhere to move stuff to. I'm
giving atmail.com serious consideration for mail/calendar/contacts, as the
first is reasonably easy and the latter two _seem_ to be working on desktop,
android, and ipad. They don't do XMPP though, so I need to figure out a
solution for that.

~~~
Nux
Roger that.

I would still recommend Zimbra or Zarafa, as far as open source goes. Or if
you can't be bothered with running your own, both products list a series of
partners who do this for you.

------
kushti
Another fail of Evil Corporation. However, we need for a new generation of
Internet Jedis to smash the Dark Empire of Darth Serge and Darth Larry

------
jfoster
Not sure about the headline, but the tone of this message from the FSF
suggests a maturing of their efforts. Notably missing is a lack of the word
"evil" or strongly worded demands. It makes them far more credible, I think.

------
Semaphor
Good thing I never gave up my @jabber.org address… To think I recently held
them up as properly working with the XMPP consortium to add official features
unlike facebook.

------
mwcampbell
For those who would like to move away from Gmail+GTalk but don't want the
hassle of running a VPS, Dreamhost offers XMPP as part of its web hosting
service.

------
cooldeal
Interesting Google week so far.

1) Discontinuing Google reader

2) Retiring CalDAV support

3) Removing ad blocking apps from the Play Store

4) Replacing Android chief Andy Rubin with Sundar Pichai.

5) Dropping support for sending chat invites to Google chat users from other
domains.

Wonder what else is up for the next couple of days.

~~~
InclinedPlane
It's not surprising, google has been on the road to becoming the next
Microsoft for a while. Though it is depressing.

It's interesting though, so much of google's core business relies on trust,
and they've been making a lot of moves lately to just throw away as much trust
as possible. Maybe it's sustainable, but I'm skeptical.

~~~
mehrzad
Well, it seems to be creating growing support for Mozilla and the FSF, which I
think/hope we can get behind.

A bunch of people disagree with the FSF though.

~~~
mh-
Google is far and away the largest source of funding-revenue for the Mozilla
Foundation.

~~~
eloisant
Google is not subsidizing Mozilla, they have a search contract that is as much
in the interest of Google as Mozilla.

* Mozilla can switch to a different default search engine (like Bing) and still get as much revenue without depending on Google.

* Google can't act like it controls Mozilla, because they don't want to lose the partnership either

~~~
takluyver
Is there any evidence that what Google pays is the market rate? I.e. if they
cut ties, would Mozilla be able to get about the same amount for default
search engine position from Microsoft?

Now that Yahoo is powered by Bing, there are only really two big multinational
commercial search engines. Both of them are owned by companies that also offer
browsers to compete with Firefox. If neither of them paid, Firefox would still
have to include a default search engine. I hope Mozilla can keep going, but
honestly it does seem rather dependent on Google's good will.

------
andyl
"Don't be evil"

et tu, Google?

~~~
camus
At least Microsoft folks do not pretend they are not evil , Google is even
more devilish when they claim they are not evil ... Well they are at least as
bad in PR as Adobe , that's a relief.

