Not just failed. Google betrayed XMPP. You can blame Eric Schmidt for it. He gave some lame speech about how other instant messaging services aren't playing fair, so Google should also become a walled garden. And that was it. They made Hangouts and stopped caring about federated IM idea.
It's a wonder we can send e-mails between many servers. Imagine someone like Eric Schmidt driving it. We'd be stuck with incompatible AOL and Compuserve forever.
And about XMPP shortcomings - sure, it's not perfect. And if Google thought they can do better, why didn't they propose some IM-next as an open federated or P2P protocol? Because "don't be evil" is off the table I suppose.
People tried proposing to use Discord for me, but I'm really not interested in another walled garden closed protocol, without FOSS clients and servers.
Microsoft's one-way integration was them not exposing presence and typing notifications to services they federated with (but they were "taking" that data from others).
Larry Page says there he would like to see more open standards used, and he is sad the industry isn't moving in that direction. So why isn't any of their new IM technology an open standard?
Because being constrained by existing standards, rather than being free to make the changes to the protocol that you think will make the product better, is a huge trade-off. The benefit is that your users get that interoperability with e.g. MSN Messenger or Skype or whatever.
However, if no one else is doing that, if no one else is playing fair (like Microsoft accepting things like typing or presence notifications but not sending them in return), then it turns out you're being constrained, and possibly doing additional work to add features, and the person benefitting the most is someone else's users and not your own.
I really doubt that Google were constrained for technical reasons. XMPP is completely extensible. I've worked with it for over a decade and there's absolutely no technical reason why Google couldn't have extended the XMPP specification and put out their own XEPs (XMPP Extension Protocol) which would have given them all the functionality and features they needed.
XMPP's only major downside at the time was mobile battery life due to XMPPs limitations over cellular. This could easily have been resolved by using a non-XMPP accepted handshake (as WhatsApp did) or by using a completely different protocol for mobile as Facebook did for mobile (MQTT).
Google decided to go for a walled garden approach as a competitive advantage. There's no technical reason for it.
Because their old one was an open standard, and they were effectively the only one open one (and the last open one standing). So I find it totally reasonable that they stopped caring about it (and probably would again if another major player did).
I disagree. I can understand why you think so, but an analysis of the situation will point out something different.
Google got talk working almost by accident. And it was the most feature-complete chat app for years, at least. And then, they abandoned it for years. This isn't to say it had many features, it had few. But it worked well, was integrated with gmail, was compatible with XMPP and geek's IMs, ...
After that, they killed all the compatibility before making an app that was yet again a lot more feature complete : hangouts. And hangouts got the best features for quite a while.
And then Hangouts started wildly bashing it's weight around. Forcibly taking SMS was one thing. Refusing federation. Forcing Google+ account and lots of extra info. And so on, and so forth. Hangouts was a good app, but created a lot of ill will in the process.
And then it was seemingly abandoned. Feature frozen, with the excuse that all these features had resulted in an extremely difficult to maintain app that they couldn't add features to. Chatting without having a gmail being a big one (whatsapp allowing you to chat with "everyone in your phonebook"), status, reliability (and showing clearly and timely when it isn't/cannot be reliable because disconnected, not 3h after connectivity gets restored), video call quality, adding non-gmail users or just screens to video chats, not allowing bot interaction, apps in the video chat, ...). And this lasted for years.
Inside of China, some chat apps demonstrate how chat apps can be monetized in a way that users appreciate : allow chatting with companies and give those companies the ability to show interfaces for transactions. E.g. buy a coffee. Line at starbucks ? Open up whatsapp, because you're in starbucks it shows starbucks as a contact (you can add it permanently) and there's a button "order coffee". Select what kind of buvaranicpoppafrappadongieccino you want, you pay through the app, and you've just skipped the queue. Next time you do it 5 minutes before arriving. App takes cut of transaction value (just like credit card payment does).
Next, other apps turn up. Whatsapp, Lyne, ... and so forth. And they caught up with Hangouts. Surpassing it in some ways, behind in others. Mostly they're superior in letting you find the people you can chat with. And then they passed it by. And then they left it far, far behind. It's not (yet) the case that they truly dominate hangouts in features, but it's getting close.
And lo and behold : people switched to the (sorry) better apps. I'm not entirely sure why anybody is either surprised or complaining. These changes look a lot like they're making it worse, not better, but we should give them the benefit of the doubt.
Microsoft does support full federation. As long as you federate Skype servers. I know that's a bit lame, but since they bundle most of their products with most of their other products in licensing, no one in business seems to care.
Google doesn't do even that, as they don't offer anything on-site (anymore). They have seriously failed on business markets. I don't know about consumer market, because I mostly just send dick pics for the giggles to random people I don't know.
That's the big thing about SaaS, mobile and Cloud.
I don't get that people complain about Microsoft software being closed, and anti-competitive ... well any cloud software is more closed and anticompetitive than Microsoft ever was on it's worst days : it's not that you can't get good documentation that the application you're using uses ... you can't even get at the data at all, in most cases.
Google is the best at this out of all the cloud players, but it's still abysmal. And yes, you can download exports. And there's Google takeout. Is the data there all the data Google apps use to work ? No. Quite simply, no, it isn't. There's plenty more they won't ever give to you. And that's ignoring the fact that there's 1000s of developers making apps for Google and about 5 working on takeout. Why should Google be forced ? Well, that's the standard we're applying to microsoft, isn't it ?
And this is by far the most flexible cloud provider exporting data. They're not being jerks. I don't even think they're trying to be anticompetitive. Nevertheless, they're far more anticompetitive than Microsoft.
And of course, conveniently, the DRM-breaking is now a crime on things like iTunes and Hulu and ... because it involves actually creating access into someone else's computer system or at least changing their code (even though you're paying for it to do things and only getting it to let you exercise rights granted to you by law, like copying a movie). And the result is the same. iTunes, Hulu, even Play are ridiculously expensive, idiotically restrictive (buy movie on iTunes for a trip, go to Europe, log in to hotel wifi ... no movies. But no worries ! I copied that movie onto my device first. Nope, it won't play. WTF).
Same thing goes for app stores. There Apple is the big instigator of forced-incompatibility (or otherwise: your app won't run without Apple's permission hardware). Remember the flood of articles scaring people that microsoft might do this ? They're still at it, with the BIOSes that supposedly won't run anything but windows.
Perhaps this is the point we need legislation to require that IM services with above X users in the relevant jurisdiction must provide a full, open, protocol and enable interoperation via the published protocol? That's the only way I see the companies involved would move to enable cross-platform communications?
Aside:
>It's a wonder we can send e-mails between many servers.
I have a shared account on a small ISP, Microsoft won't receive our emails, even when replying to an email from a Microsoft user, even when user whitelisted. [It's probably the same with some other suppliers?] The only way to reply to people using Outlook/Hotmail/Live (I'm a user of hotmail for something like 16 years) is now to send via MS servers - basically had to make an account for this purpose.
The sending IPs aren't blacklisted with Senderscore (or other blacklists, I mention senderscore because they apparently provide domain reputation services for MS); some "related IPs" have medium reputation scores (senderscore 72).
If I send a new person an e-mail, I have to contact them on Facebook and tell them to check their spam folder. Fuck Google and Microsoft and their terrible e-mail filtering. Just because people are so dumb they will open any ransomware sent to them doesn't mean you drop all e-mail you don't trust with no way to account for it.
How long have you been sending mail from your latest IP Address?
I run some websites that have their own mail servers. One thing you didn't mention which can help is to be sure to use an email signature. Basically short emails are more likely to be flagged as spam, especially if there's an attachment.
In use 12 years, only ever used to reply (except when messaging businesses), never done mailings. Of course there is a footer with bricks & mortar address, phone, unsubscribe link.
Had problems with hotmail before, it's just associated IPs being factored in too strongly. Same ISP has an IP address that spam has been sent from; support responds with won't fix. In their favour at least they responded.
Domain has SPF but not DKIM.
Thankfully no other mailbox provider blocks us. We could pay to be added to the third-party "friendly" domains list of course ...
I'm surprised they wouldn't whitelist our domain but leave it on a tight leash.
From the fine article he linked: SPF, DKIM and DMARC are all domain verification systems for validating e-mail’s origin to prevent spam. I have all three records set in DNS records for all the domains I send e-mail from, verified they were correct using testing tools, and I still get flagged as spam.
I'm not a big fan of legislating technical standards, but in this case I think I could support it. IM has become a fundamental communication mechanism. Imagine how held back society would be if we had 50 different voice standards rather than the singular phone system we have now. IM needs to be forcefully standardized.
Korea mandated through legislation that all online shopping had to go through ActiveX controls, because that was the only way to guarantee secure encryption at the time. Now it's practically the least secure way to online shop and even Microsoft recommends not using ActiveX, but it's a huge amount of work to strip that infrastructure out and make everything work 'properly' so it's still an issue even today.
In short, legislating a solution is a workaround that can turn out to be a huge detriment not too long down the road. I'm no libertarian, but it seems like a bad idea to me.
That's a good point - especially important if someone's thinking about legislating a particular technology. On the other hand, maybe in this case it would be enough to legislate a requirement for openness of the protocol? I.e. whatever it is, if it has enough many users, it has to be fully available to third party integrations.
What about SMS? Everyone has it, it works, and is usually fairly instant. It's not very feature rich, but could there be an SMS 2.0 protocol that was backward compatible?
Except we still do have many different standards for the phone system. As an example the main codecs used for encoding voice today are μ-law (North America), A-law (Europe), GSM (2G/3G), AMR (3G/4G), iLBC (VoIP), G.729 (VoIP).
That is just for voice, now imagine you are a phone company and want to send a call to another phone company. There must be so many different things you need to cooperate on to make it happen - billing, switching, physical connections, numbering, codecs, etc. Yes the actual protocols may be standardised, but the main part is cooperation.
It's not standards we need, it's companies that are willing to cooperate and work together. None of the big tech companies want to do that now though. Everyone wants their own walled garden so they can say they have more users that X competitor.
(I'd argue that XMPP is good enough, or with a few extensions could be made good enough, for a company who really wants to push open IM)
It's not something needed. It's a plus, even if it influences society a lot, it's not something like food or healthcare, where if someone plays unfair people die.
And it's not like net neutrality either, because you are not costrained by space (putting cables underground).
People should be free to choose a service, and if the server want to stop providing the service or change the way it does ut, it should be free to do so. Anything else you can use contracts, or change provider, as long as there is net neutrality in place. And it's not something that messes with your health, so no justification for regulation aside from an arbitrary and unprovable "society being held back". Being unable to make something because of regulations, that's some real and provable holding back of innovation.
Don't we already have GSM and LTE, and who knows what else? And I'm not aware of any legislation for the use of those over others. As far as I know, as long as I have the right to use the frequencies, I could start my own cellular telco that transmits whatever format I want.
Spectrum is owned by the public, and regulated as spectrum is a limited resource. IP messaging isn't by any practical means a limited resource.
> As far as I know, as long as I have the right to use the frequencies, I could start my own cellular telco that transmits whatever format I want.
No. Spectrum is regulated in terms of what technology you can use, and kind of usage is supported. You can't buy LTE spectrum and broadcast radio on top of it, and vice versa. You can't (in most cases) take 2G spectrum and deploy LTE on top of it.
Also, bandwidth is a function of how much spectrum is available (for example between 1900MHz and 2000MHz there is a chunk of 100MHz) and the spectral efficiency (for example, 1 to 2 bits per Hz available).
For example:
If a carrier "owns" the block between 1900MHz and 1940MHz, and let's say LTE supports 2 bits/Hz using Frequency Division Duplexing (each direction, up and down, gets a chunk of spectrum), with 3 sectors per cellular tower, the carrier can support a total of 40x2x3 = 240Mbps per tower, or 120Mbps each way.
If the tower serves 1000 users (they serve far more in dense urban areas), each user will have 120kbps of capacity, and if they all use it at the same time, that's the speed they'll get. If there's a single user, the maximum down speed for that user would be 120/3 = 40Mbps.
> As far as I know, as long as I have the right to use the frequencies, I could start my own cellular telco that transmits whatever format I want.
Point I ("I have the right to use the frequencies") directly depends on point II ("whatever format I want"). When applying for using a certain frequency band, you have to specify usage and purpose, and you can't just change your mind, i.e. you can't apply for GSM frequencies and say you will be running a GSM network, but then run the PavelLishinOverAir protocol.
>When applying for using a certain frequency band, you have to specify usage and purpose, and you can't just change your mind, i.e. you can't apply for GSM frequencies and say you will be running a GSM network, but then run the PavelLishinOverAir protocol.
While this is mostly true, in practice (at least on the 450-470 MHz band I work with), you can give fairly vague purposes, like "telemetry" or "SCADA", and it doesn't matter what vendor or what protocol you use. In fact, there is no section of the licence that says "I am going to use protocol X on equipment Y manufactured by vendor Z" or any part thereof (emissions designators are a different story).
I don't have any experience will cellular licences, though. Given that carriers frequently repurpose spectrum as technology changes, I can't imagine their licences say "GSM only" or anything like that. The band that used to be 3G GSM is being used for 4G LTE on my phone right now.
This likely differs a lot between countries (though there should be a common core due to ITU of what is being regulated how by whom); in my country the cellular frequencies were auctioned off for dozens of billions of euros to the cellular service providers, with strings attached, e.g. that the networks had to at least use and support 3G.
The same regulatory body (BNetzA) is also responsible for licensing telecommunications providers in general. This is probably the crunchpoint - if you're not a licensed telecommunication provider, it's simply illegal to offer telecommunication services.
Right, the point is that you don't have the right to use whatever frequencies you want because the usage of the frequency spectrum is highly regulated, and for good reasons.
China has effectively mandated IPv6, and IPv6 certainly is popular in China (I think all mobile phones in PRC use IPv6, I may be wrong) - but let's not forget that China is promoting IPv6 because APNIC's allocated IPv4 space for China was comically undersized.
Unfortunately this is not the fact. While some ISPs are experimenting with IPv6, the vast majority are not on that train. Indeed, IPv4 address space has practically run out in China already, but many ISPs are taking the nasty NAT-like approaches, Carrier-grade NAT [1] for example. And it is particularly common in mobile. So far, IPv6 is only (relatively) well accepted among universities (CERNET [2]).
The Great Firewall likely affects the reliability of everybody's stats, but China seems to be way behind the US in terms of IPv6 deployment. (<3% versus 30%):
You do realize how firewalls work, right? You have iptables and ip6tables. Sure you need entries for both, and sometimes ISPs forget, but it's easily implementable in all major routers and operating systems.
Firewall rules are not yet implementated on ipv6 network in China by ISP. It is a fact, and actually people are using ipv6 network as a means of bypassing the wall.
I don't think it needs legislation as such but could be worked from a licensing point of view.
Everyone moans about AGPL. Fine. Make something better that says if you're going to interoperate with a service, you better damn well actually interoperate. Make the license viral like GPL and hopefully when we all push in that direction, everyone needs to join up.
No, it's not. What you're proposing would be like the government saying you can only use this specific service, which may be run by the government, to communicate with others. Standardizing IM is making is so that if you want to talk to your friend across the country you're able to using different applications but the same protocol. Forcefully standardized would be mandating any messaging app that's used in the country would have to follow the protocol. It's almost the same thing that happened to fax machines in the 70s and 80s before the Japanese government stepped in.
Keep it simple. All protocols for communication and data formats for storage must be published as open source or public domain. This is to prevent or reduce lockin, walled gardens, etc. No other change. They can still compete in numerous other ways.
Although this is an appealing idea, I don't think it is realistic. Lobbying would destroy any legislation almost immediately, and there would still be the issue of competing standards.
I think a more achievable version would be that any online service which maintains user data must provide a method for a user to retrieve that data in an open-source format. Some companies would deliberately obfuscate formats, such as calendar appointments as JSON rather than iCAL or another existing standard, but I think it would give users more control than they currently have.
"Lobbying would destroy any legislation almost immediately, and there would still be the issue of competing standards."
Oh, I agree. You're describing the effect of a corrupt government allowed to prevent or eliminate laws meant to protect its voters. I was talking about what law would be necessary if voters cared enough to push one.
And the result is OOXML, which is well known for being a trainwreck and which nobody - including Microsoft - has been able to implement properly, despite being an "open standard".
The result of open-sourcing all data formats and protocols would obviously be all the data formats and protocols released as open-source. OOXML is not the only one in existence.
Which would be obvious, flout the law, and allow for legal action to be taken.
The company itself has to use the format for live message exchange, have to publish a full working protocol document.
With OOXML Microsoft aren't required to be able to read & write a files correctly when it's written to the protocol. As I envisage it the requirement would be to open the protocol actually used "on the wire"; again with Word Microsoft don't use OOXML as primary.
I think it's workable except perhaps in requiring political will. It's an internet age equivalent of allowing third-party car parts manufacture.
Will it be obvious? How? Browsers often fail to implement open standards like CSS, even when they have nothing to gain, it's just that shit's hard.
Microsoft don't use OOXML as primary.
Sure they do, that's what .docx files are. They've been the default since Office 2007.
Microsoft was compelled to create and implement the standard so that governments can comply with their requirements to use "open standards". And they followed all the steps, including getting it past ISO, despite the mess that the format is. And so governments can now tick the "open formats" box when purchasing Office. It's all a sham, and I don't see how it'd be otherwise with this proposed law.
Oh yeah. They'll probably try that. That will either be defeated with tons of sweat as it largely was in products like OpenOffice or the legal recommendation modified to attempt to deal with it. It might also be handled in the courts if the law includes language that specifically instructs to attempt simple, readable protocols to "facilitate interoperability." Obfuscations could be argued as the opposite of that in court.
That will either be defeated with tons of sweat as it largely was in products like OpenOffice
Then why do you need open standards? Every major proprietary IM protocol has been re-implemented by open source software, including Hangouts: https://bitbucket.org/EionRobb/purple-hangouts
It might also be handled in the courts if the law includes language that specifically instructs to attempt simple, readable protocols to "facilitate interoperability.
XMPP, the thing Google is being roasted for dropping support for, is often decried as being very complex and hard to implement correctly. If genuinely open protocols often end up like this, how can you possibly punish companies for writing complex, hard to implement protocols?
"This is to prevent or reduce lockin, walled gardens, etc."
What I said. I'll add that it's hard to copyright API's to sue over compatibility when the API's and their implementation are released as open-source software.
"If genuinely open protocols often end up like this"
What are you talking about? There's all kinds of results from protocol design. You chose one of the worst ones. I was a PSYC fan over that if I had to choose an IRC replacement or open chat standard. A test for new ones might be trying to run it through LANGSEC's Nail, Cap n Proto or ZeroMQ to see if they handle it easily. Anyway, it's easier to do compatible implementations if you have source and can't be sued/imprisoned for imitating it.
i kinda doubt anyone sees protocol violation as a criminal offense.
it's wild to me how quickly "we should find a market-external pressure source for forcing interoperability given the profit motive seemingly guarantees lack of federation because it relies on competition as its motive force" becomes "we must frogmarch innovators to the death pits"
The correct way to solve this problem is through raising awareness of privacy issues and supporting efforts to create competitive and attractive open source communication protocols, not legislation enforced by a prison/military industrial complex. What you're suggesting is to hand the State further control over our choice in communications platforms and trust them to never abuse it.
And what if they don't publish the protocol? Do we block downloads of their client?
Even if we assume they are American companies, and we can sue them if they violate the rule; how on EARTH would we define an IM protocol broadly enough to avoid an easy avoidance of the law, but narrowly enough to not basically block all proprietary protocols?
If a company doesn't publish the protocol in use within, let's say 3 months of a valid request by a third-party then they'd be liable to a fine, perhaps something like $1 per day per registered user, plus costs of government and registered third parties in pursuing the interoperability request.
Small companies might escape, but if Facebook tried it they can't really run away. You'd need to be able to issue notices to ISPs to block those convicted of breaking the law, such orders are already part of UK law (against torrent discovery websites for example). Again you'd get some leakage but big providers like Microsoft couldn't feasibly expect enough customers to move to VPNs to bypass such a block.
IM client definition could be hard but in practice you'd have clauses requiring a judge to decide if the traffic exchange amounted to an IM communication. I see no problem with having a broad definition, if the definition were narrow then it would just get worked around. The definition would probably include email as a subset.
Were there other person-to-person communications protocols you are concerned would fall in the scope that definitely shouldn't be open?
> You'd need to be able to issue notices to ISPs to block those convicted of breaking the law, such orders are already part of UK law (against torrent discovery websites for example)
And most of us in the US hope we never lose the right to open access in the way citizens of the UK have. I can't think of a worse thing to happen to the internet. The UK has proven that once you allow a government to censor the content of the internet they will aggressively and rapidly expand that power to censor and control.
I can think of a few... one example that comes to mind is a video game. The communication protocol to synchronize the game could fall in scope, and I think video game companies have the right to have their own proprietary game network protocol.
I have a personal mailserver box in a cheap colo, and gmail won't put messages from me to my gmail-using friends in their inbox. It's shameful what email has become.
Yeah, that's one of those problems which are hard to solve without global collaboration, and current major IM participants are too selfish and greedy to understand that concept.
It was a product driven decision. The number of active federations was absolutely tiny and the number of messages sent over those links even tinier. The amount of XMPP spam was catastrophic, and for most users the only knowledge they'd ever have of XMPP was when they got a friend request from buy.herbal.viagra@cheap-online-meds.com. XMPP federation was a great idea, but it never had adoption, and at some point Google had to decide between UX and software purity - With one side having a clear high cost and unclear benefits.
(I was a Google SRE, sitting near the talk team. Everyone wanted XMPP to succeed, but the numbers didn't even come close to working out)
> XMPP federation was a great idea, but it never had adoption,
And your alternative @ google was yet another closed source protocol that doesn't work with any other closed source protocol(obviously, because they are closed source). Now imagine if the internet worked that way, Google wouldn't be where it is today.
Product driven or not, that decision was backward, not forward directed. And as I said, if they thought XMPP was deficient and couldn't address spam or what not, where is their better federated and open alternative? They never made one.
If you notice serious problems with federated messaging, it's an entirely valid choice to abandon federation instead of trying to make federation work. There's no evidence that a system that's (a) federated, (b) open to everyone and (c) non-spammy is possible in the first place; maybe it is, but even more likely you have to choose two out of three at best.
I don't see anyone repeating this fallacy w/regards to email.
The only difference between email and federated IM is email started out (mostly) federated and IM systems started out (mostly) not.
Only inertia is what drives this skepticism of federation in IM.
Just as there is no technical reason GateKeeper couldn't succeed on iOS just as it does on macOS, there is no technical reason federation couldn't succeed in IM as it does with email.
Firstly not nobody. Secondly, how can you grow it if instead of moving it forward, they roll it back? That's the opposite of progress.
> Ultimately most users just don't care about federation. Them's the facts.
Not really. Everyone is annoyed by this issue. But there isn't much they can do. People need to register on N different services and use N different clients to communicate with users of those networks when they need to. Do you think they appreciate this mess or find it highly convenient? But they don't have any other option (except may be not communicating with some of them at all).
People care about convenience, not federation. Google Talk and Facebook Messenger both supported it for multiple years and the number of people who used it rounds to zero. This is where we ended up. There was time for it, just no demand.
That work asynchronously even when I'm not online, that I can check later, that have functionally replaced almost all of the communication that used to happen over email...
It's technically and functionally identical to non-federated email.
Slack doesn't replace email. That's a silly marketing slogan. Imagine for a moment if it really did. What if Slack became so popular that everyone stopped using email? That would be horrifying. You wouldn't be able to communicate with anyone without using Slack because it doesn't federate with any other service. Every website would have to integrate with Slack for user account registrations. Etc. Decentralized federated communications is the backbone of the internet. The fact that IM services have never broadly adopted federation dramatically cripples the potential of the medium with horrible fragmentation and balkanization.
Surely the right answer would have been to enable users to report spam invites, and to kill spam-sending domains. Kinda like what is already done for email. Could probably repurpose a lot of the existing infrastructure.
The world made it impossible to roll your own server, not Google. The proliferation of spammers in email has necessitated a cottage industry of security and filtering and all manner of mechanisms, automated and otherwise, to deal with the avalanche of bullshit streaming through that protocol.
Google's the blame party for a lot of things, but not the loss of email freedom.
I don't know a whole lot about XMPP, but since it's federated, wouldn't it have the same avalanche? It seems like your choices are to use something interoperable and get spammed, or use a walled garden and you don't get spammed because the gardener has pitchforks.
Since this is IM the most you could do would be to generate a lot of requests to put the spammer on someone's roster list. In the unlikely event that a particular rogue XMPP managed to convince anyone to let them add some bogus users to their roster before it was blacklisted, it would take literally seconds to resolve the issue by removing them from your roster. So there isn't really any incentive to do spam over something like XMPP.
I have been on the federated XMPP network pretty much as long as it has existed and I have not gotten a single spam attempt. I guess I could post my jabber ID somewhere to make it more sporting (they need that to even try) but I really don't want to add random people to my roster. Email is intrinsically different that way.
This is not true. There are email standards out there to help prevent spam. But right now Google simply ignores all this even if everything is setup correctly. I think Google has a responsibility in helping me debug _why_ an email is spam and give tools to fix it. Currently, there is 0 information, 0 support. The webmaster tools simply say everything is awesome and dandy. And you can't do anything else.
Here is today' world work flow:
1. Buy a domain
2. Have to use Google because your mail will end up in spam with any provider (unless it's outlook or some special provider).
I've never heard of Google being particularly helpful in debugging literally anything. According to their "support", Google software is perfect and if it's not working for you you're doing something wrong.
If you're looking for a future in federated chat: matrix.org. I think there's real potential in the protocol and Im slowing moving most of my chat activity there.
Matrix looks interesting, though I'm not sure why they use HTTP for transport. Sounds a bit counter intuitive, since HTTP was never built for proper duplex communication. XMPP (in the browser) had to use BOSH to work around that, then came WebSockets, WebRTC and so on.
But actually they replaced email with gmail. Now you can't send email from your own server because it will be blacklisted on gmail by default and noone on gmail will receive these emails. So you have to use "cloud email providers". On Android they changed "Email" client to "Gmail" client.
I built my own mail server, and use it for every day communication. If you do everything correctly (mainly setting up DKIM and SPF), you get pretty much trusted by gmail automatically. You might have to make sure your IP address isn't already blacklisted, though I didn't have that problem.
I’ve had proper DKIM, proper SPF, my IP wasn’t blacklisted anywhere.
But I still ended up in Spam.
So I had to send for about a year emails to friends every few days, replicating our discussions on WhatsApp etc (so they’d have organic content), and I’d ask them to mark them as "not spam".
Now I end up in their normal Inbox.
EDIT: Why the downvotes? Please comment instead what you think is wrong – or how else do you think I should get my personal domain trusted?
> EDIT: Why the downvotes? Please comment instead what you think is wrong – or how else you think I should get my personal domain trusted enough?
I don't understand the downvotes either. So I upvoted your comment to counteract them.
I've found myself doing that more and more lately. I'll upvote a comment I disagree with if it was made politely and in good faith, and it appears to be getting downvoted unfairly.
UPDATE: In case someone ever wants to know more details about why my server failed verification, here are checks for that domain I took when changing DKIM keys last (today):
The thing is, and this is the current state of email in general, aka not just google, using an email server other than from [big cloud], like say from my local ISP or a large shared hosting provider, you will run into issues. Usually fine, but every few months your (my) server's ip will randomly get blacklisted for a few days, and for a small biz you'll have customers occasionally not receive your email.
Random blacklisting maybe is less of an issue if you run your own server with a unique ip only you use, but for most people it's getting harder and harder not to use gmail/outlook/lycos.
Yeah, people can still run their own mail servers without too much work. I've had one for a couple years but I ran into issues with Google tagging my mail as spam at first. It was a while back but I think they want domain owners to verify their ownership with Google's webmaster tools.[1]
My server worked better after adding a TXT record with google-site-verification + all the normal DKIM stuff.
I plan to roll my own email server in the near future, due to legal and privacy concerns. As I understand it, in the US email archives aren't protected under the Fourth Amendment because they're asking for data from a third party, and the Fourth Amendment only applies to self-incrimination.
As I understand it, in the US email archives aren't protected under the Fourth Amendment
Currently, as I understand they are not. But, there is legislation [1] which proposes extending Fourth Amendment protection to e-mails and communication stored on your behalf on third party servers. I've already written to my Congressperson in support of that bill.
5th amendment is self-incrimination, 4th is unwarranted search and seizure.
I don't know the current law, but it used to be that e-mails left on a server for more than 6 months were treated like abandoned property, which has a lower-bar to clear for the 4th amendment. I seem to recall hearing that this was changed a while back though.
How the 4th amendment applies to any case is complicated though because it's very much a sliding scale of intrusiveness, with higher levels of intrusiveness requiring more cause for searching.
Managing your own server is in general pretty hard (for sending or receiving). Unless you want to invest a lot of time in security and so on. It's simply way overwhelming. And if you don't invest in security, you are inviting trouble.
Sure. I've been running my own mail server for almost 20 years, but you seemed to imply that the bare act of sending email from a personal server to Gmail was a bad idea for some reason.
I've been running my own email server since... forever. In 1985 it was uucp ! paths and sendmail. I got a "for real" domain just for email in 1993, still over uucp dialup. DSL always on in 1999. But it does become harder as time goes on.
On the list of computing tasks I do every day, running my email server is WAY down there on the list of difficulty. Email servers are probably even easier than web servers.
Which doesn't necessarily mean you take necessary steps to seriously protect it. Or may be it means you are an expert and all that is trivial for you. But in general it's clearly not.
At least Wave was an open protocol including federation, even based on XMPP in fact. It maybe was the last thing before Google transitioned into a mostly walled garden.
> Now you can't send email from your own server because it will be blacklisted on gmail by default and noone on gmail will receive these emails.
I run a small mail server for some years now, no issues sending to gmail.
"On Android they changed "Email" client to "Gmail" client."
No, they didn't. Those are two separate apps. The Email one is part of AOSP. The Gmail one is part of Google Apps. Although many OEMs roll their own email client, as part of their custom skin.
On my Nexus 4, (Android 5.1.1), when I click on what used to be the email icon, I get a screen that says "Email has moved. The Gmail app now lets you view all of your email accounts." It has only one option (and a help link). That option is "TAKE ME TO THE GMAIL APP".
They gave up on Federated XMPP and never implemented video using XMPP. If they hated it so much, they could have proposed open changes. Now we're in a terrible IM landscape. No federation at all. E-mail is federated, but is been rendered terribly unreliable simply due to spam filters.
Open federated protocols with no obligation for providers to upgrade as the security landscape changes are a betrayal of user security and privacy of the highest order.
HTTPS is only becoming reasonable because Google can use its monopoly power and forced upgrade mechanism to bully website owners into adopting better practices. If we had a fragmented ecosystem of many open source browsers with user-consent upgrades and similar market share like the open web people wanted, certificate transparency would remain academic.
Personally, I consider the median advocate of open federated protocols to be more culpable for the wholesale violation of user privacy than the median NSA staffer.
No, closed ones are a betrayal. If someone doesn't upgrade, you can easily exclude them from communication and even build such logic in the protocol itself. It will pressure them to be up to date. I don't see an issue here.
You can if you care about security. That's exactly how the very same Google Talk was cut off from the rest of the XMPP universe, even before they decided to shut it down (they refused to support server to server encryption).
It's a wonder we can send e-mails between many servers. Imagine someone like Eric Schmidt driving it. We'd be stuck with incompatible AOL and Compuserve forever.
And about XMPP shortcomings - sure, it's not perfect. And if Google thought they can do better, why didn't they propose some IM-next as an open federated or P2P protocol? Because "don't be evil" is off the table I suppose.
People tried proposing to use Discord for me, but I'm really not interested in another walled garden closed protocol, without FOSS clients and servers.