
How Tutanota replaced Google’s FCM with their own notification system - grammers
https://f-droid.org/en/2018/09/03/replacing-gcm-in-tutanota.html
======
yegle
Is it possible for every app developer to stop using FCM? Yes, in China.

The result is a chaotic market. App devs will have to use multiple push
machinism for different device: on Huawei phones, Huawei's own push service is
the most reliable; on other phones, you might want use SDKs from
Tencent/Baidu/Alibaba so their app (Wechat/Baidu search/Taobao/Alipay) can
"wake up" your app to receive push. Battery life become miserable, push become
unreliable. It's to a point where the government start "regulating" the market
(the Unified Push Alliance,
[http://www.chinaupa.com](http://www.chinaupa.com), is lead by CAICT, a think
tank of Gov of China).

~~~
ktosobcy
But do we really need "push providers"? The problem is (IMHO) that app
developers don't know how to use the connection efficiently causing the radio
to wake up constantly (IDLE socket connection has very low battery impact).
Instead of pushing every darn detail queue less important ones and send them
out when something more important arrives; your client app can signal the
server, that app is in the background (or phone is going to sleep) to send
less frequent updates. But - companies will abuse everything they can to
notify users about absolutely everything. Currently I'm running phone without
google play services and it works just fine (with exception of Viber, which
can't deliver messages correctly without push services, but it's not that
important for me).

~~~
ucaetano
If you don't have a push provider mediating the many apps trying to get
notifications, you end up in a tragedy of the commons scenario, where every
single app will try to implement its own push service and will prioritize its
own push service, quickly resulting in a race to the bottom where the device
is constantly connecting to a different push service, draining battery and
exhausting your data connection.

It isn't very different from preemptive/cooperative multitasking.

~~~
iad
Why can't the device just chose a single wake schedule and communicate that to
each push provider? Hopefully UDP can be used so there's no need for any
keepalives or handshakes during the wake periods?

Are clocks too unreliable or is there too much NAT because everybody's too
lazy to implement IPv6 still?

~~~
izacus
This actually happens on Android (read about Doze mode), but the issue with
your approach is that you don't get realtime delivery - only periodic sync.
Which is actually less efficient, because the device must wake up on interval,
activate the radio and poll for changes.

FCM (and Apple push) run a single keep-alive TCP connection and incoming
packets will cause radio and application process wakeup only when necessary.

This change significantly improved Android battery life (remember that HN
Apple lovers really loved to bash on Android battery life before this FCM push
happened). The fact of the matter is that a lot of developers didn't care one
bit about battery life and abused Android background services when there was
no need to do so (e.g. schedule periodic wakeups to check for a preset alarm
needed once 24 hours) and if we learned anything by this experiment is that
you simply can't trust developers to respect users anymore.

~~~
nicoburns
Does the 1 tcp connwction notnrequire the radio to wakeup to receive pushes?

~~~
izacus
It does, but not really periodically - the radio is just listening and not
transmitting until a packet comes. Then it wakes up the application processor
to handle it.

~~~
int_19h
I guess the question is, if a single listening TCP connection is not a problem
because everything can sleep until a packet arrives, what's the problem with
multiple apps each opening such a connection?

~~~
izacus
Increase amount of chatter, more keep alives, problems with NATs (see the
other post in this thread) which don't whitelist non-Google/Apple endpoints,
etc.

But majorly - it makes little business sense for Google to support that. It
can only make Android look worse for little gain to them.

------
Findus23
This is really the chance for a great open source project. So many open source
apps (Signal, Mastodon clients, riot.im, telegram-foss and more) have to do
hacks to be able to deliver push notifications without GCM/FCM

What if there was an open source software one could setup on a server that
would provide push services for all these apps and would interact with one
open source client running on android.

That way one would have the energy saving benefits of only handling one server
connection, but the privacy benefits of a private server for
oneself/friends/people one trusts.

This is definitely not easy and would require coordination between many open
source projects and also additions in Android to run on a system level (maybe
in lineageos and similar), but I really think it would be worth it.

~~~
subway
I wish there were an open standard for push notifications with some sort of
mux/proxy support. Let me run a daemon (or use a service provider) that
collects all my notifications in a data-center, then send a datagram, sms, or
even pocsag message to let me know notifications are available for pickup.

~~~
zielmicha
There is an open standard - it's called Web Push ([https://www.w3.org/TR/push-
api/](https://www.w3.org/TR/push-api/)). It's already used by Firefox and
Chrome to implement push notifications for web pages (via service workers).
The notifications are proxied via Mozilla/Google server respectively, but
there is no reason you could not run your own server.

------
xg15
> _Wouldn’t it be great if the user could just pick a “push notifications
> provider” in the phone settings and OS managed all these hard details by
> itself? So every app, which doesn’t want to be policed by the platform
> owner, didn’t have to invent the system anew? It could be end-to-end
> encrypted between the app and the app server. There’s no real technical
> difficulty in that, but as long as our systems are controlled by big players
> who do not allow this, we have to solve it by ourselves._

Fking this. ideally, this would also involve a standard API on the backend for
how to send push notifications. E.g., something like:

1) App on phone queries OS for selected push provider.

2) Phone OS returns some metadata about the push provider to the app,
including a a backend URL for sending messages.

3) App sends that URL to it's own backend server.

4) When a push message should be sent, the app's backend server invokes that
URL with the message to be sent in some standardized way.

If - which is likely - push servies require that developers register their
apps with them before use, this could be expanded by the phone OS returning a
list of providers instead of just one.

~~~
metildaa
I would encourage app developers to take a look at the network architecture of
the major US carriers while determining how to give their app notifications
from the server. In the US, the 4 major LTE cellular carriers operate IPv6
only networks with NATed IPv4 access tunneled in as an unsupported
afterthought.

From my testing with IPv6 enabled VOIP for inbound calls, compared to FCM with
calls coming from an IPv4 only server, only the former setup would reliably
get calls to a test device on the first ring. If you need very low latency
push notifications, IPv6 and FCM/Apple Push Notification are needed for best
performance. Thus far, inter-carrier roaming generally does not support IPv6,
so you still must support legacy IPv4 to a degree.

~~~
StudentStuff
To this end, Apple has been leaning on developers to support iOS devices that
are on IPv6 only networks:
[https://developer.apple.com/support/ipv6/](https://developer.apple.com/support/ipv6/)

------
Boulth
People migrating from GCM to FCM need to be aware that in FCM analytics are
turned on by default:
[https://github.com/siacs/Conversations/issues/3041](https://github.com/siacs/Conversations/issues/3041)

~~~
closinginverse
From your link:

>I’m closing this issue. I think current master successfully disables
analytics. Thanks for bringing this to my attention.

~~~
seba_dos1
"Current master" obviously refers to Conversations, not FCM libs.

------
emilfihlman
>SSE fits our needs better than WebSocket would (it is cheaper and converges
faster, because it’s not duplex). We’ve seen multiple chat apps trying using
WebSocket for push notifications and it didn’t seem power efficient.

This is just not true unless there is some serious fuckery going on with the
websocket implementation. Both require full duplex communications on the
transport layer.

------
hawski
I'm wondering how it works with different messaging apps. I'm using mostly
Facebook Messenger, because of the network effects. A few months ago I tried
to use several apps to communicate with my wife. Both Signal and Wire failed
to reliably send messages in under 30 minutes or so. More often then not the
message would be delivered after a few hours. That's totally unacceptable. If
there are some magic settings in Android they were not properly advertised.
Our phones are on quite pure 8.1 and 7 versions. So I'm back to Messenger.

Reading this it seems that there is a known solution to this problem. Does
Mastodon offer something like private messages? Or is there a messaging app
that doesn't use FCM, but works in a way described in article?

~~~
icebraining
If your phone has Google Play Services running, Signal should use GCM/FCM;
they have a websockets mode, but only for devices lacking the former.

I don't use Signal, but I don't think that delay is typical, could there be an
issue with the connectivity to their servers?

~~~
metildaa
Nope, this is not normal behavior for Signal, it is a sign that the sending or
receiving phone has FCM and data connectivity issues.

I have seen this issue on Motorola Android phones with known dying LTE Modems
(eg: these phones couldn't maintain a data session for more than a few
minutes, GPS off WiFi is also unreliable).

------
EGreg
_Final thought: Every user should be able to choose a “Notification Provider”
for every app_

At Qbix, we thought deeply about this problem. Ideally, to maintain anonymity,
each notification would come from some random endpoint on the cloud. But
that’s not how the Internet works these days, you have to connect periodically
to SOMETHING. So you can choose your own notification providers.

The trouble is if you have many apps, then your device is constantly connected
to many servers.

What we settled on for now is a background process with WebSockets, but
perhaps this works better. The operating systems weren’t designed for this use
case and the phones aren’t optimized for it. For example, how would you do the
same on iOS?

However, what IS possible is tunneling through the native iOS VoIP
notifications support and encrypting your notifications. You can even process
them on the client side with some “IFTTT” type logic. To do that for Android,
however, we had to implement this background process approach. But it’s a
hack.

~~~
cobookman
each app having its own notification provider cones at the cost of extra cpu
cycles, network calls, and interrupts. Worsening battery life.

It's likely in the users benefit to only have 1 notification provider

~~~
EGreg
Right. I think they should have one other mechanism “in the cloud” somewhere
and no need to have too many, one for each domain.

I am starting to realize that DNS is main source of problems on the Internet.
We need to replace it with a DHT using something like Kademlia.

It was originally designed to have human-readable domains on the Internet. But
it’s become a glorified federated search engine, essentially, whereas we can
have lots of different search engines, one per app or location. Today, there
is very little benefit to having human-readable URLs. If it’s anything much
longer than a hostname, you won’t even say it or read it. So it only works for
a tiny subset of internet resources but serves to centralize control in the
hands of a few large websites. It leads to centralized databases that attract
the NSA and advertisers — and all the stuff from Cambridge Analytica to the
Equifax breach are downstream results that can be fixed by apps switching from
DNS to a DHT.

~~~
wongarsu
Even if we ignore the readability of URLs (which I wouldn't), DNS is a nice
service discovery protocol with aligned incentives. To find out which server
to contact to get hacker news, you ask the DNS server for .com, which has
incentive to answer honestly because hacker news pays them to do so, and then
the DNS server for ycombinator.com which has incentive to answer honestly
because it's operated/payed for by the people running hacker news.
Alternativly you can ask the DNS server of your ISP which has incentive to
answer honestly because otherwise the ISP risks loosing customers (assuming
working capitalism, i.e. competition).

In a DHT, anyone with very different incentives can add as many nodes as he
likes. If the node id assignment isn't designed very carefully any attacker
can even dominate whatever part of the DHT he is interested in and deny or
modify information or just log information.

Sure, the DNS system isn't immune against state actors, especially the US. But
DHT is vulnerable against everyone willing to spend some money on spinning up
hosts.

~~~
EGreg
Why would you care about readability of URLs, when each client can just show
the cached metadata? Which is what happens anyway, kind of, in your browser
history and anywhere you share the links. Most people won’t read the url.

Now, as for the incentives. In fact, DNS does get poisoned and that’s why
DNSSEC was invented. The same exact thing can be done on a DHT with private
keys and certificates authenticating the author of a document. Except you
don’t have the kind of dynamics that lead to centralized databases and
breaches. Instead, we can even have content addressing, so the same exact
resource can be cached even if it would have been “on a different domain”. And
you know that resource hasn’t been changes from under you. Talk about
incentives — there is a huge incentive for hackers to access these giant
centralized honeypots and change up what’s being returned from a URL, or do
phishing based on a slightly similar domain name.

Look up the MaidSAFE project and tell me, how exactly is the Kademlia DHT
there vulnerable against everyone willing to spin up hosts? You can still use
certificates and domains can still sign everything. The domain system is
basically a search engine, which also has its own incentives.

And finally, Capitalism is the 2nd best system for organizing people and
information, the first being open source. The Web, Wikipedia, WebKit, MySQL,
PHP and Linux have long ago beaten AOL, Britannica, MSSQL, ASP.NET and Windows
NT Server.

------
southerndrift
What's the deal with push notifications if the modem has to be switched on
every couple of minutes to keep a connection alive? I was under the impression
that push notifications were implemented similar to a call handshake that only
activates the phone if there is a call.

Are the FCM service and the iOS equivalent also IP based or can they use some
lower level, more energy efficient protocols to wake up the phones?

~~~
repolfx
They are IP based.

There are several problems that make replicating FCM hard, which is one reason
why Google tries to push you to using it (wordplay definitely intended).

It all boils down to IPv4. IPv4 address space is exhausted, so carriers have
had to deploy carrier NAT in front of cell towers, often multiple levels of
carrier NAT. Well NAT requires translation tables to be held in RAM and
carrier-grade boxes are very expensive in general, so to keep the machines
alive they need to aggressively garbage collect dead translations. Otherwise
they'd run out of RAM.

This is the origin of the 'keep alive' problem - NATs want to close your
connection to free up their own resources, and you want the connection to stay
open so you can receive push messages. So phones have to wake up every so
often and send keepalive packets or do a connection rebuild.

Google and Apple have an interesting solution to this problem .... it's a mix
of learning and data analytics to figure out what NAT timeouts used by each
carrier are, and cutting deals with carriers directly to adjust the timeouts
for their IP ranges specifically. Therefore you _cannot compete directly with
Google or Apple on energy efficiency_. This is something a lot of hackers
don't realise. Getting to the level of efficiency FCM has is very hard and
takes a lot of work, and basically requires you to be a giant company. This is
why it's an OS level service.

They also use very tight protocols, batch things together and of course
provide oodles of server side disk space for buffering messages to disk until
the devices return, lots of other MQ type things that are hard to do at scale.

IPv6 solves this problem by allowing carriers to lose the NAT boxes. No more
machines that need stuff in RAM for every TCP connection, which means
connections are no longer a scarce resource that must be culled from time to
time. If you only have to maintain connections to devices that support IPv6
you could theoretically maintain very long lived connections if the kernel,
radio and server cooperate. Of course there are still timeouts: the TCP
connection requires _some_ state server-side, so the servers will kill off
connections from time to time because devices may roam across different IP
addresses. But it should be a lot more stable and not require cutting deals
anymore.

~~~
lucb1e
I doubt the accuracy of what you are saying. Treating certain IP ranges
differently would not be net neutral, which is exactly why we have net
neutrality: otherwise it's impossible for new entrants in the market to
compete. Also, most ISPs don't run CGNAT (except historically notoriously many
in Italy, and by now also a few in the USA). And "oodles" sounds like a lot
but running a push server doesn't sound like it would require that much disk
space. At Google's level, sure, but for your average app... it can easily be
run without Google.

The real problem (afaik) is that people change address whenever they change
network (so it doesn't all boil down to v4, either), so you need regular
keepalives. If every app starts doing keepalives individually, your phone will
be using its radio every 30 seconds 24/7 (assuming you have four apps that
need push, and those devs want a maximum delay of 2 minutes, and their timers
are roughly evenly distributed). By having a single server to keepalive, the
issue is avoided.

~~~
jmgao
> Also, most ISPs don't run CGNAT (except historically notoriously many in
> Italy, and by now also a few in the USA).

My understanding is that essentially every mobile network in the world uses
CGNAT. I happen to have AT&T, T-Mobile, and Verizon SIMs handy at the moment,
and all 3 of them are behind NAT.

------
pndy
> Wouldn’t it be great if the user could just pick a “push notifications
> provider” in the phone settings and OS managed all these hard details by
> itself?

Having the possibility of choice is a good thing but in this particular case I
don't think it would be good if 3rd party notifications providers were
available. This should be an integral part of the OS, not a field for
competition; knowing that Google itself can harvest data from notifications we
can't exclude the possibility that there would appear companies interested
only in that activity, at the same time enticing users with pretty and simple
design of notifications or whatever else.

~~~
GranPC
Well, any Android app can harvest data from notifications right now, by
setting up a notification listener. I don't see how this would be any
different.

~~~
lucb1e
The difference is that it's server side. I'm not aware whether the third party
server logs my messages, but I am aware when an app has local permissions (you
can see it in the permissions overview) to read my notifications.

------
snipem
_Another difficulty was caused by the Doze mode, introduced in Android M. The
Doze, which is turned on after a period of inactivity, among other things
prevents background processes to access the network. As you can imagine, this
prevents our app from receiving notifications._

 _We mitigate this problem by asking users to make an exemption from battery
optimisations for our app. It worked fairly well._

This is why we were migrating to FCM from our own custom built MQTT solution
that was invented back in 2012.

~~~
craftyguy
MQTT still requires a network connection to publish to (or receive
subscription events from) a MQTT server.. so you still have the exact same
problem with Doze preventing that. What are you doing to get around that
problem, if not exempting your app from battery 'optimisations' ?

Edit: Would anyone care to explain how I am wrong in addition to downvoting
me? If Doze blocks network connections to your app when it is backgrounded,
how can it receive MQTT events?

~~~
TeMPOraL
GP knows that. They said they're migrating _away_ from MQTT, to the Google
solution.

~~~
craftyguy
Ah, somehow I missed that :(

------
gustavmarwin
This is a very exciting development, it's great to see more developers take
this centralisation issue seriously.

On this subject, many people including myself have been failing (hard) to
convince Moxie that dropping FCM (or GCM or whatever the name) is the way to
go for Signal, if anyone want to give a hand...

[https://www.reddit.com/r/signal/comments/9ekawn/moxie_drop_t...](https://www.reddit.com/r/signal/comments/9ekawn/moxie_drop_the_gcm_requirement_already/)

[https://github.com/signalapp/Signal-
Android/issues/7638](https://github.com/signalapp/Signal-Android/issues/7638)

------
azinman2
Wow reading this seems like android push is a total mess (outside of FCM).
Things are much simplier on the iOS side...

~~~
kumarm
As an User and Dev of both iOS and Android, I can tell you Push Notifications
are miles ahead on Android compared to iOS.

The article refers to not wanting use Google Solution for non technical
reasons.

