
Dear Google Cloud: Your Deprecation Policy Is Killing You - bigiain
https://medium.com/@steve.yegge/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc
======
fh973
I think Google couldn't do better, even if they wanted to.

The reason is the engineering culture and the promotion system. People expect
to switch to new projects, and work towards to a promotion within 2-3 years.
This shows in all products, and presumalby also the reason for the
deprecation. New people come on the project, want and need to do a major
overhaul, and do not have resource or incentives to support the old stuff.

The other half is that most projects are done efficiently by relatively small
groups of engineers. I remember that Android Market was still a few people in
MTV, with no payment possibilities outside of the US, when Apple already had
significant revenue from apps. And that shows everywhere. I'd argue Google is
incapable of tackling projects that require more than a few hundred people for
more than 2-3 years.

But as a billion dollar company, that does not work. Throwing money at obvious
new billion dollar opportunities like Microsoft did with gaming and cloud, and
following through over 5 years is not possible for Google.

~~~
mamurphy
My armchair thought: what if the CEO said "wow, Yegge is right, we've gotten
really off trail here. you need to show leadership in maintaining a 3+ year
old system at Google for promotions?" Could that happen and would it work?

~~~
Traster
The promotion system isn't the issue, the promotion system was the issue 10
years ago. For the last 10 years Google has been hiring and promoting and
keeping people who prioritize shiny new things more than maintaining old
things.

If you start rewarding maintaining old things you aren't going to suddenly get
what you want, you're just going to lose a tonne of people because they don't
want to do what you're telling them and you won't be able to hire good people
with your priorities because no one is going to believe they can have a good
career at Google by prioritizing stability and long term support. It's all
downstream of culture, and culture change is very difficult.

Of course if you do buy into this idea that this is the issue, then GCP has
already lost. It would be years before they could make a dent in that culture,
and more years after that for external customers to know and trust Google's
new culture. By that time there's going to be a dominant player anyway. So
maybe actually it's best to stick with the current culture - try and win on
your strengths than try to fix your defficiencies.

~~~
zimbatm
It might be even deeper than that. I don't know if it's representative, the
few Googlers I talked to were there only for the money. Their view of the
company was actually quite negative.

Keeping backward-compatibility requires people that care. That have enough
pride in their work to counter-balance the grind.

~~~
corin_
Obviously having people that care about the work they do is a positive thing,
but there are plenty of people in plenty of industries managing to do a good
job despite only being in it for the money.

It might not be a good business decision for Google, but surely if they create
enough positions that are solely focussed on the boring stuff but are paid
better to make up for it, there is a price point at which they'd get enough
interested people.

~~~
hdfhu
That would be a career suicide to become a support engineer like this. 4 years
later, when his stocks dry out, he'd have to switch the company and he'd need
to tell a convincing story why the new company should pay him the new market
rate. And "I supported legacy code" is not such a story. There's always an
option to go to Microsoft and support legacy stuff for life, but beware that
MS pays peanuts (relatively speaking) and with the MS pay you'd be priced out
of the housing market.

~~~
d1zzy
To show that you deserve whatever pay you need to show impact and complexity.
I don't think you would have a problem showing either when talking about
maintaining Google scale products...

~~~
hdfhu
Now we are talking. The thing is, only few work on those big projects. The
rest are doing god knows what and have to jump ships often. If your resume
says you spent 5 years maintaining a smallish noname project, your career is
at risk. I don't believe that Google keeps all its 100k employees busy with
complex and high impact projects.

------
sitkack
> Let’s say hypothetically that Apple was dumb enough to pull a Guido van
> Rossum, and declare that Swift 6.0 is backwards-incompatible with Swift 5.0,
> much in the way that Python 3 is incompatible with Python 2.

Damn, I was waiting for this. The whole essay was a setup for this paragraph.
Reframes the argument using a shared traumatic experience for everyone. This
humanizes the effect, we all know someone that lost someone to the Python2/3
transition. I can't count the number of Python friends that turned into
Gophers, joining some cult of middle class bearded tech dad with a side
business distilling pear brandy.

Best Yegge Essay Yet! Welcome back Steve.

 __edit, finished the essay. He _smoked_ GCP in this one, damn damn hard.
Hickory.

~~~
Avamander
I don't get the Python3 bw-incompatibility hate. Getting rid of cruft is
literally the only way to get rid of cruft. It's literally the Pythonic to not
have cruft.

I dread the day Python will be as bloated as C++ or Java, neither that dare to
remove things.

~~~
overgard
Probably less to do with backwards compat then how it was handled. The first
few versions of python 3 didn't really provide any compelling benefit, (indeed
things like performance were worse, not better), and most people didn't _care_
about the cruft it removed. The "print" statement being a great example. Was
it kind of weird? Sure. Did that weirdness have any consequences at all? Not
really. Was it worth it to force people to replace thousands of lines of code
for something that was essentially a minor aesthetic issue, if that? No way.

Also the python 2 string type was more in line with how other unix utilities
work, so while I do think the 3 string type is generally better for the web,
it wasn't an obvious improvement for the other large usage of python. While
it's a very versatile language, I would say the major niches of python are:
web dev, shell scripting, and scientific computing. The changes in python 3
were (somewhat) helpful for web dev, but really were inconsequential and/or
actively harmful for the other uses.

Mixed with how condescending the python dev team acted about it, and how it
really wasn't until about python 3.6 that there was any compelling reason to
move to python 3,you can see where most of the hate comes from.

And by the way, even if Python is very popular today, I do think that rift
really hurt the language. I worked at least at one company where Python was
considered for a project, but eventually shelved because of management's
confusion over the 2/3 issue.

~~~
_dps
Agree with everything you said here, but let me riff on this

> The "print" statement being a great example. Was it kind of weird? Sure.

I am reminded of the Emerson quip "A foolish consistency is the hobgoblin of
little minds". The Py2 print statement is special/weird because it is a very
important and special behavior, _especially_ for newcomers. It's not randomly
special, it's special because it models something that deserves special
treatment.

Py3 is more consistent, but less humane. I use it over Py2 only because the
market has moved on, but every time I type print with parentheses I curse
Guido.

~~~
Grue3
The funny thing is, if you type print with space it _recognizes_ what you
meant, and still wouldn't do it, raising an exception instead. It would've
cost them literally nothing to support both ways to write print, since the
parser has to detect statement print anyway.

~~~
Avamander
Yes, but it would be a different way to do the same thing. It'd be anti-
Pythonic.

------
userbinator
It's odd to see an article emphasising backwards compatibility, but not a
single mention of Microsoft (I even ctrl+F'd the page source to check!)
They've regressed a bit in the recent years but I'd still consider MS the gold
standard for back-compat. I have Win32 binaries I last modified over 15 years
ago, small (or perhaps _tiny_ \- size measured in KBs) utilities that I use
daily. They worked perfectly on Win95, and still do on Win10.

I wonder when/if the software industry will ever stabilise. It's possibly the
only industry where frequent and disruptive change continues to happen, and is
even welcomed by many in it (not me).

~~~
jimmaswell
The absolute worst in my small experience with it is anything touching the
node ecosystem. Try to follow any guide or use any library not updated in the
last week and there will be breaking API changes and deprecation warnings
everywhere.

~~~
kungato
The sad part about the js ecosystem is that even the biggest corps like Google
can't keep their own js ecosystem stable on the last version. I have a small
Angular app with Firebase backend and every time I come back to it to update,
the libs are out of sync

~~~
joepie91_
Google is notoriously bad at maintaining (or even designing) their open-source
stuff. I wouldn't say "even the biggest corps like Google" \-- Google does
considerably worse at this than many hobbyist maintainers! I just don't use
Google libraries anymore, if I can at all avoid it.

See also my other comment[1] - Angular is an example of such a magical does-
everything framework.

[1]
[https://news.ycombinator.com/item?id=24168279](https://news.ycombinator.com/item?id=24168279)

~~~
RonanTheGrey
We've abandoned an app because of the inability to keep Firebase working on it
(JS libraries). Between out of sync Typescript definitions, broken APIs,
broken interfaces, "this version of some other dependency works with this
version of Firebase thanks to some random transient dependency", it just
became too much. We won't use the JS libraries at all anymore.

Angular is the only one that has remained even remotely usable, but certainly
not stable. It's never as simple as running the upgrade utilities, changing
the version, and being done. It ALWAYS takes at least a day to find all the
"little" things they didn't feel fit to mention in the upgrade docs.

It's aggravating. I genuinely don't like using Google software, at all.

I've felt for a long time that Google is coasting on the momentum of the early
web and the awe people felt for what they built early on. That hasn't been
Google for a long, long time.

------
lowiqengineer
The answer to why this is, I’m beginning to realize after talking to dozens of
Googlers working at Google as acquaintances, friends, former coworkers, HN
comments and twitter personalities is - _Google has an active disrespect if
you’re not part of their in group_. If you haven’t passed their tests, you’re
basically an object of derision - a mark, an object just used for incrementing
a CPM metric, or an incompetent, low-class buffoon who doesn’t deserve to have
their jet setting luxury lifestyles. And unfortunately, everything trickles
down from that.

~~~
wh-uws
(warning rambling rant)

I feel like this was super true in the pre Facebook days but they were humbled
A LOT by the massive flop of Google plus.

I honestly don't feel like I get the my s __* doesn 't stink vibe from them
anymore.

Especially because everyone knows their Achilles heel now.

But for their ad revenue what would they be?

Everything else material from came from an acquisition or is infra stuff.

It's cool but they also had to learn the hard lesson of Hadoop too.

Yeah they built big table... But the rest of the industry standardized on
Hadoop (at the time) because they never released an implementation and so they
started losing out on good hires because they didn't want to get locked into
proprietary Google infra.

I think you see them recognizing this with intiatives like all of the stuff
around kubernetes.

But hey I wouldn't build anything on their (non open) stuff and get more
scared by the day over my dependence on El Goog because they've shown time an
again how little they care for us common plebs.

But I feel like they get better when they lose but they've only started losing
more recently

(Rant over)

~~~
thethethethe
> Yeah they built big table... But the rest of the industry standardized on
> Hadoop (at the time) because they never released an implementation and so
> they started losing out on good hires because they didn't want to get locked
> into proprietary Google infra.

Hadoop and Bigtable are not the same thing. Bigtable is a noSQL database.
Hadoop is a big data processing framework. Hadoop is actually an open source
implementation of MapReduce, which was developed at Google and which has since
been replaced by Flume

~~~
manigandham
Hadoop is a collection of things: HDFS (filesystem), Hbase (keyvalue
database), MapReduce (data processing), and YARN (resource scheduling and job
management).

Hbase was open-source implementation of Bigtable while HDFS was a copy of
Google File System. The divergence appeared with other processing frameworks
like Spark but now there's Apache Beam to unify it all.

------
crazygringo
This very long article doesn't describe even a _single_ instance of anything
GCP has deprecated.

It's a long rant with a bunch of f-words, and the claim that he gets
deprecation e-mails "about once a month".

Can someone who is informed point to actual GCP services that have been, or
are being, deprecated?

From the article, I have no idea whatsoever if this is some huge actual
problem with core services... or pulling support for ancient versions of
Linux... or sunsetting experimental/beta services that never had guarantees to
begin with.

I'd find it pretty surprising for a platform as relatively new as GCP to
already be deprecating anything meaningful, so I'd really appreciate if anyone
has actual facts here.

(Also I know a lot of people who use GCP, and I've never heard anyone
complaining about services being deprecated, so to hear someone complaining
about it happening monthly is pretty surprising.)

~~~
mtlynch
Python2 on AppEngine is a huge, painful Google GCP deprecation.

I have a friend who built an AppEngine app within the first year of the
platform's release, and he's spent the better part of the past 9 months doing
almost nothing but managing his company's migration to Python3 AppEngine.

It wasn't just migrating from Python2 to Python3 syntax. Google used it as an
opportunity to also turn down tons of APIs and services they no longer felt
like maintaining.

One big difference was the Users API. Python2 AppEngine had a pretty simple
way to authenticate users that you could set up in an hour.[0] In Python3,
they've dumped that entirely and tell you that you have to move to a totally
different solution, and they offer no help with that migration.[1]

In Python2 AppEngine, you essentially got a memcache for free. The ndb
library[2] allowed applications to read/write to Cloud Datastore, but it also
automagically managed the cache for you so that you didn't have to set up a
separate cache service. In Python3, they've dumped that and force you to
maintain your own cache service and handle all the caching logic yourself.[3]

It looks like the ndb library is pretty complete at this point,[4] but when I
was looking at a migration earlier this year, the ndb library wasn't even out
of beta. Meanwhile, they were breathing down their customers' necks about
moving off the Python2 version.

I had a fairly small (~8 KLOC) Python2 AppEngine app that I wrote in 2018, but
when I realized how much of a hassle it would be to migrate to Python3, I just
rewrote it using Gridsome and moved off of AppEngine entirely.[5]

[0]
[https://cloud.google.com/appengine/docs/standard/python/user...](https://cloud.google.com/appengine/docs/standard/python/users)

[1]
[https://cloud.google.com/appengine/docs/standard/python/migr...](https://cloud.google.com/appengine/docs/standard/python/migrate-
to-python3/migrating-services#user_authentication)

[2]
[https://cloud.google.com/appengine/docs/standard/python/ndb](https://cloud.google.com/appengine/docs/standard/python/ndb)

[3]
[https://cloud.google.com/appengine/docs/standard/python3/mig...](https://cloud.google.com/appengine/docs/standard/python3/migrating-
to-cloud-ndb#caching)

[4] [https://googleapis.dev/python/python-
ndb/latest/index.html](https://googleapis.dev/python/python-
ndb/latest/index.html)

[5]
[https://whatgotdone.com/michael/2020-04-17](https://whatgotdone.com/michael/2020-04-17)

~~~
AnssiH
But Python 2 on App Engine is not officially deprecated yet (e.g. not listed
here:
[https://cloud.google.com/appengine/docs/deprecations](https://cloud.google.com/appengine/docs/deprecations))
and they have not sent the "[Action Required]" email (of which OP talks about)
for that yet, so I'm not sure it counts. They are "just" recommending
migration.

I have a ~60 KLOC AppEngine Python 2 app (started in 2011), but I'll probably
migrate that over only when the deprecation actually occurs, as:

\- that gives the python-ndb library more time to mature (performance and
bugs),

\- easier migration paths for some of the non-py3 services I use (e.g. Images,
Users, Memcache) might appear when deprecation actually occurs, either from
Google or 3rd parties,

\- I'm in no hurry,

\- I will be surprised if Google gives less than 1 year of deprecation notice
(Python 2.5 got 4 years from deprecation to termination).

(If the APIs I use were available for py3, I would've probably already
migrated.)

------
john-shaffer
> [Google] contracted with a company called Bitnami to create a bunch of
> “click to deploy” solutions, or perhaps I should write “solutions”, because
> they don’t solve a fucking thing. They’re just there as checkboxes, as
> marketing filler, and Google never gave a shit whether any of them worked
> from Day One.

I've had similar experiences with broken Bitnami images that makes it seem
like they aren't even tested before being released [1][2]. Their WordPress
images do work, but the way file permissions are configured doesn't play well
with the way they have WordPress set up [3]. Have I just had bad luck, or is
this a common experience?

[1] [https://community.bitnami.com/t/cant-authenticate-to-
fauxton...](https://community.bitnami.com/t/cant-authenticate-to-
fauxton/71354) [2] [https://community.bitnami.com/t/couchdb-server-log-gives-
wro...](https://community.bitnami.com/t/couchdb-server-log-gives-wrong-
username/71595) [3]
[https://github.com/WP2Static/wp2static/issues/598](https://github.com/WP2Static/wp2static/issues/598)

~~~
Ysx
Bitnami broke their MongoDB Helm chart. We're careful to pin versions, and
hadn't anticipated someone would re-write history and break published code.

Can't get too upset, we don't pay them anything. Not in a rush to use them
again though.

~~~
ithkuil
Have you reached out to (community) support? My colleagues there tend to do
their best to help the community.

EDIT Disclaimer: I work for Bitnami; not in that team though. I reached out
internally to see if this is a known issue, but it's Saturday...

EDIT 2: does the downvote mean bitnami community support is so bad or did my
wording offend anybody?

~~~
john-shaffer
There are lots of random downvotes for no real reason. Sometimes perfectly
ordinary posts are dead. I try to upvote gray posts if they're not worthless.

That said, my take (and that of siblings) is that I'm much better off building
my own containers than serving as unpaid Q&A for Bitnami. It would be smart to
use containers built by people more expert than I am. Containers that are
broken by default do not meet that criteria. So, "reach out to community
support" is not helpful to people that have already decided to ditch Bitnami.

> We're careful to pin versions, and hadn't anticipated someone would re-write
> history and break published code.

This is exactly the kind of thing that will make me ditch a company forever
and never look back.

~~~
ithkuil
It's not clear what happened though.

We use immutable tags like "4.2.8-debian-10-r50" which we never overwrite.

Then we have "semantic versioned" moving tags, like "4.2.8", "4.2", ... which
resolve to the latest image matching that prefix.

Moving tags are what they are; I'm personally not a fan; all major popular
images have that
([https://hub.docker.com/_/golang](https://hub.docker.com/_/golang),
[https://hub.docker.com/_/python](https://hub.docker.com/_/python), ...)

You can read more about Bitnami tagging scheme at:
[https://docs.bitnami.com/tutorials/understand-rolling-
tags-c...](https://docs.bitnami.com/tutorials/understand-rolling-tags-
containers/)

If you want to pin your image you should either use the most specific tag or
just use the digest:

    
    
        $ crane digest bitnami/mongodb:4.2
        sha256:8650c2d92eea97732eae359a140ee86ee3923a2a19b19443e1dc01ec20d5387d
        $ docker run bitnami/mongodb:4.2@sha256:8650c2d92eea97732eae359a140ee86ee3923a2a19b19443e1dc01ec20d5387d
    
    

Now, we might have introduced a regression between some version of a container
and the next minor version; shit happens; it's hard to tell without more
specific information though.

> > We're careful to pin versions, and hadn't anticipated someone would re-
> write history and break published code.

> This is exactly the kind of thing that will make me ditch a company forever
> and never look back.

On the other hand, assuming that instead of a misunderstanding, what you saw
is actually the image behind an immutable tag such as 4.2.8-debian-10-r50
being replaced, this is a serious security issue; somebody could have hacked
docker hub, or crafted a valid certificate for docker hub and MitM'd you,

I'd also ditch a company forever and never look back if they honestly don't
care about that problem, which I assure you it's not the case.

We'd greatly appreciate reporting such cases to security@bitnami.com /
security@vmware.com .

------
causality0
Google is an absolutely fantastic company from which to get free stuff. Gmail,
Maps, Docs, Youtube? Mmm-mm good!

Buy things from Google, though? You mean, give Google money and in exchange
expect them to adhere to some kind of standard of behavior, support, and
customer service? Go get a coffee because you're clearly not awake yet.

~~~
Jabbles
This is the exact opposite of the usual complaint: that if you don't pay for
support you can expect Google to arbitrarily shut your account and scoop up
all your data - however paying them actually allows you some control.

~~~
fauigerzigerk
Many of the frequent horror stories on here about account suspensions and
completely unresponsive support are coming from people or companies with paid
accounts.

I do believe that there is a level of spending where you can actually get a
hold of someone competent who can look into things. But that level appears to
be rather high.

~~~
fallenpegasus
About 5 years ago, Hewlett-Packard tried being a paid GSuite customer, paying
for tens of thousands of seats, with a path to onboarding the entire company.
We could not get support, or even a TAM who would answer their phone.

After a year, HP dumped GSuite, and went to O365 instead. We got a responsive
TAM, full multi-tier support, and white glove handholding for migration. And,
I think it was cheaper, too.

There is NO level of spending as a customer at which Google actually gives a
shit.

------
aaanotherhnfolk
It is a total hassle to keep up with Googlers changing everything constantly.
It's not just GCP it's every platform they control. Try keeping a website on
the right side of Chrome policies, a G Suite app up, a Chrome extension
running.

Thousands of engineers chasing promotions by dropping support for live code.
If it was their code they wouldn't do it. The org is broken. If you want to
see what mature software support looks like, check out Microsoft. Win32
binaries I wrote in college still run on Win 10. Google looks unimpressive by
comparison. But they all got promoted!

~~~
MaxBarraclough
I'm reminded of a blog post I once read, which annoyingly I now cannot find,
on how the job of a library maintainer is to not break the API. If you break
the API, you've failed at your job. This mindset makes more sense than
treating API breakages as a routine case-by-case decision. As the article puts
it, breaking changes are breaking the library's commitments. (The
library/service distinction doesn't matter much here.)

I remember that same old blog post also lamented overhearing a conversation in
which a developer bragged about how their idea for a change was so good that
everyone agreed to break the library API to implement it. This reminds me of
how a startup being 'highly disruptive' is sometimes fetishised as an end in
itself, but of course it's even worse than that: they're celebrating
undermining the library's dependability.

~~~
jeffffff
sounds kind of like this famous linus rant:
[https://lkml.org/lkml/2012/12/23/75](https://lkml.org/lkml/2012/12/23/75)

~~~
geranim0
Is he always like that? That's toxic af

~~~
fallenpegasus
Complaints about "toxic" are a screen to hide the fact that you can't say
"incorrect".

------
gravypod
> I have been inexorably pushed away from GCP, towards cloud agnosticism.

On a side note I think this is what all of us should be aiming for. Every time
someone on a team I'm working for starts trying to use some very complex tool
that a SaaS/cloud has built for us I always urged them to look for a simpler
solution.

Do you really need AWS Lambda/kinesis/etc or can you write your code and
deploy it onto our cluster as a normal service/api/control loop?

If you make it really easy to write code, deploy it, and monitor it, you'll
often find it's way less devops time to write your software as apis/control
loops than it is to use a large amalgamation of cloud services. I also haven't
found a good way to unit test cloud services and their interlinking.

~~~
Illniyar
On the contrary, every time someone comes to me with a suggestion to build
something in-house or use an open source solution when a aws service exists I
say the hell with that. I don't want to maintain it, build features into it or
operate it.

I want full in vendor lock-in, it saves an immense amount of time.

You just need to pick a vendor that gives a shit about it's customers to be
locked in to - and that's simply not google.

------
Illniyar
> I’m not actively developing on AWS, so I don’t have as much of a sense for
> how often they sunset APIs that they have previously dangled alluringly
> before unwitting developers.

SimpleDB still works, it's not even deprecated or grandfathered to not accept
new users. There is really nothing else you need to know about how AWS treats
deprecation than that single example.

Deprecation is practically unheard of in AWS. If it does happen you can be
damn sure it takes years to actually remove obsolete functionality and there
is a fairly easy migration path to something of equivalent functionality.

~~~
dividuum
Another example: There was a change to how objects within S3 could be
addressed by http(s): [https://aws.amazon.com/blogs/aws/amazon-s3-path-
deprecation-...](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-
plan-the-rest-of-the-story/)

Initially the (justified) change gave you ~18 months to implement necessary
changes. Seems feedback was negative regardless, so they modified it in a way
so current usage remains unaffected.

~~~
aynyc
They didn't have a choice, their internal teams weren't ready for S3 SigV4.
For example, EMR was not ready for SigV4. That means ton of customers can't
use EMR.

------
lazzlazzlazz
Ten years ago it would've been inconceivable to me, but I have to admit that
my default assumption is that Google cannot follow through on any of its
projects. It is no longer a "live player"[1] and lumbers along purely on its
draining intellectual endowment and incumbency.

Network effects and increasing anti-competitive practices will keep Google at
the top for years to come, but without a cultural reinvention it cannot
possibly regain the prestige it once held as an institution.

[1] [https://medium.com/@samo.burja/live-versus-dead-
players-2b24...](https://medium.com/@samo.burja/live-versus-dead-
players-2b24f6e9eae2)

------
rst
Can't help wondering if some of the fall-off in Rails usage is due to a
similar attitude toward deprecation of formerly supported APIs, which has made
upgrading large Rails apps to a new major release a pretty significant
project.

Things they've removed have included the entire original ActiveRecord query
API, and the RJS templating system for Javascript. Those two required rewrites
of significant portions of any app that tried to bridge the transition using
only supported APIs. And every new point release (say, 5.1 to 5.2) brings a
enough more minor deprecations to make an upgrade not a task, but a project.
That's forcing a lot of significant rework, and every one of those gives
people a chance to just take their project elsewhere.

(There's actually a third alternative. The release after these two major
components were removed from core Rails, they were still semi-supported as
add-on libraries. In both cases, core support for those was ultimately cut off
-- but people with old apps could try to keep at least portions of them
working to minimize the damage. I've done that to help avoid rewrites in
components of an older Rails app. I didn't support all of either old API --
for example, rewriting uses of deprecated options to `belongs_to` was easier
than keeping them on life support. But on balance, it was easier than the
rewrites.)

------
walki
I have personally used a large number of cloud compute services: AWS, packet,
Alibaba, Digital Ocean, nimbix, Azure, Oracle, ... I still use all of these
services from time to time. If you carefully look at my list of cloud vendors
you realize that there is only one of the big cloud vendors missing: GCP. Why?

When GCP came out I wanted to switch from AWS to GCP because AWS was costing
me a lot of money and GCP was marketed as being cheaper. So I went to their
website and signed up. Or at least I tried to sign up, because GCP did not
care about individuals, you had to be a company! No other cloud vendor I
tested had this requirement! So up to this date I have never tried to sign up
with GCP again.

~~~
packetslave
Let me guess, you're based somewhere in Europe? I don't remember the
specifics, but the "business only in Europe" thing was 100% the fault of the
EU's (tax?) laws. Not sure if that ever got resolved, actually.

~~~
walki
Yes, I am based in Europe. If this was EU's fault then why didn't any of the
other cloud vendors have this requirement?

------
physicles
The mishandled Python 3 transition sure created a lot of pain (I experienced
some of it, even though I only started with Python in 2016), but I never
realized how much it hurt the language’s momentum. This must be one of the
reasons why JavaScript is so popular today. JS benefits here from being run in
browsers, which the author points out are fantastic at back compat, even as
the dev culture surrounding the language doesn’t value it at all.

~~~
Animats
_I never realized how much it hurt the language’s momentum._

Yes. The combination of a botched transition with a high arrogance level was
hell. Here's something I wrote in 2015 after porting a medium-sized system
from Python 2 to Python 3.[1] I listed a number of package problems
encountered during the conversion. Ones that indicated those libraries weren't
being used much in production. I discovered that the SQL connector broke if
you loaded a database of tens of thousands of records, for example. And oh,
did I get hate mail. You can see the comments below mine.

That's what led me to Go. Go is a mediocre language. That's a strength when
you just need to get something done. It comes with a set of libraries which
are heavily used internally within Google. So, if you're doing something that
is server-side for some web-related task, the libraries for that are probably
both present and well-debugged. And they don't seem to be deprecated rapidly.
I'm not using them for any Google-specific services, so I can't speak to that.

[1]
[https://lists.archive.carbon60.com/python/python/1187081/?pa...](https://lists.archive.carbon60.com/python/python/1187081/?page=1;mh=-1;)

~~~
physicles
Yeah I have the same thoughts about go, which I've been using every day for
the last four years. It's super rare for an upgrade to break anything (has
happened though). When it comes to running stuff in prod, boring is good.

------
skohan
I've been hiring dev ops engineers lately, and the consensus seems to be in
the candidates I've talked to that _all_ of the cloud providers are somewhat
painful to work with.

I've worked the most with AWS so I'll use it as an example, but it seems like
they sell on "new services", so they are incentivized to release quickly,
which leads to services with a lot of holes and poor documentation, making it
almost essential to get paid support to work around these issues.

It makes me wonder what it would look like to build a minimal cloud product
which focused on a few core services with an emphasis on reliability,
performance and developer ergonomics.

~~~
OzzyB
The only option I can see as an alternative "minimal" cloud is one that runs
Kubernetes on one of the "mid-tier" hosting platforms.

DigitalOcean, Linode, heck even OVH now have K8s offerings with ready-to-go
(and free) control node/panel instances.

Of course K8s isn't shallow but at least when you master their way of doing
things you're free to go and do whatever you want.

It's cheaper too.

~~~
raesene9
One small note of caution there is that using managed k8s solutions has some
consequences for flexibility of what you can do.

For one example, in most managed k8s setups, you can't change the admission
controllers used on a cluster unless the cloud provider exposes this via an
interface. Some cloud providers do more of this than others.

So with a minimal k8s managed option you might get stuck if your cloud
provider doesn't support an admission controller or other API server option
you need.

------
1024core
I have heard that at Google, you get promoted/rewarded for launches. To me,
that seems like an obvious perverse incentive to continuously deprecate
things, so new things can be "launched" to replace them. There are no
"launches" if you're just maintaining backwards compatibility!

Maybe GCP should take a hard look at their internal incentives?

~~~
gberger
That's one of the points of the essay. Steve also argues that it's not a GCP
problem, but a Google-wide problem (excluding Android and Chrome).

------
KingOfCoders
"Fuck yooooouuuuuuuu. Fuck you, fuck you, Fuck You. Drop whatever you are
doing because it’s not important. What is important is OUR time."

Most NPM packages.

~~~
sidkshatriya
> “[...] Drop whatever you are doing because it’s not important. What is
> important is OUR time”

> Most NPM packages

NPM packages tend to be free and open source so morally they don’t owe us
anything. So yes what is important is the time of the authors of the packages.
I am grateful someone took the trouble to release them and many popular ones
are actually well maintained.

But cloud services that you are paying for are a totally different ball game.
The company does need to maintain backwards compatibility for a good amount
time and they need to value your time.

~~~
murgindrag
I don't care about moral debts or obligations. I care about whether stuff
works and I can get shit done.

Other ecosystems (like Python) don't have the same culture as NPM. There are
clean, well-documented, maintained packages.

Google doesn't have a moral obligation to support GCE customers either. It's
just that everyone I know who has used GCE would never use it in production,
and loudly advises everyone they know not to do so either. They'll be bleeding
money at some point, not to mention they're already bleeding reputation.
Google went from smartest-people-in-the-room to smug-incompetent-arrogant-
douchebags. That's hard to get over. That's about good business, not some kind
of moral obligation.

~~~
KingOfCoders
Same for Java.

------
pdonis
"GNU Emacs, which is a sort of hybrid between Windows Notepad, a monolithic-
kernel operating system, and the International Space Station."

The article was worth a read for that quote alone.

------
remus
I'd be interested in some concrete examples of things GCP have depreciated.
I'm an (admittedly fairly light) user of GCP and haven't come across any
myself.

~~~
larrymcp
I saw that he mentioned a few of them in the article. He writes:

 _I can tell you that virtually everything I’ve used, from networking (legacy
to VPC) to storage (Cloud SQL v1 to v2) to Firebase (now Firestore with a
totally different API) to App Engine (don’t even get me started) to Cloud
Endpoints to... I dunno, _everything_, has forced me to rewrite it all after
at most 2–3 years._

~~~
habosa
I work on Firebase. His characterization of Firestore is incorrect and lazy.
It did not replace Realtime Database (which is the product he’s calling
Firebase) they are two products which live side by side and both are being
actively improved.

People assumed that because we made a new database we were killing the older
one, but that was never true.

~~~
lclarkmichalek
Well, that’s a novel take on the customer is always right

~~~
joshuamorton
"the customer is always right" doesn't mean every individual customer is
always correct and you should never disagree with any of them, it means the
customer, in aggregate, is correct and you need to react to customer opinions.

But also idk what else you'd want them to say "the thing you're taking shit
isn't deprecated" seems like an important response to "you mishandled this
deprecation".

------
MrStonedOne
Google thinks it can operate the same in paid user bases as it does in free
user bases.

You aren't a customer, you're a user who paid to bypass a rate limit. There is
a difference.

------
jrott
I really want to like google cloud. Every time I use it things seem better set
up than in AWS. I feel like I can't even bring it up at work though because of
the issues with deprecation.

~~~
code4tee
I hear you. At the end of the day though a little more effort on front end
setup is infinitely better than the time wasted later on when you have to
rebuilt everything every time Google decides to get rid of another product /
service / platform / setting. Burned by that one too many times and moved on.

------
9nGQluzmnq3M
So much ranting, yet not a single example of a GCP product that Google has
actually killed. Got examples?

I mean, there's stuff like Python 2 support being sunset on managed services,
but as Yegge himself points out that's on Guido, not GCP.

~~~
joepie91_
The old Firebase API. No longer possible to create new databases that can be
used from the old API.

Which in practice meant that "migrating to a new database under a different
account" translated into "rewriting half a codebase due to a cascade of
breaking changes in the surrounding tooling".

~~~
zwily
Are you talking about the Firebase RTDB? You can’t create new ones anymore?
That’s news to me.

------
staticassertion
Scathing. Well written. Extremely hard to argue with.

GCP will fail, I think everyone knows it, and even if they don't, they _fear
it_.

------
sfifs
I had Google cloud break twice this week. First their "AI Notebooks" images
broke some dependency, then some IAM change broke the cloud console interface.

Though our engg. team is on it, Pointless hours of productivity lost :-(

------
djhworld
Has AWS ever sunsetted a service?

I know they deprecated SimpleDB but I don't think they ever actually shut it
down for existing customers, that might have changed in the last few years
though.

~~~
code4tee
As you mention my understanding is that that there’s one service no longer
offered to new customers but they are still supporting it for those that were
using it. This is a big difference between AWS and Google’s approach to
customers. A number of friends’ companies got hit really hard when Google
suddenly decided they no longer wanted to do something anymore (regardless of
what their customer thought). That list is big and growing: killedbygoogle.com

------
em-bee
i think this is the key point of the article:

 _it actually winds up being less DevOps work, on average, to support open-
source systems running on bare VMs, than to try to keep up with Google’s
deprecation treadmill._

~~~
disgruntledphd2
I honestly feel like this is a better idea in general, at least in data
science stuff.

For context, I tend to get dumped with undocumented poorly written untested DS
code bases every time I move to a new job.

The difference between setups where there's a bare VM/server and it runs using
standard tools (command line scripts) or at least some open-source
orchestration and any kind of proprietary technology tied to infrastructure is
massive.

Additionally, the server stuff can be maintained and debugged by a _far_
larger number of people, thus reducing bus factor.

Like, maybe in the web dev world there are things that everyone knows that are
AWS services (maybe S3 counts, I guess) but in the DS world, I just know that
I'm in for a whole lot more pain if they haven't just used bare servers.

~~~
em-bee
this is the classic advantage of Free Software and Open Source over non-free
tools. you often get a mix of commercial services and inhouse custom
development that is no better documented or supported than FOSS alternatives.

------
rocketraman
There are always trade-offs though.

1) If you're busy maintaining backward compat, you're not busy building
innovative new things -- Microsoft. 'nuf said.

2) You have a built-in excuse to not make your new stuff equal in
functionality or greater to the old stuff, because hey, customers can just use
the old stuff right? I built some software that talks to O365 Sharepoint. Ok,
I have two choices of API: the older Sharepoint API or the newer Graph API.
They recommend Graph, ok. Build the product and put it into production. Get a
new customer requirement for fine-grained auth, and then find out that Graph
doesn't handle fine-grained auth -- only the Sharepoint API does that. Oops.
Ask Microsoft when that capability is coming to Graph? No timeline, because
the Sharepoint API isn't deprecated.

3) Keeping old stuff working may be easier, but building _new_ stuff is harder
because the landscape is muddier -- how many times have you looked at
Microsoft docs and found multiple ways to do things with no idea if the docs
you're looking at actually apply to the approach you're using now?

I think there are ways to have (mostly) the best of both worlds. Linux does it
by bring as much stuff in-tree as they can -- they break compat all the time
for out-of-tree stuff. Ever try to keep a proprietary VMWare module building
without errors? In-tree KVM never breaks because they're super careful with
the ABI. Some languages have explicit mechanisms by which they attempt to keep
things current, but make it easier on users -- Kotlin is an example of this
[1] but its a young language so time will tell whether they can actually
thread this needle. My experience so far is yes, I think they can -- I've
updated code from Kotlin 1.2 to 1.3 to 1.4 with relatively little pain.

[1] [https://kotlinlang.org/docs/reference/evolution/kotlin-
evolu...](https://kotlinlang.org/docs/reference/evolution/kotlin-
evolution.html)

------
tomc1985
I can hear cries from tenderfoot developers whining about supporting multiple
options and backwards compatibility...

"But that's sooooo muuuuch too teeessstt!! Whaaaaaaaaaaaaaaaa!!!!!!!!!!"

But that's your job dude. So what if it costs more money and time. Fucking pay
for it anyway.

I feel like startup software dev needs a swift kick in the ass or three with
that phrase.... "do it anyway!"

Who fucking cares if these founder assholes only make an 800% return instead
of 900%? Do it right.

~~~
dmitriid
You didn't read the article, did you? Startup developers don't mind supporting
multiple options and backwards compatibility. Often they have to because they
don't have the time and the money to rewrite everything, so it's common in
startups to run and support multiple versions of APIs, clients etc.

It's _Google_ who doesn't provide backwards compatibility, upgrade paths and
migration options.

> But that's your job dude. So what if it costs more money and time. Fucking
> pay for it anyway.

So why doesn't Google pay for it anyway?

> I feel like startup software dev needs a swift kick in the ass or three with
> that phrase.... "do it anyway!"

You mean Google devs.

~~~
tomc1985
I read the whole article. My reaction is such because I feel that much of
software dev is plagued by the same bullshit that the article describes about
Google

I've been on the web since the mid 90s. The last 15 years have had more cool
things taken away from me because of sunset bullshit than the first half of
that time period. Software (even web apps) should be eternal, there should
never be "features removed" or "EOL" announcements. If they don't want to run
something they should open source it and let the community take over. What the
hell is the point of hoarding all this wonderful IP if it serves noone?

------
jedberg
And then there is AWS, which has depreciated exactly one service, SDB, but
only after they created a much better replacement (DynamoDB) and offered to
help migrate all of their large customers, so they could get the usage percent
so low that almost no one was affected by the shutdown.

~~~
Aperocky
Disclaimer: AWS SDE

The focus on consistent behavior and once launch, never deprecate practice has
caused many headaches, I personally know of control planes specifically
written to accommodate old behaviors that a few customers had years ago even
when the entire service has been updated. But IMO this is still much better
than deprecating production services and behaviors on the fly.

~~~
yjftsjthsd-h
But by making in Amazon's problem, the customer never experiences breakage,
and that's a _huge_ win.

------
irrational
This puts into words what I have suspected for a long time. When we were
considering which cloud platform to move onto, we considered Google Cloud for
about five seconds before someone said “How long until they kill it?”. And all
of us knew exactly what they meant. It feels like Google’s main mode of
operation is killing projects. We ended up going with AWS.

------
Glyptodon
Besides AWS and Azure what's the third major cloud provider he's referring to?
(I can think of others, just haven't really thought any were close to
Amazon/Microsoft/Google.)

~~~
gundmc
Was wondering the same thing. Alibaba? IBM?

~~~
lowiqengineer
Last I checked Alibaba was 3rd in the international market and Oracle cloud
was growing. Steve might be referring to Oracle getting 3rd place.

------
ec109685
“ it’s well-known that they refuse to host (as a managed service) any third-
party software until after AWS has already done the same thing and built a
successful business around it, at which point their customers hold them at
gunpoint. But that’s the bar, to get Google to support something.”

This isn’t true, of course. They supported managed Kafka before AWS did:
[https://cloud.google.com/confluent](https://cloud.google.com/confluent)

~~~
baskire
Gcp doesn’t have a 1st party Kafka offering. Confluent cloud is still 3rd
party just with some nice integrations with GCP.

If you have an issue it’s not GCP support or GCP’s eng job to fix the cluster.
There’s real service uptime implications of this.

------
antoncohen
People are confusing new versions of things, or new APIs, or new products, or
not recommenced anymore, with _actually removing products_. From this thread
people are complaining about:

\- Datastore -> Firestore in Datastore Mode: This was a seamless transition
with no API change, Google changed the underlying tech.

\- Firebase -> Firestore: Never happened. They are different products. The
Firebase Realtime Database still exists.

\- gcloud CLI updates: It is a CLI tool, it gets new versions.

\- App Engine with Python 2.7: It still exists, you can still use it. There is
a new App Engine v2 that supports Python 3.

\- Cloud SQL v1 to v2: They had a tool to automatically upgrade, it set up
replication and switched over.

AWS has new versions too. RDS Aurora v1 (MySQL 5.6) can't be upgraded in-place
to RDS Aurora v2 (MySQL 5.7)[1]. AWS Lambda runtimes get deprecated[2].

[1]
[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html)

[2] [https://docs.aws.amazon.com/lambda/latest/dg/runtime-
support...](https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-
policy.html)

------
rcarmo
I had a few laughs with this, but (maybe because I have been using Python
since 1.2) I just can't relate to the Python bits.

I migrated all my "working set" stuff to Python 3 piecemeal over a couple of
weeks and never looked back (except when I needed to pick up some older
project and take an hour or so to refactor it).

The "There Will Come Soft Rains" reference, however, was brilliant. I hadn't
thought of that story for _decades_ and it hit home.

~~~
erichocean
> _I just can 't relate to the Python bits._

I work in visual effects/animation, and 2020 is literally the first year
anyone is even _trying_ to use Python 3 (and Python 2.x) is still supported
everywhere. Prior to 2020 Python 3 couldn't even be used at all (and I mean
that literally: could not be used outside of isolated toy examples on alpha
software that wasn't actually running in production).

We (as an industry) have written thousands and thousands and thousands of
business-necessary logic/scripts/libraries in Python 2.x that have to be
updated to Python 3 _that work across commercial tools_ so literally every
tool has to be updated at the same time in tandem to Python 3, so much so that
the VFX Platform[0] project was created in 2014 _specifically to address how
badly Python 3 had fucked everyone in my industry_.

It's all make-work too: literally none of us actually _need_ Python 3's new
features.

Javascript's "use strict"; is the way to do backwards compatibility if you
really really really need to change the semantics and keep existing code
working as-is.

[0] [https://vfxplatform.com/](https://vfxplatform.com/)

------
xenihn
>Businesses are taking a mercenary’s accounting of their dual mobile teams
(iOS and Android), and starting to realize that those phony-sounding dog-and-
pony-show cross-platform development systems like Flutter and React Native
have real teeth, and using them could cut their mobile team sizes in half, or
alternately, make them twice as productive.

Is there much Flutter and React Native dev going on at big companies outside
of Google and Facebook?

I know plenty of smaller companies are using them because they don't have what
sometimes feels like near-infinite money to throw at mobile engineers, but I
can't think of a single Big Name On Blind with iOS or Android position
listings that mention Flutter or RN. They're all native.

Even at Google and Facebook, usage of either of their respective frameworks
seems to be an exception and not the norm, and they're still just components
in a larger native project. I wonder if the goal is to eventually re-write
everything? But that seems impossible given the size of these applications'
source.

------
justincormack
I do hope the intermediate solution he mentions, tooling that upgrades your
code for you, starts to be used more. Go did this in the early days before
1.0, but it is fairly rare still outside the internal Google tools. It is a
big chunk of work obviously but it saves a huge amount of downstream work, and
lets everyone move forward.

~~~
dhanna
Right? This is a strategy/trade-off that depends on migration tooling that
exists and is user friendly.

------
dangoor
While I agree that GCP's deprecation policy is painful, I don't think he's
painting a compelling picture that the deprecation policy, specifically, is
the reason GCP isn't ranked higher.

Taking two of the examples of "bad platform moves": Python 3 and Apple.

Where I work, we are simultaneously dealing with Python 3 and GCP
deprecations[1], and it sucks. But Python is more popular than ever![2] Yes,
some people (including us, at least for our website code) left Python, but
lots and lots of people have moved to Python 3.

I've seen lots of people say "oh, the move from 2 to 3 wasn't that bad because
of the tooling." That's probably true, and we probably wouldn't be moving to
Go were it not for the fact that we had API changes _and_ Python's changes to
deal with (and Go gives us other benefits).

And then there's Apple. Apple has a long history of maintaining compatibility
for a time, and then getting rid of the bits that are holding them back. There
are tradeoffs.

Maybe Android doesn't break things for developers, but Apple makes it so that
iPhone users can keep updating their phones to the latest OS (and apps) for
much longer than the typical Android phone. Because of deprecations and a
general desire to make apps that fit the platform, developers of popular apps
on iOS put the time into updating them, and these apps are really nice to use
as a result.

I think fit and finish on non-first-party native Mac apps tends to be better
than those on average on Windows, though that's just my opinion.

Carrying backwards compatibility comes with tradeoffs, and it's possible to be
successful by making different tradeoffs than maximizing backward
compatibility.

[1]: [https://blog.khanacademy.org/go-services-one-goliath-
project...](https://blog.khanacademy.org/go-services-one-goliath-project/)
[2]: [https://redmonk.com/sogrady/2020/07/27/language-
rankings-6-2...](https://redmonk.com/sogrady/2020/07/27/language-
rankings-6-20/)

------
discordance
One thing that I don't understand about all this deprecation talk, as pointed
out here:

> So when it comes to shouldering the burden of compatibility, you need to pay
> for it. Not us.

Surely if there are shoppers/customers affected by a deprecated service on GCP
(or any cloud provider), then that would indicate a paying customer behind it.
That payment should at least cover costs of running that service (infra,
development, software and maintenance). In that case, why deprecate at all? -
version the API and continue running it as long as it economically makes
sense?

------
rewq4321
TLDR of the article:

> In the Emacs world [...], when they make an API obsolete, they are basically
> saying: “You really shouldn’t use this approach, because even though it
> works, it suffers from various deficiencies which we enumerate here. But in
> the end it’s your call.”

> Whereas in the Google world, deprecation means: “We are breaking our
> commitments to you.” [...] It means they are going to force you to do some
> work, possibly a large amount of rework, on a regular basis

It's fine if they want to go hunting for their Perfect API v10™, but just
leave the old APIs alone - build abstractions under them if needed. Don't push
that work onto your customers.

------
nonbirithm
Two things this essay made me think about.

First: Why don't more programs or services go the `make-obsolete` route and
allow people to use deprecated APIs forever? If maintaining compatibility is
the goal then I can't see how removing a function in a new API version will
ever work. Maybe it's because companies are expecting people to be dumb and
call support when they use this deprecated API that was specifically
designated as unsupported. "But I can use it," they'd say, and still complain
that it breaks or something. Maybe that workload is what they're trying to
avoid. But maybe they'd silently move away instead if continuing to use the
platform meant rewriting everything.

Second: Why don't people usually have automated API refactoring tools? When I
was trying to port a Minecraft mod to a later version I found a bunch of
things had changed in the engine. Okay, go to MCP and look at what changed.
Guess what? It's all a bunch of Markdown. They painstakingly went through the
process of labeling every single API change and method renaming and so on, and
they put it all in _Markdown._

There was so much missed potential there. Imagine if you had a policy
encouraging your developers to put down every API change you make in your
program in a machine-readable, schematized format that could be printed to a
webpage or Markdown or something else, but could _also_ be combined with a
static analysis tool to at least print out the parts that need changing, if
not refactor them, just like Google's internal tooling? If developers don't
notate those changes by hand then that information is going to be permanently
confined to unparseable changelogs and obscure IRC conversations. That's the
kind of thing that a semver increment can never capture the nuance of. Why
guess what broke or manually parse changelogs based on a major version
increment when you can have a computer do all the work for you?

Maybe it's because a changelog is "information to throw away," that you'd have
the painful time trying to read through the moment things break and then can
forget about a couple of days later when the refactoring is done. But the
changelogs and migration steps are essentially a bridge closing a chasm, with
your entire userbase on one end trying to get to the other, and if it's not
made easy enough for them to cross they will either give up and leave or stay
on the other side forever.

------
bickeringyokel
> One fun bit of trivia about Bigtable is that they had these internal
> control-plane entities (as part of the implementation) called tablet
> servers, which had large indexes, and at some point they became a scaling
> bottleneck.

Anyone understand the technical jargon here? I'm a bit lost.

~~~
dilyevsky
Check out the original bigtable paper and it’ll make more sense. I think the
author maybe misremembering some technical details here though.

~~~
jeffbee
Maybe he's talking about the way that bigtable metadata1 is itself hosted on
bigtable? The statements don't make much sense.

------
dilyevsky
Clearly author is making things up - there were no gummy sharks in micro
kitchens. Sergey spoke at lengths about this.

> I considered using Google Cloud Bigtable for my online game, but it costs an
> estimated $16,000/year for an empty Bigtable on GCP. I’m not saying they’re
> gouging you

Ok, but they are

~~~
rrdharan
[https://techcrunch.com/2020/04/07/google-cloud-makes-it-
chea...](https://techcrunch.com/2020/04/07/google-cloud-makes-it-cheaper-to-
run-smaller-workloads-on-bigtable/)

~~~
dilyevsky
Cool so i can build a three node cockroach/tidb/tikv cluster that will have
much higher reliability with the same performance characteristics for less
money. Sounds like they still gouging you.

------
micimize
There are other tradeoffs between cloud providers that are important to keep
in mind here. In my experience, GCP has been easier to get started with, and
has offerings that are by-and-large more comprehensible. One thing I really
appreciate is that a lot of their client libs have automatic auth within GCP
(last time I used AWS this wasn't the case or I didn't know about it).

As someone who hasn't been burned by it personally, it is hard to quantify the
actual risk and maintenance cost of GCP's deprecation policy, but I know
decision makers at larger organizations that are rightfully afraid of it.

~~~
manquer
Easier to get started ? Their UI , IAM architecture and firebase being kind of
separate is so confusing .

Aws used to be simpler , while they are screwing it up now, the recent route53
upgrade was terrible , a single click action has now become 4-5 clicks they
are still better than GCP.

Azure has the really the best interface (outside of DO) , it is easy to go to
related and nested resource in one freaking app. Both google and aws end up
forcing you to open three four tabs to setup interconnected apps .

DO interface is really really good ,however they don’t have the complexity of
features that the big three do , so not a fair comparison

~~~
micimize
That's just been my experience – I remember having much more pains trying to
get AWS IAMs and permissions to work, and I really like that they give you
suggestions like "this machine could be smaller" or "this role never actually
uses all these permissions."

Also I haven't worked _that_ much with firebase, but it seems like a great
example of the benefit of using GCP. Firebase is a cohesive and accessible
solution to a lot of what can be fairly nightmarish technical problems. This
kind of thing will always depend on the project/team/team size, but I'm just
trying to say that there are significant benefits to GCP that should be
considered.

~~~
manquer
Google are not competing with DO for developer mindspace, for large
enterprises stability matters a lot more than new features and enterprise
support matters too, for startups serving enterprises they don't do much
either, what's left is consumer focused companies like SnapChat who can thrive
on just innovative new tech.

I can get Microsoft on a call anytime, I have account managers responsible who
talk to me atleast once a month, reach their product teams, get preview
access, get the MS account manager of my customer to help with a deal, put
them in front of my customer, help with compliance, even get their sales guys
to recommend my product. To a lesser extent I can do a lot of that with AWS
too, with even sub $100k/year spends they will still put an account manager
for you. I am not sure any of this was possible with GCP for most customers.

It is _extremely hard_ to get a human from Google to talk to you even at
$100k+/year GCP spends. Google has contracted a lot of partners to do all the
heavy lifting in support for them so they don't have to do the hard work. It
does not work, the partners can not do much beyond what is available on the
portal or clarify beyond the documentation.

Azure serves enterprises really well, and has really made effort with
developers and their support for startups in fantastic, AWS is not as good
yet, however it feels like they are really trying and their tech popularity
works in their favour and they care about backward compatibility _a lot_ S3
API _from 2006_ still works. With Google and GCP it does not even look as
though they are trying at all.

~~~
EE84M3i
>S3 API from 2006 still works

An interesting note is that Amazon actually planned to EOL part of this vis-a-
vis what path to address objects at, and then walked it back (for buckets
created before a certain date):
[https://aws.amazon.com/blogs/aws/amazon-s3-path-
deprecation-...](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-
plan-the-rest-of-the-story/) I think the behavior is still considered "old
style" if not explicitly "deprecated", but is "supported".

~~~
manquer
Yes. That decision to continue to support is costing them a ton of money every
day. The older API is _very_ expensive, This is also why backblaze originally
did not add a S3 compatibility layer.[1]

This is exactly the kind of commitment I expect from Microsoft or Amazon, it
is why enterprises pay premium for a product.

In a similar situation I imagine Google would have just sent a note and with a
window of few months and shutdown the old API. Great for innovation and
keeping your tech cutting edge, not so much for the customers who are not as
agile as they and can only move slowly if at all.

[1] [https://www.backblaze.com/blog/design-thinking-b2-apis-
the-h...](https://www.backblaze.com/blog/design-thinking-b2-apis-the-hidden-
costs-of-s3-compatibility/)

------
waheoo
I recently migrated off of digital ocean into aws.

I was originally planning to go to go for the native k8s experience.

Then I read somewhere online that was doesn't sunset services where as gcp
does.

I was half way through the migration process to gcp and bailed on a dime.

~~~
dragonwriter
What I really want is a cloud provider that has AWS’s “we don't kill anything
anyone is still using” and Google's probability of the documentation being
both where you expect, having what you need, and being current and accurate
(and, yes, I know Google is far from perfect on that, but dear God is AWS
utterly _horrible_ and that's just for the services that they are actively
pushing.)

------
znpy
I would like to offer some kind words for the author of this post but in all
honesty I can't.

Why is that guy still giving money to GCP if it's so bad?

Go back to AWS. There's a reason if AWS is still the market leader.

------
sabujp
Google messed up when it sold boston dynamics (where's my cooking,
dishwashing, and laundry (folding) robot?), didn't buy github, and sold
motorola (i just bought my wife a motorola edge)

------
kanzenryu2
So what is the 4th best cloud platform that he thinks might overtake GCP?

~~~
iforgotpassword
Oracle, or globally, Alibaba.

------
doonesbury
Well, hmmm, if only there was one or two more FUs _then_ I would have agreed
with the Op.(sarcasm).

A phd quit my employer and went to google. On last week we were discussing
relative differences. He pointed out searching for _X_ really had no wrong
results and even if you didn't like it who you gonna call? Google probably is
a little siloed from customers in a way many other business are not. Our IT
systems deal in money. There's nowhere to run and hide from customers.

------
lordleft
This is why Microsoft, for all its faults, has huge mindshare in the
enterprise: they don't give up on anything, and when they do, they give you
years to make the transition.

------
pragmatic
What is the fourth cloud offering that may be number 3 soon.

I've used AWS and Azure gig.current uses GCP (don't get me started on that)
but what is the fourth?

------
k__
Weren't GCP users running around telling everyone _" Google only shuts down
consumer products and not its cloud services!!"_

~~~
tedd4u
We’ve been in GCP for 5 years and this really hasn’t been an issue for us.
Good price, solid performance and reliability, some fancy stuff you can’t get
elsewhere.

------
luord
One thing I was left curious about after reading this is, what is that third
cloud that Google could fall behind of? I know they're after AWS and Azure,
but there are several candidates for that fourth place.

~~~
hyperdimension
Alibaba, I think. Probably the 'natural' choice for Chinese-facing developers,
but I think Azure has a region there as well.

------
sukilot
The main problem of Google Cloud is simply that that no amount of money can
pay off a 10 year late start in a race against the #1 incumbent.

Microsoft still wins on the desktop. The only way to beat them was to compete
in new emerging markets.

------
tschellenbach
Android is the top example of this by far, everything is half finished &
deprecated.

------
bilater
Am I in the minority being a GCP fan? Articles like these leave me conflicted
because on the one hand he's right but on the other hand I actually really
like GCP and Firebase is a godsend. It would really suck if GCP goes under.

------
lowlevel
I've been saying this about Google for a decade... everyone thinks I'm nuts.

------
blickentwapft
A perfect, spot on analysis.

The only thing that’s weird about this is that it’s about 3 years late.

If this had been written 3 years ago it would have been topical.

But really it’s old news that google deprecates everything and leaves it’s
developers in the lurch and that as a result no one wants to use their stuff.

It’s not surprising to hear this from Steve Yegge cause he’s a super switched
on guy, but it’s surprising to hear this from him NOW.

Google’s cloud ship sailed years ago, they just haven’t yet got to shutting
the whole thing down, but the writing is on the wall.

Google seems totally oblivious to how much developers distrust them, which is
probably because dump trucks turn up every hour or so and dump cash onto
everything which probably makes everyone feel that everything they do is
awesome, even if it isn’t.

~~~
te_chris
I mean maybe I’m not their dream customer or whatever, but we’re happy with
google cloud for our purposes - GKE, CloudSQL, Redis.

I really enjoyed the essay and surely is some truth, but these products have
all been pretty good for us.

~~~
blickentwapft
No doubt what is there works and works well.

The issue is that developer confidence is very low due to google being in a
never ending relentless drive to cancel stuff.

------
ryanobjc
I used to work with Steve. He’s a good story teller. But a story teller tells
good stories by having a fairly casual relationship with facts and the truth.

For example, emacs lisp suffers from bitrot. Some core things haven’t changed
in forever but also you can’t pick up elisp config from 15 years ago and it’s
fine today.

All live systems evolve and change. In this case emacs is just so slow that
Steve thinks that it doesn’t.

~~~
jiggawatts
Nobody pays EMACS a few million a year and runs their core business through
it.

------
neop1x
That's the price you pay for being lazy and using closed, proprietary
technologies and acting on their marketing. Like GCP and AWS were the only
clouds there. Like you had to have everything hosted by them under "planet-
scale" buzzword.

------
EE84M3i
Honestly I just use GCP for everything only because AWS console logs me out
all the time and 2faing with TOTP is a pain.

