
Bringing Pokémon GO to life on Google Cloud - bryanmau1
https://cloudplatform.googleblog.com/2016/09/bringing-Pokemon-GO-to-life-on-Google-Cloud.html
======
jcastro
A bunch of the comments are already pointing out the launch issues Pokemon Go
had, and it's well known that a rep from AWS was also throwing jabs at them
during launch for their issues.

It would be naive for everyone to assume that a high traffic launch is all
about the cloud underneath and only that.

The article didn't mention any of the technical details of the Pokemon
application itself, for all we know the infrastructure was humming nicely and
the application itself didn't scale. Or the other way around, or a combination
of both or one of the other of thousands of moving pieces it takes to launch
something.

~~~
spacehunt
> it's well known that a rep from AWS was also throwing jabs at them during
> launch for their issues

That's pretty low. FWIW Simcity 2013 is on AWS and the launch was far more
disastrous. Doesn't prove anything.

~~~
mpdehaan2
Programming architecture/problems in SC, yes.

I think Pokemon scaled remarkably better than any app many of us have ever
dealt with, seeing the crazy demand spikes. You would run into a lot of app
issues, data choices, a lot of things that don't just get solved from simple
uses of autoscaling.

~~~
surge
It scaled better than Twitter, it hit the same active user count (or close to
it) in a month that took Twitter years to reach, and Twitter would sh*t the
bed all the time with the infamous "fail whale" popping up for users
constantly even with its steady increase in growth over time and time to
adapt. Twitter was pioneering at reaching that scale, that many now use as
guide/lessons learned now, so its forgiven, but how many companies see that
much growth, that fast?

~~~
rryan
To be fair -- Twitter is a much harder architectural problem (significant fan-
out and fan-in problems to solve).

Pokemon Go is less complicated in terms of the conceptual model. There are
only a few pretty simple ways that different players can interact (gyms, lures
on pokestops, that's about it).

~~~
chickenbane
On the other hand, Twitter is dealing with strings < 140 character. Pokemon Go
is tracking all the trainers and Pokemon at all times, not to mention the gyms
and lures.

~~~
andy_ppp
Sure, but you can partition based on country, city and even block, it's clear
how this should work.

None of Twitter can be partitioned like that because every user follows
different people.

------
richardlblair
> Not everything was smooth sailing at launch! When issues emerged around the
> game’s stability, Niantic and Google engineers braved each problem in
> sequence, working quickly to create and deploy solutions. Google CRE worked
> hand-in-hand with Niantic to review every part of their architecture,
> tapping the expertise of core Google Cloud engineers and product managers —
> all against a backdrop of millions of new players pouring into the game.

IMO This is the most valuable thing in this article. It essentially says what
others are pointing out. You can't just press a button and have scale. It's
not that easy. You have to tackle many layers. Considering they 50x'd their
worst case scenario it would have only taken a few bad queries to fuck shit
up.

~~~
user5994461
> You can't just press a button and have scale

Actually you can... if the system was designed for this purpose.

~~~
bottled_poe
Technically true, but it doesn't make sense from a business perspective.
Anticipating and accounting for every scaling issue costs a lot of money. Most
businesses would decide that satisfying 90% of scenarios is sufficient.

~~~
user5994461
Business is irrelevant once you're sure that the service will go into
production AND will have some amount of users.

It's known that some things will break from time to time (e.g. servers, hard
drives, networking). If you've designed the application poorly so that a
single minor failure can snowball to the entire system, it's guaranteed that
your site will fall apart periodically, costing you lots of money each time.

At this point, the best technical AND business converge to "pick an option
that can sustain [minor] breakages". Either you've done it like that from the
start, or you're in the process of rewriting/redesigning your site.

~~~
bottled_poe
> Business is irrelevant once...

Not really. In the case described they had accounted for some maximum number
of users and instead got 50x that number. It is reasonable to design a system
which handles just some planned number as long as there is a strategy in place
to manage a situation beyond those limits. In this case the strategy was to
apply engineering effort. That seems like a valid risk management plan to me.

------
mooman219
Currently on Cloud here at Google. I would like to elaborate "Google CRE
seamlessly provisioned extra capacity on behalf of Niantic to stay well ahead
of their record-setting growth."

Just because you have the resources does not make for a well scaling service.
As outlined in the post, "Google CRE worked hand-in-hand with Niantic to
review every part of their architecture, tapping the expertise of core Google
Cloud engineers and product managers". Look past the chart, this wasn't just
an estimate for Google, it was also for Niantic. You don't have unlimited
development resources, not every aspect of an application may have had the
time to flesh out a scale-able approach.

Netflix didn't scale out overnight. I'm sure we've seen their techblog about X
extremely specialized framework/tool they've built out over the years. I'm
impressed with how quickly Niantic achieved a playable experience.

~~~
Fiahil
> I'm impressed with how quickly Niantic achieved a playable experience.

I have mixed feelings about this, didn't they drop more than half their peak
user base to achieve playable experience?

~~~
Kiro
Where did you read that? Pretty sure peak was after all the problems.

~~~
Fiahil
I experienced it myself. There was chunks of 6+ hours where you could not
connect at all, and you had to restart the app every 10 minutes (or every
third pokémon), and wait while it was draining your battery like an hungry
walrus on its bucket of fish.

However, it became perfectly playable after their "tracker patch" (which was
the trigger for the massive exodus that followed).

~~~
Kiro
What tracker patch are you referring to? They changed it several times and I
hardly doubt there was any massive exodus due to any of them. The loud
minority on reddit is not representative of the game's community in large.

When the game experienced the most instability it hadn't even been launched in
most countries.

~~~
Fiahil
The first one, when they dropped third parties and removed the already-bugged
step count under close-by pokémons.

It's purely anecdotic, but after that release, popular spots (with multiple
pokestops and arena in one place) usually crowded with more than 50 people
playing actively were down to one dude on a bench.

Sure the "loud minority" is not representative of the whole game's community,
but they were it's core players. They shaped the game's community (and niantic
didn't helped) from the beginning. And it didn't help when those people
decided to stop playing.

Back to server's issues, Google CREs did a very good job during these times.
They can't be blamed for trying to scale an app that was absolutely not
designed to get this high.

------
AaronFriel
This is pure marketing that might convince decision makers and execs that
didn't play Pokémon Go.

No doubt, the CRE program could prove valuable. But in this case, they are
congratulating themselves on a rocky and widely panned launch of a product on
their platform. One might wonder, "If this is what deploying a viral app on
Google Cloud Platform looks like when you have help from Google engineers,
what chance does anyone else have of getting something right on their
platform?"

I think that's probably the wrong takeaway, but it's not difficult for me to
imagine that being the only conclusion one has.

~~~
itcmcgrath
"Niantic phoned in to Google CRE for reinforcements" -> Note this was post-
launch on the world's largest ever mobile launch.

~~~
AaronFriel
Article states that it was still prior to the US launch date.

~~~
itcmcgrath
"the next day". Looking on the graph shows there wasn't a lot of time between
initial launch, and exceeding the world-wide worst case estimates.

------
kylecordes
From a user point of view it did not go nearly as smoothly as it is made to
sound in this post. I would love to see a follow-up post about what went wrong
along the way, and whether the hype surge might have continued longer and
stronger if more people picking it up for the first time would've had a smooth
experience (rather than lots and lots of errors talking to the servers).

~~~
Lewisham
Look at this chart:

[https://4.bp.blogspot.com/-GLc5ve5_djc/V-ysZgW6uDI/AAAAAAAAD...](https://4.bp.blogspot.com/-GLc5ve5_djc/V-ysZgW6uDI/AAAAAAAADIY/-tVg2iC0qQcWBwjc0vBVBxVcJ9Yhbij6gCLcB/s1600/niantic-1.png)

The worst case scenario was 5x, which was a factor of 10 off. If you are that
far off when you do your capacity planning, you can be pretty sure you've got
problems throughout your entire stack.

~~~
MBCook
That shows their estimates were _terrible_.

The article says (in one of the only bits of real information) that they
blasted past their estimates with only Australia and NZ just 15m after launch.

Whoever came up with those numbers must have had some serious methodology
flaws. I know they couldn't predict that it would become the biggest online
game ever for a while, but the initial demand prediction was clearly _way_ off
even before it started growing like a rocket.

~~~
bduerst
>That shows their estimates were terrible.

Pokemon Go became the most-downloaded app in history within the first few
days.

Are you saying that's the bar that people should estimate at launch?

~~~
kmm
It shows that they were over their worst estimate total volume in one day.
Were they really expecting less than a few million user-days in total?

~~~
bduerst
From the article, it seems like a few million users/day is a typical
expectation, but record-breaking isn't.

>Throughout my career as an engineer, I’ve had a hand in numerous product
launches that grew to millions of users. User adoption typically happens
gradually over several months, with new features and architectural changes
scheduled over relatively long periods of time.

------
ShakataGaNai
I feel really bad for this article. It is the Kobayashi Maru of sales pitches.
Working in IT/DevOps/Servers/Software Dev/Etc all my life, I understand that
even if you have the servers, scaling can be hard and time consuming. I also
can't even imagine supporting the number of people they have. So they did an
awesome job.

However, the Pokemon Go player in me says "Wow, even with all of Google's
resources, they still couldn't manage to get this remotely stable for several
weeks?".

I'm sure there was many amazing technical feats that occurred, and from a
deeply technical level this is a good sales pitch. I'm sure a good sales
person could spin it even better "50x your expected traffic? Google Cloud can
do that!". But beyond that... most people will probably see this as a failure.

------
wnevets
On one hand handling such huge amount of traffic is crazy hard and an amazing
accomplishment however the tone of the blog is off putting because of just how
much a trainwreck it was from the users point of view.

~~~
dmix
Yeah the graph indicates resource demand rather than user playability. This is
why it's critical for KPIs/metrics to be grounded in reality. Reality being
that data reflects your business objectives - which in this case is the user
was able to enjoy the experience.

From watching my partner play the game (from Canada), the scaling issues went
on for days and it serverly hampered her enjoyment of the game. This should be
a cautionary tale of having worse case scenario planning at launch time, which
a large organization like Nintendo should have factored in (when planning with
Niantic).

This article doesn't fairly reflect that. But maybe they are acting confident
now because they _will_ be better prepared next time? They can point to
Pokemon Go to why you need experts who have been through the trenches of a
rocky high-traffic launch.

~~~
brazzledazzle
It's probably because their KPIs/metrics are in fact grounded in reality, the
reality of being the cloud platform provider. This isn't a blog post by
Niantic.

~~~
dmix
Right the KPIs were Google's not Niantic... there's an interesting story here
that's being glossed over as a result.

------
lnanek2
> paid off when the game launched without incident in Japan, where the number
> of new users signing up to play tripled the US launch two weeks earlier.

The "without incident" part is hilarious. The game was unusable for over a
week when they added more countries. There were memes all over the place about
Niantic execs ignoring the burning server and pushing to launch in more
countries anyway. I wonder if any of them actually tried to play as a user on
the public servers and spent hours trying to logon and it failing, or locking
up soon after for a week.

Not to mention they never even got the original tracker functionality (1
footstep, 2 footstep, 3 footstep for anything nearby) working again after
that, they had to replace it with a lower load knock off where you just see
what is around a certain location that isn't very popular. So not only did
they not even keep login working, they cut features too.

~~~
cbhl
Those countries were all side-loading US APKs anyway, so not launching in
those countries wouldn't have reduced load on the servers much.

~~~
Tepix
There were no Pokéstops, arenas or spawns in Asia after beta and before launch
so even if you had the APK it was of no use.

------
daveloyall
To those that played and were not impressed with game/system stability: okay,
okay. I wasn't there, I don't know.

...But, according to the _nightly news_ , it was a tremendous success. The
word 'ever' came up a lot.

PMG was the first successful overnight/viral planet-wide/client-side launch.
People who had never heard of it saw it on the news and then visited their
local app store in response.

And according to the googleblog, it took a tremendous expenditure of money,
hardware, electricity, skills, and knowledge to pull it off.

Makes me wonder... Did some other game/app go almost global, but fall short
for the want of those very resources described in the blog post?

Something for app devs to think about.

~~~
enolan
The broad consensus in the games press was that Pokemon Go is a great example
of a strong gameplay loop overriding a massive technical failure.

Google's post is weird because they seem to think the game was a technical
success. Google may have done great, it's impossible to tell from the outside,
but the actual user experience is - or at least was when I played it - awful.

~~~
kevincox
I think what this article is saying is that the game was relatively
technically successful. It admits there were some problems but scaling your
infrastructure to 50x what you expected is pretty amazing and most people
would have expected more downtime for a similar situation.

So not total-success, but very successful for the situation at hand.

(disclaimer: I work at Google but not on GCP)

------
neves
This was one of the most downloaded apps from all time. Went from zero to
gazillions of requests in a single day. Nobody could have planned this. Com'on
these guys are great.

~~~
zzguy
Yeah the negativity here is overwhelming. Which, isn't surprising, HN's
comment section isn't the cheeriest place on the internet. But seriously, for
a tech news aggregator you'd think more of the users would appreciate at least
the difficulty of scaling an app from nothing to THE most popular app ever, in
a matter of days. Yeah they had/still have issues that they could've mentioned
in the article, but it doesn't take away from what they DID do.

------
Declanomous
I really appreciate this blog post. It gives a great insight into what is
going on behind the scenes. I was really surprised by how low their their
worse case scenario was. Absolute worst case would be every single person
capable of running the game playing. Obviously this wouldn't happen, but for a
brand with as much recognition as Pokemon, I think "What if everyone in the
world started using this" is a good place to start. Obviously this won't
happen, but it's important to think about why it won't happen. "What if
everyone who has played Pokemon or wanted to know more about Pokemon
downloaded this game?" is still unlikely, but it's less unlikely. It's
probably not far off from what actually happened.

I don't want to criticize their model too much, because it's obviously
simplified for our benefit. However, it appears that their worst-case scenario
was "What if we become the next bejeweled or [insert popular F2P game here]?"
It's a ridiculous assumption, because Pokemon has a much broader appeal than
any other casual game, cause the IP is so insanely popular, and the game still
appeals to people who just want a casual game. I know it is a lot easier to
get fired for spending too much money than it is for not spending enough, but
it's a stretch to say their launch traffic was beyond imagination. Niantic
should start looking for new analysts now if their current analysts honestly
thought this traffic was outside the realm of possibility.

I don't consider the server issues to be much of a problem though. It's hard
to ensure everything will work perfectly under that kind of load. You have to
accurately predict who will be playing, how much they will be playing, how
they will interact with the game, and so much more. However, I do think they
need to figure out their communication with the fan base. I know that there
will be a vocal portion of any constituency that hates everything. That isn't
a good excuse for communicating poorly. Good communication will help almost
every relationship.

~~~
jdcarter
> I was really surprised by how low their their worse case scenario was.

There's no Y axis on the chart, so we don't know exactly what their estimate
actually was. Regardless, I'm pretty sure Pokemon GO exceeded _any_ reasonable
expectations of popularity, even accounting for the brand and marketing
efforts behind it.

From my own experience, lots of people that never engaged with mobile games
before started playing Pokemon GO within days of its release. My entire
extending family was playing the game. Local bars have become arenas for
Pokemon fights. The adoption of this game was absolutely _crazy_.

So even given the scaling problems, the features they had to remove from the
game, and the bugs they introduced, I think this is still a solid win for
Google CRE.

~~~
Declanomous
I'm assuming the Y-axis is to scale. Assuming it isn't, the numbers they give
imply what actually happened is 10x worse than their worse case. Unless
Niantic defines "worst case" as "1 in 10 chance of happening", being an entire
factor off in a worst case prediction is really bad.

For example, one of the highest estimates of daily US users I saw was 25
million. That's way more than I would have guessed off the top of my head, but
it's no where near 10 times what I would have guessed the "worst case"
scenario would have been. Pokemon Red and Blue sold 9.85 million copies on
gameboy alone. My worst case scenario would assume that literally everyone who
played the game as a kid would want to check out the game as an adult. Our
worse case is massively more accurate than theirs just using the Gameboy
numbers alone, with the assumption that each cart sold was only played by one
person.

The assumption that people who have never played mobile games before wouldn't
be interested in Pokemon is kind of crazy as well. I can't think of another
mobile game that has as much name recognition before it released as Pokemon
go. It's a beloved franchise, whose popularity spans generations. On top of
that, cellphones are far more ubiquitous than the gameboy ever was. There are
people who are playing Pokemon go who have never been in to mobile gaming, but
I doubt the number of people playing Pokemon Go who have never played another
video game in their life is much, much, much smaller.

We had our team building event for my company at an barcade last year. Every
single high score was set by someone older than 50. Literally every single
person at the company had played Ms. Pac Man before, including some people who
literally have never owned a cell phone. The people who don't own a cellphone
are obviously not playing Pokemon Go, but video games are not niche, and have
not been niche for a long time now.

I will give Niantic massive credit for recognizing how much of a problem this
was going to be immediately. I'm sure I would have under-allocated resources
on launch day too, and I don't see myself handling that issue as well. Still,
there's no way in hell what actually happened would be 10x worse than my
actual worse case.

~~~
zardeh
>Pokemon Red and Blue sold 9.85 million copies on gameboy alone.

That was two games, and that was "by the end of their run".

>The assumption that people who have never played mobile games before wouldn't
be interested in Pokemon is kind of crazy as well. I can't think of another
mobile game that has as much name recognition before it released as Pokemon
go. It's a beloved franchise, whose popularity spans generations. On top of
that, cellphones are far more ubiquitous than the gameboy ever was. There are
people who are playing Pokemon go who have never been in to mobile gaming, but
I doubt the number of people playing Pokemon Go who have never played another
video game in their life is much, much, much smaller.

You're aided by hindsight. PKGO grew faster than any prior mobile game or any
previous pokemon IP, by a large margin.

------
Tepix
Nice, but it's disappointing that they do not mention any hard numbers such as
concurrent players, requests per second, traffic, etc.

That makes the article a lot less interesting and worthwhile.

~~~
lucb1e
Or anything technical whatsoever. The article sounds very much written by a
marketing person.

~~~
BinaryIdiot
I would agree. No hard numbers are mentioned so it's hard to understand the
context or scale. However Google did mention their new service offering,
Google CRE, that gives you access to this same exact service that the post
discusses.

So yes this is a PR piece. Not very technical at all.

------
Perixoog
>... Google Cloud customer...

That's pretty misleading - I believe Google's parent company still own part of
Niantic. So other customers shouldn't expect the (implied) same access to
Google resources.

~~~
bitmapbrother
Google is just one of many series A investors in Niantic:

Alsop Louie Partners

Cyan Banister

Google

Lucas Nealan

Nintendo

Pokémon

Scott Banister

You & Mr Jones Brandtech Ventures

~~~
eloisant
Yes, but being a spin-off from Google I imagine most of their employees are
ex-Googlers and as such have a good network inside Google.

------
KirinDave
I can't be the only person who looked at that graph and burst out with a
cackle that startled everyone around them. The deep and inescapable dread of
that fire burning around you even as you make history must have been quite a
feeling.

Or in the vernacular of youth: "This is fine. Everything is fine" as a scaling
graph.

------
daok
So their estimation was that nothing would increase? I am not sure I trust the
two parallels lines in the graph. The estimation should have been a spike at
the release date a small drop and some grow over time, no?

~~~
Lewisham
It's just an upper-bound. "At its worst peak it'll be 5x"

------
Thaxll
It's easy to blame on the cloud where the application server probably had a
lot of issues.

------
acallahan
Related: a similar story of scaling the mobile app "Draw Something", where
usage was doubling every day.

[http://www.gamasutra.com/view/news/168799/Scale_Something_Ho...](http://www.gamasutra.com/view/news/168799/Scale_Something_How_Draw_Something_rode_its_rocket_ship_of_growth.php)

~~~
foxylion
Thanks! From a technical perspective this is a lot better written. And not so
much marketing.

------
randomsofr
Pokemon GO really disappointed me, i'm a big fan of Pokemon and this app
really sucks. It is really buggy. I stopped playing two weeks ago because of
the GPS instability. I hope they get it right some day. But i'm glad they
fixed the server issues.

------
kriro
Interesting read but a bit too positive overall. I think the biggest failure
of the launch was not learning from the initial launch zones before launching
the other zones.

The launch in Europe was a catastrophe imo (constant crashes and freezes). I
don't know how much of this is to blame on the cloud infrastructure but I
suspect it's not nothing. I feel they didn't provide nearly enough
infrastructure given the data they should have had from Australia/USA.

All that being said I think they smoothed out everything and the system seems
to be running very nicely now given the scale. It's certainly a positive
engineering tale overall.

------
Cyph0n
Anyone know what kind of backend Pokemon Go is running on? I'm guessing Java
or Go?

~~~
puddintane
As the following aggregation states possibly Java and Google Cloud [1].

[1] [https://www.quora.com/What-server-infrastructure-is-
Pok%C3%A...](https://www.quora.com/What-server-infrastructure-is-Pok%C3%A9mon-
GO-running-on-that-is-performing-so-poorly)

~~~
Cyph0n
Looks like it, thanks for the reply.

------
ksec
Software Scalability issues aside. I am not sure if Pokemon Go would ever be
possible if it not on the cloud. How could you get instances up this fast. It
had explosion of players in very little time. There is no way you could have
planned this resources ahead of time. And it die down fairly quickly, which
means you would have lots of unused server if it were not for cloud.

------
pingec
Would be really interesting to read up on some of the bottlenecks they had
identified and how they optimized them away. I know at some point they had to
turn off the "location where a pokemon was caught" map and the radar to keep
the thing responsive :)

------
dom96
I'm really disappointed with Pokemon GO. Despite the issues with the launch I
still had a lot of fun playing it initially. But the tracker breaking together
with the lack of communication from Niantic killed it for me.

Interestingly, the one time they did decide to communicate was when they
announced that they banned a bunch of third parties from accessing their
server[1]. Of course, just like in this post, they show a graph with a missing
y-axis which tells you very little about the traffic they actually received.

It's surprising that this wasn't mentioned in the Google blog, since according
to Niantic it was thanks to this ban that they were able to launch in more
regions.

1 -
[http://pokemongo.nianticlabs.com/en/post/update-080416/](http://pokemongo.nianticlabs.com/en/post/update-080416/)

~~~
dom96
I don't do this often but I am confused so I will ask: can anyone explained
why they downvoted this?

~~~
puddintane
Unsure, it seems only posts that reflect the negativity of the application are
being down voted.

Being this type of site I wonder if a lot of the employee's at Niantic have
accounts and could be down voting?

Either way it would be nice to hear why other users are down voting these
because the users comment is within the guidelines.

------
AndrewKemendo
_When issues emerged around the game’s stability_

I'm most curious about how they identified issues. Obviously crash reports are
a thing, maybe traffic to a login error page, twitter complaints etc...

I wonder how they managed prioritizing issues.

------
rokm
The article says containers were used, but the D word is not mentioned?

~~~
scott_karana
Kubernetes doesn't necessarily imply Docker.

~~~
kevincox
IIUC it pretty much does. When I asked for the option to just run an arbitrary
command on a host (instead of a container) the answer was pretty much to
create an empty container that mounted the host root to the container root and
pretend the container wasn't there.

------
dozwang
the Pokemon Go Cloud story in Chinese.
[http://www.ithome.com.tw/news/108719](http://www.ithome.com.tw/news/108719)

------
ihsw
I'm more interested in how far popularity has crashed.

I was previously interested in the game but Niantic has shown zero interest in
being responsive to the community's concerns.

~~~
jandrese
There have been a few numbers about the crash. It's there, but Pokemon Go
still remains one of if not the most popular mobile game on a day to day
basis.

Even with the servers stabilized the game has plenty of other issues that need
addressing. The "nearby Pokemon" window is still broken and shows no sign of
ever getting fixed. People in suburbs and rural environments are still stuck
playing in extra grindy hard mode. The distance tracking has a speed limit
that is way too slow (10.5kph) and noisy so it largely fails if you jog and or
get on a bike. The gym battles are comically out of balance (slow defensive
Pokemon are blatantly overpowered).

They shouldn't be hard problems to fix but Niantic doesn't seem interested.
They toyed with a half-fix to the nearby Pokemon window, but only deployed it
in their hometown and then seemingly forgot about it. It's clear the client
has enough data to do the launch style footsteps (see: all of the PokeRadar
apps on the store), but for some reason Niantic has no interest in re-enabling
it.

~~~
shkkmo
> It's clear the client has enough data to do the launch style footsteps (see:
> all of the PokeRadar apps on the store), but for some reason Niantic has no
> interest in re-enabling it.

I've always suspected that was because they quickly realized that getting
pokemon to not spawn in dangerous spots was an insurmountable task, so having
tracking that encourages players to go to those locations was asking for even
more lawsuits that what they face currently for that issue.

edit: added quote I was responding to

~~~
dilap
that's quite sad if we live in a legal environment that games that tell you
"go here" basically can't be made. :-/

~~~
wingerlang
Or.. just don't point people to private property? Plenty of public space
available.

~~~
jandrese
Finding a map for the entire world that denotes what is public and what is
private property is going to be difficult.

------
bkjsbkjdnf
This is just a worthless advertisement.

------
heh
This all doesn't really matter when you lock out anyone who has a rooted
phone, with no warning. Sure, a few people were cheating with them, but anyone
running cyanogenmod or any other custom firmware, or anyone who wants to get
rid of a god-awful OEM skin, is now screwed out of the game. Ironically, this
hasn't stopped the actual cheaters or people using AutoMagisk.

~~~
lucb1e
Wait, they did that? That's like blocking anyone from playing any games when
they have the administrator password to their computers. Being very pro-
rooting (for independence reasons, how do you "own" a device that you don't
have access to?) this is a good reason to boycott the game on any phone.

~~~
makomk
Yeah, they did that using a very aggressive Google-provided tool called
SafetyNet designed exactly for that purpose, which downloads a program from
Google's servers and runs it with system-level privileges to do the checks.

~~~
heh
Incidentally, as I understand it SafetyNet is intended for verification of
device integrity when you use Google Pay - i.e. as a NFC wallet.

------
treve
Also interesting that it said "WAS the biggest Kubernetes deployment". This
tells me that they've tanked hard enough for that to no longer be the case.

~~~
itcmcgrath
When you write blog posts reflecting on something that happened you often
use/think in past tenses. Don't try reading too much between the lines on
things like that.

------
revelation
It's been 10+ years since WoW launched, and that scaled better than Pokemon
Go. It didn't need no fancy cloud or autoscaling or the few moore iterations
Google got, and it's a game that you know.. actually has a use for networking.

You could make an offline Pokemon Go version and not notice any difference.

~~~
yazaddaruvala
"OMG there is a Dragonair! Do you see it?" "Na, I'm not on your server.."

Blizzard scaled using appropriate technologies for its time. i.e. sharding
basted on "Realm" and enforcing active player limits. This scaling mechanism
was used by every MMO I knew about until EVE.

EVE pioneered the modern MMO. It is no longer acceptable for an MMO to be
sharded by server. Even WoW has been aggressively trying to scale its
architecture and infra such that it can re-connect these shards into more
cohesive worlds. With the most credit to them, they have a lot of technical
debt in this regard, but the fact that some servers are labeled "full" with
wait queues to join, implies that they still don't have the technology to
scale let alone connect all their servers[0].

[0] Note, there are also other reasons (i.e. in-game economical and political
reasons) why they are not aggressively connecting more shards.

~~~
elsonrodriguez
EVE still uses sharding in a sense. One system cannot scale over more than one
server. So when you jump through a gate or bridge, you are probably connecting
to another server. This limitation shows itself when a system suddenly becomes
the flashpoint for a ridiculous battle, and CCP has to scramble and move the
system to a beefier server.

~~~
yazaddaruvala
Totally true. If it came across like I meant EVE has no sharding mechanism, my
bad.

> One system cannot scale over more than one server.

In this regard, Blizzard's dynamic realms are really interesting. Ideally, we
can move to a world, where even single "systems"/"zones"/"areas" are able to
scale horizontally.

I have zero experience with it, so I don't know if it can practically scale to
solve this problem, but
[http://www.paralleluniverse.co/spacebase/](http://www.paralleluniverse.co/spacebase/)
has a really cool set of features that MMOs could take advantage of.

------
lanestp
Seamless would not be the word I would use. I would instead argue that what
Pokemon Go proved was that Google's Cloud is not of the same quality, at
scale, that AWS is.

~~~
sofaofthedamned
What exactly shows that?

What it probably shows is that using the cloud isn't a magic bullet to making
you scale indefinitely, you need to work on your code to decouple from the
infrastructure too.

