
Pokémon Go Fest attendees to get refunds as technical issues break the event - janober
https://techcrunch.com/2017/07/22/pokemon-go-fest-attendees-to-get-refunds-as-technical-issues-break-the-event
======
yoodenvranx
As a long term Ingress player i am not surprised by this _at all_. Niantics
management has proven itself again and again and again as being completely
incompetent at everything they do. (They even to manage to ban their own
official event photographer after he had GPS problems while traveling to
Paris).

I used to be angry at them for ruining Ingress (one of the best games ever
made) but nowadays I am just amazed at the level of sheer incompetence.

They managed to learn _nothing_ from 4 years of Ingress and I think now it's
time that the whole upper management is fired so that the company can start
fresh.

~~~
FreakyT
I realize this statement probably won't go over well here, but Niantic's
behavior strikes me as the kind of thing that happens when a company is
entirely run by software engineers who refuse to accept relevant advice from
anyone who isn't a software engineer.

There are people who understand event planning, there are people who
understand how to build F2P mobile games, and I get the impression that
exactly zero of these people were consulted, or, if they were, their advice
was immediately ignored.

~~~
awalton
Anyone who's played Pokemon Go or Ingress can refute this; their UI design is
abysmal at the very best, the game suffers a million bugs and patches
introduce at least as many as they fix, etc. A company ran by Software
Engineers would have hired competent UI developers and QE engineers and would
have demoed the hell out of the event before putting it into production...

No, I'd argue Niantic may originally have been engineering driven, but once
Ruthless cut them from Alphabet, they went straight for the greed play,
developed with an IP that would sell billions with next to no effort, and are
now tightening the screws on the player-base with Pay-to-Win gym tickets and
events requiring as many Lures and Incenses as you can buy to continue
extracting income.

~~~
FreakyT
> A company ran by Software Engineers would have hired competent UI developers
> and QE engineers and would have demoed the hell out of the event before
> putting it into production

UI developers? Testing? You have apparently worked with very different
software engineers than I have! Most I've worked with have followed the "if
users don't understand the UI it's their fault" UI design philosophy, and all
testing is handled in production because no one wants to put emphasis on
testing when they could be working on the next feature.

------
fencepost
Just as glad I didn't get a ticket ($20? sure. Scalper prices? Heck no) and
waste the time. This whole thing is going to turn into a real fiasco for some
time due to the number of people who bought resold passes (so the original
seller sold it and now gets their original $20 back), people who traveled (I
know many got hotels, not sure how many flew in), and questions of how they're
going to get that in-game currency to people (reportedly some are still in
line to get in, while others are waiting in a not-quite-as-long line just to
get out).

Each person attending has a uniquely-numbered wristband pass, and apparently
on entry they're getting an envelope with a patch and a unique-to-them QR code
which needs to be scanned after spinning a special "Pokestop" and effectively
checks you into the event. Since the game is crashing basically at startup for
most folks, that makes it a bit of a challenge just to get in. Still, I
suspect the in-game credit is going to be based on people who checked in with
that code since the ordering process didn't involve providing a game login.

Also, I've seen some folks questioning why they didn't have WiFi set up, but
I'm pretty sure that the way WiFi works would make that almost impossible
anyway (2.4 would be a bad joke no matter what, 5GHz might be possible over
most/all available channels, APs that can handle hundreds or thousands of
simultaneous connections are rare and probably require major advance planning,
AP broadcast power would probably have to be quite low to allow large numbers
of APs in a physically small area, etc.).

~~~
maxerickson
People in a stadium aren't going to be quite as focused on their phones, but
lots of people in a tight space isn't an unusual occurrence for networks.

~~~
fencepost
~15k people in a small section of Grant Park all trying to play a somewhat
data-intensive online game that requires constant communication with servers
is a VERY different activity stream than people in a stadium. Might be closer
to think of those people in a stadium all trying to stream video of the event.

~~~
wfunction
Is the game that data-intensive? What data is there to send/receive that
you're comparing it with streaming video?

~~~
Schampu
The game uses google's protobuf protocol which makes communication more
efficient and the game updates (I can only tell from <= 0.35 analysis) all
10-30 seconds like loading pokemon spawns and map entities of the current map
area.

The most expensive data in 0.35 was:

\- Pokemon models (They were dynamically loaded but cached then)

\- Pokestop images (Server sends image url and client searches for a chached
version)

\- Syncing the player state with the client (Inventory, rewards, egg states..)

\- Updating the map (Future pokemon spawns, pokestops/gyms in view..)

If a player is idle and doesn't move, just keeps farming a single pokestop and
battles sometimes, then it should be relatively low cost since the game
doesn't use a realtime communication (It's actually HTTPS based _lol_ ).

~~~
RugnirViking
The pokemon models are loaded over the network? Whatever for? Seems like a big
waste of precious data

~~~
dimillian
So you can add/update them without doing an .ipa/.apk update ? Seems really
convenient, and a one time deal if it's cached.

~~~
Schampu
Last year I developed a server emulator for the game - On the server's first
run, it cloned all models from the official servers to a local folder. The
download urls and the models itself were encrypted, also their urls got
disabled after ~24 hours. Other informations like how big a model gets
displayed inside the game world was declared inside a GAME_MASTER file. When
connecting to the custom server, the client asks for the GAME_MASTER file and
the models as needed. It was pretty fun to edit these and send the players
custom models like a GIANT snorlax or a red pikachu. I'm not 100% sure, but I
think the GAME_MASTER file even contained a "model signature" to trigger a
forced model update, so it's possible to overwrite already cached models. In
the first weeks Niantic broke the model of Starmie (it turned full black) and
I'm pretty sure they fixed it without a full game update. You can checkout the
emulator project here:
[https://github.com/maierfelix/POGOserver](https://github.com/maierfelix/POGOserver)

------
free_everybody
>When Niantic’s John Hanke took the stage, he was greeted by an audience a few
thousand deep, many of them chanting “FIX YOUR GAME” or “WE CAN’T PLAY!”

Can you imagine how stressful this would be? It's one thing to get
disappointing feedback data on bugs, UX, etc. Standing in front of thousands
of angry people screaming at you to fix your software this instant? Yikes!!

~~~
capkutay
I remember Google Cloud was doing a ton of press about how Niantic was using
their platform for Pokemon Go. I wonder if that will backfire for them now.

~~~
dmoy
These are completely unrelated, no? One is about scaling the servers and stuff
behind the scenes, this thing is about cellular data networks failing and
dying.

------
kayoone
Even at big music festivals, mobile networks are usually totally overloaded
just by the sheer amount of people logged in at the same time, without even
doing anything. Even recieving an SMS can take minutes. I don't assume they
would do an event like this without letting providers know in advance to
upgrade capacity, but it seems like this.

~~~
iDemonix
I've always wondered why cellular is affected by this, anyone have a link?

~~~
TrickyRick
There's a limited amount of "space" in the air, ie not everyone can be
transmitting at the same time. This is fixed using a few different techniques
depending on the cell network but for GSM something called TDMA (Time division
multiple access) is used. This basically gives your phone a certain amount of
time to broadcast on before it should leave the airwaves for someone else to
transmit on. Too many people in one location means not enough time for
everyone to transmit.

~~~
vvanders
That's not really quite accurate on modern networks. Most of them use CDMA
which allows many devices to transmit/receive at the same time, assuming that
they can modulate their power with respect to distance from the
tower/etc(which is a requirement for the "xor" to cancel out).

That said there's still a bunch of other stuff you have to sort out if you
want to support that many devices in a small space.

~~~
TrickyRick
> Most of them use CDMA

Really? I thought CDMA was something only Verizon used for their network. I
tried looking into it a bit prior to writing the original answer but I
couldn't find anything other than a mention on the TDMA Wiki [1] which has a
[citation needed]. Do 3G/4G networks across the world use CDMA as well?
Because I know that in Sweden carriers are buying up large chunks of frequency
bands to be able to increase 4G coverage, that shouldn't be needed with CDMA,
right?

[1]: [https://en.wikipedia.org/wiki/Time-
division_multiple_access](https://en.wikipedia.org/wiki/Time-
division_multiple_access)

------
intopieces
Anyone who has ever tried to send a photo at Soldier Field during any large
event could have predicted the crippling of the network that happened here.
Maybe I missed it in the article, but did the event organizers not set up an
ad-hoc network for this?

~~~
Hydraulix989
This is a situation where a mesh network topology (like Firechat) would be
necessary.

~~~
azernik
Mesh is unfortunate for these situations; the limiting factor at these scales
isn't infrastructure size, but EM spectrum. The accepted solution is mandating
players go onto WiFi, where you can get them to much lower emitted power by
deploying enough APs, close enough to any player location, that no clients
need to shout to get their traffic heard.

------
cloudkj
Are there any Niantic employees active here on HN? As an engineer, I would be
very interested in hearing some first hand accounts of some of the technical
problems Niantic's engineers have had to deal with, not necessarily just for
the Go Fest, but for the game in general. To the extent that they're allowed
to talk about it, it would be very interesting to get an AMA style take on
some of the challenges.

~~~
davidmurdoch
As the lead engineer on an up coming game of a similar scale, YES! This would
be so awesome!

------
dEnigma
Niantec just posted an update[0]. These are the steps they will be taking:

 _All registered attendees will soon receive an email with instructions on how
to receive a full refund for the cost of their ticket. These instructions will
be sent to the email addresses associated with your Pokémon GO account.

All registered attendees will receive $100 in PokéCoins in their Pokémon GO
account.

Special Pokémon, Eggs, and check-in PokéStops appearing during Pokémon GO Fest
have had their range increased to a two mile radius surrounding Grant Park
through Monday morning, July 24. These Pokémon and Eggs will only be visible
to Pokémon GO Fest attendees who validated the QR code they received when they
entered Pokémon GO Fest. Attendees who were unable to validate their QR code
during the event can do so through the special PokéStops through Monday
morning.

All registered attendees will have the Legendary Pokémon, Lugia, added to
their account._

[0][http://pokemongolive.com/en/post/pokemongofestupdate](http://pokemongolive.com/en/post/pokemongofestupdate)

~~~
kendallpark
> All registered attendees will have the Legendary Pokémon, Lugia, added to
> their account.

Lugia? That’s some damage control right there.

~~~
mizzao
Beg for forgiveness and give out free stuff, that generally makes people much
happier :)

------
NightlyDev
I guess that's no suprise? Downtime with every single event som far, so why
should this one be anything different?

WiFi is a thing and should he able to handle it just fine. Heck, they could
even block anything but PGO and get away with low bandwidth too.

When voulenteers can manage to get WiFi working for thousands of nerds at
indoor LANs... Let's just say that Niantic should be able to avoid this with a
little bit of planning.

This is the company running one of the most unstable and buggy games I have
ever played, so I would expect no less. At least the game is very fun for a
lot of people... When it works.

~~~
make3
Don't large lans use.. LAN ? (Ethernet). I could be wrong, but as many
desktops still don't have wifi and you know where people are and it's so much
more stable large scale.. it would make sense

~~~
NightlyDev
Ofc ethernet is used, but WiFi is often used for phones and additional devices
like game consoles as it's mostly 1 port/person. The gathering and Dreamhack
are two good examples of events with WiFi scaling to many thousands of
connected devices without much problems. More or less every AP provider has
solutions for this.

------
vortico
This game has been so much of a trainwreck, I don't see why Game Freaks
licensed Niantic to use their IP.

~~~
sddfd
Niantic seems to have expertise in Mobile Gaming from ingress. Which other
company has?

If your comment is based on this event, it seems unfair: Who else has
successfully done an event with similar bandwidth requirements before? Also,
my understanding is that the cellular network is the problem, but you might
correct me if I'm wrong.

Besides, Blizzard had spectacular problems in the past with WOW launch events,
simply because no one had ever done something of that /scale/ before.

~~~
gambiting
It sounds like they planned an event for 20k people without telling mobile
networks they are doing it. For festivals mobile networks usually deploy extra
capacity to handle larger than usual number of people. It sounds like this
could be handled by large number of local WiFi spots as well.

~~~
wfunction
How do you just "deploy extra capacity"? Do they install new antennas just for
an event?

~~~
dungle6
Deployment of cell sites at events like stadiums for sporting events and
outdoor music festivals is pretty common for years and years. Unless you're a
hermit you almost have certainly seen a COW at one point or another. Cf
Wikipedia mobile cell sites, especially COW.

~~~
wfunction
I've seen them around alright, just not realized how common or easy-to-set-up
they might be for an event like this, or how they work from an inter-provider
standpoint. Do they not cost a whole ton for the organization setting up the
event, and wouldn't every provider (T-Mobile, AT&T, etc.) be required to set
one up? It seems like such a massive headache and cost for something like a
Pokemon-catching crowd that might not even generate much revenue(?) that I'm
confused how everyone is saying "just deploy extra capacity". Like do
providers just follow crowded events for free and put antennas wherever people
are as if it's no big deal?

------
kermittd
The event looked interesting and the game is a great concept. Sorry to hear
about this fail as well.

------
monksy
The weather in Chicago since 5pm till early this morning has been filled with
storms. Storms were also threatening this afternoon, but surprisingly they
didn't come. I had to cancel a beach meetup that I was looking forward do over
this.

------
ShabbosGoy
Could they have used a mesh network to help load balance connections to WiFi
APs?

------
em3rgent0rdr
The silver lining is that all these people would would only interact with each
other virtually then had a perfect opportunity to talk to each other face-to-
face, fwiw.

------
jbmorgado
What surprised me in this article is that people still gather to play Pokemon
Go.

I personally never got into the game, while my cousin was totally into it, but
where I live (which is a medium city for Europe), the fad has totally passed
in less than 6 months after launch.

