
Moving the NYT Games Platform to Google Cloud With Zero Downtime - jprob
https://open.nytimes.com/play-by-play-moving-the-nyt-games-platform-to-gcp-with-zero-downtime-cf425898d569
======
chatmasta
> We found that some web customers were unable to access the puzzle, and found
> the cause of the problem to be App Engine’s limit on the size of outbound
> request headers (16KB). Users with a large amount of third-party cookies had
> their identity stripped from the proxied request. We made a quick fix to
> proxy only the headers and cookies we needed and we were back in action.

That’s pretty funny. Users of NYT are sending request headers with _sixteen
kilobytes_ of tracking data. Maybe that’s the real problem eh?

I wonder which news website has the largest amount of trackers. If I let the
CNN home page sit open in chrome, I can come back an hour later and find
_thousands_ of requests blocked by uBlock.

~~~
godzillabrennus
If the website is free then you are the product. Not sure why it’s a surprise
these companies aim to maximize the value they can derive from their product
(aka our data).

~~~
freeone3000
It's not free. NYT relies on a subscription model.

~~~
tgb
The mini-crossword is free as well as some of their full crosswords.

~~~
iamthirsty
_Some_. I actually cancelled my subscription with them because of the
horrendously large Google Home ads on all the _paid_ crosswords. When I asked
them about it I got about four non-answers — it's ridiculous that even if you
pay they still serve you ads.

~~~
dragonwriter
Newspapers have always included ads even if you pay; not sure why anyone would
expect “on the internet” to change that.

~~~
iamthirsty
Because I pay _specifically_ for _only_ the crossword part. I bought a book of
50 (of NYT puzzles) for the same price, and it came with no ads. Much better
deal, if you ask me.

------
duijf
Is anyone else curious why the NYT uses Medium? Their own website is literally
about reading stuff

(Sorry if this is off-topic)

~~~
dgritsko
It's an engineering blog post, which usually serve double duty as being both
informative and also useful for recruiting ("Look at the cool stuff we are
building! Come be a part of it!"). Case in point, the post ends with "we’re
currently hiring for a variety of roles and career levels".

In addition to not being really appropriate for nytimes.com, I'm guessing that
publishing content there brings along a lot of extra cruft that is probably
not necessary for a post like this (advertising, paywall system, isolating it
from the "real" NYTimes content, etc.). Easier to just throw it up on Medium
and call it a day.

~~~
natural219
This is really fascinating to me. Is this because engineers who they want to
recruit dislike the New York Times brand, or because readers of the New York
Times don't want to read things as informal as transparent blog posts about
internal NYT decisions?

It's very easy for me to see something like blog.newyorktimes.com with a
similar design / community philosophy as Medium, but would that somehow
cheapen the experience for NYT readers? Or does NYT just not see itself as a
"hip tech company" like Medium? I have endless questions about this, haha.

It seems to me like there's a lot of unstated assumptions hiding in "not
appropriate for nytimes.com". Some things mentioned include -- "advertising,
paywall system, isolating it from the "real" NYTimes content, etc.". This is
absolutely baffling to me! I would be much more inclined to read regular NYT
content were it not for these things.

~~~
untog
I think you're over-thinking this. There isn't a huge crossover audience for
engineer blog posts and general NYT audience, and I imagine the engineering
blog posts do not go through the same editorial process content on nytimes.com
does. That alone makes the case for using a different domain.

~~~
natural219
I'm being a cheeky detractor of NYT here. I think "candid, engineering-style
blogposts" are the future of news, and ancient vehicles like New York Times
are long dead. I think the "general NYT audience" is participating in
#FakeNews, and they should radically reconsider their information diet.

As vivid example of this, compare James Birdle breaking the "Youtube
exploitative kid videos" story way before, and in greater depth, than in any
major publication. This is _actually_ the future of news, and pretending like
aging institutions like the New York Times are remotely relevant anymore is
longshot wishful thinking.

Editorialization, fact-checking, and cultural leadership have important roles
to play, and I'm excited to see these features unbundled into separate
services. I'm long on services like Verrit and Snopes, and wish that I, as an
independent publisher, could pay an intern to get official statements, cross-
check narratives with history, and perform some of these functions. As is, I
think people are operating under the delusion that ONLY NYT-style institutions
can perform these functions, which baffles me.

(Actually, the future is probably more like James posting on jamesbridle.com,
and then aggregating it through sites like Hacker News. But what do I know,
I'm just a millennial who doesn't understand all these big partisan topics
like modern journalism)

[https://medium.com/@jamesbridle/something-is-wrong-on-the-
in...](https://medium.com/@jamesbridle/something-is-wrong-on-the-
internet-c39c471271d2)

~~~
untog
Hmm. I'm going to disagree with that! I think "candid, engineering-style
blogposts" are and will continue to be great for an engineering audience, but
I'm very skeptical that they will be great for a wide audience. For instance,
I read and was fascinated by the YouTube kids post, but I do not know anyone
outside of the tech industry that read it. And you're wrong to say he reported
it way before, the NYT published this two days previous:

[https://www.nytimes.com/2017/11/04/business/media/youtube-
ki...](https://www.nytimes.com/2017/11/04/business/media/youtube-kids-paw-
patrol.html?_r=0)

and Birdle's post itself links to reporting by New York Magazine from 2016. I
don't dispute that his post goes into more detail, I just dispute that longer
automatically equals better. Someone with domain knowledge reporting a story
in great depth and a major publication reporting a simplified version for mass
consumption is certainly not a new model.

I'd also strongly disagree that NYT is an ageing institution unable to adapt
to this modern tech reality. John Herrman writes some of the most perceptive
pieces about the state of tech out there:

[https://www.nytimes.com/by/john-herrman](https://www.nytimes.com/by/john-
herrman)

(and a minor quibble: I don't think the post linked here and the Youtube Kids
post are in any way comparable. The engineering writeup is not news in any
way, shape or form, it's just a guide to how NYT implemented something)

~~~
natural219
Thank you for this good response. I didn't notice the NYT covering this story
before, because I have cut NYT out from my life due to their malicious,
partisan reporting. So maybe I should be less bold about my evaluations of
them and just continue to enjoy my personally-curated, high-information-dense
feed.

------
marksomnian
Doesn't this lead to vendor lock-in? All these Google proprietary services
seem like they would be a big issue if they decide for whatever reason to
migrate away from GCP.

~~~
Top19
It’s actually double lock-in, so 2x worse.

You used to have to just be afraid of lock-in, which I don’t think is as big
an issue as it sometimes seems.

But with Google, you’re not only locked in but might be LOCKED OUT when they
kill your product.

~~~
chatmasta
Don’t be ridiculous. Google killing a feed reader is a way different from
Google killing a cloud service with paying customers and SLA agreements.

~~~
un_montagnard
Like the QPX Express API?

~~~
chatmasta
Interesting point. However that’s not a google cloud product and never had an
SLA (the QBX FAQ says “we do not guarantee support”). It’s also a unique case
because of its reliance on third party data vendors.

If google starts killing their cloud products, I will eat my socks. Just let
me wash them first.

------
dnr
If anyone wants to try collaborating on crosswords in real-time, try

[https://squares.io/](https://squares.io/)

You can upload .puz files or let it download from NYT with your subscription,
then share the link with friends.

(Web only for now, sorry.)

------
johns
Since there are people from NYT here, can you point me in a direction to help
figure out why my streaks have been all messed up the past few months? Not
sure if it's an app bug or something on the backend, but puzzles are
retroactively being marked as being completed perfectly when they're not.
Email in bio if you'd like to discuss more.

------
amelius
What does zero downtime mean?

No interruptions of services?

Or just that people could still log in all the time?

~~~
degenerate
Pretty sure they mean people could login again after the switch.

I can't imagine what the purpose would be of capturing session data for each
logged in user and transferring that over... I wouldn't even expect that of a
fortune 500 company moving platforms.

If that is what they did, it warrants a post on its own.

~~~
everyplace
Imagine if you were half way through the puzzle when the cutover happened, and
then you lost your entire puzzle state and the board was reset. For the die-
hard crossword players, this would be devastating.

