
Reddit was down for over two hours - bluedino
https://reddit.statuspage.io/incidents/4q8mfp969hky?u=3mcg5t0ys8mn
======
bitwize
Y'know, the toll that all this activity takes on Reddit's servers gave me an
idea for a sort of distributed Reddit. News links and stories assorted into
topic-based groups, with the option for users to comment or respond to others'
comments in a threaded discussion. It would be completely decentralized and
belong to the users, not some central media company. We could call it Users'
Network, or USENET for short.

We could implement it with blockchain.

~~~
manigandham
Except UX is everything. 100s of billions in corporate value are based on
nothing more than better UX.

If only everyone who talks about usenet, IRC, and the rest could actually be
bothered to update the experience to what modern users demand.

~~~
Klonoar
Citing UX in a thread about Reddit, the company with an almost universally
hated redesign that moved away from a mostly text-based condensed format is...
interesting.

~~~
manigandham
Every user engagement metric has increased since the redesign. Unfortunate for
some of us, and it doesn't preclude the metrics measuring the wrong thing, but
that's reality on the ground.

Same reason why Gmail is a bloated mess, because that's what the metrics show
is wanted by users.

~~~
AnIdiotOnTheNet
> Every user engagement metric has increased since the redesign.

Doesn't that just mean that users are spending more time on the site because
it's so much more of a pain to get what they actually want out of it? Like,
you could increase "user engagement" of a shopping mall by turning it into a
labyrinth and rearranging the shops every day.

~~~
manigandham
Reddit is a corporation with a large amount of VC investment. Profit is a
primary goal, and engagement that shows more time and thus more revenue is
exactly what they want.

~~~
ryandrake
It would have been more honest of them to come out and just admit “yea, this
new design is terrible but it juices our metrics and makes us more money so
deal with it if you want Reddit to survive.”

Also, Protip: in case anyone still doesn’t know, go to old.reddit.com for the
previous, better interface. Who knows how long it will be kept alive though.

~~~
realusername
> Also, Protip: in case anyone still doesn’t know, go to old.reddit.com for
> the previous, better interface. Who knows how long it will be kept alive
> though.

I'm using an extension to automatically redirect to the old one. When the old
site is gone, I will probably leave Reddit, the redesign is too unusable for
me.

~~~
SN76477
Yep.

I like HN but I miss the diversity of conversations. /r/synthesizers
/r/starwarssleaks etc.

------
notori0us
I noticed this outage coincided with elevated timeouts in one of our EC2 us-
east-1 availability zones.

We had recovery by removing that zone from our Auto Scaling Groups, but I
haven't been able to find many others affected online (other than Reddit).

Did anyone else experience something like this?

edit: Our errors started at 13:35 Pacific Time.

------
quickthrower2
No problems from Australia in the last 2 hours (including just now.)

For example
[https://www.reddit.com/r/menwritingwomen/new/](https://www.reddit.com/r/menwritingwomen/new/)
shows a post from just now (verified the timestamp).

~~~
LilBytes
It's 'up' for me but, I'm regularly having problems opening comments and
posts.

~~~
tossAfterUsing
i miss the fail whale.

IMO, their javascript UX is awful... impossible to see all the comments in a
thread! what sort of nonsense is that?

old.reddit.com ftw

~~~
quickthrower2
[https://i.reddit.com/](https://i.reddit.com/) ft-mobile-w

~~~
spraak
Whoa, that is orders of magnitude faster than the default mobile site. Thank
you for sharing, I didn't know about it

------
Deimorz
They were doing another test run of their new live-streaming service "RPAN"
today, I wouldn't be surprised if that's connected to the downtime.

If you can get the page to load:
[https://old.reddit.com/r/pan/comments/d5paue/a_midweek_treat...](https://old.reddit.com/r/pan/comments/d5paue/a_midweek_treat_rpan_will_be_onair_tomorrow/)

------
opportune
I wonder what Reddit devops is like, I find it so strange that such a
trafficked website can’t handle rollouts without what appears (on the outside)
to be a full reboot. And also strange that the rollout takes so long. Makes me
think they are doing something naive like running on naked vms and having one
guy running a script to take them down, update os, redeploy new binary, etc.
all at once. Then probably encountering issues and live debugging them until
it works again

~~~
gbrayut
some details at
[https://www.youtube.com/watch?v=z7TIzCAEo0M](https://www.youtube.com/watch?v=z7TIzCAEo0M)

------
bonerman69
Does anyone know why they tweeted this???

[https://web.archive.org/web/20190918200317/https://twitter.c...](https://web.archive.org/web/20190918200317/https://twitter.com/reddit/status/821409753)

~~~
hobabaObama
I believe it was on r/worldnews . They just tweeted it

Its still showing up on reddit.

[https://old.reddit.com/r/worldnews/comments/6l17g/though_bla...](https://old.reddit.com/r/worldnews/comments/6l17g/though_blacks_are_only_24_percent_of_new_york/?st=k0q71tmt&sh=dbb3f82e)

------
aldoushuxley001
They've been having lots of crazy errors all week (like e.g. all comments
posting 3-4 times, other downtimes, etc.). Something's broken on their end for
sure.

------
Animats
And US productivity increased 15% during the outage.

Maybe social media should be turned off on Tuesdays and Thursdays, as
suggested in Ready Player One.

~~~
0xdeadb00f
I'd like to see how that would go.

------
banana_giraffe
Number of new posts every 15 minutes in the past 24 hours:

[https://imgur.com/a/HqUsYGn](https://imgur.com/a/HqUsYGn)

It's interesting to see how it fails. Clearly some users can still post. I'm
sure there are things that still work (maybe the phone apps .. got me), but
also very clearly, there's a sharp dip in new posts.

------
aceon48
I dont know how, but reddit's tech actually feels WORSE now. The old site was
simple, but it was fast, clean, and it worked for the one thing it needed to
do: upvote downvote content.

Now the newer version is always trying to kick me into either Chrome or the
App. I don't want the f'ing App. And the mobile experience is supppper slow.
Its literally just a text webpage it should load near instantly, if they didnt
have to redesign the whole thing in some new web framework.

------
paganel
At first I thought it was of me refreshing /r/soccer so many times this
evening because of CL matches.

------
ropiwqefjnpoa
Stuff like this is great, you can always point to it when management demands
an explanation why the company email server was unavailable for 5 minutes.

------
frequentnapper
That explains the strange uptick in my productivity chart during that time.

------
OrgNet
reddit is going back to the reliability that they had a few years ago

------
kryogen1c
Webroots website started acting up a couple his ago

------
StreamBright
The productivity of IT teams skyrocketed world wide.

~~~
rapfaria
I remember reading that when a judge ordered Whatsapp to be taken down in
Brazil for a few hours, the GDP through that small period went up by 2%.

------
JohnBerea
And nothing of value was lost.

------
dymk
Reddit is the only major social media/aggregator I can think of that actually
has to go offline for scheduled maintenance still. I can’t imagine the kind of
shitshow it must be internally.

Imagine if Facebook or Twitter had to go down regularly for “maintenance”.
Heads would roll.

~~~
mike_d
"We are a startup that inherited a 15 year old website" was one of the things
that resonated when I worked there.

Facebook is Alexa #5 in the US and has ~40,000 employees. Reddit is #6 and has
~500 employees. Twitter is #17 and has ~4,000 employees. (Numbers taken from
quick Google searches)

They have some super talented engineers that work miracles with band aids and
bubble gum.

Edit: Yes I understand not all employees work on everything. I was trying to
give a sense of scale. Reddit is a tiny company with a massive website and no
way to make money. If you have ideas - they are hiring.

~~~
opmac
Meh... they could have rewritten the site from the ground up, including any
sort of data migration, multiple times over by now. I have never worked there,
but outside looking in, it seems like there is a significant lack of technical
vision from them.

Other big players like Facebook, Twitter, etc, are literally inventing pivotal
parts of the modern web technical stack. What has come out of reddit?

~~~
joatmon-snoo
> including any sort of data migration, multiple times over by now

Someone's never done a data migration while serving 1M+ qps!

~~~
sethammons
I've come close to that many, many times with zero downtime. It does take
planning, more than many folks would think. But it can be done as long as you
think it through.

~~~
arcticfox
I've never done anything close to that scale of migration, but I also can't
really imagine taking an approach that _feels_ like a huge migration.

I feel like I would try to strangle the old app (putting proxies in front and
redirecting endpoints as they're rewritten) rather than swapping it out
wholesale.

At that point I don't know if it really counts as a migration as much as just
development.

Any thoughts on that approach? Is that what you have done, or was it a more
holistic swap over?

~~~
joatmon-snoo
That isn't a data migration, that's a release.

------
a-wu
Marked as resolved now, mods should probably update the title.

