
Reddit Is Down To One Developer - citizenkeys
http://techcrunch.com/2011/03/18/reddit-is-down-to-one-developer/
======
trotsky
I took a job some time ago (back around when the internet was morphing from
uucp to ppp) as a consultant to an "independent agency of the United States
government" which seems to enjoy placing big blue boxes all over the
landscape.

It was my first real job outside of the university system, so it is safe to
say that I was rather green. This job basically involved being an apprentice
to someone who oversaw the code supplied by vendors that sorted a tremendous
amount of non machine readable mail, i.e. it assisted humans using chord
keyboards sorting mail with pneumatic arms and big chains and slots. Back
then, a fair amount of mail traveled this way - stuff your grandmother wrote
that couldn't be OCR'd, postcards, etc. The whole apparatus was similar in
size to a semi trailer [1]. Each station dealt with a piece of mail every
second (give or take), with two fifteen minute breaks and a half hour for
lunch. This was around the time of the postal shootings [2].

Two weeks into the job, my boss quit. Just never showed up again. Not exactly
sure what happened. Apparently, I had become the most knowledgeable person in
the organization on this software which was currently being rolled out widely.

Unsurprisingly, there were some serious code quality issues. This was QNX real
time sorta unix in C and ASM with RTOS daughter boards. The vendor's code
jockeys, apparently, had become unavailable.

Main takeaways: _Systems seem to keep working despite everything looking very
fragile. Read tons of code. If it's working, don't poke it._

A year later I was able to make some form of industrial dance music by
coordinating the openings and closings of the mail slots and the chain with
the software. Bad situations can sometimes lead to greater clarity and new
opportunities.

[1]
[http://www.usps.com/postalhistory/images/photogallery/equipm...](http://www.usps.com/postalhistory/images/photogallery/equipmentpix/APPP136_fuji280cropped.jpg)

[2] <https://secure.wikimedia.org/wikipedia/en/wiki/Going_postal>

~~~
bhrgunatha
It seems like a common situation in the tech world for those with a little
capability to deal with some legacy system that you somehow get lumped with or
become the "expert". Maybe it's inevitable, people do get bored with doing the
same things all the time.

I'd love to hear some of the music that came out of your experience - anywhere
to listen to it?

------
redthrowaway
I wish the TC article had done more than merely mention that keltranis is
"joining Alexis". Hipmunk now has both reddit cofounders, their first hire in
keysersosa, and their most senior programmer in keltranis. I wouldn't be
surprised to see jedberg join them before too long, and then hipmunk would
have almost all of the old reddit team on board.

~~~
jedberg
Me too. It is a total dis-service to hipmunk to not mention Chris or Steve.

But for now I think their tech team is full, so they won't be needing my
services. :)

~~~
redthrowaway
Does that mean you'd consider joining them if they asked you to? Also, given
the concerns expressed at [1], once you get a healthy dev team would you guys
look into maintaining redditOSS a little more thoroughly? I imagine personnel
shortage has made that impossible to date, but if you had a sizeable group is
that something you'd do?

[1][http://www.deserettechnology.com/journal/reddit-the-open-
sou...](http://www.deserettechnology.com/journal/reddit-the-open-source-
software)

~~~
jedberg
> once you get a healthy dev team would you guys look into maintaining
> redditOSS a little more thoroughly? I imagine personnel shortage has made
> that impossible to date, but if you had a sizeable group is that something
> you'd do?

Yes, very much. We would like to make the open source software MUCH easier to
run, so that it is easier for others to develop on. It will probably be one of
the first projects for a new hire, because it will make it easier for THEM to
develop too.

------
jarek
Unless they're working on new features which are needed tomorrow, one dev
seems fine in the interim. The number of sysadmins is more important to
keeping a site running, but of course that's not what's in TC's headline.

~~~
thezilch
Reddit's sysadmins neither have to maintain DC racking (nor travel), maintain
hardware lifecycles, nor maintain vendor relationships. They certainly appear
up to their teeth in working around AWS (EBS) deficiencies. They're not going
to control a lot of their hardware, and most "fixing" of AWS pitfalls are
going to come in the form of software.

Now if we are talking devops, I can agree. Engineering teams this small need
to wear many hats; I think jedberg has been said to have helped in development
efforts, if my memory of past blogs serves right.

~~~
jedberg
Yes, I do dev too. I just haven't done it in a long time.

A lot of our sysadmin time is spent tracking down bugs in the stuff we use
(like Cassandra) and doing tuning and automation.

Also, I do a lot of the other business stuff, so at best I'm a part time
sysadmin right now.

We do pretty much do "devops" though. A lot of the work of Neil, our
programmer, is tracking down bugs due to data inconsistencies due to hardware
problems.

------
jprobert
This makes me shiver in my boots. I couldn't imagine having to run that site
myself. Neil is inline for a big pay raise.

~~~
jedsmith
That's a bit unfair to jedberg, who actually runs the site with his operations
team of two (one of them a new hire). This article and the comments are
putting far too much weight on the developer, as if Reddit is a site that's
constantly iterating new features. That's not to downplay his contribution,
but:

    
    
        * He was hired in November.
    
        * He works on code, and didn't get woken up during
          the last major downtime until the end (from their post
          mortem, presumably due to the DB eating itself). The
          word 'even' in "we *even* woke up spladug" is telling,
          that indicates to me that he doesn't wake up often.
    
        * Reddit is pretty much a done product, and the last
          features I can think of were meta (Gold comes to mind)
    

He has a positive outlook[1], anyway, which the TechCrunch piece didn't seem
to notice.

Doing operations for a site the size of Reddit with a staff of 2, on AWS, is
far more impressive to me. Keeping an operation like Reddit afloat in front of
that many uniques must be a pretty significant challenge. There's not a lot of
depth for on-call, so I feel for their sleep.

On top of that, jedberg is the sole voice actively fighting for Reddit's
reputation in public, since (let's be honest) the exodus looks a little
suspect. Watching his comments[2] in the face of a perceivably sinking ship --
not to mention former employees who are saying quite a bit in front of an
impressionable user base[3] -- is interesting, to say the least.

They're definitely undergoing a challenge, and I'm interested to see how they
come out of it. (Aside: Today has apparently been discuss-Reddit day, looking
at my threads.)

[1]:
[http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...](http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l7i4m?context=1)

[2]:
[http://www.reddit.com/r/reddit.com/comments/g6d7i/three_form...](http://www.reddit.com/r/reddit.com/comments/g6d7i/three_former_reddit_employees_ketralnis/c1lagp0)

[3]:
[http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...](http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx)

~~~
jarek
Sorry! I accidentally downvoted while intending to upvote. Tricksy fingers...

------
nkassis
This is the second TC article today I see where they claim news site like digg
or reddit are in competition with twitter. I don't think they understand that
it's two very different type of stuff that is complementary. Or they are paid
to say twitter in every article.

~~~
colkassad
Tech journalism has a hard on for Twitter and it's not so surprising why.
Twitter is all about self promoters. I see people using reddit all the
time...I have never seen Twitter in the wild.

------
starnix17
What happens when the last one leaves?

Is Conde just going to run off to Elance or something?

I can't even find a jobs page on Reddit.

At least it's open source.

~~~
mcantelon
Ironically, a month ago they were looking for devs and asking them to invest
considerable time writing demo apps (front end engineers, for example, were
tasked with writing an Javascript Reddit clone using HTML5 storage to keep
track of votes, etc.) to prove their worth:

[http://www.reddit.com/r/blog/comments/fjgit/reddit_is_doubli...](http://www.reddit.com/r/blog/comments/fjgit/reddit_is_doubling_the_size_of_its_programming/)

My guess is this may have repelled some experienced devs that, otherwise,
might have applied.

~~~
zach
That's how you reach out to inexperienced devs. And frankly, if they're
posting on Reddit, they know what they're getting into. They have to set up a
pretty serious mediocrity filter.

If your resume identifies you as a badass programmer, you just toss them a
copy of your resume. You know how it works, and they know that insisting that
you do a programming challenge before setting up an interview hurts their
chances of landing you, so they'll give you an in-interview test instead.

It's the same with Facebook and all the other places that have programming
challenges -- they're there to qualify leads.

~~~
mcantelon
>That's how you reach out to inexperienced devs.

Not sure why an org like Reddit, which maintains a small team, would be
interested in hiring inexperienced devs.

~~~
redthrowaway
Nobody from the original team had any experience before they started there, to
my knowledge.

~~~
officemonkey
...and now, one of the most popular sites on the internet has 6-hour outages.

Coincidence?

~~~
uxp
Outage, singular. You are talking about the 6 hour outage, not the 6 hour
outages. Reddit is prone to go down more than some other high profile sites,
but you are talking about a single occurrence of it going down due to a known
issue that is explained in the article this thread links to.

No, it is not a coincidence. Please don't sensationalize a comment about an
article that clearly explains what went wrong. Go to Reddit for that.

~~~
seabee
As someone who hangs around in the reddit dev and downtime IRC channels,
several-hour outages do happen on a monthly basis, as well as loss of some
features more e.g. commenting. Reading between the lines on each outage, it
was due to building the site on shaky hardware foundations. It just wasn't
explicitly mentioned until now.

[http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...](http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx)

------
grncdr
OT but I found this amusing: competitor Digg (which has 42 employees and has
raised over _$40_ in funding)

~~~
swirlee
That's as many as four tens!

------
jrockway
This is what happens when your primary demographic is people with no jobs and
no money -- no ad revenue because nobody clicks the ads, and no possibility of
selling subscriptions. Compare Reddit's target demographic with Bloomberg's
target demographic. I bet Bloomberg has more than one developer.

(Incidentally, what a scam. Bloomberg terminals cost some insane monthly fee,
_and_ they have ads! And all I've ever seen anyone do with them is check
sports scores and read celebrity gossip. Google should have put them out of
business years ago.)

~~~
jedberg
It's not a money problem. We've got plenty of money. We just weren't allowed
to spend it on developers until recently. Now we have a bunch of open
developer positions that we are in the middle of hiring for, and will probably
have more soon.

~~~
quicksilver03
I'm sorry, but if you have money and you're not allowed to spend it on what
you do need the most, then actually you _don't_ have money and you've
definitely got a money problem.

------
citizenkeys
In theory, if they can get by temporarily with just one engineer then the site
was engineered well to begin with. There's some good management lessons from
"The Mythical Man-Month" being applied here:
<http://en.wikipedia.org/wiki/The_Mythical_Man-Month>

------
gersh
How many developers does hacker news have?

~~~
cookiecaper
HN is a much, much simpler program, both in terms of features and internals.
Internally, Reddit is a gigantic mess.

~~~
shii
Isn't HN basically news.arc[1] plus some custom mods and caching with
content/stories saved as flatfiles?

Reedit has many more sophisticated load issues to about. The two aren't really
comparable it seems.

[1]:
[https://github.com/nex3/arc/raw/042990a55d8c880a0a88476eb52b...](https://github.com/nex3/arc/raw/042990a55d8c880a0a88476eb52b169e47f35d32/news.arc)

~~~
cookiecaper
Yeah, that was my point. Reddit is much more complex and much messier.

------
fedd
my bet is that there is some 'technical debt', so the devs leave when said
'business is over your tech details'! :)

------
rjhackin
This is not good given that they just had a recent 6 hour downtime. Why would
folks at Conde let this happen ?

~~~
cookiecaper
Conde really doesn't know what to do with reddit. They probably acquired it
"knowing" they had to have something in "the social media space" (the
acquisition predates the salesman's current buzzword of the hour, "the cloud")
so they could compete in the "digital age" and all of that, but since they've
acquired it my understanding is they've basically starved it and reddit has
been running on fumes until they implemented Reddit Gold last summer. They're
keeping it because they think they need something on this new-fangled internet
stuff, but they don't what to do with it, and it's not generating much
revenue, so they're just trying to leave things alone until they figure out
something.

Note that this is just speculation based on what I've read, I don't have any
inside knowledge.

~~~
JoachimSchipper
Apparently, they have enough money but internal politics prevent them from
hiring more people (it's a different budget...) Hence AWS, which isn't that
cheap but does reduce personnel costs.

See the big reddit thread on this topic, or HN's thread about that reddit
thread.

------
dhughes
Wouldn't it be surreal if Kevin Rose went to Reddit?

------
phillco
Interesting. This sure is boosting their pageviews/developer ratio.

~~~
redthrowaway
Which is currently sitting at 1.1B : 1.

------
heed
Hmm.. Would now be a good time to build a new reddit?

~~~
aberkowitz
Building a new reddit is easy - building a community, and scaling to support
1.1 billion views a month is not.

------
barista
Does this mean that the social new bubble has finaly burst with reddit down to
one person and digg winding down?

------
diptanu
HN seems to be doing fine with just one awesome developer :-)

~~~
cookiecaper
HN has been struggling to keep up with load recently. Remember the pagination
of comments a few weeks back needed to keep the server from falling over?
Yeah. I think news.arc/HN could really do with some extra scalability work; it
still slows down significantly at peak times.

~~~
bluesnowmonkey
Server? There's one server?

~~~
tomjen3
Yes one server, but that isn't entirely fair because pg has Robert Morris to
tweak it.

And he is a kick ass sysadmin.

------
anacrolix
Scrap it. Rewrite. It can't be that complicated and if it is, it needs to go.

~~~
marcc
Rarely a good answer. All of that complexity wasn't added for fun. Making a
comment like this really shows ignorance in software development. While we all
want to scrap and rewrite some projects, most of us know that it isn't as easy
as it looks and you will be spending your time on the wrong thing.

Also, I don't have any idea what to make of your comment "...and if it is, it
needs to go.". Are you just trolling or do you really suggest that they shut
the site down because it's complex?

~~~
grammr
I think he's suggesting that if any part of the reddit codebase is too
complex, then it needs to be ripped out.

------
pdaviesa
It appears that Reddit's days are numbered. Sure they may be able to get by
with one developer for a while, the product is fairly mature and content is
driven by the user base. But how long can a website remain stagnant, with no
innovation until the user base starts to migrate to the next best thing? Also,
being owned by a large, old media company doesn't necessarily give one much
confidence that Reddit is backed by visionaries who are willing to push the
envelope and take risks.

~~~
Raphael
And what is the next best thing in your opinion? It's one of the few places
that does nested comments properly and the lightweight up/down voting always
keeps it interesting. The only thing I can see replacing it is an outright
clone, but no one's stepped up and done it in all the years Reddit has been
around.

~~~
officemonkey
>And what is the next best thing in your opinion?

I'll know it when I see it.

I was a daily user of Slashdot from 2001 to 2006 and then I found Reddit. I
completely abandoned Slashdot less than a year later.

What could beat reddit? Well, if there was a facebook-y site which allowed me
to associate with people who had my interests (rather than family, friends,
and people I went to college/high school), I'd be down with that. (I love my
mom, but I don't think she cares about Minecraft, old-school RPGs, French pop
songs, Asa Akira, and Jules Verne novels.)

Twitter is close, but the character limit, url shorteners, and "twitter-speak"
(intentional bad spelling/grammar in order to "get under the limit") just make
my teeth itch.

