
Peter Molyneux’s staff work 36 hours non-stop - beedogs
http://ilovecrunch.co.uk/?p=230
======
jpatokal
Jeebus, what a way to turn a molehill into a mountain. @22cans:

In fact @pmolyneux BEGGED the team to go home but they wouldn't have any of it
:) <https://twitter.com/22Cans/status/268387590368030721>

22 Cans is a tiny startup and they were expecting 10,000 concurrent users tops
-- they're at 2 million and counting now, and they have to capitalize on that
momentum and not piss off their obviously fanatical players. So they pulled an
all-nighter to build and migrate from a single server to a clustered
architecture, and hell yeah, if I was in that position I'd do that too.

Seriously, is there another option? If you think the answer is "clock out at
5:01 PM and go home", you shouldn't be in a startup. Obviously pulling 36 hour
shifts on a regular basis is not sustainable, but this is _precisely_ the type
of situation where it's warranted.

~~~
fatbird
The question is why did they assume 10k concurrent users, or plan for that to
be the cap handled under the current architecture? Any competent project
manager should have said "let's pretend we're massively successful... how will
it work then?" and have some contingency plan in place so that they're not
trying "experimental" fixes.

The flaw here isn't the heroic efforts, it's that heroic efforts were needed.
In the days of cloud based architectures, it's a lot easier to be ready for
massive horizontal scaling than it used to be, so much so that being unready
is malpractice.

~~~
Negitivefrags
Assuming massive success is a stupid move. The majority of the time you are
not that successful. The ground is littered by projects that spent vast sums
of money (and more importantly time) designing for load they never needed to
handle before release.

In about a month my company will be releasing an online game. We think we will
get about 15-20k concurrent users in the first few days, but we will be
testing the realm for as high as 50k concurrent users using load testing bots.

What will happen if we suddenly get 100k or 200k concurrent users? Even
throwing more hardware at the problem might not solve it immediately if there
is some new bottleneck we haven't found before.

Testing the realm for more than 50k concurrents and then spending engineering
effort required to make it work will consume time and effort that we can't
afford to spend. All for an outcome that is unlikely to occur.

~~~
fatbird
Assuming massive success is stupid, but testing for it seems prudent. Given
how easy it is to spin up EC2 instances generate bot-load, even just to see
what'll happen or to find the collapse point, seems so straightforward as to
be routine. And as the commenter below observes, the Curiosity interaction
model requires high server load for all interaction.

~~~
btilly
The problem with your theory is that there are lies, damned lies, and load
tests.

Any sensible app is going to have caching layers. A simple load test is going
to just be a really good exercise of the caching layers. By contrast a complex
load test with lots of calls to do really random stuff is going to stress you
out in entirely unreasonable ways. And finding a realistic balance of traffic
is difficult unless you've simply sampled real traffic - which you can't do
until after you've generated some.

~~~
bowyakka
This!

I cant agree with this more, unless you approach me with an experiment that
would put most science to shame I do not value your use of
Jmeter/Tsung/Siege/Rando-Python-Script. If you test plan does not have pauses
in it, and you are not engineering a nuclear reactor I probably wont even
start the conversation.

.... At $DAYJOB, when we were putting together a new search engine we spent
some time upfront putting in a lot of tooling to worry about and test
performance every step of the way. Some of this tooling was built around
jmeter and focused on generating realistic test plans (which we could derive
from the already operational existing search engine). It was a neat
infrastructure it tracked everything by commit, captured all manner of metrics
and used huge datasets, mined from the logs, built out with markov models.
Everything we reported into an install of the software that runs
<http://speed.pypy.org>, we thought we had it conquered.

Yet it was so flawed! Data would get stale, models would break, caches
invalidated datasets and much much more.

It was funny at the same time I wrote a tool that we now know as darklaunch,
this tool is stupid simple, it captures incoming traffic to the production
system with pcap, and then (after a little mussing with it) sends it through
to a always on test cluster. Yet its this tool that produces all the load, has
all the weird user behavior, never gets out of sync and can run load, soak,
endurance and impact tests.

Guess which one we use to test out new software, or new features, or to tune a
garbage collector in as close to production as possible ?

I have not started Jmeter now in over a year, its been no great loss; sure we
have microbenchmarks where they matter but for everything else you often hear
the words "darklaunch it"

Now for sure we have the luxury of having real traffic to use and abuse, but
if you don't have real traffic then realise you are using a scrying device
anyhow and I will bet you good money your assumptions are wrong.

------
jpxxx
Industry bullshit artist approaching has-been status... super high-concept
game design without any actual game... his nephew as producer... five staff
total working on a massively parallel realtime cross-platform title with novel
mechanics...

What could _possibly_ go wrong? popcorn.gif

~~~
supersaiyan
Peter Molyneux in a nutshell : <http://www.youtube.com/watch?v=DkGMY63FF3Q>

But aside from the hate 22cans is doing some impressive stuff

------
MortenK
To "just work harder", in this case 36 hours, is the easy decision to make
when something goes wrong. It smells deliciously of heroics, bravado and top
performer mentality. Most importantly, the soul drenching, exhausting late
hours, gives the illusion that "we are doing all that is humanly possible to
resolve the problem". How tough are we, right?

But as the author points out, heaps of literature clearly show that
"crunching", like it's affectionately called in the games industry, is contra-
productive. One guy working 8 hours, going home for a good sleep, and coming
back to work an additional 8 hours will vastly outperform the developer-
turned-zombie that's been at it for 24 hours straight.

At that point, short term memory is gone, stupid mistakes (blunders even) are
rampant. Fixing simple issues takes extraordinary amounts of time, because the
ability to focus is all but lost. You effectively have tunnel vision and your
reactions and logical skills are reduced to that of a drunk.

This is not anecdotal - there are an immense amounts of studies done on the
subject. But time and again, the knowledge gets discarded because it's just
easier to "work through the night" at a frantic pace, rather than take the
necessary step back, let people rest and do the work in an orderly and planned
manner. By burning the candle in both ends, at least no one can challenge your
commitment to solving the problem.

While it may sound counter-intuitive to go to bed while the house is on fire,
it is, at least if 40 years of research is to be believed, actually the only
correct thing to do in a software development crisis. Note this doesn't mean
"stick 100% to 8 hours and always go home at 17.01", but it does mean if you
are still there at 23.00 you are much better off going home, getting sleep and
resuming work when well rested.

The games industry seem to be especially vulnerable to this problem. It might
be due to the general inexperience level caused by games development being a
young mans game, or it might be due to a failure to understand that games,
while glorified in the industry, is "just" software, and therefore also
subject to all the established rules of software development.

In any case, I agree with the author that Molyneux should know better than to
tweet from the rooftops how many hours everybody is working, but I guess he is
not thinking entirely straight after 36 hours of work.

~~~
jacquesm
It depends on whether or not a fix is perceived to be further away than what
you can accomplish in that one night.

Being down for yet another business day when you don't have to be ('house on
fire') is usually not an option.

There is also a class of problems (chasing subtle bugs) that you're much
better equipped to tackle when you come in fresh in the morning or when you're
actually not looking directly for them, the flip side is there are those that
it takes forever to get the mental model set up that maps the circumstances of
the bug with enough resolution that you have a chance of cracking them that it
pays off to just sit there until it's done, even if that means staying up well
past your bed time.

The rule seems to be 'go home if you can no longer positively contribute
towards solving the problem', not just 'go home'.

~~~
MortenK
That's a reasoning that is prevalent and very often catch project managers and
developers alike: It is "not an option" going home, because of whatever crisis
is occurring. Because if we aren't working through the night, we are not
committed, and we are certainly not doing the most we can to resolve the
crisis.

The outcome is almost always stereotypical: Early in the morning, the fire is
doused, and all is good. Until suddenly three more fires spring up out of
nowhere, previously undetected due to fatigue and tunnel vision on the one
particular issue that got it all started.

Slow and careful consideration of the problem, the potential fixes and
especially the side consequences of implementing those fixes, gets thrown out
the window after 24 (or even 12) hours of coding.

I agree in theory that the best solution would be if you could somehow tell
when you are no longer positively contributing, and only then going home. But
it is exceedingly hard, if not impossible, for an individual to accurately
determine when he is no longer positively contributing towards solving the
problem (i.e. Creating more problems that is being fixed). The outcome is you
have people valiantly soldiering on, in the false belief that they are
actually contributing, but with actual, measurable setbacks being incurred due
to fatigue.

~~~
jacquesm
Some things just can't wait. If you have never worked in an industry where
there are no commitments to outsiders then that's all fine and good but
production (banks, networking, airlines, critical infrastructure such as
electricity plants, water pumping, hospitals and a very large number of other
services that society relies on) problems take precedence over a lot of
otherwise completely valid concerns.

If that is a recurring theme then there is definitely a deeper problem but on
occasion things go wrong and 'going home' really isn't an option.

Count your blessings if you are not stress resistant and are working in a line
of business where these things don't matter.

Production problems have a habit of not announcing their arrival dates, and
some of those can be surprisingly hard to track down to their root cause.

Working until the problem is fixed in those situations has nothing to do with
'showing commitment' it has everything to do with being committed.

There is a right time for knowing when to call it quits and there is a right
time for pushing on until the lights are back on.

You simply can't categorically state that it is wrong to push on, it depends
on what is on the line and the capabilities of the people involved.

~~~
MortenK
In the case you mentioned, with a clear perception of a problem being fixable
within a nights work, I of course agree, that's the time to keep on working.

But I think we will probably disagree on the right time to call it quits.

In my book, that is when the situation is more diffuse, like the hard-to-find
bug on a real-time enterprise system with many integrations, and your team is
fatigued, having spent 20 hours already trying to fix it, with no clear
outlook on when it will be done.

If you keep on at this point, you run the very real risk of turning a
manageable bonfire in the kitchen into a wildfire that takes down the whole
neighborhood. I.e. direct contra-productivity in the form of unintended side
consequences and breakages introduced by bleary-eyed programmers. Such issues
can often end up putting the system down for even longer than it otherwise
would be.

In a standard, non-live development situation, the effect will be less
immediate, like slipped schedules and high defect counts, sometimes turning
into the classical "deathmarch" project where there is a constant feeling of
one step forward, two steps back.

The risk and impact of course varies a lot from situation to situation, so
just to be clear I am not categorically stating anything here: Calculated
risks can be okay, and everything has exceptions, especially when it is a
production / sysops crisis versus a development deadline crisis.

------
astrodust
Unless you're battle-hardened and able to endure multiple 36-hour runs in a
single week, which few of us can withstand before something completely _snaps_
, this is probably the worst possible idea.

You do a big push like this as a last resort, not as standard practice.

It reminds me of an anecdote about Icelanders and how they do their off-road
driving: You keep your vehicle in two wheel drive. If you get stuck you can
always use the four wheel drive to get back out. If you're always in four
wheel drive, when you're stuck you're really stuck.

~~~
biscarch
_Nowww you knowww_

But seriously. Interesting point about Icelander off-road driving style. It
makes sense for terrain that could stick you while in four wheel drive, but
it's never occurred to me to drive like that. Ex: Chicago snow storm about 2
years ago, I just drove super-careful but many people had to abandon their
cars on the side of the road.

------
ispivey
For all of you crapping on Peter and his team:

What if the employees were founders with equal shares of the company, and they
launched a product and it fell over? Would you blame them for pulling an all-
nighter to scale it up in the face of unexpected traffic? Should they give up
at 10pm? Midnight? 5AM?

If they're not equal partners, should they immediately give up? What if they
own 10%? 1%?

What if they don't have much on their resumes and are excited to make this a
successful project, instead of a laughingstock that fell over in the face of
scaling challenges? What if they care about each other and want to make this a
success for their friends and coworkers?

What's the function into which you plug all these variables and decide when to
go home?

It's not black-and-white, pun intended, and if you think it is you're being
obtuse. I hope they feel happy and like they accomplished a lot in the face of
long odds, and you should too.

~~~
Deestan
I'd rather have a colleague come in to work _drunk_ than pull an all-nighter
to work on core functionality.

 _Yes_ , we really need to get this thing improved as soon as possible, but
having effectively brain-damaged people introduce bugs all night _and_ then be
utterly useless the next day is not helping. Take your financial lumps and get
over it instead of burning out everyone and ruining the codebase.

These kind of masochistic hero stories perpetuate the destructive belief that
every hour your face is planted in front of the monitor is equally productive.

------
JanezStupar
I did a 36 hour push once. I did it because there was really no other way to
achieve the deadline, since there was about three days worth of bug squashing
to reach a deadline and I only had two days left.

I am proud that I managed to pull it off, however I tell it as a tale of
caution not as a tale of inspiration.

The work I did was objectively atrocious, by the 24 hour mark the simplest
operation took me 5-10 minutes. My short term memory was completely gone by
then and I had to rely on writing down each step of an operation as simple as
refactoring a block of code into its own function, just to be able to pull it
of in a "reasonable time frame".

Thus I would call bullshit on anyone claiming that his team made "huge
improvements" and that after such a push some piece of software "started
behaving like it should".

Unless that means that they introduced a plethora of bugs that they would be
busy fixing for months from now.

~~~
babebridou
The experience of going for an all-nighter is akin to watching a full season
of 24 in one go, only you're the protagonist. It's memorable, and yes,
especially if you manage to pull it off.

I tell it as a tale of inspiration: do it, _once_.

Whether it's good or bad for business/health/morale is a whole other story.
Hint: it's not good.

~~~
JonnieCache
One of the best things about going to university is you can get all of that
stuff out of the way without hurting anyone else.

~~~
anthonyb
Well, until you graduate and continue your old habits...

------
mikeklaas
When we launched Zite, I (along with several other engineers), were awake for
about the same amount of time keeping the servers alive.

Sometimes a heroic emergency effort is required. As long as it is
(extraordinarily) rare, it's not in itself sign of a dysfunctional
organization.

~~~
hrktb
> _heroic emergency effort_

Completely agree that it'something required sometimes. But it's nothing
heroic, it is a (temporary I hope) dysfunctionment, and shouldn't be put in
midly positive light.

Planning failed (not assuming you might have 2m users is sensible, but it's
still a failure in it's way), and shit hit the fans. It's the kind of failure
you might wish you had, but next time you should think hard about a way to
deal with this kind of situation with minimal emegency, and no need for
'heroes'.

------
borplk
It's all our own fault.

We glorify the noodle-eating couch-sleeping code-generating-machine hacker and
let the corporations profit by exchanging our health and personal lives for
stupid hacker-perks which costs the companies nothing.

Woohoo! come hack for us, we give you a fancy chair and large monitors to
stare at and even food! what else do you want out of your life?

We need to see that we are being taken advantage of.

We have raised expectations so high, that you can't land a decent job unless
you have dedicated your evenings to patching bugs in open source projects for
free.

What do you think that says to an employer? Sorry to break it to you but it
roughly goes like this, "The poor bastard has no life. When he goes home, he
does the same crap _for fun_. He'll be more than happy if I pay him 2 peanuts"

Have you asked yourself why other professionals don't have "open source" like
we do?

If programmers are expected to work _for fun_ to develop your email client,
web browser, server softwares etc... it is only fair if lawyers were also
expected to _contribute_ to cases for free.

I'm too damn tired of this crap. We have put ourselves into an infinite loop
of competition that only hurts ourselves and brings profit to the
corporatoins. The guy next to you has to sacrifice his life because he needs
to compete with you to stay relevant and you do for the same reason and so
on...

We are smarter than this. We deserver better than this. We need to bring this
to a stop.

~~~
philbarr
In other professional fields (like with lawyers), doing work for free is
called "pro bono". I would think that if you're being interviewed for lawyer
position they would ask you how much pro bono work you've done.

So I disagree that doing this stuff for free is inherently bad, but I fully
agree with the sentiments of the rest of your post. Particularly this: "come
hack for us, we give you a fancy chair and large monitors to stare at and even
food! what else do you want out of your life?"

An old manager of mine was once on the phone to a recruitment agency trying to
hire some developers complaining that they hadn't made progress, at one point
he screamed, "it's software development, it's not fucking rocket science!" He
did that in a room full of developers.

For some reason many non-technical people get to become managers of technical
people and have that kind of opinion. This then leads to the, "well if it
doesn't work you can stay here all night until it does" mentality.

~~~
toadi
To be very honest. Software development is NOT rocket science. Writing
corporate CRUD code is not that difficult...

Writing a new 3D game engine is off course more daunting but still not rocket
science.

~~~
Codhisattva
One difference software engineering has with aerospace engineering is that the
people building the rocket get all the time they need. Since it's rocket
science it's pretty much a given that if it's rushed or a PM or CEO puts the
pressure on (with people working triple shifts for instance) then things just
literally blow up.

In software the coders rarely have the option or will power to push back
against unreasonable deadlines and fantasy launch dates. This is especially
common in corporate cultures where a chain of command makes those calls and
large scale plans are built around dates pulled out of the air. At best the
coders are given lose-lose choices: miss the date or ship a broken product.

------
wallflower
The danger of incurring so much sleep debt so quickly is that you will start
paying compound interest on it immediately - in terms of ability to avoid
accidents, struggling with mental coherence, handling relations with loved
ones, out-of-it attitude, and dealing with unexpected situations gracefully.

------
dpeck
I like Mr Molyneux's imagination, but I have to say that I enjoy his
deliverables significantly less. Whether intentionally hyped or not he has
become, in my mind, quite the bullshitter. Seeing this I'm not surprised by
the disconnect between his vision and products.

I'd advise anyone against purchasing something from his group without having a
very generous return policy.

~~~
damian2000
I read here (see below link) they are soliciting user donations in order to
speed up the bug fixing of the server code... so presumably to help pay the
overtime of the coders. First time I've heard that a game user has been asked
to pay to fix bugs in their game. I congratulate Molyneux on his chutzpah.

"The popularity of the app has now forced Molyneux to ask for donations so
that they can help improve the experience, but we’re not too certain how well
this plea will be received." \--
[http://www.phonesreview.co.uk/2012/11/13/curiosity-app-
game-...](http://www.phonesreview.co.uk/2012/11/13/curiosity-app-game-
by-22-cans-peter-molyneux-social-cube/)

~~~
jpatokal
Srsly? The Curiosity app is free. The donate button isn't even in the app
itself, but on the developer's web page.

------
damian2000
My advice is for Peter to have a read of "Peopleware: Productive Projects and
Teams" by DeMarco & Lister.

~~~
markokocic
My advice to _anyone_ in our field is to read this book. Even if you yourself
are not manager, and never have managed any team, it will at least make it
easier to spot some common pitfalls and be aware of the consequences before
it's too late.

------
bryceneal
You're making a lot of assumptions here based on a tweet. Also calling these
employees "idiots" is pretty irresponsible in itself.

~~~
beedogs
Frankly I just copy-pasted the title from the post. And it looks like it's
been 'corrected' now.

------
cwh
as one of two 'server coders' that scaled Draw Something from 0 to 35 million
DAU, a 36 hour day sounds nice.

~~~
miahi
What was your motivation?

~~~
cwh
quite simply, to see just how far we could take it.

------
jrockway
The author of these tweets should read a book about literacy. He could learn
something worthy of cerlibretion.

------
michaelgrafl
I'm a big proponent of working (i.e. laboring) as little as you possibly can
in order to leave as much room as possible for convivial activities and
personal growth. But I think it is disingenuous in this case to paint Molyneux
as a slave driver.

They were ill-prepared for the launch and stuck together to scoop the water
out of the boat. It's probably a bad idea to do it for 36-hours straight, but
it's understandable. I would certainly stick my teeth in there till the
storm's passed. Especially in a company this small, where the line between
collegiality and friendship are probably quite blurred.

I'm much more interested in what the RSI is a result of. That shit needs to be
taken care of.

------
lyricalpolymath
Does anyone know to what kind of books or reference is the author talking
about when talking about the "15 years of free literature on how to run large-
scale online projects"

I've seen in the comments that damian2000
(<http://news.ycombinator.com/item?id=4781706>) has suggested "Peopleware:
Productive Projects and Teams" by DeMarco & Lister.

any other suggestions?

------
seivan
I'm a gamer and this is very subjective of course but Peter Molyneux hasn't
actually done any good games. Overrated junk ~ and it's not his works, it is
the engineers, designers, illustrators, sound engineers and actors who've done
the game. He just makes enough noise to get attention and praise.

~~~
vacri
His early games were interesting, fun, and original, pretty much all the
Bullfrog catalog. I'm also not sure you can get away with saying it's not his
work when he's the lead designer and executive producer on those titles. Yes,
he didn't do it alone, but those roles stamp a considerable amount of
personality onto the product.

Lionhead produced nothing but crap.

------
jheriko
... and to think all they needed to do was hire one real programmer in the
first place.

------
gobengo
No one should be pressured into this type of work, and it's important to
always maintain that.

However, if you haven't worked a 48hr+ stretch in a young startup, check your
pace and/or commitment. It may be fine, just think about it.

------
politician
Anyone else find Curiosity intriguing not for the game (which is arguably
about vanity), but for the technical brain-trust that operating such a service
creates? Here's hoping they all write books...

------
BklynJay
On what planet does this make sense? Here on Earth, this is generally
considered "unwise."

------
antidoh
We hope none of those people are driving when they finally go home.

------
rprasad
Judging from the reviews for Curiosity in the Google Play store, it was not
time well-spent. (I would provide my own comments on the game, but it insists
on crashing before I can do anything with it.)

If it is a joke...Mr. Molyneux would have better spent the time on fixing his
abysmally bug-ridden game.

