
Google Wave and the mythical man month - michaelneale
http://rethrick.com/#mmm
======
rachelbythebay
I was there when "Walkabout" started being bandied about. It irked me to no
end that I could jump through all of the hoops and finally get behind the NDA
curtain by being hired and then be told "nope, you can't hear about that".
That was a severe jolt to the internal culture of the whole company.

There had been other projects which had been kept under wraps successfully.
Chrome and V8 were demoed in Seville long before anyone on the outside heard
about it. They said keep it quiet, and we did. Google TV had been knowable for
_ages_ before it went anywhere externally. There was no reason to think
someone would leak Wave.

Instead, they started playing the "we're special and you are not" card, and
that started a sense of resentment growing. Not even the infrastructure teams
which were going to provide services to them were let in on what was really
going to happen in there.

Then after far too long, demo day of Wave arrived. I only stayed long enough
to see them hit backspace and have it echo out to everyone else who was
connected. I remember my exact comment at the time: "packets". As in, lots and
lots and lots of packets flying around to generate RPCs for all of those
deltas. Then those turn into XML or whatever going out to web browser clients,
and ... yeah. SO many packets. That right there worried me greatly.

So then I see this thing about it not scaling properly and choking JVMs and
suddenly it all makes sense.

Oh well. All of the secret code depots and restricted access areas must have
been practice for what is now happening with Plus. Entire floors of buildings
you can't open as a full-time employee? Yep. More code depots being locked
down? Yep.

~~~
hyperrail
> All of the secret code depots

DepotS ? So I assume Google finally gave up on making their one Perforce
server scale, then?

(Unless Google is crazy enough to actually do multiple depots on a single p4d
instance...)

~~~
twoodfin
> (Unless Google is crazy enough to actually do multiple depots on a single
> p4d instance...)

Why is that "crazy"? AFAIK, depots are just an organizing mechanism, they
don't have any effect on scalability.

<http://kb.perforce.com/article/684>

~~~
hyperrail
"Crazy" is probably the wrong word for me to use (I have been known for my
hyperbole in speech). What I meant was that, from my point of view, having
multiple depots is almost the same as having multiple directories at the root
of a single depot. The Perforce Support article you linked to says the same
thing.

Hence, I feel that having multiple depots on one Perforce server simply eats
up client names and causes confusion among those who haven't learned the
distinction between "depot" and "server".

------
cletus
It's an interesting post and raises several cogent points but it misses the
biggest problem with Wave: it was a solution in search of a problem.

I say this simply as someone who was, at the time, outside looking in. I know
little more than that but it had all the hallmarks of what happens when
engineers are running the asylum. Here's this communication medium in which
basically all other communication media can be implemented (Email, IM, forum
posts, Twitter, etc). It's the kind of general solution that engineers come up
with it.

I read a post from someone else (can't find it now but I think it was on
Quora) who was familiar with the matter and they were saying the risk-reward
thing (which this poster mentions in passing) was all messed up. Basically the
incentive structure rewarded mediocrity.

I can't speak with any knowledge of those matters but I can believe it. After
all, in a startup what happens if the startup fails? You find a new job. There
is a strong incentive to make your runway last and get to your next funding
round (or, Heaven forbid, profitability). Inside somewhere as cashed up as
Google, those incentives (IMHO) disappear.

If the "startup" fails, what happens? You just move to another part of Google.
What do you think the odds were that with Wave going away, any extra Wave
incentives became worthless (as would happen in a startup)? Basically zero
(IMHO).

~~~
josephg
Every time I tell an engineer that I worked on the wave team, I'm greeted with
the same response: "Oh, Wave! You know, the _real_ problem with wave was X".
And then I stop listening, and tell them this:

I have heard that for all the values of X. Here are the most common:

\- It was too hard to use / I didn't know what to do with it

\- It didn't integrate with email / didn't have email notifications

\- It didn't support IE

\- I started using it, but none of my friends were using it, so I stopped

\- It was too slow

Too late, the team spent a bunch of time talking to actual users and finding
out how they were using the product and what their real pain points were.
People use wave to communicate in small teams. IIRC the biggest pain point our
users cited was that wave didn't support printing. The people who did get over
the adoption hurdle ended up using wave _a lot_. (I still wonder if maybe wave
might have survived if we had charged for it.)

Wave died because we were too slow to make it good. The code base consists of
around 1M lines of java. With 1M lines of code, you need a giant team to get
anything done, and a giant team working hard on features tends to add even
more code. The wave team burned too much money too fast. And we were still
working too slowly to get explosive user growth. It made lots of people sad,
but Google was right to kill wave.

If I could go back in time and give advice to the team, I would say:

\- Don't use Java

\- Don't scale past a few engineers. You will have a much longer runway that
way, and you'll need it if you want to replace email.

\- The network effect will kill you unless you integrate with email from day
1.

\- You don't know what you're building, so talk to users early. Like, today.

\- Don't optimise for scalability until you know what the product is supposed
to be

\- Don't try and reinvent the scrollbar, you idiots. (So much stabbing.)

~~~
rphlx
I am truly boggled that Wave contained 1M lines of code given the
functionality.. No wonder it was hard to make it good..

Feels like two high-end devs could implement the 80% of it that people
actually used in a couple weeks and 10-50k LOC.

Maybe I am missing something, but it sounds like Large Company culture doomed
it.

~~~
josephg
It would take more than a couple weeks, but 10-50k LOC sounds about right.
Since leaving google, I've rewritten wave's concurrent editing algorithm in
coffeescript in just 3k LOC.

[Blatant plug: <http://sharejs.org/> ]

~~~
bane
Was there any particular reason why it ended up so huge then? Trying to handle
each and every possible use/edge-case completely? Why not Java (from your
previous answer)?

I _loved_ Wave despite its warts and managed to run several extremely
successful small team document writing collaborations through it, completely
replacing IM, email, groups and lots of early document drafting in Docs/Word.
There was nothing that anybody on the team had experience with that came close
to that. We went from hundreds of emails a day and hours of IM'ing to
absolutely zero of each within a week of moving to Wave.

My only real complaints were that the web client was dog slow and going from a
Wave of draft-edits to a final document was kind of painful.

Never did find a use for lots of the other bells and whistles (embedded
widgets things, channel bots, etc.), perhaps that's where a lot of the extra
LOC went to...

The entire Wave story would make an amazing case study if Google would ever
let the entire story out.

~~~
josephg
I don't think there's any one single reason why the code base is huge. Its
written in proper idiomatic java, so a lot of the code is bureaucracy
(interfaces, factories, managers, assertions, etc). The code that was
opensourced is reasonably representative of the rest, if you're curious:

[http://code.google.com/p/wave-
protocol/source/browse/#hg%2Fs...](http://code.google.com/p/wave-
protocol/source/browse/#hg%2Fsrc%2Forg%2Fwaveprotocol%2Fwave)

I think we should have spent more time keeping the code clean; but thats
always a tough call when you have features to ship. Its easy to point to lots
of things and say they were mistakes. But I worry that, had the project been
successful, we might point to the very same things and talk about how clever
we were.

A huge effort was put into making the client more responsive in the few months
just before wave got cancelled. I'm not sure if its still the case, but at one
point wave was loading faster than gmail. We were a month or so away from
shipping server-side rendering of waves with progressive enhancement for
editing. That made waves load in a snap. (The guys finished that piece of work
and put it in the opensource wave in a box project.)

You weren't alone in getting a lot of value out of wave. We used it internally
for everything. When we had meetings, we shared the agenda in a wave. People
edited the wave with meeting items, and during the meeting people
collaboratively annotated the agenda with what was decided, turning into the
minutes. It was a thing of beauty. I still think there's a need for something
like wave - it solves real problems.

When wave was cancelled, there was a series of honest retrospectives
internally on different aspects of the project. I hope they get eventually get
published eventually too. A lot of people were quite career-sensitive after
wave. Most blog posts you see by wave developers are written by people who
have left google anyway.

------
johnfn
> And this is the essential broader point--as a programmer you must have a
> series of wins, every single day. It is the Deus Ex Machina of hacker
> success. It is what makes you eager for the next feature, and the next after
> that.

I liked the whole article, but this paragraph really stood out to me. It's
something I've never thought to put into words, but is so true. Going for days
and weeks and not seeing any success in what you're doing is horribly
demoralizing. It's happened to me only once, and that was enough to reconsider
my appreciation of programming entirely.

It's because the real appeal of programming is, like he said, the incremental
gains, the constant moving forward, the iterative process. It's why small side
projects are so much fun, because there's nothing getting in the way of your
next small accomplishment. It's what draws people (like myself) to programming
in the first place. And in its lack, it's the slow killer of large projects.

Thanks for the article.

~~~
SkyMarshal
While true, I suspect the reason is that it creates that dopamine release we
all find so pleasing and addictive. It's why people also do drugs, have sex,
play MMO's, etc. Our built-in carrot.

 _> And in its lack, it's the slow killer of large projects._

It's interesting that one reason for large project failure may be failure to
stimulate dopamine release in its programmers.

------
robfig
"Now, I don't mean to imply that Wave did not have some very smart engineers
working on the UI, we certainly did. But talent is different from experience.
The latter is a guard against 3.5MB of compressed, minified, inlined
Javascript. Against 6 minute compiles to see CSS changes in browser. Against
giving up on IE support (at the time, over 60% of browser market share)
because it was simply too difficult. Against Safari running out of memory as
soon as Wave was opened on an iPad."

I wonder how much of the failure was this sort of thing vs just having a
product that people didn't understand. (I definitely agree that if the UI had
been simple and snappy it would have been better)

On the bright side, I feel pretty confident that some real startup will take
the open sourced Wave technology and do something good...

~~~
josephg
> On the bright side, I feel pretty confident that some real startup will take
> the open sourced Wave technology and do something good...

I don't share your confidence. Making even trivial changes to the 350k lines
of opensourced java takes a lot of time and a lot of skill. There are only ~5
part time developers working on it. I doubt that it will be useful anytime
soon.

 _[Disclaimer: Some of that code is mine]_

~~~
bane
Sounds like it would almost be easier to start from scratch with the protocol
spec and build something lighter weight and not Java.

Then in the client only expose a subset of features until they are really
good, then another, then another. Keeping the client codebase small as well.

~~~
josephg
I agree, although if I had the chance I would change the protocol spec too.

The client-server protocol uses protobufs encoded over JSON, and the
federation protocol uses protobufs encoded over XML, via an XMPP extension. As
others have said, there weren't accepted standards for this stuff just a few
years ago. Today, you could build whip something together reasonably easily
using JSON and socket.io or something.

------
heat_miser
Fantastically generous post. I wish more engineers were as publicly honest
about their own, and their team's failures so that other engineers and
engineering teams can learn from the experience.

~~~
hello_moto
.. will happen only if the society appreciate and give a bit reward to such
situation...

These days we have plenty business bullies that sing the song "Win big or go
home in shame" or "Second place is the first place loser"...

------
dilap
_You need the same mix of experienced talent working in the UI as you do with
traditional "serious" stuff. This is where Apple is simply ahead of everyone
else._

Not only does Apple not shunt n00bs at the UI, it actively hires extremely
brilliant people to do UI invention/R&D.

E.g., consider the CV of the (obviously brilliant, IMO) "Up and Down the
Ladders of Abstraction" guy, Bret Victor (1).

That's about two lightyears removed from "UI is boring and easy; make the
junior programmers work on it while we do the algorithmically hard stuff on
the backend."

(As an aside, Holy Shit is BV impressive and refreshing -- incredible tech
chops combined with awesome aesthetic/design sensibilities, all wrapped up in
a humanistic focus on usability...just, damn.)

(1) <http://news.ycombinator.com/item?id=3099595>,
<http://worrydream.com/#!/cv/bret_victor_resume.pdf>, and
<http://worrydream.com/Bio/>

~~~
sceptre
Impressive.... but damn, why do i feel like a junkie

------
jamieb
_UI is hard_

hear hear. ui is harder than back-end stuff in my opinion.

~~~
vecter
UI is hard in the way that art is hard. It's difficult to understand what
people really want or will value.

Backend is hard in the way that building a skyscraper is hard. You need a
solid design and framework, but you also have to make sure every wire runs in
the right place. Managing complexity is really the problem there.

Which is harder? I don't think they're very comparable, in the way that I
don't know if creating beautiful art or building a skyscraper is harder.

~~~
jamieb
Yes, that's the mistake most people make: back-end is engineering, front-end
is art. No wonder we have such shitty front-ends.

Sure, there are a whole slew of "applications" that will suffice with a basic
form-driven UI accessing a rails or servlet back-end. And indeed there are
applications like twitter where the entire complexity is in the complexity of
the back-end. But then there are all the apps that simply aren't web-apps yet,
because nobody has figured out how to do complex, rich, and above all
Interactive, applications using fucking javascript. We're getting there, but
people are writing the code to do it as I type (jquery, backbone, etc).

So this is why UI is hard: its art _and_ its engineering. _And_ nobody thinks
about the engineering part. _And_ when they do, they have to find an engineer
who can talk to the designer without scaring him/her (very often "her").

~~~
danmaz74
In a word... UI is architecture (in the original meaning, like "designing
buildings where people live/work", not the IT one).

------
xbryanx
Please stop trying to reinvent the browser scrollbar on your blog. Can't page
down with this junk.

~~~
rachelbythebay
Worse than that, there's no content if you have Javascript off. Talk about
tedious!

~~~
spydum
At some point, people are going to have to admit the web is not as interactive
without javascript, and will need to stop complaining when they choose to
disable it. Those people are welcome to revive gopher if that is what they are
after.

~~~
xbryanx
Are those people supposed to turn their eyesight back on, through some magical
switch, as well?

------
mathattack
Great post but I don't follow the connection to the Mythical Man Month which
is about having the wisdom to avoid tossing bodies at a late project.

The article seems to be more about making sure you have experience on the GUI.

All this said, i shouldn't complain about the title - rather i should just be
happy the author shared hills lessons.

~~~
rdouble
One of the deeper messages of the Mythical Man Month is that large teams in
software don't work. The author of the post mentions that he should have known
the project was doomed when the hiring manager didn't think 26 employees on a
"startup" project was too many.

~~~
frossie
Yeah. According to the Mythical Man Month (or was it Peopleware?), when
someone gives you 100 software engineers to do a project, what are you
supposed to do? Set up 10 teams of 10 people, don't tell them about each other
and make them all do the same project.

The argument is that you are going to be better off doing that and picking the
best one at the end, than you are running the project with 100 engineers in
the first place.

The OP reads totally true to me.

~~~
snth
I don't remember that from the Mythical Man Month.

------
nostrademons
I really wish Larry would read this, and the Mythical Man Month in general.

------
acak
The 20% time policy that Google allows for one's own projects has probably
served as a nice way to experience those "small wins" he is referring too
(though the win may not be in the main line of work). I wish I had a similar
policy at the place I work - I feel I could have avoided a burnout phase.

I'd be curious to know if Google revokes that policy for focus teams like the
ones that worked on Wave and Google+.

------
Hitchhiker
Great post. Reminded me of the following for some curious reason :

" If the land mechanism as a whole is good, then every part is good, whether
we understand it or not. If the biota, in the course of aeons, has built
something we like but do not understand, then who but a fool would discard
seemingly useless parts? To keep every cog and wheel is the first precaution
of intelligent tinkering. " - Aldo Leopold

------
johnx123-up

       At the end we were close to 60 engineers
    

I guess, this is the reason for the failure. Probably 2-7 would be ideal for
this kind of experimental projects.

------
brok3nmachine
I really appreciate this post. I'm going through a very similar experience
myself. Was at a startup, which was recently acquired by a larger corporation.
Since I've previously worked at both small(and fast growing) and large
companies in the past, I thought I possibly had the experience to make the new
larger team I would be working with more "agile". But taking months to do what
I normally accomplish in a week or two definitely is hurting personal morale.
I'll try to achieve some daily small wins though=)

------
j_baker
With apologies to jwz:

 _Some people, when confronted with a problem, think "I know, I'll hire more
engineers." Now they have two problems._

------
bane
Start small, grow organically, don't use Java.

------
vegai
One of the lessons: Do not use Java for production applications.

~~~
wmf
Do not use Java for production applications if you don't know how to use Java.

~~~
rryan
You're implying that Wave engineers don't know how to use Java. But Wave
engineers are Google engineers. Google engineers are hired somewhere in a high
90th percentile of programming ability and experience.

You've proven the GP's point -- don't use Java because the only people who
know how to "use it" are 1% of programming experts that you probably don't
have working for you.

~~~
maximusprime
Only in the bubble of language fashion that is Hacker News, could the failure
of Google Wave be attributed to their language choice.

Billions of successful projects run on Java.

~~~
vegai
>Billions of successful projects run on Java.

I will give you billion dollars if you can prove that.

My personal guess (using my personal metrics) is that there are 10^3 to 10^4
successful Java projects and 10^9 failed ones.

~~~
maximusprime
In any event, the failed ones don't mean anything useful.

Successful ones mean "You can succeed using this". The failed ones mean "This
failed, could have been for a ton of reasons".

------
gunz_rozez
Like most acquisitions a great concept gets stuck in neutral....not sure
why....wave fundamentally is a fantastic idea....but a complex problem to
solve....the fact that no one else has solved what wave attempted to solve is
in itself a testament....but having said that I don't think it was an issue of
programmers coming up with a problem and then trying to solve it....I think
virtual communication as we know it is still pretty bad....and this post gives
you a good insight into how complex problems cannot be solved by adding more
people to the team....that just adds more complexity to an already complex
problem.

------
perfunctory
> I don't regret a single moment of being associated with it

This is a human nature. It's very hard to admit you just waisted your time.

------
yuhong
I wonder if Google Wave inspired creation of Chrome Frame, and if IE9 would
had enough HTML5 features to support Google Wave.

~~~
josephg
<ex-wave team engineer>

Early versions of wave ran on IE7. The problem wasn't HTML feature support, it
was a lack of engineering time. In the leadup to the IO announcement, the
client team dropped IE support in order to get wave ready for the demo.
Dropping IE support was supposed to be temporary, but everybody was too busy
to fix it. "Just tell people to install Chrome frame" was the regular excuse.

It was being worked on when wave was finally cancelled.

