I've got a bone to pick with people that have a bone to pick with waterfall, especially when they don't realize that whatever competing newer workflow they swear by is actually waterfall in disguise.
Let's just ignore for a second that a waterfall process sent men to the moon and continues to build skyscrapers, with each project having a set of stakeholders the size of a small city and all while adjusting to a shifting landscape of nearly infinite variables. Especially the first few moon missions, when no one had done it before. It's why all of the original astronauts were test pilots - not just because of their bravado but also their experience in co-designing unfinished machines along with the engineers to bring it to a workable finished state. The roots of the modern HCI and human factors industries comes from analyzing and optimizing the usability of cockpits. The astronauts even had their own usability jokes about a model of training jet that was so user-unfriendly that the emergency instructions for ejection were printed on the side of the canopy, which is nuts because step one was to blow the canopy (along with the rest of the instructions) up off into the sky.
These weren't dumb monkeys strapped to a giant tin can and blown into space, and neither were the engineers that sent them there. With waterfall.
So you use agile/xp/scrum/whatever and swear software is way different than building bridges. I've worked on and with several of those teams, with varying levels of strictness, and guess what? It's still basically waterfall. But instead of PMs owning the process and spec'ing things out in advance, engineers own the process and the spec is written and re-written during development on a bunch of index cards with sharpies like a grade school art project (or the digital equivalent), designers get barely a week ahead of each sprint or run around like caffeinated janitors cleaning up after each sprint, and documentation gets written last or not at all.
And before anyone does ad hominem attacks on whether my teams were doing agile correctly, this is the same dynamic I witnessed at the most famous agile and rails development shop you can think of. You probably only need one guess to figure out who it is.
What matters then, if it's not process? It's called expertise - a broad collection of various techniques and the context-specific awareness of when to use each one. Instead of blindly following a process and trying to hammer it into every situation, take a tools-instead-of-rules approach to using whatever techniques are appropriate for each project.
For anyone interested in this kind of flexible thinking instead of linear GTD productivity tools that leave you stuck in robot mode, there's an excellent book called Pragmatic Thinking And Learning that goes into a lot more detail and pulls inspiration from how the nursing industry overhauled their education system in the 70's.
I'm totally not getting why you're saying agile approaches are waterfall in disguise.
The essential difference to me is in length of feedback loops. Microsoft releases a new OS every few years; some place like Etsy releases new code many times a day. A developer or product manager at Etsy can know within a few hours whether some change is really a good idea; Microsoft not so much.
You seem to be saying that because there are the same sorts of people involved, and because one has a giant spec document while another has a stack of index cards that might be called a spec, then they're the same. Which isn't making any sense to me.
Nope, it has nothing to do with the commonalities in people or roles, sorry I wasn't clear. Let me try again.
It's a knock on all rigid processes and dogma, not agile in particular. I trimmed the rant down by taking out the sections that rip into UX firms, with their Big Upfront Design and cocky rockstar designer personas, but it could just as easily been about them. I only cut it because I thought the parallels were obvious with the diagram on the Epic Consulting website.
What I'm saying is that if waterfall means people are grouped together by specialty - programmers with programmers, designers with designers - each group is run by one dominant process suited to only one of the specializations, and each group tends to work in different stages, then both traditional UX design and agile development are run as de facto waterfall teams, whether by explicit hierarchies or simple stakeholder buy-in.
Consider this - what's the difference between:
1) a PM spec'ing things out in advance and basically telling everybody what to do and in which order.
2) an agile shop coding things up on the fly and telling the designers to run after them to clean up the visuals without making major changes, so they might as well be building to a spec.
3) a UX firm mapping out how the product works before handing off to peon engineers that can't make major changes and might as well be building to a spec.
To end on a positive note, there is one other solution to expertise, by the way. It's grouping teams by product, not role, and figuring out an interdisciplinary way to work together, rather than running the entire team by the methodology of one specialty. That way you're picking good ideas from across different processes and adapting to the company culture to boot. For example, TypeKit does daily stand ups at 10:05am every day and that's an aspect of agile that they've liked enough to use for the whole company, but I'd be surprised if their designers were checking their art assets into Pivotal Tracker or the programmers were writing up UX journey maps of each user persona.
Steve Jobs always liked to describe Apple as the biggest startup he knew and a more accurate description would be that it's a collection of startups, with each one focused on a single product. Seems to work pretty well for them.
This is the paper that was the basis of Scrum: The New New Product Development Game, Harvard Business Review, 1986. [1].
The crux of the paper is that you should be "grouping teams by product, not role, and figuring out an interdisciplinary way to work together, rather than running the entire team by the methodology of one specialty".
If the word "Agile" freaks you out, then by all means don't call it that. And certainly there are a lot of bullshit artists plying their trade out there. Indeed, if someone tells me they are "Agile", by default I don't believe them. But as this 1986 paper shows, there is a lot of good data and good theory out there if you but look.
I agree that "agile" teams that group people by specialty are clueless. Done right, the teams are, as you suggest, cross-functional. That was the original intent for the people who coined the term Agile, but much of that has fallen by the wayside as larger organizations decided Agile was fashionable and consultants rushed in to sell them 3% dilutions of XP or Scrum.
That's a great writeup! I know exactly how you feel except my disappointment was doubled, because I was a hybrid developer/designer for years before completing the switch from developer to designer and I lost the faith in both agile and UX at about the same time.
I wish I had the courage to write up my disappointment like you did, but I'm happy that I've moved on to a meta-process of borrowing from different techniques as needed.
Agilistas like to pretend that if you're doing waterfall you can't ever go back to the previous step.
Whereas in practice, if you find something wrong or confusing with the specification, you just hunt down the business analysts and ask them what drugs they were on when they pooped out the spec. And then after walking them through their mistake, you get a new and better spec.
If you do it often enough, they will start to catch on and actually do their job better. Don't think of that as a poke at BA's either, everybody puts more effort in if they know that someone gives a crap about what they do. They may not like it at first, but you're actually empowering them.
"Agilistas like to pretend that if you're doing waterfall you can't ever go back to the previous step."
The Agilistas are correct.
The reason for this is that Waterfall was constructed deliberately as a strawman. The very point of constructing Waterfall was to draw a distinction between the process as it was being presented to the VP, or in this case as it is presented to the customer, and the fact that in practice it doesn't work at all and therefore it could not have been what actually happened. Therefore, given that it isn't a possible process anyhow, we should embrace an iterative approach, because ultimately there's no alternative. We should face up to the truth and thereby learn to take proper advantage of it, instead of sneaking it in the back door, hiding it in shame, and deliberately doing stupid things that can't work because management thinks it makes them happy. And in this case, there's no possible way that that is an accurate description of what happens in any significant project that company undertakes.
If you don't understand that Waterfall is not and never was a "real" methodology and don't grasp the history, which it seems still almost nobody does, then you don't really get what's going on. And attempting to rehabilitate Waterfall into a real or viable methodology is metacontrarianism [1] run amok. There's no point in performing CPR on a scarecrow.
I thought waterfall was a strawman too, until I read The Checklist Manifesto and started learning more about other industries.
For example, large-scale construction projects like skyscrapers are run by gantt charts and committees, which sounds unbelievable and like a disaster waiting to happen. It's like the opposite of every startup I've worked at in the last 4 years. Except it actually works and they keep the everything running smoothly through:
* (at the top level) real time adjustments to huge room-filling charts that shown all the dependencies.
* (at the grunt level in the trenches) an abundant use of checklists to keep work mistake-free and to avoid groups stepping on each other.
* (at all levels) constant and real time communication. Lots of it.
So yeah, I think it's hard to write off waterfall as both ineffective and non-existent if a large industry uses it to such a degree.
Large-scale construction projects are also notorious for delays and cost overruns, so I wouldn't make too much of that.
Even granting your point, though, that doesn't tell us much about software. Nobody expects to start a building as a small house and work it up to something the size of the Pentagon. You can't automatically replace every screw in a building in minutes. Buildings aren't made out of words, and they don't have requirements that change continuously. Buildings aren't expected to respond quickly to competitor's new amenities. The construction of buildings is not 100% automatable with modern technology. So I wouldn't be too hasty in suggesting that the Waterfall approach is made more plausible by a similar-sounding process being used in an entirely different industry.
And honestly, real buildings aren't even much like what you're imagining. Stewart Brand's book "How Buildings Learn" examines actual buildings and how they adapt and change in ways very different than what people with GANTT charts think should happen.
Waterfall has no such history in the software domain. Analogizing software to construction has a long and worthless history, and it has failed to produce one actionable insight that I'm aware of. (It has produced some really dangerous pronouncements about how software engineering should be done, of the kind that you can crash and wreck a company on, though, so there's that.)
The problem is that construction has a completely, radically different cost model than software. If they could possible throw up one building today, tweak it, and throw up another tomorrow, they'd go iterative too. Instead, they have horrifying implementation costs that have no analogy to software; if Waterfall "works" for them it is only because they have no other choices. That's hardly a ringing endorsement, as wpietri observes.
Interesting outlook on things... I've always thought (and argued when possible) that the key to any software/computer issue is smart people - not 'rockstars' and these developers who think they're the center of the universe, but a team of capable people, familiar with a broad range of technology and capable of solving problems, using whatever tools are needed.
But I have long thought that waterfall just couldn't work - but I suppose it could if you have management and executives that understand the 'product' at the end will be iterated.
I agree with you that any kind of process sucks when a team doesn't really know what they are doing.
And by using a methodology you are saying just that "we have no fucking clue, so we will try it this way that person X sol us".
I have debated agile proponents about whether agile would work in their respective organizations on multiple occasions. Usually I argue it would fail just as much as any other approach?
Why? Because people try to adhere to it strictly and completely miss or ignore their current context, which is usually THE problem they are trying to solve in the first place.
And you are dead wrong about waterfall. Waterfall is not even a methodology. Anyone selling himself as an "methodology expert" while trying to sell you on "waterfall methodology" is a quack and a fraud. Anyone talking about waterfall as a possible methodology is a quack and a fraud who knows nothing about methodologies. Because waterfall is not a methodology, just like adding rat poison to kids candy is not a business model. Waterfall has always, since the very beginning of the term, been an anti pattern a "anti-methodology".
Let's just ignore for a second that a waterfall process sent men to the moon and continues to build skyscrapers, with each project having a set of stakeholders the size of a small city and all while adjusting to a shifting landscape of nearly infinite variables. Especially the first few moon missions, when no one had done it before. It's why all of the original astronauts were test pilots - not just because of their bravado but also their experience in co-designing unfinished machines along with the engineers to bring it to a workable finished state. The roots of the modern HCI and human factors industries comes from analyzing and optimizing the usability of cockpits. The astronauts even had their own usability jokes about a model of training jet that was so user-unfriendly that the emergency instructions for ejection were printed on the side of the canopy, which is nuts because step one was to blow the canopy (along with the rest of the instructions) up off into the sky.
These weren't dumb monkeys strapped to a giant tin can and blown into space, and neither were the engineers that sent them there. With waterfall.
So you use agile/xp/scrum/whatever and swear software is way different than building bridges. I've worked on and with several of those teams, with varying levels of strictness, and guess what? It's still basically waterfall. But instead of PMs owning the process and spec'ing things out in advance, engineers own the process and the spec is written and re-written during development on a bunch of index cards with sharpies like a grade school art project (or the digital equivalent), designers get barely a week ahead of each sprint or run around like caffeinated janitors cleaning up after each sprint, and documentation gets written last or not at all.
And before anyone does ad hominem attacks on whether my teams were doing agile correctly, this is the same dynamic I witnessed at the most famous agile and rails development shop you can think of. You probably only need one guess to figure out who it is.
What matters then, if it's not process? It's called expertise - a broad collection of various techniques and the context-specific awareness of when to use each one. Instead of blindly following a process and trying to hammer it into every situation, take a tools-instead-of-rules approach to using whatever techniques are appropriate for each project.
For anyone interested in this kind of flexible thinking instead of linear GTD productivity tools that leave you stuck in robot mode, there's an excellent book called Pragmatic Thinking And Learning that goes into a lot more detail and pulls inspiration from how the nursing industry overhauled their education system in the 70's.
http://pragprog.com/book/ahptl/pragmatic-thinking-and-learni...
I know, from the book publisher that sells the pickaxe Rails book and stuff on agile. Oh the irony!