Assuming facts not in evidence error at line 1: Presumption that Agile ever ascended.
The basic problem with the Agile movement is that it took a few proven helpful practices -- almost all of which were well known and often used by experienced software developers before the buzzwords were attached, even if managers didn't realise -- and instead of promoting those, it wrote a "manifesto" so vague that it would make a political speechwriter proud, which didn't actually say anything at all about the practical steps that were useful.
Then concrete examples of "Agile" processes started to appear, and a few less proven or even discredited practices were added to the basically good ideas. Suddenly, despite valuing people over processes or some such vacuous claim, it's all about The Process again.
The predictable result was something non-technical managers thought sounded good but didn't really understand, and technical people who didn't drink the Kool-Aid looked at sceptically while asking where the evidence was that it was better than what they had been doing already.
And the only answer the Agile community has collectively agreed on in the ten years since then seems to be that people aren't doing Agile properly in most cases, but look at the successes where smart people have made it work! Of course, those smart people probably could have run a successful project starting from any other process, too, because if it didn't help they would have thrown it out in favour of things that did, just as they always had. And if most of the case studies of your supposedly superior process don't show significant benefits, you start to lose credibility after a while by claiming that they're all doing it wrong.
And now, we're doing the same thing all over again with the not-at-all-pretentiously-titled Software Craftsmanship movement, and the result will be the same: lots of people selling books and charging consultancy fees to peddle snake oil mixed with just enough actually proven ideas to be credible for a while until the results come in, by which time they will have moved on to the new hotness.
I am not normally a fan of Zed Shaw's style, but in this case, I sympathise with his point of view:
That is approximately the opposite of how it happened.
Several different groups of people invented processes that worked for them (e.g., Scrum (1995), XP (1996), FDD (1997), DSDM (1994)). They all liked what they had done, but discovered they had commonalities that were worth talking about. They got together for a long weekend and came up with the Agile Manifesto (2001).
Having lived through the transition, I promise not all the good practices were "well known and often used". Frequent releases were generally considered dangerous, and weekly iterations were seen as heretical. Unit testing was done by a small number of projects, but mainly in safety-critical areas. Test-driven development was unheard of. Continuous integration was certainly rare, as was using refactoring (supported by high test coverage) as a way to manage change.
Personally, I think the Agile movement is irredeemably doomed at this point. It's basically a question of incentives:
However, the current sad mess happened in spite of the good intentions and major discoveries of the people behind the methods that got rolled up into Agile(TM).
You're not the only one who lived through the transition. And IME the good practices that are used in Agile processes mostly are the ones that were well-known and widely-used.
People have written automated tests for decades. People have been testing at the interface level since the dawn of OO and before. Put those together and you've got what is now called unit tests.
People have been making small changes to improve the design of code without changing its behaviour forever, too. Fowler published Refactoring in 1999, but even then it was common practice among experienced programmers to work that way. I assume you're not seriously going to suggest that no-one thought of doing things like moving a method up and down a class hierarchy or splitting a long function into a higher level control function using lower level functions for the details before the book was out. (I'm not disputing that Fowler did a good job of pulling a lot of common knowledge together in a way that is helpful to those just starting out.)
I also assume you're not seriously going to dispute that long-term projects have been managed using shorter-term milestones for a very long time. That's hardly unique to software development, never mind to Agile processes.
Every project I've ever worked on professionally has seen developers naturally pair up or form small teams around the same computer or whiteboard to work through difficult problems or review each other's solutions.
People have been doing what is called continuous integration today for as long as there have been source control systems with check-in hooks.
I first had the idea of organising a big project by keeping a pool of little projects and picking out the most important one to do next when I was about 13 or 14, so I rather suspect that plenty of managers were using the same idea to prioritise long before anyone called it a backlog and worked through it in sprints.
I could go on, but I think you get my point.
Now, there are plenty of other practices required by various Agile processes that weren't (in my experience) almost universal among good programmers before Agile came along. They include:
- TDD
- Full-time pair programming (same computer, two keyboards)
- Forcing short-term milestones into periods of exactly the same length
- Frequent meetings (daily in some processes)
- Relying on organic designs that evolve through arbitrary refactoring by individual developers rather than a carefully planned architecture and smaller-scale designs that are peer reviewed by multiple people
If you want to claim those as the gifts Agile has given us, you're welcome to them. As I said before, most of the good practices were already in use, and the consultants mixed in a bunch of unproven ones later.
You're again wrong on the history of the Agile movement, and I'm disappointed that you entirely ignore the meat of the point that you're replying to.
As to your assertion that everything good was in wide use before: that's not how I recall it. For example, CI: the Joel Test, written in 2000, suggests that the vast bulk of shops couldn't even manage to do a daily build, and he didn't even mention CI as an option. Also, it's not clear to me what a 13-year-old's imaginings proves about what a majority of development shops were doing in the mid 90s. My recollection is that waterfall was the majority, with most of the rest of the shops doing code-n-fix. And the first edition of McConnell's Rapid Development, which was a mid-90s book, either doesn't cover or treats as exotic most of the Agile notions.
But honestly, I don't know that it matters. I'm not sure why, but you seem eager to prove that there was nothing good about it based on your incorrect version of the history. The good news from your perspective is that what's sold as Agile has turned almost entirely into dismissable buzzwords, and as a fad I expect it will be extinct soon. There's nothing left to fight off.
But personally, I plan to keep using and evolving what I learned from it. If you want to see what a shop like that looks like (including all those practices you mention above), feel free to drop by in SF sometime. We like visitors.
There are millions of programmers in the world, and plenty of us have been doing it professionally since before Agile was all the rage. It sounds like you have too, but are you really arguing that just because you personally didn't encounter these ideas before Agile advocacy came along, no-one else did either? Or perpetuating the false dichotomy between Agile and Waterfall? I have never worked on a project that followed a strict Waterfall model without any kind of cycle or iteration in the process, not once in my whole career. Now get off my lawn!
Seriously, though, you've cited Spolsky and McConnell, but you seem to be appealing to their authority on the basis that if the techniques we're discussing had been in common usage, they would surely have been mentioned in those authors' writing in the late '90s. I would point out that the first edition of McConnell's Code Complete was published in 1993 and didn't say much about OOP, yet OOP was already in use by many professional developers, and books about languages supporting it had been published many years earlier: the first edition of The C++ Programming Language was published in 1985, for example. Similarly, the second edition of Code Complete was published in 2004, yet contains no index entry for the word "concurrent" or "parallel" (though there is a passing reference to multithreading on page 337). So perhaps it is unwise to infer too much from any absence of references in the kinds of source you mentioned.
I have never argued that "no-one" encountered those ideas, and your repeated exaggeration of my position is frustrating.
Nobody in the Agile movement claimed novelty for individual pieces. What was different was the way the pieces went together, and the shift in values and cultural norms that they required. That wasn't purely novel either; the fact that several people invented similar processes at the same time suggests that, as with most inventions, change was in the air. But what the Agile people undeniably did was to define and promote that difference in a way that gave people something to rally around. At the time, waterfall and a waterfall-ish RUP were the big processes people were discussing, and the notion that one would focus on people over process, or working code over documentation was heretical.
Also, I'm having a hard time taking your approach to evidence seriously. Rapid Development was a major survey of current software project management and a catalog of best practices put out by a large tech publisher. Splosky was also writing clearly about current practices. Your counter-evidence is the (unnamed) projects you've (anonymously) been on. Plus that one of them wrote a different book that you can pick nits on.
And honestly, I don't get why you're arguing the nits and ignoring my main points. You clearly have some big issue with your version of Agile and it's imagined history. But I honestly don't care what process you follow as long as it doesn't suck for you, so arguing it with me isn't really helping anybody.
I'm sorry you're feeling frustrated with this discussion. I'm afraid I just don't understand what your "main points" are supposed to be, if you think I'm only picking nits with the details.
I started this subthread with some fairly black-and-white statements about the relative timing of good practices being adopted, Agile taking off, and less proven practices entering the picture:
Since then, you have accused me, repeatedly, of being wrong about the history of Agile. Is your objection that early versions of a few Agile processes already existed, albeit in relative obscurity, before the Manifesto was written, but I talked about how 'concrete examples of "Agile" processes started to appear' after the Manifesto was written?
If so, I'll happily concede that point. I don't think it really matters whether you define Agile as starting from the evolution of the general approach/abstract philosophy (which had evidently started by the time of those initial processes) or the writing of the concrete Manifesto and popularisation of the Agile moniker. Nor do I think it really matters whether you define processes like XP and Scrum as appearing as "Agile" processes when they were popularised and given that label (around the time the Manifesto also became widely known) or from when the original processes that became what we now call XP and Scrum were first used anywhere. Any way you cut it, we're talking about the same groups of people, the same general period of time, and the same end results.
My main point is simply that many of the good practices that were adopted by those early Agile processes and are often considered to fall under the Agile umbrella today did not actually originate during that period of time or with the people who worked on those early processes and the Manifesto. However, many of the Agile practices I consider unproven did originate during that period of time and with those people.
While you may wish that no-one in the Agile community claims otherwise, unfortunately that is not always true. While I was researching my post with sources, I found a comment on Martin Fowler's personal web site claiming that Kent Beck coined the term "refactoring" in the mid-'90s, for example, and numerous comments from various posters on other forums suggesting that Fowler's book was the definitive original reference. I think it is important to debunk such claims and any attempt to paint Agile and Waterfall as some sort of dichotomy. That just isn't how software development works, and being "Not Waterfall" is not an argument for switching to any given Agile process.
> I found a comment on Martin Fowler's personal web site claiming that Kent Beck coined the term "refactoring" in the mid-'90s, for example
No, I didn't. Sorry. I had read something similar in several other places, and when I did a search (since Fowler's site itself was blacked out) the resulting excerpt was truncated in a critical place that made it look like the above was true. Still, there are plenty of people who have claimed that the Agile movement originated ideas like refactoring in on-line forums and the like, so aside from setting the record straight regarding Fowler's site, my original point stands.
I didn't have anything specific in mind; I'm going by my own experience, discussions with past colleagues, and conferences over the years more than anything. That said, it's not hard to turn up examples with a few minutes looking through my bookshelf and Google Scholar.
For automated testing, just stick search terms like "automated software testing" into Scholar and set the latest year to, say, 2000. You'll turn up dozens of papers going back decades that talk about techniques similar to what we would call unit testing today.
Continuous integration is another obvious one. The likes of Grady Booch were writing about the advantages of frequent integration builds by the early '90s. The term "continuous integration" appears in Object-Oriented Analysis and Design with Applications (1993), for example.
The prior existence of refactoring techniques is kind of obvious, as people had been discussing simple transformations like moving functions and data up and down a class hierarchy since the dawn of OOP. If you want a specific citation, Design Patterns (1995) mentions the specific term "refactoring" a few times, in turn citing academic papers on the subject cowritten by one of the authors as early as 1990, around a decade before Fowler's book and several years before Beck coined the term according to some Agile advocacy.
That's all I've got time for now, but I'm sure you could find eqally authoritative sources for the other things I talked about if you genuinely want to know.
I'm not claiming TDD as something widespread before Agile. I'm claiming TDD as something unproven that the Agile consultants started mixing in later. (Please see the list in my second post to this thread.)
For the third time: you are incorrect about the history. Test-first development was created by working programmers as part of Extreme Programming before the term "Agile" even existed.
I'm not necessarily disputing that, though I note that you have changed from TDD to merely test-first development, which I don't think is the same thing.
My point is that TDD, like several of the other practices in XP that do seem to have originated with those people during that time period, is to this day not proven as any better practice than the alternatives.
Organizing groups of people to put together something decent in a highly limited amount of time and with humane treatment is extremely hard. I don't see Programming, Motherfucker addressing that in any way.
I don't see much evidence of Agile methodologies addressing that in any way, either. That's rather the point.
After this long, if Agile is as superior to the alternatives as its advocates frequently claim, there should be heaps of empirical data miles high to justify each and every practice these consultants charge so much to explain by now. So where are they?
The best I've seen any big name Agile consultant cite is a handful of small-scale experiments, mostly using CS students rather than experienced industrial practitioners, invariably not controlling for using unit tests alone without the TDD process, that show that projects using TDD have modest but inconsistent and frankly unspectacular gains over projects that don't bother unit testing at all.
I'm a scientist. I believe in the data in front of me, and my conclusions on these matters are falsifiable. I will be happy to revise my opinions if and when new evidence shows any of them to be incorrect, but that evidence had better be more than a few alleged experts telling me what they think I should believe if I'm going to change the development practices of entire teams based on it.
It would be awesome if somebody would fund those experiments. But since Agile methods are mainly about sustaining high quality over the long term for teams of experts, studies with a useful n would be stunningly expensive.
And honestly, there's no great reason to believe Agile consultants. Most of the market for software process improvement is in helping the clueless and/or terminally lazy. (And there's often even better money in not helping them, but instead just making them feel special.) Plus, a lot of the benefit from particular practices depends on the existing code, the business conditions, and the people involved.
I think the right way to deal with Agile techniques is the same way one deals with new tools or new languages: try 'em for a while and see. E.g., I tried out TDD for a while and ended up liking it a lot. As does the rest of my team, such that they all were specifically looking for test-driven shops when I hired them. But even we tend to only use it on core code; test-driving most UI just doesn't pay off.
> But since Agile methods are mainly about sustaining high quality over the long term for teams of experts
> Most of the market for software process improvement is in helping the clueless and/or terminally lazy.
So are we trying to help teams of experts or the clueless and/or terminally lazy here? I don't know many clueless, terminally lazy experts, so it's got to be one or the other...
Also, experts will evolve a workable process for any given project without needing to follow any particular Agile process to start with. That's one of the reasons they're experts.
Also also, while it's convenient to pretend that no serious studies exist because of the cost, it should still be pretty clear that Agile processes have potential, because there should be a strong correlation between the time many organisations introduce them or which teams within a large organisation have adopted them and an improvement in measurable performance for the Agile groups. Correlation doesn't prove a causal link, of course, but lack of correlation is a strong indicator that no such link exists.
> So are we trying to help teams of experts or the clueless and/or terminally lazy here?
Me? I'm trying to help teams of smart, well-intentioned people. Which is why I'm happy to share what's worked for me and encourage people to try those things if they're interested. That's what motivates the early Agile people I know.
But consultants are mostly in business to make money, and the bulk of their audience is clueless, so most of what gets sold as Agile today is tuned for them. I go into more detail on that here: http://agilefocus.com/2011/02/21/agiles-second-chasm-and-how...
> Also, experts will evolve a workable process for any given project without needing to follow any particular Agile process to start with. That's one of the reasons they're experts.
True, but that ignores the people who aren't experts yet. They're the ones I care most about. And it ignores the substantial costs of learning all the important lessons yourself; it's often better not to reinvent the wheel.
> [..] teams within a large organisation [...]
Personally, despite 10 years of looking, I have never seen teams within a large organization successfully adopt what I would consider a serious Agile approach. I have seen a lot of large organizations adopt Agile jargon and get nothing out of it, though.
The problem is that Agile methods require a shift in power relations from top-down to bottom-up. Their approaches and values are basically antithetical to a lot of American business culture and MBA dogma. And successful exploitation requires deep change on the business side as well.
I think Agile failed for the same reason that Lean Manufacturing failed here: the important part wasn't the easily observed rituals, but the cultural differences. And the vast majority of the people who are "doing Agile" (whatever that means) would not accept deep cultural change.
At the risk of stating the obvious, has it occurred to you that the reason you have never seen a teams in large organisations adopt Agile successfully is that the basic philosophy of Agile doesn't scale to large projects?
You can get away with a lot of informality and doing things on-the-fly for small projects, when everyone involved can get around the same table in a meeting room and a single expert from the customer side can make all significant decisions. When your project team is made from 200 people divided among geographically and temporally diverse teams and the customer for your business management software is a 100,000 employee global corporation trying to co-ordinate the activities of six divisions in five key areas? Not so much.
Why yes, that has occurred to me. It's possible. But I doubt it for reasons I explained above. The few successful Lean Manufacturing adoptees in the US are on the smaller side as well. That's not because it doesn't scale, as Toyota is now the world's largest car manufacturer. It's because small companies can make cultural changes more easily. See the This American Life episode on NUMMI for a poignant illustration of that.
I think the answer to this question in software will come from companies that have started small with something like XP and grown into large companies. I suspect you'll end up with something like the cellular model that bodies use to scale. That's how it seems to be going at the companies I've visited. One startup I know is at 50 or so engineers and they are doing fine with a lot of small teams working semi-independently, releasing daily, and coordinating as they see fit. Or take YouTube. They have a few hundred people but they maintain a weekly (or more often) release schedule, power works in a bottom-up fashion, product management is decentralized, and the teams are loosely joined. It works fine for them.
Please provide studies to the effect that the practices you consider "proven" are, indeed, effective and references to the fact that they were widely adopted before the agile movement.
Because, so far, you've drawn a line based on personal opinions -- actually, just your own personal opinion -- about what's effective and what's not, what was common practice and what was not, said everything effective was common and everything not common is unproven, and then, based on this edifice or your own single opinion, declared that the agile movement didn't improve anything.
I'm very impressed with how much you deduced from your own thoughts, which means you are probably omniscient. In that case, you'll have no trouble providing actual evidence backing your statements.
The basic problem with the Agile movement is that it took a few
proven helpful practices -- almost all of which were well known and
often used by experienced software developers before the buzzwords
were attached, even if managers didn't realise -- and instead of
promoting those, it wrote a "manifesto" so vague that it would make
a political speechwriter proud, which didn't actually say anything at
all about the practical steps that were useful.
Avoiding creating a list of helpful practices was by design. The signatories of the Agile Manifesto were all very seasoned developers, and like all seasoned developers they knew that a practice that helps in one case harms in another.
You can kind of see how it might go wrong from there:
* Seasoned developer makes list of helpful practices,
without providing context, i.e. "Helps in this case,
harms in this case."
* Unseasoned developer doesn't understand when and where
practice is "helpful", tries to use it in all and every
case. Sometimes it works, sometimes it fails.
* Unseasoned developer who complains: "Helpful practice X
didn't work for me" is told "You're doing it wrong."
Again, no explanation of when and where the practice is
helpful is provided.
The takeaway here is to always, always, ALWAYS describe the context when describing a solution, i.e. "This is what worked for me, in this situation."
I don't notice any hesitation from those "seasoned developers" when it comes to advocating specific practices elsewhere. According to Bob Martin, if you don't do TDD you're not even professional, for example.
Yes that's what I was implying. (I guess I wasn't clear enough.) If Bob Martin said something like that, and he couldn't list even _one_ case where TDD wasn't the best approach, I'd question whether he had fully thought out the problem.
The basic problem with the Agile movement is that it took a few proven helpful practices -- almost all of which were well known and often used by experienced software developers before the buzzwords were attached, even if managers didn't realise -- and instead of promoting those, it wrote a "manifesto" so vague that it would make a political speechwriter proud, which didn't actually say anything at all about the practical steps that were useful.
Then concrete examples of "Agile" processes started to appear, and a few less proven or even discredited practices were added to the basically good ideas. Suddenly, despite valuing people over processes or some such vacuous claim, it's all about The Process again.
The predictable result was something non-technical managers thought sounded good but didn't really understand, and technical people who didn't drink the Kool-Aid looked at sceptically while asking where the evidence was that it was better than what they had been doing already.
And the only answer the Agile community has collectively agreed on in the ten years since then seems to be that people aren't doing Agile properly in most cases, but look at the successes where smart people have made it work! Of course, those smart people probably could have run a successful project starting from any other process, too, because if it didn't help they would have thrown it out in favour of things that did, just as they always had. And if most of the case studies of your supposedly superior process don't show significant benefits, you start to lose credibility after a while by claiming that they're all doing it wrong.
And now, we're doing the same thing all over again with the not-at-all-pretentiously-titled Software Craftsmanship movement, and the result will be the same: lots of people selling books and charging consultancy fees to peddle snake oil mixed with just enough actually proven ideas to be credible for a while until the results come in, by which time they will have moved on to the new hotness.
I am not normally a fan of Zed Shaw's style, but in this case, I sympathise with his point of view:
http://programming-motherfucker.com/