If only management understood this. Actually, I think good managers do understand this, and do it anyway, because if you take the general case and look at it from above, methodologies do improve upon chaos.
As a manager, the trick is to find the sweet spot between chaos and blind adherence to rules. The uncomfortable truth is that there isn't one sweet spot for all cases.
I've run teams where a very agile approach made sense (usually where the dev and users where small in number and very close) and others where a more formal phased approach made sense (usually where we need co-ordination across companies).
The simple fact is that trying to run these types of programme the same way is an exercise in futility. That's not to say that one can't extract common practices that make projects generally better e.g. it's generally preferable to get code into the wild sooner rather than later if you can do it safely, it's just that the "one true path" idea is a marketing concept not an engineering one.
> Everything I've seen seems so "In my experience..." as opposed to a formal proof.
The scientific method is ill-equipped to formally prove a generalized theory of software development. There are too many significant variables at play; too many equations to solve. We are left with little but "In my experience..." to guide us (as well as local experimentation, where science can actually be of some help).
Yup. TDD, object orientation, agile, functional programming and whatever else are all good ideas. In some cases they work, in others not.
But it seems people have a tendency to make them into ideologies/religions that when applied correctly will solve everything. I guess it's a way to exercise power over people. No individual thought allowed.
At best they can give a person with good judgement a different angle to look at their problems with.
At worst they are thought-stopping slogans that turn a gullible novice into a crippled novice.