In the startup world and in much of the tech corporate world in practice predictability is valued above all else. It's because investors, perhaps reasonably, value and therefore demand predictions.
But as we know that is fundamentally difficult. So many tech orgs make up a "hero-hustle" culture to compensate. "We're top 10% and we work all the time, this is the best we can do!"
Strong leadership is strong leadership, building the right product development culture is really hard.
The hero hustle hides all the bad stuff which will then never get fixed.
Predictability is a fantasy. It's predicting the future. No one can do it. When you hear someone say they've figured out a way to get predictability they are lying, in much the same way people who say they've come up with a new guaranteed way to make money off the stock market are lying.
Predictability depends primarily on the time scale, plus a few other factors. I have seen existence proofs of mature, disciplined organizations using stable technology consistently hitting >90% of commitments over periods of up to 3 months. Over longer periods or when more research is needed then predictability will necessarily be lower.
While perfect predictability is impossible, many organizations have lower predictability than they could. This usually comes down to lack of discipline, which is a fixable problem (much easier than beating the stock market). Insisting on high levels of discipline eliminates the need for "hero hustle".
Agile is a tainted term. For it's proponents, it's become nothing more than a vacuous manifestation of the No True Scottsman fallacy. Anything that works is True Agile, and anything that doesn't is Not True Agile (or would be True Agile if you weren't holding it wrong). The principles that underlie Agile are either entirely self-evident and can't be claimed as a particular benefit of Agile per-se, or they are hopelessly naive about the way people, and companies, actually work. Agile itself has no real original ideas, and offers nothing of value.
At this point, the best thing we can do is let Agile, all of it, die.
Core to the 12 Principles, and thus Agile by extension, is the idea of no managers. Each of the 12 items exists to get you thinking about what developers (and the business people!) need to do if there is no manager acting as the guiding force.
While no managers is not a novel idea in a vacuum, it isn't something organizations usually think about. It's just the unspoken expectation that there will be managers once you have enough people to form a team.
> hopelessly naive about the way people, and companies, actually work.
Indeed. One of the 12 principles is quite explicit that it will not work with any Random Joe; that you need very specific people who are motivated to make an organization function without management helping them. Much as you want to pretend you are not Random Joe, you are, and it literally tells you that it won't work for you.
The original agile manifesto is silent on the topic of management. Self-organizing teams are a good practice, but once you scale up and have to coordinate the work of many teams across a huge portfolio it generally becomes necessary to have some level of hierarchical management in order to prevent chaos and dysfunction.
I don't see how that "no managers" core can be true.
Many of the principles, like "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale." are valid even for solo developer projects. Agile came out of older practices, like Rapid Development, which definitely could be used by solo developers.
Several of the principles do imply the existence of multiple people - #4 implies business people are distinct from developers - but still apply to a two-person company, like married friends I knew who ran their own software company where the wife handled the business and the husband the development and there was no manager.
If there are no managers, then either everything they do is Agile (because there are no managers) or nothing they do is Agile (because "Agile" can't be applied). If the former, then a solo developer following waterfall is Agile. If the latter then what alternative principles can be applied to solo- and partnership developer projects?
Every term has that property. If there's a thing called T you can say that the good Ts are "real Ts" and the bad Ts are "not real Ts". It means nothing to dismiss an idea because of this. In the No True Scotsman fallacy, the fallacy is not the definition of the word "Scotsman", it's the use of arguments of this form.
AFAICT the "not true scotsman" is motte-and-bailey applied to the definition of a word - the fallacy isn't the original definition of the word "scotsman", the fallacy is re-defining the word to mean something different and more defensible when you're challenged, and then to redefine it back to the original definition later, to re-include all the insufficiently-burly scots.
It's easy to criticize. What specifically are you proposing as an alternative that will allow software organizations to reliably deliver value to customers?
Criticize all you like. But if you want to actually effect positive change then you'll have a lot more success with proposing specific improvements. Otherwise your co-workers will tend to perceive you as a whiner.
There is no single easy answer, and I'm not a management consultant so I have no interested in trying to sell one. You simply have to do the hard work of understanding the constraints you are operating under and exercising good judgement on how to work in that environment.
And, now I eagerly await the dozens of replies I'm going to get telling me that exercising good judgement and common sense could only mean that I am in fact following the True Agile (unless it doesn't work, then it was obviously No True Agile process).
Clients and other stakeholders often need status updates and estimates to conduct their business. It may even be worth sacrificing some speed and value in order to have better insight into when it is likely to be delivered, so plans and coordination can happen accordingly. This is still hard to do, and perhaps the agile-branded methodologies aren’t the best way to achieve that objective, but that doesn’t mean that predictability is meaningless as a goal and shouldn’t be attempted. ”Hire competent people and when they’re done, they’re done” doesn’t necessarily produce the best outcome for an organization.
That's the thing, though. Agile tries to accurately measure the progress as well as predict the time cost of the immediate future. It doesn't propose to predict the full time cost of an entire project.
Lots of methodologies do promise that, but they all seem to end up even worse. I know I don't want to go back to the waterfall days of requirements gathering for weeks, then being held to a schedule that was built entirely on hypotheticals.
What works is to have a competent team that cares and aren't hindered by bureaucracy. Agile is one such obstacle that prevents people from getting things done.
Agile might be a good way to deliver value if you have sub par teams, I'm not sure, but I'm pretty sure it hurts competent teams since now you made it their job to deliver reports rather than improving the product.
> I know I don't want to go back to the waterfall days of requirements gathering for weeks, then being held to a schedule that was built entirely on hypotheticals.
Removing constant scheduled reports and meetings all the time does not necessarily mean waterfall where you frontload all those meetings and reports. You could just have meetings and reports that fits the project as needed instead of forcing a one size fits all solution to everything.
Agile and waterfall are both examples of the same corporate inflexibility.
Okay but that's not really actionable. There are only a small number of highly competent people available in the labor market at any price. Occasionally you can get lucky and hire enough of them to assemble an entire team, and the results can be extraordinary (at least until the ego conflicts destroy it). But since no large organization can rely on that strategy they have to find ways to deliver value using average people who might be incompetent in certain areas. That usually means some kind of agile methodology to provide a level of predictability and prevent things from going totally off the rails.
I appreciate the spirit this is written in. Having fought the good fight, here’s what I’ve learned:
Leadership in most organizations have ideas about product but no idea what the actual product is that runs things.
Anything that a client or stakeholder wants is an automatic “P1.”
The senior dev team inevitably concludes that tech debt, which has been accumulated for years, has to be address due to near catastrophic levels of issues in security, compliance, performance, or maintenance.
Business analysts & PMs, almost always bright, well-intentioned people that don’t realize they are in Kafka-topia, believe that all that is missing is a clear business justification for prioritization & with calculated value to the end user. Those that go hunting for answers are soon shocked to discover their core users have no idea why they do what they do or what relevance it has to the org. All of the above drives additional, well-research product enhancements that get greenlit by leadership to be done “alongside the client P1s.”
If your pipeline is clogged, shipping value can’t happen. However little time it takes, it still takes time to change the way you work - which is what agile is about at its best: continual learning and improvement.
I learned a valuable lesson in game design - your constraints are your rules engine.
As much as the concepts and beliefs at the core of agile are an amazing “dev outward” approach to value creation, agile does not and cannot stand alone in the global dysfunction of human organizations.
Without clear leadership on priorities, without clear budgeting of investment in infrastructure, and most importantly without an actual feedback loop across all the players in a system, agile is not agile.
Agile is a way to get there, yes, but if you are in an organization that can’t answer basic questions about itself then you are probably in an organization that won’t be able to implement anything authentically. You will just have new shapes and sizes on the same lack of alignment, decision making, and insight.
Practically speaking, Corporate Agile is Agile in that it is what 95%+ of people doing Agile are actually experiencing.
As far as I can tell, the only companies doing "manifesto" Agile are startups, mostly out of necessity or out of a preference for agility (which is necessary for a startup) over predictability (which is impossible for a startup). Large companies mostly fail to do Agile not because they are incapable (well, maybe some), but because they don't want to - they value predictability over agility.
If you're about to reply that your Product/Engineering team is totally doing "real" Agile inside a traditional corporate operation, save your energy lol.
Now here is my hot take: Any organization that can reasonably hope to achieve predictability will strongly prefer it to agility and always choose it. This is why Agile never works in established corporations. No matter how much they talk a good game about "disruption", the reality is that they are just hoping to get predictable results.
Even a startup cannot do manifesto-agile most of the time IMO, at least not if they have a strong desire to succeed, because they are constrained by the possibility of running out of money in the near-future.
I am dabbling with bootstrapping now that I've hit my financial-independence targets and I think this, along with profitable+small+private+engineering-led software businesses and FOSS, are some of the only environments in which you can engage in true Agile. That assertion stems from the fact that they are not subject to real constraints in budget and resilient to business meddling in things like TTM and MVP.
Is there anything you can't substitute into "Even a startup cannot do X most of the time IMO, at least not if they have a strong desire to succeed, because they are constrained by the possibility of running out of money in the near-future." ?
Not exactly sure what you mean, but most startups operate at losses for a long time before becoming profitable. I explicitly noted that bootstrapped startups with passive income and profitable small engineering-led companies are not constrained by this problem, because their financials support operating indefinitely - but perhaps I misunderstand your point
I've seen a lot of agile shops and in my experience most of them have the right idea and relatively few are really abusing it. The friction with any project, agile or not, almost always boils down to estimation. "When will my product be done?" That's a really hard notion to disabuse regardless of process. No methodology has successfully solved for predicting the future.
of course not. but if you don't follow corporate agile, then you're allowed to make estimates beyond a single sprint. are these prophetic? only in the sense of being self fulfilling. we can do is have a rough map of the work and eyeball our progress given the team we have, and try to line that up with the endpoint by working backwards.
this should give us some general feeling about risk. we have lots of ways to repond to perceived risk of failure -
- pull in a more experienced person to get a read and possibly help with that part
- shuffle the order of development to try to swallow some of the uglier pieces early
- talk with product (possibly yourselves wearing different hats), about whether we can meaningfully throw some work off the bus without hollowing out the release
- thin out some features
and actually probably several more. what corporate agile says is 'this is all too hard, screw that, lets not try to account for individual strengths, and the nature of the development process, and the sensitivity of the markets towards particular features. everybody pick something to work on for the next two weeks...and if that didn't work. well, we tried our best'
You’re 100% right. In every case where I’ve been a part of agile, you have someone at the top that’s wanting predictability. If I told them to pick between 5 guaranteed features and 10 features but they don’t have control over the timing, they’d pick the former every time (well they’d try to pick 10 guaranteed, but ultimately they’d rather have predictability than velocity).
I feel like there are metric lurking in here somewhere. Something to do with the following concepts:
(1) the number of documented atomic changes to the codebase, be these merges or pull-requests or changesets or patches — the thing with a title and description that was code reviewed;
(2) the number of tasks in your task management system that might be bugs or feature requests — these are large and must be implemented in pieces, with testing and reviewing of assumptions along the way; and
(3) the time period on which you are required to be held accountable for actually finishing stuff.
I do maybe 30 commits for a big task and will spend six weeks on it. There might be small bugs along the way that get closed out with a 1:1 mapping between task and commit. The majority of the work is a giant slowly but surely moving set of iterations towards a top level goal. The cadence is to review on a quarterly basis.
Not so in the past though. I’ve had to do one commit per task. Multiple tasks in a big bushy tree of verbiage that overlaps too much with the actual commits (the meaningful bits!) and all done on two week cycles. It felt exhausting and piecemeal, as if software engineers were a fungible resource there to further marketing goals instead of innovate the next big things. Nowadays I demand to work for a true technology company, not a marketing company that sells tech.
It's nice to have your labor in high enough demand that you can pick and choose which types of companies to work for. Most developers worldwide don't have that luxury.
Large corporations are basically run like planned economies in communist states. 5 year plans, leadership detached from reality on the ground, tons of internal propaganda, lots of pretending that things are great, empire building by the mid levels and so on. There may be pockets of "true" agile here and there but I believe a large corporation simply can't be run in "true" agile manner. They are way too controlling to really empower their workers.
The tension is fundamental and comes from a perfectly reasonable desire to know how much things will cost, how long things will take and what they will get at the end.
Adequately communicating, negotiating and managing the uncertainty is the core part of agile, "do what you want" isn't going to fly, that's why teams don't have complete agency on process.
I agree the politics and "fog of war" in large companies is tiresome and wasteful but it seems to be almost unavoidable once you get to a certain size.
Good take. Just want to add that the problem is “how long things will take” is always a guess when formulated as a single number or date and in reality something more like a probability distribution subject to many foreseen and unforeseen forces.
That’s where the fundamental mismatch between corporate planning and Agile lies. Agile is supposed to be resilient to known and unknown unknowns, but corporate planning usually isn’t - as you mention due to very real constraints like cost and opportunity costs and so forth. It’s why you essentially can’t do “true-agile” in basically any corporate setting IMO.
Back when I was solo consulting and not inside the sausage factory I would provide cost ranges to clients who wanted fixed price (vs hourly). If they wanted a firm price they got the +3-sigma estimate. After doing that for a few years and successfully coming in within budget but not always on the timeline people were expecting I split it into separate estimate ranges for both cost and schedule. It took some time to really nail down how many billable hours there would actually be in a week, which is why cost and calendar had to be decoupled.
I do still laugh about the fact that I could take a complex problem and break it down with hour estimates well enough to not end up being significantly over or under at the project level but then struggled really hard at figuring out how many hours there actually were in a day to put the calendar together. There’s always things that pop up that rob calendar time and virtually never things that free up calendar time.
> The tension is fundamental and comes from a perfectly reasonable desire to know how much things will cost, how long things will take and what they will get at the end.
Then they should buy their software off the shelf and resist the desire to customise it. For most enterprises building their own software makes about as much sense as building their own vehicle fleet.
Your argument seems to be that these "fiefdoms" aren't providing you anything and just taking a portion of your generated value. The obvious follow up is then: why don't you quit and go solo then? Clearly if they're entirely parasitic leaving is strictly beneficial to you.
I used the word "symbiotic" and not "parasitic", in a symbiosis there are mutual benefits: while labour is exploited and I only get a portion of the generated value I also get the benefit of not risking myself going solo, nor having all the stresses of having to cover sales, marketing, product development, and so on.
It is still the same: the value extracted is only possible to exist because of many like me, doing labour to create added value which can be sold, without this there's no management class, there's no administrative tasks to perform around the work.
My comment is a retort to this mindset:
> and they're also the people who pay you.
They are only able to pay me because of the work done by many like me, their power can only exist through this conduit so bending over to view the ones in control of capital as "above me" being the "who pay you" is absurd in my worldview. They have their reason to exist but need to respect that without us they are nothing, absolutely nothing. Just like me without people taking care of a business would be in a less comfortable place.
Again, it's symbiotic and I'm just fighting the notion that someone should shut up or think of themselves as lesser because they are being paid by someone, that's only possible through a mutual relationship.
There are about a zillion other ways to avoid starvation. Start a company and run things your own way. Become an independent consultant. Buy a farm. If you don't want to be a corporate employee then you have options. This may require stepping out of your comfort zone and taking risks.
This is exactly why the conflation of "capitalism" with "markets" has been such an effective propaganda coup for the neoliberal rentiers: while there is often non-zero-sum value creation and growth in the short term, it's usually with an eye towards long-term consolidation of social power and rent extraction (converting flows into capitalized assets, representing expected returns and embedded growth obligations).
Monopoly, oligopoly, and market power are a feature, not a bug, and that includes reducing labor to a commoditized asset on the spreadsheet like any other: not only as cheap as possible, but as predictable, and as fungible as possible. (The extent to which the social power of these fiefdoms is not always a means to a "Number Go Up" end, but instead a primate drive for status and power merely enabled by rent-seeking, is left as an exercise for the reader.)
See "Markets Not Capitalism" and "Capital As Power".
It's what our political system produces, and as we all know it's "the best", and thus defended accordingly.
Mark Fisher famously said "It's easier to imagine the end of the world than the end of capitalism", well the same seems to be even more true for "democracy", the causal force that Shall Not Be Discussed Critically.
We seem to refuse to admit that companies are a microcosm of the broader cultural/political environment and politics as much as we try to avoid it, plays a big part in how we work.
There are clear conservative/progressive ideologies: Progressives are always wanting to try the shiny new thing, while the conservatives want to stick with what works, and disagreements over the languages, tools and processes we use have similar religious fervor and animosity as you see in the broader culture wars.
An individual can sometimes strike out on their own and succeed, and that's all to the good. But it simply does not scale across every sector of the economy, let alone being a viable strategy for most hand-to-mouth laborers (IIRC, something like 40% of US workers have less than $1000 in savings).
Every shareholder and investor wants a competitive moat: for corporate firms to have enough market power (including lobbying power and regulatory capture) to keep labor costs low (with an intentionally managed reserve pool of the unemployed, via NAIRU, to "discipline labor").
The fact that some individuals can beat the odds, does not change the systems dynamic: "the table is tilted, the game is rigged".
USA has 33.2 million companies, most of those aren't giant conglomerates, so the system do support people to work outside of companies, it isn't as hard to do as some people think.
The main reason people don't go and work for themselves is that companies pay so much that usually it isn't worth it. They would make money on their own but they make more working for a company, that isn't really a bad thing though.
Yes, that is life without a corporation that takes the risks for you. Wasn't that what you wanted?
You can't have your cake and eat it, you either get the risk or you take orders from someone who took on that risk for you and give you a stable income.
Fascism is basically Communism (as practiced); corporations own the means of production rather than the government, and it uses religion/nationalism rather than humanism/atheism.
They are the same in kind, but different in degree. You could even say China under Deng Xiaoping and later Xi blended the two.
Agile is many ways does smack of Communism:
- It proposes egalitarianism (skin in the game), but generally leads to micromanagement/totalitarian situations.
- If it fails it's because you didn't do the "real" Agile.
- If adherents often speak/act in cultist/religious ways.
I'd posit that this is not necessarily a feature of large corporations; I've been in smaller places (startups, to boot) where I observed almost all of what you've described.
It takes a certain combination of sociopathic, psychopathic, and apathetic individuals in the right places to make this happen -- and they're almost always present in the right (wrong?) places, regardless of size.
Does whatever you propose to replace corporate Agile give upper management fine-grained observability and control of the SDLC? No? Then it won't get adopted, especially not by large organizations. Period. It's not the 80s or 90s anymore, programmers don't get to build little fiefdoms of accountability strictly on their own terms anymore. The C suite knows that those fiefdoms are Dennis Nedry situations just waiting to happen. Software development is accountable to the business. The business has a right to track how much value developers are delivering, and to change things if it's not enough.
As a certified project manager myself, huge portion of all these practices are just pure none sense and sometimes are used only to feed the managers ego and burn the employees more while producing nothing but a garbage product. They see it done in an XY big corp and they start mimicking it blindly believing that in few years they will become mulit-billion dollars company, and I always say it, just because some approach worked in an environment with these people and that product, it does NOT mean it will work for you, it’s a tool, it fits that work it doesn’t mean it will fit yours, but holy crap how ignorant some people are when they just apply work without thinking, it is far dangerous than not knowing, the former one will think they know it all!
I recently left a long-established, mid-sized company that has been stuck in a miserable mire of Scrum that stands as an ironic antithesis to the manifesto the countless (and exorbitantly expensive) agile evangelists we’ve had slink through our doors exhort yet fail repeatedly to achieve.
The new company has no formal agile process, but effectively is kanban. I gotta say, it’s glorious. Leadership gives us a clear and actionable goal, we self organize the work, and we approach it in a way where we can iteratively build and release things every few days to maybe a week or two and get it in front of stakeholders for feedback, adjust as necessary, repeat until it’s done. If something urgent comes up, stakeholders are apprised, we handle it, and we return as soon as we can. We have a little bit of documentation to help with the bus factor and onboarding. No bullshit, no scrum master or coach, no planning poker, no fucking stand ups or parking lots or sprint planning or retrospectives or scrum-of-scrum or refinement or the dozens of other meetings that took 1/3 to 1/2 of my time at the old place.
We can afford to do this because we’re a small, focused company where everyone knows their role, we get the fuck out of each other’s way but we hold each other accountable, and the pulse of the org is built around this lifecycle. As the business gets larger, it will stress this nice and cozy arrangement and I imagine we may have to add some light process to keep it from spilling apart, but I’m really hoping this ground up culture of getting things done can scale.
That’s the thing of it. scrum arose out of people trying to boil down and extract the magic of very successful teams so we can duplicate it everywhere, but in the process of trying to make it a machine you just install and turn on, it has turned a photocopy of a photocopy of a photocopy of a Frankenstein monster where various leadership members-who may unfortunately not be as smart or effective as the folks they emulate-pick the parts they like and push it down onto their reports, whether it works organically or not. Worse than that, it gets applied piecemeal across the organization rather than the organization being built around it. And of course they change it unilaterally and don’t coordinate across teams.
I think you either have it or you don’t. If you’re waterfall and can’t do anything but waterfall, just do waterfall and don’t make everyone miserable by wasting their time trying to pretend something you're not. If your org isn’t effective, sure, give it a shot, but it’s going to take an organization-wide, focused effort to make agile work. At any rate, I can’t understand or justify the amount of money and time we throw at snake oil salesmen and charlatans in the scrum/project management space.
I'm not sure you realize all that Agile gave you that you described: kanban, a direction and goal, self-organized, iterative builds, I assume version control, daily builds possible, in front of stakeholders, adjust, repeat, necessary documentation. Probably in DevOpsy environments built regularly.
Before and outside Agile we had months of requirements gathering, nominally with stakeholders, months of break-down and design (by a different set of individuals, barely seen by stakeholders), and then devs would march to the Gantt chart. We were lucky that "make" would be successful in our own component most days, integration test with other major components were on an ad-hoc basis. Version control? Backups, if lucky. Somewhere near the end of that Gantt chart we would start "testing" by hand-delivering hand-built components to the test systems. After testing we would build for delivery. What? That step wasn't on the plan? Oops. Hand-deliver to production. Oh, and don't ever mess around on those systems because no one knows how they were built or updated.
That most teams these days do pervasive version control, iterative build, regular delivery, and system builds is simply a massive transformation of the industry. The Lean Methodology folks were on the right track but they never delivered usable, actionable practices until the Agilists started documenting them.
But as we know that is fundamentally difficult. So many tech orgs make up a "hero-hustle" culture to compensate. "We're top 10% and we work all the time, this is the best we can do!"
Strong leadership is strong leadership, building the right product development culture is really hard.