This means that we must have a standup, everyday. we must have a retro, we must have planning.
almost half a day a week is lost to pointless meetings. I just want to get on with my job. Trust me to use the ticketing system to figure out what I'm doing and raising blocking issues.
sprint: artificial crisis
end of sprint: abandonware
Scrum Master: CV credit for my planned escape to a new company
stakeholder: someone you can't get away with externalising your costs onto, because they can hammer a wooden spike through your heart
Project Manager: a job that no longer needs to exist in the astounding new world of Scrum! (Pay no attention to the project owners, product owners, story-writing users above you in the org chart or Scrum Masters behind the curtain.)
simplified: doesn't implement the actual business requirements
lightweight: doesn't implement the actual business requirements, but does so much more elegantly than the version that works
easy: project was born circling the drain (and doesn't implement the actual business requirements)
legacy: the version that works and implements the actual business requirements, though no sane human wants to touch it
user story: a valiant attempt to extract coherent requirements and bug reports; ends up being precise specifications for the wing feathers of the desired magical flying unicorn pony
velocity: a speed with a direction: skittering about following marketing's random hairpin turns
retrospective: blamestorm incoming!
stand-up: establishing blamestorm targets early
Inspect and Adapt: perhaps bong hits will fix my makefile
We'll put that on the backlog: ahaha fuck you
doing Agile wrong: noticing any of the above.
I do enjoy some of the jargon. empowered: do your own fucking job. "Could you just copy down these log files from these fifteen servers for me and put them on the shared drive? Thanks." "I'm sorry, I'm afraid you're empowered to do that."
Every good idea is turned into a bad one by the relentless management quest to Taylorise clue.
(originally posted by me at https://reddragdiva.dreamwidth.org/594955.html )
Most importantly, a product where that happens is missing a definition of done. Done includes, among other things, automated tests, good code quality, and a product owner looking at it and saying, "yes, this will actually meet the business requirements." If it doesn't get done within the 2 weeks, then it rolls over to the next two weeks. If not enough shit gets done in a 2 week sprint (and yes, I hate sprint as a concept), then you schedule less work.
That two weeks are efforts to work in small chunks (something good developers do anyway) and give a general sense of when the project will be done. That means that everyone is aware earlier that a project's scope or timeline is unrealistic. You're doing this rewrite because shit is slow to change; it's the developers' fucking job to point out that repeating the same problem again is a waste of money.
(Agile means pointing out unicorn fantasies as early as possible as to make the hard decisions before the project is 3 months late.)
Now, personally, I prefer continuous flow boards and cycle time to measure that. Then, you can skip the sprint bullshit, but that's a detail.
Finally, the problem you describe is a rewrite. If the problem is that nobody wants to touch that code with a ten foot pole, then you are "fucking empowered" to spend that same time to clean that shit up. Unless business requirements have changed and the software is no longer the right thing to do (see: implements the actual business requirements) then the right, sane thing to do is get a copy of Michael Feathers' Working With Legacy Code and dig in.
Using an agile methodology to do a dumb thing doesn't make it any more or less dumb.
Devs have to act like adult professionals though, and keep their issues updated, otherwise they deserve the long, boring meetings they so loathe :)
Every single one of those ritutuals has value.
Just because you're happy coding quietly, alone, in a darkened room with the door shut, doesn't mean everyone is.
I have yet to see an agile shop that doesn't benefit from stand-up to keep the team coordinated and on track. And retros are the forum for self-management and process refinement/iteration.
Moreover, some of those rituals (like sprint planning) are also needed by stakeholders other than the developers themselves in order to get their jobs done. Because, believe it or not, coders are not in fact the center of the universe.
almost half a day a week is lost to pointless meetings. I just want to get on with my job.
Woah, wait, wut? 90% of your time is apparently spent coding and that's a problem for you? You know, at some point this starts to sound a lot like an excessive sense of entitlement...
Every single one of those rituals has value.
For you/your company. why? because Agile is empiric. Every company is different.
The whole point of Agile, is to be nimble, and respond to evidence. I know what my customer wants, because I talk to them. Funnily enough I like my customers. I want to know how they are using my services. I like talking to people because I am human.
Because my stakeholders are different to my customers, I have to actively fight "Agile" to get my job done. I have to do the TCO calcs, I have to fight for my resources. The feedback loop is looping on the wrong part. But because the agile book says its ok, it must not be changed. But its ok "People not process".
I have once been in a company where agile was done "right". It was refined to the point where we had one meeting a week(effectively a retro).
Feedback was done through jira, which provides rich metrics. Collaboration was expected to happen as and when it was needed. If you couldn't figure it out, then you were in the wrong job. you needed to be able to talk human.
Why is that agile? because the process was mapped to the company, and tweaked until it produced measurable results. Not the other way round.
Everywhere else it feels like agile is a religion that must be observed, never questioned. Woe betide anyone that dares question the knowledge of the vicar as they mumble the softly spoken spells during stand up.
The reverse is also true. Just because a daily standup has value for some doesn't mean it has value for everyone.
My mind still boggles how nobody else questioned such a misuse of resources.
The entire team including PMs, design, business dev, etc. would be present everyday for that absolutely excruciating 1 hour long meeting.
If your standup has a dozen people and takes 45 minutes, that implies that either everyone's taking about four minutes to say what they should be able to cover in 2 or 3 sentences. That or someone's missed the point and your standup has turned into the kind of meeting that standups are explicitly not supposed to be, only without the chairs.
A 45 minute meeting is by definition not a stand-up... physically it just makes no sense.
I'm not wild about people trying to be pedantic about rigidly adhering to some idealized Agile-with-a-capital-A. But it's not rules lawyering to say that someone's completely missed the boat. And it sounds like whoever implemented the process at your former employer completely missed the boat.
Same process run by different group will feel different.
Which means, given a 8 hours working day, that you've been needing the help of Dave since the previous day, for an average of 4 hours, and you didn't ask him yet?
It's very possible the guy thought he could figure it out, but discovered he was wrong and mentioned it in the stand-up because it was the first opportunity that day to bring it up (remember, in a lot of places stand-up happens fairly early in the morning in order to give coders the largest block of contiguous time for development).
Exactly. Which means that he's saying that at the standup not because of any special advantage given by it, but just because it's the very first appointment of the day.
Had there been no standup, he would have asked for help anyway.
1) the standup covers up a fundamental malfunctioning of the team, namely that people can't communicate freely during the day, or
2) useful stuff that people say at the standup is stuff they would have said anyway at that time of the day.
The nature of building with stone is such that you never truly know what resources you have to work with. The shape, color, texture of stone is all variable. This means too that the way each component fits together is unpredictable.
For the eye of a master builder, who constructs edifices with each relationship perfectly balanced, each component placed intentionally, who can see every step of the process and what each choice means for the integrity of the whole, such a craftsman understands that every day brings with it a whole new set of variable inputs, which can never be planned for, spec'ed out, or tasked to laborers who do not see the whole.
Modern construction cares nothing for such craft. It will be the domain of robots. Software development too will follow this path, as we humans are suckers for predictability. But such a road will be littered with people suffering in jobs that make them feel like robots, projects that fall flat, and endless repeating boxes of sameness.
Agile came from business and finance, where everything is uncertain and even not well understood.
If everything goes right, yes waterfall. Everything never goes right.
All you have to do is make a giant list of all the things to do, make sure people are working enough, and on the right things, and use the appropriate level of design -- sometimes a little, sometimes more.
I think the hardest thing is getting business stakeholders to get out of the "fixed project" mindset, though: here's what we want, here's how much money we have, how long will it take/how much will it cost? When the real question ought to be, what should we do first, given the budget that we have, that will most help the business?
Also, devs should be treated as adults. No "planning poker", forced daily stand-ups, or any of that nonsense. Just keep your stories updated in the issue-tracking system, deliver iteratively, and keep on shipping.
So much of what gets blamed on agile feels like bad org politics/poor stakeholder management.
Extreme programming had the de-emphasis on documentation in favour of verbal communication, which got garbled into "just don't document anything".
Agile has sprints, which has been twisted into "crunch time every two weeks for two weeks".
In the parent example the management should have known enough about building to realize they were a creating a clusterfuck, and the builders should have had enough knowledge and input to the business to figure out what they were supposed to be building. When both sides assume the other is too stupid to work with and avoids meaningful interaction, it shouldn't be a surprise when the result is a hacked together mess.
In my current team, it's followed religiously. Stand-ups are daily and people use it as a time to fraternize between talking about actual work, I just want to get back to work. There's no camaraderie, every retro is this one guy bitching about how things aren't done the way they were in his previous job (and usually they aren't because it wouldn't make sense). The time between demo and sprint planning is a full day's of wasted time per developer, because there's nothing to do. Every developer writes hacks and things that shouldn't pass code review get passed because there's not enough time in the sprint to implement the known-good solution.
I've caught flak from going beyond the story requirements because I know what the final solution actually looks like. I also catch flak regularly because I run out of work and I take more tickets that are not in the sprint. I've never not finished a task when I did that. Every other meeting I get to hear a spiel from the project manager about how great Agile is.
I'm sure there's cases where Agile is the perfect solution. But it's not every time, and the rigidity that people want to apply was never meant to be there. I wish it would go away, but it's the perfect tool for management to be able to submit reports and say "look how consistent we are!"
So, here are the few quick fixes I see:
1. point out in retro that standups as currently constructed are a waste of time. Tell them you will utilize the law of two feet if they can't be tightened up.
2. Suggest a definition of done. If the product owner doesn't agree it meets requirements, code quality isn't high, and tests aren't there then it's' not fucking done. That means your sprints are implementing too much.
3. Don't pull in more work if other members are struggling to finish something. How can you help them out to make sure their code quality doesn't suffer? If your piece is adjacent to another that is in desperate need of refactoring/code quality improvement, just say that you needed to clean that up to implement your solution. Use that time to de-hackify.
I see what you're saying in your last point, but on that particular sprint, someone didn't want to do something they had assigned to themselves and pulled in a different story/ticket. I have no issues picking up anything that is still unassigned, and I had mentioned during the daily that I was available if anyone needed help with anything. No one got back to me and I honestly didn't know until later that day that the other story/ticket hadn't even been started. I don't think I should have to talk to individual developers to see if they need help; we're adults and if you can't manage your workload or can't communicate that then I don't have a solution for you
So, not Agile then...
I think expecting business to know what they want is a mistake. When my dad was making software in the 60s, 70s, and 80s, we were mainly automating existing business process, so we could draft behind all the hard work done working out the right paper forms and reports. But modern software comes from a process of experimentation and discovery, competition and evolution. As the people who understand the technology best, we should be as part of that process from beginning to end.
This analogy is misleading because software gives us power that we don't have in the real world. If we start with port-a-potties and want real bathrooms, we just have to change the description of a bathroom and the computer will do the rest. In software, if you are disciplined about automated testing, refactoring, and keeping up with technical debt, an evolving architecture is totally doable.
I think where people fall down is that they don't believe that managers will give them enough time to keep things clean as they go. And I agree, they won't. You have to take the time. But if you do it from day one you can do it gradually and incrementally. You can gradually redirect management pressure away from developers and toward making better product choices.
So please let's take control of our working conditions back. We're professionals, and we can say, "No, I'm still writing tests. That's a best practice, and it would be negligent for me to leave that out." But we can do that while delivering incrementally, so that no week goes by without the business getting some new benefit in production.
I'm not much of an agile true believer, but as a former developer who is now "management" (shudder), I have never-ending frustration when teams expect full, perfect, set-in-stone requirements from day 1. When you're developing a new product in a new field, that's not how it works. We have to build something that minimally works, get it in front of real customers, and then make adjustments based on their reaction. Lather, rinse, repeat. This isn't "management can't make up their minds." This is "management are not clairvoyant."
To those who hate requirements changes, I'd love to know how you are able to determine with 100% certainty the exact product the market wants before you build it. If you can do that consistently, you should be a billionaire.
If you do understand that we don't know for sure what needs to be built from the beginning, you have two options. First, I can give you my current best guess at what we need to build, and we can work together to figure out what the right starting point (mvp) is, with the understanding that the rest may change later. Or I can figure out the mvp myself, and only tell you that, preventing the situation in which I told you about something that later has to change, but also robbing you of the opportunity to see the big picture and contribute to the plan. Which do you prefer?
Fr example can u imagine telling a plumber no the bathroom is now going to be 30 feet to the left. The plumber says ok but that will cost u a week of work. How many people would say no, u have 1 day to do it and u will be held accountable for my estimate and not ur estimate because I don't trust u when u say a week. Now assume the walls are closed and it's only after u open them u learn there are other issues. Doesn't matter, one day. This is what most de s deal with. I assume ur not like this but this is how most devs have to work. It's frustrating. If u would be given the week then u would be fine, but instead ur told the timeline and/or cost. U may even be told no u can't use those tools they r too expensive, but u need to complete the task in the faster then even people who can use the tools.
My whole point is it's often not the change requests that are the issue, it's what it entails, in that the schedule and costs are not adjusted fairly.
That said, I think "give us the perfect requirements up front" is a reasonable defensive reaction. Because the first thing a lot of developers get asked is "When can you have this done?"
In most places, if the (made up, arbitrary) deadline is not hit, it's not managers who say, "Damn, our bad; we should have been clearer about what we were asking for." It's almost always an inquisition into either a) why the estimates were wrong, or b) what's wrong with the developers that they aren't delivering on time.
I now have a bag of tricks for keeping this kind of trouble to a minimum. But I think managers and execs have to stop asking, "When, to the exact hour, will this (hazy and ever-changing) thing be done?" It's not a sensible question, and if we pile blame culture on top of that, it's no wonder most projects quickly go off the rails.
1. Just this week, I watched a manager wait until after I was on vacation, and then publicly chew out my employee for slipping a deadline. I would agree he should have pushed harder, BUT he slipped it due to our peers being very rigorous on his CR (he is a new employee) to bring him up to standards. (I would add we would have had even MORE buffer had the person in question not continued to change the spec 2 months into a 3 month project, which can happen, but should be acknowledged as a cost) .
The manager in question found this (the deadline slipping) regardless unacceptable and gave him grief in front of the whole team about it, and this is the second time I've watched the manager place these meetings when I'm going to be gone because otherwise I come down like a bag of bricks on someone going after my employees. But to why I mention that last sentence regarding my nature, NO AMOUNT of this seems to motivate positive change in the long run, _especially_ since I'm "managing up" and having to do this as an engineer myself, any big "push" I make costs a lot of effort and karma, and doesn't seem to stick long term despite my best efforts.
2. More specifically, I had a conversation you summarized above (taking TIME to dev well) just a few weeks ago with my manager. Over the last few years we've repeated a pattern, I take longer than he'd like, he gives me shit for it, I tell him it'll pay dividends long term, he grumbles, but then it does. (bug rates, iteration speed under frameworks I write, reuse due to good modularity/code cleanliness, these are all benefits you see rendered on a very short time scale, even just keeping with the most broad basics.) Now, I will give him credit for saying "you have a point and I'll leave it to your judgement" but that took me _two years_ of very incendiary 1:1's of my trying to put my money where my mouth is against someone who has been at this a lot longer than me and with some very strong opinions about that. I frankly don't see a "shortcut" after that experience, although the conclusion was more positive.
So let me tie up this ramble with addressing your conclusion: I'd LOVE to take control of my working conditions back. But when the same people (PMs) who lead to my just throwing my hands up at even TRYING to get work done at work due to constant context switching/interruptions _SCHEDULE MEETINGS_ asking us why we're so slow (and then spend the whole meeting complaining about how WE pull them into meetings they don't need), it seems like there are just massive gaps of understanding or lack of shared incentive between the various powers that be. I don't know how to take control of my working condition back, but I've damn well been trying, and would be curious to hear any anecdotal suggestions from you in working this through more "political" environments.
I also think this is tied in with current American managerialist culture. The notion of controlling work by setting arbitrary hurdles and deadlines is so pervasive that many people can't imagine an alternative. This very easily turns into pure fantasy, even on the billion-dollar scale.  I expect that there are plenty of companies that would literally go out of business before they would change their ways.
So in your shoes I'd take a hard look at your company and ask what its true values are. If they really value appearance over results, politics over substance, dominance over cooperation, then I don't think that's something you can change on your own. I'd personally try to find someplace better, and if I couldn't then I'd mainly play to survive, not to win.
But if you think the company has its heart mostly in the right place, then in your shoes I'd start with weekly team retrospectives.   Each week, you'll make note of problems and possible solutions. Each week you try to make things incrementally better. And gradually you'll build up solid data to say things like, "For the last quarter, the most common thing in team X's weekly top 3 productivity drains has been insufficient contiguous time to focus. If you want us to go faster, please help us to limit meetings and interruptions to the mornings for the next quarter as an experiment."
If possible, I also strongly recommend finding ways to ship early and often. The longer managers go without experiencing actual progress, the crazier they get. If you release new business value at least weekly, then they'll focus much more on picking what's next then on yelling at people. At my last startup, even with a small team, we were releasing a few times a day. That meant that the CEO wanted something urgently, he could always get it soon just by telling us what thing currently in progress was less urgent. He always felt in control. It also meant that he was kept busy looking at new results and thinking about what came next; he didn't have time to fuss much with our work.
Related to that is building cross-functional teams. To me, a team is a group of people that win and lose together, a group that can play the game basically on their own. So I'll try to put a few developers physically together with a designer and a product manager and whoever else is vital. If PMs think of themselves as their own team, then it's hard to counteract those bad incentives. But if the PM only wins when the project and the rest of the team wins, they'll be much more collaborative.
And I'd also encourage you to make good use of every disaster that comes your way. The best luck I've had changing culture and process has always come when there's some mega-fuckup that's fresh in people's minds. That's especially true when you already have a successful small-scale example going. E.g., I remember one retrospective after a team spent 9 months building a big upgrade that was a flop. One of their peer teams was at that point releasing every few days and quickly responding to business data: killing bad ideas quickly, doubling down on the good ones. The pain of failure plus the envy of colleagues doing well was a powerful combination.
Sorry for the ramble in reply, but basically my answer is: as problems present themselves, solve them in ways that move you toward greater agility. By which I mean more frequent releases, faster feedback loops, closer collaboration, less work in process, more responsive to business and customer needs. To it continuously on a small scale and regularly at the large. If you can make a 1% weekly improvement, things will be 68% better by the end of the year, and 181% better at two years.
Does that help? If not, feel free to reply here or email me. Glad to help as I can.
 For example, the dumbfounding mess in Mississippi, which we know about because of one stubbornly honest engineer: http://www.nytimes.com/2016/07/05/science/kemper-coal-missis...
 A good book is Larsen and Derby's Agile Retrospectives: https://www.amazon.com/Agile-Retrospectives-Making-Teams-Gre...
 Also handy is the Retromat: http://plans-for-retrospectives.com/
To make the comment more productive and to echo some sort of thoughts on your thoughts:
Would agree re: managerial culture. I've seen a very out-of-wack ratio of deadlines set vs. deadlines that provide productive bottom line or user facing value.
Secondly re: values, I've come to a realization that I love my coworkers but find that what I love in them seems to get diluted the higher you get in the organization. It puts me in a conflicted spot about fit; and in the interim your survive v win advice is likely wise if only from a hedging-risk point of view.
The emphasis on small ship cycles is something I certainly need to improve. I've pushed hard to turn my code/commit style into this manner, so it seems lacking that I would drop that from overall feature delivery.
Cross functional teams however, although appreciating wholeheartedly (had a 4 man dream team at my last job that I really miss) I have very little power to change. The PM org is as you suggest _very separate_ (at least from my perceptions of their incentives and alignment). I can't be entirely pessimistic, there is certainly shared value when the shared feature ships well, but for whatever reason the focus seems often much more on asserting control and less on delivering value, for whatever reason.
Disaster handling as well is something that where I see the value, although pragmatically, you're usually in "put out the fires, damage control" mode and holding ransoms isn't really looked kindly upon. To their credit, the powers that be have been _reasonable_ (although as a grumpy ex-sysop I always like to push harder) in assigning time to postmortem/address issues, but for incentives beyond even THEIR control, it's not a priority.
In short; I've seen some "large company" problems at multiple places I've worked and in the lack of "true power" to oppose it your advice is very valid, if an uphill battle.
I agree with you on some people favoring control over value. That's especially true for people whose primary job function is seen as control. Project managers and middle managers especially are often rewarded for the appearance of action, not the quality of the business result. This is part of why I strongly favor approaches where cross-functional teams mostly self-manage the work and other details normally given to project managers.
As to disaster handling, I should be clear that I'm not talking about asking for big changes during a crisis. Instead, there's this window after trauma where people are open to doing something different. Before the pain, nobody has reason to change the status quo. And eventually, they forget. It's sort of like how when people have a really bad hangover, they might say, "I'm not drinking again for a month." In that moment you can say, "Wow, I've been wanting to do that too. Let's try it together."
Good luck with your efforts! I wish I had a general-case solution for you. But my approach has been to find or create contexts where I can get a relatively independent team focused on value delivery. I think current managerial culture is inimical to that, so I expect that's an uphill battle at most places.
The problem is not the specific body of techniques used to schedule tasks and manage time; the problem is the way that the hierarchical nature of traditional management structures diminish agency and discourage transparent communication. This leads to misapplication of effort and destroys motivation, crippling productivity.
Empower people to act responsibly and they generally will. Exercise empathy and people will trust you. Give people the information they need and they will reciprocate. Agile development was an attempt at doing these things better, but it has been reduced to a collection of code words and empty rituals by managers who don't realize that it is actually their philosophy about the purpose and nature of management itself that needs to change, and not merely their choice of techniques.
If they had, or if hey had bothered to read a book or something, then they would probably have seen that nice chart that's invariably somewhere close to slide (or page) 4 that lists all the things that $agile_process is and is not good for, accompanied by warnings about the need to be adaptable.
It's part of why I'm uncomfortable with name brand agile practices, because they make it way too easy for folks to interpret the guidelines as rules, or to interpret a useful tool as a silver bullet. For a while it bothered me so much that I was inclined to throw the baby out with the bathwater. I don't like to think that way anymore, though. Incompetent leadership will cause a team to be dysfunctional regardless of how it's organized. Blaming X or Y organizational technique for your boss's bad behavior is a gross misunderstanding of the situation.
Agile is simple, but it isn't easy at all. Without management buy-in to sit through the bikeshedding to find the technology gems among their technical talent's ideas (that they did a good job of recruiting and compensating, right?), Agile is going to turn into a swamp where mediocrities fool each other into thinking they are doing something.
Doing this while engaging low-bid outsourced implementation resources is 10X harder. Unless you pay through the nose for True Believers like ThoughtWorks, you're going to get fake Agile.
Tl;dr - if you want software to be more reliable, insist less on reuse, and allow copies to diverge more.
Thus if you want a wall of a certain size and shape, you have a list of materials to pick from depending on the weight is to carry.
The reason people insist on upgrading to the latest and "greatest" software is because there are no rigidity to APIs etc.
The Linux kernel gets more use than GNU/Linux user space because the kernel devs are held to a more stringent API rigidity. This makes kernel behavior predictable.
Similarly, Windows, for all its warts etc, gets used in business because Microsoft built a reputation for predictable APIs. Only with 64-bit Windows can you no longer run 16-bit binaries (dating back to Windows 3.0!), and that's because the 64 and 16 bit CPU modes are mutually exclusive.
Some of those requests are very important and will benefit with actual real monetary results, but he has to personally escalate it through hierarchy and force dev team to break their sprint rules and force them to work on it.
In other words he has to make them be agile by force.
A well-functioning product development org has processes in place to decide what gets done first. These processes take into account tons of inputs including bug feedback, enhancement requests, potential for outages if something isn't fixed, long- and short-term competitive and strategic concerns, customer requests, and internally-sourced architecture requests.
Then some selfish jackass comes along (from whatever department) who just can't wait because WE NEED TO GET THIS DEAL, so the entire process is thrown out the window to satisfy the whims of one customer, who may not even be large, high-revenue, or strategically important, and the wrong works gets done.
Please, stop the madness. Don't go around the product team, and if you're in eng, please don't tolerate this bad behavior.
That has nothing to do with agile. That's a basic failure in product management, and would happen regardless of the dev methodology.
If those features really do have such a large business impact, they should be getting prioritized (hint: the backlog isn't a rigid, time sequenced FIFO).
If they're not, either the PO sucks, or the features aren't as important as the sales guy thinks relative to other items in the backlog.
Because it turns out everyone is making those same requests and the product owners are putting "we'll put it on the backlog" as a soft no. A backlog needs to be shit that will actually happen.
Alternatively, the wrong things are prioritized. The backlog is not "first in first out." It's a "here are the next 5 most important things. Why is your special snowflake more important than that?"
Better to allocate a single full-stack dev to experiment for a couple weeks. He also will interact with the business owner daily. Let him focus on the business logic and also the core user interactions.
On top of that MVP foundation (i.e. on week 3), you can put the rest of the team to refine backend, frontend, graphic design and so on.
I've been trying to push this kind of thinking recently. Most successful big projects grow out of a successful small project.
This is probably true, one could even say that forming a big team from day one is cargo-culting others' success.
This statement rings so true to me that I really hope there exists a fleshed out version of this theory, with examples to prove it and maybe a catchy name, etc.
Where I am now, I've literally participated in an identical thing being built in two ways at two different times:
- The first time, I was the initial full-stack dev, along with one other guy (we were more of a duo... we paired a lot and worked pretty fluidly, it was kind of a 2-person equivalent of a single-person experiment.) The result was finished in 3 weeks and was flimsy but cohesive. Eventually the team was reduced to just me and I carried out the project for a few more months before a proper "team" was on board.
- Then, the company noticed what we had done, decided that a bunch of other people had done similar things (it's a large company, after all), and then a 250-person organization was formed to write an MVP that was largely identical in scope. I wasn't in any leadership position in this new org, so nobody really cared that I had experience in this. I was reduced to writing a tiny insignificant part of the larger whole while our director drove the requirements. We're 10 months in and haven't shipped the first milestone yet.
In the first example, we were almost too successful. The resulting product was used by thousands of people, for important things, with a lot of satisfaction and kudos from the users. But the total size of the team never really exceeded 5-10 people, so it got to the point where we were burned out supporting it, and there weren't enough reserves to call in to help because of the "second system" distracting everyone:
In the second case, we don't have our first customer yet (although we've only kind of "soft" launched.) We did a massively half-assed attempt at agile that was basically "waterfall, but more vague, and please work harder", where we kinda half-assed came up with requirements, split everything up into 10-15 major "tentpoles", each of which was developed by separate teams. 6 months later we decided to start "integrating" all of it. 4 months after that, things finally kinda sorta work end-to-end.
I've gone through a ton of self doubt through the process. In the first case, it worked only because there was almost no communication: we just did what we felt like, explored, and we were lucky that the end result worked. "Clearly, this will never scale, and is no way to run a real software product", I remember thinking.
When the org got huge and we did our second system, I thought "Finally, I can learn to do what real software devs do and work for a big team with communication and requirements, etc". And inevitably when the result is a big pile of shit, I can't shake the feeling "this would've been better if I knew how to do real software development and work with a large team." Self loathing ensues.
So I witnessed the pendulum swing each way. The first way was massively successful, but I suspect isn't sustainable (the product is withering due to lack of support now.) The second isn't successful at all, and is a complete mess, but maybe it'll be better with time.
The only thing I know for sure is that few companies in the world can afford the amount of wasted time and energy we've spent on the second system. I suspect the real world works more like the first case but I don't have enough experience to know. I can't shake the thoughts of "I haven't seen real software development yet", and I don't know how much validity there is behind those doubts.
> Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
It sounds like the system started off as agile and the second system lost the plot.
Thanks for sharing.
The idea that requirements are set in stone, or - more specifically - the idea that not setting them in stone is breaking conventional wisdom is just running us in circles.
Requirements are just a name for detailed goals. They change as goals change. Diffs can be applied them just as easily as to the codebase. Maybe a team has them over emphasized and too rigid, maybe too undocumented and in flux. But prescriptive development methodology doesn't give teams tools to figure out what works for their projects.
Development processes aren't really improving with every new trendy thing that comes out, they're just finding new ways to package and shuffle things around.
Modern (ie. post-XP) software methodologies, yes. Traditional engineering? Not at all. Software is unique in that it's essentially free to convert your design into your end product.
It's one thing to iterate weekly when your build process costs $0.02 of electricity and half an hour of server time. It's another thing entirely when you have to obtain many different parts (some with long lead times, others custom built, and small batches always cost far more per unit) and then assemble them manually before testing a physical object.
Agile is good for startups building a product where they don't already have a known product/market fit. It's understandable in that situation.
A client of mine uses it to unburden itself with upfront design for a 3rd gen system that is well known and could easily be built with waterfall. There have been (what I would call) two false starts already. I don't see why agile is beneficial in this situation.
It seems to me, the there problem isn't agility, it's a lack of professionalism.
Here's the manifesto:
"We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more."
This is deep stuff and an incredible (and rare) set of pragmatic philosophies set down by folks who'd been suffering through the industry for a while.
Processes are intended to minimize risk and ensure reproduction of already achieved work. If you aren't making yet another copy of something that already exists, you don't want a process.
(sorry if this comes off a little braggardly), I've been known to run incredibly effective teams that produce tons of work very quickly. I use what I call an "Agile-ish" approach based on some personal meditations on getting software done. I call this approach the "Get Shit Done" model.
1) Is this new work or reproduction of old work (should I explore and evolve or reproduce?)
2) If this is new work, the goals are set down as very coarse milestones with exit ramps in case things are working out.
3) If this is new work, work is broken down into a check-list in a pipeline with the following steps: To Do->Doing->Paused->Done
Work is first filed as tickets in To Do. When it's being worked on it moves along to Doing. If, during "Doing", new work is identified, it's created and put into "To Do". If necessary the current work that's "blocked" is simply moved into "paused".
Once Paused work is unblocked, it moved back into "Doing". It either finished, creates new work tasks or is paused again. Rinse and repeat.
Kanban boards work pretty well for this, but you can do it with a spreadsheet in a pinch just fine.
4) If this is new work, work is only considered "Done" if it's checked in, tested and documented (and builds). Everybody hates it when they're doing it, everybody loves it when they need to figure something out later and the customers drool over getting rapid delivery of already documented, tested and validated stuff.
This scales pretty well for teams of 1-to-100 (but starts to strain on the larger sizes).
Teams are organized into hierarchies each with their own pipelines. Each team has a team-lead. I meet once a week with all the team leads and require all team leads to meet with their teams once a week. During crunches or abnormal events, we meet more often. This creates a cascading "clock" that identifies things that are taking too long and might need extra attention. I give team leads extraordinary autonomy to figure out the details at their level. This also gives people a week to be create and figure out ways around issues and problems. Team members can generally float or matrix if needed. Work moves along a general "discover, productionize, deploy" model.
What's amazing about software is that it's almost always new work. If it's not new work, you still don't really need a process to reproduce it. Just copy the code into the new place, configure it and start it up. This is unlike any other engineering endeavor in human history and it takes entirely new ways of thinking to get it done.
What I've found is that many many organization simply want to repurpose waterfall models and apply "Agile" sounding names to everything. I once turned down work because the customer had a 500+ page manual on their formal "Agile Software Process Methodology". No, no, no, NO. You don't get it.
Here's another protip: Sprints are for engineering, Kanban is for Research. If you know what you are doing, use Sprints to force a development clock. If you don't know what you are doing and need to figure it out, Sprints just make everybody angry and create useless artificial pressure that accomplished nothing. If you're engineering, you already know what to do, Sprints help provide motivational energy.
The roots of agile come from the bad old days when the typical software project was a consulting gig with an elaborate negotiated spec and schedule and a huge disconnect between the buyers and the developers, let alone the users. And when the "software crisis" was in full swing, and many software projects failed to deliver anything of use, at all. Back in the days when even using source control wasn't common, and when regular builds and smoke tests were even rarer.
Agile ensures a few things:
It ensures that you use an incremental development strategy: start with something simple that works, then iteratively add to it over the course of short periods of work to move it closer to the desired state of functionality and capability. Because you routinely have workable builds this means you will always have something to show for your work at the end, because you have a pattern of having something working all the time.
It pushes the team toward continuous integration and regular builds with testing. Which avoids building up invisible technical debt due to undone integration work as the project progresses. Agile avoids the "big bang integration" problem.
It pushes the team toward visibility and interaction, so they get to know what everyone else is working on and it forces collaboration related to integration ASAP, all of which helps increase team cohesion and leads to higher code quality. People get to see the products of other's work routinely, making it easier to see what they're good at (and not) and aiding team mates in learning from one another.
It avoids unnecessary over-complications and the tendency to procrastinate, such as the notorious "second system effect". It ensures that people stay concentrated on real milestones and real work not pie in the sky perfect systems with schedule estimates off by orders of magnitude or not even feasibly possible.
It gives the "stakeholders" more visibility into the work. The product itself becomes the ground truth for the product, rather than a spec or a memo or a contract. If those folks have concerns they can be addressed and taken care of before it is extremely costly to fix. And if they want to make changes to the design relative to their initial thoughts, that too can be done fairly easily, resulting in an end product that is more suitable than would be the case otherwise.
And it ensures that sprint by sprint, work gets done, the product moves forward. With good teams this isn't a problem, but in the "enterprise trenches" with demotivated teams of average or below average coders this can be a real problem. It can be difficult to get anything done, let alone anything more than just spinning their wheels.
All of these things are for the most part good things, and they are the reason why the concept of agile has become so pervasive. The problem is that today a lot of the agile basics just go without saying (continuous integration, routine builds and testing, iterative development, short sprints for work, and so on) while some of the other aspects end up being applied in a cargo cult manner with no comprehension for why they were used as well as when they are warranted and when not.
Even if agile practices were being used correctly most of the time, which they aren't, it still would not be a good one size fits all practice for all circumstances. Moreover, the modern perversions of agile have done a lot of damage. Removing opportunities for use of initiative, creativity, and expertise by many coders, taking responsibility for even the tiniest decision out of their hands and transforming them into children. In some ways agile is like a remedial course in software engineering, but a lot of high caliber developers are finding themselves stuck with it even though it is restrictive and harmful, because management is thoughtless.
Working in Agile shops can be a lot like watching lab mice running a maze. You try to say, "hey, so this path you are working on, there is a map for it when it was solved in <discipline x>." and the response was, "this is Agile, we don't take grand designs or waterfall requirements, and we don't need anymore chickens in our standup."
Or you say, "hey, so this thing you are building, for someone to buy it, we're going to have to explain to them why it's good and how it meets their expectations, can we create a sequence diagram or a model so I can plug in how it meets standards and regulations and code and such?" and the response is, "we don't need documentation, the code is documented, and if you had provided us with the requirements we could prove it in our unit testing."
The trouble I have with a lot of engineering and dev culture is what it lacks, and often irrationally rejects, is balancing the technical understanding with a top down view. The best engineers have or at least appreciate this, and they bring it with them. Unfortunately this gets them immediately promoted to roles where they aren't actually writing code.
I've used this house building simile in Agile shops before, where any bunch of carpenters, electricians, plumbers and brick layers can build a fine house that will stand up, but the matter of whether anyone will want to live in it, buy it, insure it, legally sell it, live anywhere near it, or trust that it won't be knocked down by local climate or regulations - is an equally complex problem you need someone with expertise to solve.
Everyone has complains, devs about managers, managers about devs, it's an age old problem. A lot of Agile gets used to formalize an adolescent culture. I look forward to the pendulum swinging back to a more balanced equilibrium.
We never had a good designer on staff, so that was a huge part of the problem. A couple times we used outside designers but they didn't have a holistic view of the app, so their designs created as many questions as they answered.
You cannot abstract away the need for common sense.
(BTW, he is very opposed to the use of Agile as a noun ;).