Hacker News new | past | comments | ask | show | jobs | submit login
Maximizing Developer Effectiveness (martinfowler.com)
427 points by edoloughlin on Jan 16, 2021 | hide | past | favorite | 162 comments



Here I am, a productive developer eager to deliver maximum value to my customers and apply my innovation to company goals.

Open up JIRA and pick up a unit of work to produce today -- gotta stay faithful to those story points!

Somehow, a magic team of spherical devops in a vacuum created an environment where things are just green, predictable, and are never broken.

Another theoretical team of angels from a parallel universe has materialized into this one, updated all the docs, and beamed back into their universe of productivity and effectiveness.

Unit of work defined, I then proceed to produce it for a few hours completely uninterrupted, since if another human being were to make contact, I'd simply explode and format my hard drive, losing all my productivity.

It's break time -- I ingest 2 story points of coffee and join a game of table tennis. The ball bounces back and forth without ever touching the ground. My movements are productive, and value-adding, just like my partner's.

Back to producing units of work, only half a story point left (sike! no such thing as half a story point, and no such thing as "left", as it's a measure of effort, not time. Almost got you there!).

Git push! Automatic CI gains consciousness and validates that my work unit has no chance of causing a user to break our software. It gives me a thumbs up and discards its human shell to fade back into the ones and zeros. A QA engineer cries out in the distance.

A metric of my productivity is logged to the OKR database.

Nothing breaks, and nothing unexpected happens, as the universe is completely predictable.


It is true that the effective developer environment sounds a bit too good to be true. Also, there seems to be a contradiction in it. We get an answer from the other team very quickly but on the other hand there are no interruptions. It seems a bit hard for both of these things to be true at the same time. On the other hand one does have to notice that one can either be moving towards the effective developer environment or towards the ineffective one. It is a very important question in which of these two directions the actions of the team are headed. E.g., if the nightly build fails is somebody doing something about it or not?


The article specifically suggests there's a dedicated support developer, which means you're sacrficing one member of your squad (per week) for the productivity of the rest of them. We can debate whether the idea is good or bad, but it's not pretending that its magic.

On the wider article, I think the litrature is a poor subsitutde for reality. And honestly I question whether a lot of these agile consultancies actually have the required experience to teach us how to be highly effective as opposed to what we can learn directly from orginsations that are highly effective, but I do believe they have enough experience to tell us that effective development teams invest in development tooling, dev focused user experience, and a dev focused culture.

Who and how you'll build this is left as an excercise for the reader, which is unfortunte because we now have a legion of consultants who've never produced a line of code making a career telling us "culture is hard" and then forcing us into scaled agile frameworks and tooling that are contradictory to the ethos.

Doesn't mean the underlying point is bad though. Engineering teams need some time to work on developer experience instead of just being feature factories; bigger org can hire teams dedicated to helping with this. This is a good thing, but many orgs fail at understanding and allowing the investment.


I'm torn about this article, I love the part about feedback loops and optimizing the parts of your day where you are in a tight loop for a long time; but for most of the article it seems he is _almost_ saying something profound, but there is too much clutter and buzzwords around it. I know these abstract consultancies have a use, but they seem to target what managers and executives _want_ to hear instead of the truth.

e.g.: "The promises that were made to executive leadership about the latest technology are not coming to fruition quickly enough."

It feels like he then proceeds to talk about how management can get the developers "organized" to make sure they get the job done. Having a developer doing support for the day kinda sorta seems like a nice idea until you realize that "Isn't that what the manager is supposed to do? Understand his team and get what they need/negotiate with other teams so that they can just get their work done?" It's kinda like he's telling managers how to outsource their work to someone else. Pass the buck and live in a fairyland.

I know people who idolize Thought Works, Martin Fowler, Uncle Bob, and others in the "consulting set"-- but it seems like they fill a particular niche, and it isn't the phenomenal tech experts, it is more of the "how management views tech from an academic level and how we can get these interchangeable programmer cogs to get our project done w/ the least amount of effort". They rarely give hard advice, it is more of soft exposure to "new tech" and "here's something that works for some people, you might be that person". "We'd love to sell you on new tech that will be a silver bullet for all your problems"

In any case, I learned some interesting stuff from the article, and I do think it is a valuable read.


Indeed.

The lack of hard advice is great for them because they're essentially never wrong but you'll also never fix the problem with the advice alone. Keeps the gravy train going; if they solved our problems with hard facts in blogs, we wouldn't need them.

That said, the advice itself is still pretty good if you have the chops to follow through, but I honestly don't think you can build the chops by hiring in the consultancy to tell you. In fact, it's actually a bad sign that you leadership both need obvious advice, and think they can enact change by hiring the consultancy.

The reality is, you'd probably have better results changing the leadership.


Asking for advice from a contractor is not bad.

Asking for leadership from a contractor is bad:

1. They don’t understand what you need or how to get it to you, but they sound like they do, and they look professional.

2. There are no great contracting options for something you can’t do yourself and that there aren’t requirements for.

Fixed contacts may start off looking good but they swap resources out and you end up incomplete or halfassed, overtime and maybe over-budget.

A renewable contract may blow really pretty smoke too and look like a high speed train, but that’s not what was needed, they never finish, or when it’s forced to completion, it’s incomplete or halfassed, overtime and over-budget.

If you don’t take care of your health on your own, you can’t expect a doctor to do that for you. Similarly, don’t expect a contractor to solve all of your team’s development problems, though some can give good advice or assistance.

Steps towards making yourself healthy may include exercise, eating better, adequate sleep, regular checkups, etc., while steps towards fixing your development team’s problems may include raising those to your leadership, own problems, foster trust, facilitate, and change your job if problems are not resolved.


Maybe.

Funnily enough I was a contractor providing leadership and helping orgs for the last 5 years. I did my best to varying degrees of success because the leadership themselves would either let you run with it or not. My 1 man band is very different from thoughtworks though.

Now I am leadership and it's way easier to just make decisions.


Sure, there’re exceptions.

But if you don’t understand the requirements, choose to pay someone to figure it out, and they finger-paint your business, you may have been better off without it.

Nothing sells better than crap. By that I mean literally nothing, like you could sell emptiness more feasibly.


Yeah, reasonble points.


To be fair, they’re selling to those executives. It’s unlikely that devs on the ground have the authority to sign off a large consultancy bill.


In my current team (and my previous one from another company) we have what we call "goalie" duties. Each week one engineer from the team is responsible for answering inquiries, not necessarily they have to know the answer but they are the ones to be interrupted and handle accordingly.

It works pretty well, our devs are capable of assessing urgency to either bring it up on Slack, wait for the next stand-up or call to arms if it's something critical.

It improves rapport with stakeholders and other teams; it improves morale, it's predictable who will be interrupted during the week, alleviating that from the rest of the engineers. And as a side-effect it helps to force knowledge sharing over time.


Isn't the typical responsibility of the manager or the team lead?

Inquire often is followed with more requests, if each developer serves as the weekly-speaker-of-the-team, will the external partner eventually figure out "John will do anything for us and Steve is mean"?


This is actually a great idea and something I'm likely to bring back to my own teams!


In my current company we have a daily devops and dev duty. With current dev team size each gets around 3 duties per year.

Dev duty takes care of rollouts, crises, rollbacks, keeping an eye on performance and error monitoring. It's a great approach that works for us and allows every developer to get familiar with rollout process eventually, but I don't know if there are downsides for smaller teams or for projects that are split into several independent services instead of one monolith.


How large is your team? 3 duties per year sounds incredibly low.


And how do you resolve this contradiction? Teach your employees that if you don’t want to be interrupted you better write good documentation. And have one dedicated team member each day responsible for supporting other teams. I think both of those are mentioned in the article, but it was a little a while since I read it.


"Teach your employees that if you don’t want to be interrupted you better write good documentation."

And reward them for doing it.

If your company do "360 degree" evaluations where your peers praise you for your quick answers to their questions, you are effectively incentivised to NOT document and instead be interrupted...


One does not preclude the other. Your quick answers can come in form of sending them links to the relevant parts of the documentation you wrote earlier.

And if their question is not something that can be answered that way, you can improve your documentation so that next time the question is asked, you do not have that problem.


It's not something to be taught. For majority of devs an average working day is too short to do everything they theoretically should. So they make choices based on feedback/motivation which rarely favors writing good docs over writing more code. On top of that many good devs are simply unable to write comprehensible documentation because its very different kind of skills, or just because English is not their native.


I would be inclined to say that interruptions to help your colleagues should be accepted. Writing good documentation is actually hard and if you not willing to invest the time to do it well and to make sure the documentation can be found when needed you had better not write documentation at all.


Thanks.

This summarizes and completes perfectly the ThoughWorker bingo. They are always dreaming about the ultimate factory line for knowledge workers, but they don’t get much done other than blog posts.


Last year I got to work on a codebase done from scratch by ThoughtWorks India, it was a convoluted piece of shit anywhere you looked... and, again, it was from scratch, they made a 1 year-old codebase feel like a giant ball of legacy spaghetti.


Even by the HNs/SV measure of success they just got 700 million in funding and a 4.6 billion valuation... Just from writing blog posts apparently.


Well, sell each consultant hour ~100 and pay ~30. After the initial project implementation swap the whole team to India, move the existing people to another projects. Keep everyone in a cult like mindset, profit.


Do you work somewhere that pays you exactly the same as the value you generate?

Are you saying ThoughtWorks bait and switch their customers? Do you mean without their knowledge? Or are they convincing them to accept dev teams from India?


Anecdotal but that's exactly what ThoughtWorks did for a company I work for, and then we entered protracted legal battles around Intellectual Property for the work/crap they had produced.


I don't know about what they actually do because I've never interacted with the company other than reading what they publish. I have to tell you that if you apply the things they share, where previously you had a home-grown "process" or lack thereof, your engineering organization will produce better results and your engineers will be happier. I can't vouch for every single thing they ever wrote, but I think this style of rebuttal must be coming from someone who's had to work in disfunctional places with overbearing product management.


The pendulum can swing too far both ways. Having worked for a startup that consisted of probably half ex-thought workers I can attest that yes some of the development processes and infrastructure we created were commendable. Some others were just absolutely frustrating. However I feel the only metric that’s really worth mentioning is that after a couple of years, there was practically no product to show of, not a single paying customer and the company slowly burned through its entire runway and faded to (more) obscurity. Yes I’m convinced that the ideological stances such staff took were directly correlated with the business failure. I think the only positive result was focusing on building an ivory tower meant everyone’s CV looked more impressive.


I think you overestimate the sophistication of the average Dev shop. There are many orgs where "having a basic CI process" would represent a massive leap forward in developer productivity and happiness.


This reminds me of this wonderful piece of art from Krazam; The Hustle.

[1]: https://m.youtube.com/watch?v=_o7qjN3KF8U


His microservice video [1] is also pretty awesome!

[1] https://www.youtube.com/watch?v=y8OnoxKotPQ


My favorite, Resolving Conflicts: https://www.youtube.com/watch?v=FDoH15ylAeo


Why haven't I seen this before?

Why am I on Hacker News at 1 in the morning on a Friday (Saturday?)


Probably better than being on Hacker News at 1 in the morning on a Tuesday.


Your reply is fun to read and I really enjoyed it. Of course the descriptions of the "typical day" in the original blog post is idealized. I suppose this idealization is there to make the contrast between the highly-effective day and the low-effective day. That is just the blog post.

Leaving aside this funny idealization and its comic interpretation of the blog post, in real life there are good habits and bad habits that can make a team work more efficiently and less efficiently. I have had first-hand experience with developers from ThoughtWorks, the consulting company behind that blog, and I can tell that they are indeed trying to apply those principles and strategies in their projects.

The fact that they are a consulting company gives them an advantage when trying to apply these principles. They only need to worry about the "small picture" of the project they're consulting on. They don't have to deal with all the "big picture" issues that might affect the employees working within the company they're consulting for (e.g. company politics, technical legacies, etc.) So IMHO perhaps some of these techniques and strategies to become a more effective development team are less applicable if you're working in $BIG_CO rather than in $CONSULTING_CO.


I belly-laughed for a good while after reading this, thank you!

I have had jobs ranging from entry-level phone tech support to Silicon Valley FAANG (E5 equivalent) to "the ops guy" at a New York startup. Like many here, I've been doing "DevOps" since before the term existed. Every job will have both kinds of day as described in the article. The only thing that matters is the ratio of good to bad, and how much you enjoy your work, your boss, and your coworkers.


"Psych" not "sike" please people! It's short for "psych out."


Huh, I always figured "sike!" was the slang version. I have to say, I feel like "sike" says something that "psych!" doesn't.


Yeah that was intentional. It’s weird to write “psych!”


There was literally a TV show called "Psych" that was a play on the phrase. It is quite normal to write the word correctly. I think it would be very weird if the show had been called "Sike."


I think the way it's used these days, it deserves it's own spelling.


Absolutely. Here's the thing though, in this hellscape you might as well take advantage, and if these units of work can be generated like so, then its quite easy to become a "top performer" by adjusting your commit habits.


You could always be the one to squash merge said unit of work and put your name as the committer. You automatically become a 10x dev. /s


I keep my github squares a deep shade of green.


Thank you for reminding me to read more fiction.


> It gives me a thumbs up and discards its human shell to fade back into the ones and zeros.

Me saying goodbye to the bus driver before walking into the office.

Thanks for writing, 'lxe.


Excellent post, thank you! I loved "spherical devops in a vacuum" :)


This comment felt like the beginning script for this video: https://youtu.be/_o7qjN3KF8U


Upvoted, of course, and also wanted to say that you made me smile, almost laugh. That's how good your writing and wit is.


I think there is this duality in SW development that some people tend to believe in, that there are two types of work, one highly creative, like product or architecture design, and then the mundane one, like coding and testing. (I actually realized this reading a recent HN post: https://news.ycombinator.com/item?id=25775872)

I don't subscribe to this, I think the work is ultimately very similar in creativity (just different skill set), both on high and low level. If something seems mundane, it is either a sign that there is lots of hidden entropy that you failed to capture (and thus risk that the high level understanding is wildly incorrect), or it means that we are doing things conceptually wrong (with more effort than required), but we don't know how (and so there is potential for innovation).

And from that duality is derived the idea, that if only we can perfectly specify the mundane work (for instance through "acceptance criteria"), it's just a matter of getting enough bodies to do it according to some grander plan. From this misconception derives the history of software development methodologies.

So originally, people thought, this duality is the same as in building a house, you have an architect, he does the design, and workers (coders) will build it. So the waterfall was born, where the emphasis was on the plan, and it was also the weak point, because in the real world of SW development, it turns out, the actual implementation influences the plan (so the architect cannot just throw it over the wall, as they say).

Then, from the disappointments, Agile movement was born, pretty much out of the idea that all development work is creative, and it cannot be easily formulated/predicted, so we basically need to iterate quickly and hope for the best. Just treat everything like research.

But, as it happens, people promptly misunderstood this (where are my metrics now??). Instead of understanding that the belief in the duality itself is the problem (a kind of wicked problem, really, because it can manifests in many ways), they looked at superficial recommendations of "Agile practices" - Scrum rituals, "embracing change" and foregoing proper planning, etc. (For me, good summary of this discussion is http://steve-yegge.blogspot.com/2006/09/good-agile-bad-agile... and https://simpixelated.com/two-year-work-retrospective)

One can draw an analogy to this duality in other industries, it's a problem older than software. It seems that with technological and process automation, other industries are actually becoming more like SW development (aka "software eats the world"). So elsewhere, this belief in duality gave rise to Taylorism (https://en.wikipedia.org/wiki/Scientific_management), and the rejection of the duality gave rise to Demming's philosophy (https://en.wikipedia.org/wiki/W._Edwards_Deming) and the related management methods (which were, again, misunderstood, I am sure). There the fundamental notion coming from the duality is that it's the management (or anybody else not doing the actual production job), not the workers themselves, to decide how to effectively organize it.

And still, to this day, some people intuitively cling to this duality, and try to apply Taylorism instead of something like Kaizen or Kanban to SW development (and certainly not only that). So you have JIRA tickets and people beancounting story points..


I think that complex organizations tend to develop caste systems. The caste hierarchy could be arbitrary, or based on imaginary metrics such as being "closer to revenue." It could have some basis in fact, for instance doctor hours are more valuable than nurse hours, which are more valuable than patient hours.

The upper castes have to be responsible for the work environment, otherwise it degenerates: The best people in the lower castes switch jobs to join a higher caste or a company with a better caste system. Or they unionize, either formally (like nurses) or informally by forming silos. Thus the castes become self fulfilling.

As an aside, one thing that HN has taught me, reinforced by this thread, is that I would not have survived in a high throughput coding environment, and am glad I didn't steer my career in that direction, though I love programming. I don't have the self discipline, and I'd have burned out quickly.


I can only applaud this.


Martin fowler is the one true God of this universe.


Patterns on the most effective teams I’ve worked on:

- high degree of trust and emotional safety between team members. The team can safely share feedback and risk sharing our crazy ideas

- high degree of care for the craft. We hold each other accountable to quality

- ships, regularly, to real customers

- little status seeking - goes with emotional safety - few individuals on the team need to be “in charge” or hold arbitrary titles.

- the individuals are more than fairly compensated and the company shows their love for the team in big and small ways

- the team talks to their customer and has a lot of empathy for their problems. They want to be accountable to them

- the team has a lot of empathy for new team members and works hard to make on boarding easier

- some willingness to “get in trouble” with the broader org because you know what you’re doing is right ultimately for the customer

- high self starters: instead of complaining, people feel empowered to solve problems or prototype ideas without a permission structure

- not too much catering to “super stars”. 1-2 heros does not a team make, the senior people make it their job to lift everyone up. The team doesn’t obsess over their high performers.

There’s a lot more emphasis on emotional intelligence and empathy in an effective team than on any specific process.


Rot sets in when:

- teams no longer have a real customer, such as “big rewrites” that will ship in 3 years

- teams accept low quality and slack off. One PR isn’t called out and that gives a permission structure for lower quality

- a bad egg gets on the team the wrecks the feelings of emotional safety. The brilliant narcissist the company feels they need to let his/her abusiveness slide

- we throw new hires “into the deep end” because they have to “take their lumps” like we did

- people get territorial over code because they’re insecure and maybe not great developers.

- the customer is ignored to chase the latest cool new thing

- hiring standards are lowered and anybody with a pulse that walked by the tech section in the bookstore is hired

- good developers stop seeing peers, and realize they do all the work, so leave for a healthier team

- tech illiteracy outside the team trumps tech competency in the team. When the manager says “stop writing unit tests, just get the feature done”

- any rumor “hire cheaper labor” causes your best to flee and your least qualified to jockey for status


- Bureaucrats who mask their power hunger with fake noble goals take over the team and call high performers brilliant narcissists.

- Bureaucrats cannot code, therefore they needlessly churn around and associate themselves with other people's creations. People objecting get called territorial and lose the political battle to the mediocrity.

As you see, it all depends on the exact circumstances. Sometimes your points are correct, sometimes mine.


What’s worse is when some of these make their way into a company that has barely gotten off the ground. Especially the narcissist.


>not too much catering to “super stars”. 1-2 heros does not a team make, the senior people make it their job to lift everyone up. The team doesn’t obsess over their high performers.

>good developers stop seeing peers, and realize they do all the work, so leave for a healthier team

These seem like alternative framings of or reactions to the same situation.

>teams no longer have a real customer, such as “big rewrites” that will ship in 3 years

"Real customer" is a luxury. In consumer facing tech, the core stuff was all written years ago, and most teams are responsible for some features of / enhancements to the overall product. UXR can put together surveys and focus groups with people in your treatment group, but you are lucky if they even clearly remember the part that your team changed. Most of their opinions will have nothing to do with you.


Can you expand on "any rumor hire cheaper labor"? Im too new to understand that one.


Likely rumors that the company will be hiring or contracting overseas.


"Now that you're all working from home anyway, I might as well outsource your job to India! That will save me a truckload of money too!"


Our company recently did this and our best developer left. I now have no one to talk to about best coding practices, new technologies, or other fun concepts. Just a bunch of overseas ticket jockeys trying to avoid lashings.


Sounds like that's your cue to leave if you ask me.


> ...not too much catering to “super stars”. 1-2 heros does not a team make, the senior people make it their job to lift everyone up. The team doesn’t obsess over their high performers.

Amen.

A team is only as fast as its slower member.

h/t The Goal, theory of constraints

https://en.wikipedia.org/wiki/Theory_of_constraints


"little status seeking"

It only takes one that gets away with it to start a race to the bottom.


amen. I've found that pushing on all these levers to improve things makes me happier and more productive, and to a lesser extent everyone else happier and more productive, too.


> There is an overwhelming amount of good advice, practices, tools, and processes that you should use to improve.

I disagree. Advice is contradictory, practices and processes are often orthogonal, and tools quite literally don't exist. I say this as someone that's worked in large companies and saw how lengthy not only process feedback loops were (especially developer ↔ product team), but also engineering feedback loops (testing, deployment, dev environment setup, etc.)

And I also say this as someone that works on side-projects by myself. The tooling simply isn't there. Setting up fast engineering feedback loops (multi-tiered deployment, test, dev, local, prod, etc. environments, heck even hot reloading or debugging) is needlessly difficult, or janky, or simply impossible. This goes for just about any language/framework out there: from Java, to Javascript, to Go.

It's not surprising that FAANG often designs their own internal systems that handle this sort of thing (from DevOps, to automated testing, to A/B testing, to deployment). I'm a believer that this is also a space that's ripe for disruption. I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now.


I'm going through this right now. I'm helping a customer deploy Azure App Service (similar to AWS Elastic Beanstalk).

Jesus wept. It's sooo fiddly to set up everything. The prod and non-prod environments need dozens of parameters that are all slightly different. Some of these are exposed as resource properties, making them obvious, some are environment variables that are nearly undocumented. Everything is off by default and has to be turned on. Logging. Monitoring. High availability. Secure parameter storage. Staging. On and on....

Oh, you want high availability? Sure, if you pay an extra $3K per month and switch to internal only networking, build Azure DevOps "agent" VMs in a zone-redundant set, and register a bunch of private DNS zones. Good luck automating this. There are template samples available, but they've been broken for years.

Regional disaster recovery? Just duplicate everything! Okay, but not like that! Slightly differently because replication is asymmetric and you may not want active-active. Then you have to layer stuff on top to actually fail over. But not one thing at a time! That's a performance disaster. All or nothing, but that's not an Azure feature, so good luck with that...

This is what I thought the public cloud provides: Some place to upload a ZIP file of code that "just runs" without me having to baby-site the infrastructure.

What's actually provided is a bunch of infrastructure parts that the public cloud vendor will patch for you. Everything else is your problem.


Things are not perfect anywhere but Azure is a bit of a special case. It seems optimized for unskilled people being able to stumble in the right direction by using IDEs and clicking in portal; not for efficiently streamlining development processes.

ARM system in particular is deeply broken in so many ways.

All cloud providers could have been a lot better but Azure is worse than average.

(Yes I have to usw Azure a lot)


Conceptually the resource manager part of ARM is actually fairly elegant.

The implementation on the server-side is a bit messy. Things like read-only properties being mixed in with read-write properties, and internal code-names used for final production systems.

Then the actual client-side tooling is a disaster. The ARM template language is just raw JSON, but isn't case sensitive and includes comments and a bizarre expression language. It's very weakly typed and then even ignores the weak typing that it does have. I've lost count of the times I've been forced to put a number into a string because... reasons.

They're working on a replacement language called BICEP, but IMHO they should have gone down the same path that Pulumi did.


Azure product teams could do with a bit more customer obsession IMHO. And not just the existing customers you’ve had for 20 years with stale IT departments


I recently opened a bunch of support tickets with them because core product functionality is either missing or broken.

For every ticket they've come back with: It is like that by design, go complain on the feedback forums.

Like the other 10 people that also had the same issue and did complain? Their "feedback" was marked as rejected...


> I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now.

Sometimes I conjure up the inspiration to actually implement an idea that's been on my mind. Then I spend half a day trying to setup an environment that just fucking compiles.

I'll spend several days kicking at an open door before I actually write any code. In most cases I just give up before this point.


Do you work with C++? Because that’s how it feels to me every time I decide to go with the language


Are you thinking of the build system specifically?


If you don't mind me asking. What stack are you working with? And how do you approach things?

My experience has been very different (was able to validate ideas and mvps extremely fast, sometimes even in a day/night - but I am very product driven which I think makes the whole difference (meaning that I care to see if the idea works from an end-user perspective first, the rest comes after and only after the project has grown for a while))


Heroku has been the dream all along.

I'm very spike on fast-compiling, statically typed monoliths for orgs of most sizes - especially for product-shaped companies versus infrastructure.

Once you go distributed, so much becomes somuch harder, and the amount of dev effectiveness focused engineers required to keep the effectiveness high becomes prohibitive until you hit a much larger scale. Logging, debugging, scaling, keeping dependencies up to date, A/B testing and feature flags, and perhaps the biggest one: developer environments.


> I want environments to just work. I want code to just compile.

Only mature tools can be complex and still work. But many projects and ecosystems increase complexity much faster than their maturity can set it. I think this loss of balance between maturity and complexity is the root cause of most of the brokenness we experience as developers.


> It's not surprising that FAANG often designs their own internal systems that handle this sort of thing

This is one reason why I've loved using Notion. I maintain a few open source repos, and our team's process has grown organically over the past few months. Being able to use a Kanban board for one set of information, then decide that it's just better as a bulleted list; storing meeting notes from quick bullet points up to detailed step-by-step guides on how to fix some issue people had that week...

The low friction to high flexibility combo is great!


In my experience for any particular aspect of software development, if you ask for advice, you will find 100 people saying about 10 different things.

It sounds confusing and contradictory. And it's likely that 1 of those ten things is much better than the others.

However, the gap between what you are doing right now and the worst of those ten things is almost certainly much greater than the gap between the worst of the ten things and the best.

IOW, just go ahead and implement one of the potentially contradictory improvements that people on HN, for example, have suggested. You will already be way better off than before.

And once you start doing this often enough you will get to the point where you're able to identify the best 2 or 3 of the 10 options presented almost immediately. And the gap then between #3 and #1 is small enough to possibly not even matter.


> However, the gap between what you are doing right now and the worst of those ten things is almost certainly much greater than the gap between the worst of the ten things and the best.

I can't really get behind this sentiment, since in my experience the worst of those ten things is frequently not only ineffective (which would be bad enough since switching costs are very real), but actively harmful compared to whatever you're doing. Chances are you didn't arrive at your current processes entirely in a vacuum, and it's already a mishmash of ideas from the zeitgeist, filtered to not being the worst on all dimensions by the fact that you're still around to observe it.

You really need to be able to pick out the good ideas from the bad ideas, and that doesn't come readily to everyone.


Well, my assumption is that if you're seeking help (a) what you have isnt currently working, and that (b) you're seeking help at a place that usually provides decent info such as HN or Stack Overflow.

Or maybe I've become experienced enough to distinguish between the good and the bad ideas without realizing it and don't really remember the earlier days when I had mentors who would help me navigate those decisions.


A huge unsolved problem in software engineering is that we lack a good way to measure effectiveness, let alone figure out if any practice actually results in improved effectiveness.


> I want environments to just work. I want code to just compile. I want containers to just run. Without me having to start digging through documentation, looking at thirteen Stack Overflow threads, and cobbling a solution that will inevitably break 3 months from now.

Because we are always shifting frameworks and approaches, there are few consistent targets for tools like that. What you might call the "Developer Experience" tooling ecosystem, is woefully immature for this fact.

I would way rather use old approaches with mature developer experience tooling than new approaches with constantly breaking environments and pipelines. But we constantly start all over again with new approaches, and never quite make it to the mature tooling phase...


I'd agree with you, but people were saying this exact thing about automating deployments and then Heroku came along and changed the whole game.

Terraform is trying to solve this but it's still a far cry from "Heroku for infra."


Have you heard of BackStage [1]? A few companies have been adopting it to help make teams and departments to help devs. Target is a notable example of going from a really unproductive environment to much more supportive of devs [2]. Including handling building out infrastructure stacks you mentioned. * 1. https://backstage.io/ * 2. https://tech.target.com/


I think the biggest factor is in this is that so many different tools are being used. Often tools are brought in to make something easier or solve some problem but nobody seems to be thinking about the huge cost that this entails in terms of keeping all of these moving parts working. I think staying with older and more solid stuff that is not in a rapid flux of so-called improvement is often a much more pleasant experience. I think the barrier for introducing some new kind of tooling should be much higher than it often is in practice.


I find it depressing. People talk of an AI revolution but it is super hard just to setup some baseline infrastructure on these "AI" providers infra.


Productivity and efficiency of people is one of the topics that's always fascinated me, and I've both read a lot of material on the topic and have had the opportunity to observe hundreds(!) of organisations first hand thanks to being a consultant roaming from place to place.

From both scientific studies on the matter and anecdotal observation of the same, I can unequivocally state that the top three priorities are:

#1) The high-level goals of the organisation must be rapidly and clearly passed down the management hierarchy so that everyone can "row in the same direction". Ambiguity about goals, or uncertainty in whether one's work is even worthwhile is an astonishingly effective destroyer of productivity. Never treat employees on a "need to know" basis, feeding them the bare minimum that you think they need.

#2) There is no known method more effective at improving overall product quality than mandatory peer review of all code. Not only does this have the obvious direct benefit, but it also helps cross-train senior staff, bring junior staff up to speed, and helps improve consistency.

#3) There is no known method more effective than checklists at improving the quality of manual processes. Sure, if you plan to do something a hundred times, automate it. But you know what is really easy to turn into automation? A checklist. For non-automated processes, a checklist allows junior staff to confidently produce quality. Checklists are a training aid as much as they are a tool for quality improvement.

Of course, there's a very long tail of things one can do on top of those basics to improve productivity, especially of junior staff. Use an IDE. Use a code linter. Use automatic tests on checkin. Etc, etc... But none of those hold a candle to the above three in terms of raw effectiveness.

To put things in perspective: I'd rather have those three things and no source control than source control and none of those three things. You can meticulously track every version of a piece of garbage and it'll still be garbage...


I agree with #1 and #2 and will add the following:

#1 - on the flip side, you need employees who give a shit about your company mission and your clients. Just like a good organization is eager to share these goals, a good employee is eager to receive them. Whenever someone doesn't care, it's a big indicator to write them off.

#2 - Likewise, people have to give a shit and be critical. I used to run code reviews on my team like this - person A reviews person B's code (together, in person) and I sit behind them, not to judge B's code but to judge A's review. It's shocking to me how people want to give code "the benefit of the doubt" and don't flag things unless they are truly egregious. It doesn't help anyone when people hold back like this.


Very good points and eloquently put. I want to add something though regarding #2 - having a code review process is not enough. You need to train people how to do good reviews.

I’ve seen countless times people treat reviews “an approval” by both author and reviewer and for various reasons. People don’t consider it worthwhile, or they don’t want to offend, or they feel offended.

There is countless amounts of advice out there on how to write good code. But there is precious little on how to review code, and even less on how to write “reviewable” code, which is also a thing.

But if people understand the PR review process as more of a conversation, meant to align people on one direction, PRs are incredibly awesome.


Liked your post, do you have any reading recommendations on the subject?


+1 excellent advice, please take the time to add references if you can.


I would really like to, but some of these references are papers I had come across a decade ago, so it would be difficult to track down. I don't research this stuff formally, it's just something I like to read up about.

I did manage to track down this New Yorker article: https://www.newyorker.com/magazine/2007/12/10/the-checklist

I know it's not exactly Nature, but the point made stands, and I did track down some papers related to error rates and different methodologies used to reduce them. There's a reason surgeons and pilots both use checklists. It's because a failure of quality in their profession kills people.

Lives don't have to be at risk for quality to matter!


I've brought this up at work many times, and even had a team adopt it for a 6 month period - but there were so many people on our team that didnt want to do the work of filling out a simple checklist, it was amazing. Great idea, and this is one of the few books I truly remember, and try to apply, all the time (Checklist Manifesto) !


I use literally just Notepad, because it's always available on every Windows desktop and server. On my laptop I use a decent text editor, but trust me, Notepad is sufficient.

I have a simple style that is easy to fill out:

    [.] Commenced high-level task.
        [o] Completed sub-task
        [.] Incomplete sub-task
        [ ] Not yet started
    [-] Cancelled or not-applicable
    [?] May not need doing at all, or pending advice
    [!] Issue or error
    [ ] Include command-lines under the tasks:
        wget https://test.com/foo
    [o] Record output after the tasks if needed -- "Task ID: 929123"
Of course, the aim is to slowly convert something like this into a script or a service desk workflow.

In my experience, it's almost impossible to correctly automate a complex, high-level task given no other information that its English description, such as "build a new web server for a developer". However, if you've built 5-10 such servers manually, diligently recording the tasks in a checklist each time, then the script practically "jumps out at you"...

As a practical example, most engineers will say that to install a new physical server (not including the OS) is about 5-10 steps. I counted 50-100 after developing a checklist!

E.g.: Did you remember to:

Check that the firmware version matches the version of the other servers in the same cluster?

If not, did you update it?

Check the NIC firmware too?

RAID firmware?

System time, date, and time zone?

Secure erase / factory reset the BIOS?

Upload an SSL certificate for the management port?

Register it in DNS?

Register it in the CMDB?

Register it in system-management software?

Burn-in test the memory?

Configure LACP on the NIC?

On the switch too?

I have personally witnessed major issues, even outages, caused by the above being skipped by careless engineers.

"Oops."


> I often help engineering organizations that are in the midst of a transformation. This is typically both a technology transformation and a cultural transformation. For example, these organizations might be attempting to break a core monolithic system into microservices, so that they can have independent teams and adopt a DevOps approach.

It's amazing this fad of microservices still hasn't subsided yet. It is the bane of every company that isn't a FAANG but has management who wants to pretend they might someday have FAANG-level scaling/development problems.


They're doing microservices because they can't make a good monolithic codebase. For some reason, CTOs are very happy to start doing microservices with high level of documentation and rigorous interfaces but completely unwilling to have a well documented and rigorously defined monolith.

I don't fucking get it. It's like one layer of abstraction on top of folders. Instead of breaking up ownership by folders or whatever bullshit - they decide to breakup ownership by github repo.

I'd love to be sold on it otherwise but that's what I've been told directly as the reason for the move to microservices by many companies and what I've seen at my own. Can't do a good job because your business practices suck? Fuck it - microservices, baby!


I think a big reason is that microservice architecture forces teams to establish clear boundaries rather than leave it to team diligence.

For less centralized organizations, I think it can be a useful forcing function.


Clear boundaries on multiple levels as well. I believe a _very_ big part of why people feel they need microservices is because of poor architecture and a lack of boundaries in a monolith. Every single monolith I've come across has had issues with circular dependencies and no real contracts between "apps" (as they are called in Django).

I believe people see ORMs such as Django's as a way to perform one big query as opposed to many small queries. The big query spans 3 apps and 8 models. You have these things sprinkled throughout the codebase and you end up with a big ball of mud.


But then why not just introduce a rule: Django ‘apps’ can only talk to each other via pure Python dicts like an interface? (rather than sharing models)?

This situation is a necessary conceptual step before decoupled micro services and far far easier (because it introduces none of the ops problems of microservices).

But that never seems to happen. I think it’s because codebase discipline rules are harder for management to ‘see’ than API rules.


> But then why not just introduce a rule: Django ‘apps’ can only talk to each other via pure Python dicts like an interface? (rather than sharing models)?

1. One person or group needs to decide on what the rule should be, what the boundaries are and how to enforce those boundaries. This is usually not an easy process, ymmv.

2. After a decision is made, you'll have to decide on how to put the rule in place. Are you going to stop everything and make sure all code paths are working according to the new rule? Are you going to do it incrementally? New features = new rule, old features = common sense? Another variation?

3. You then start running your new rule and maybe you enforce it with some custom static checks, and when that fails you hope your code reviewers will always catch this every time.

4. You still need that one person or group to make sure this actually gets done and to assess if it actually has a positive impact on velocity and reliability.

Or you can put in an implicit "rule" that new code happens in a separate repository, where you don't need to care. A developer that has to do an API call because they simply cannot "import" another module or "query" someone else's data has no choice but to do the right thing - though many opportunities to do a lot of other wrong things too, this is not a panacea. But when we're talking about more than ~5-6 teams, the coordination effort of making sure everyone is aligned on a monolith can be much tougher than to go to multiple services (hopefully not micro).


I think your points are about lacking solid engineering leaders at an organisation that could provide guidance and manage execution of the "modular monolith" approach. If a company does not have leaders who can ensure consistency and discipline across a few team within a monolith, I am afraid of the mess they will run into with multiple services.


None of these problems are solved by microservices. Now instead of undocumented unstructured interactions between modules, you have undocumented unstructured interactions between standalone applications. None of your problems are solved, except now you have a much bigger cloud hosting bill.


That’s not been my experience working with microservices for the past 5+ years. It’s in a team’s own interest to document their APIs and design them well, because if they don’t then it’s their own time sink when other teams integrate.

Microservices have problems but the way they’re often portrayed doesn’t match my experience.

I agree with the sibling commenter who said that a microservices architecture exchanges an organizational expense for a technical expense. For certain organizations, it can be the right call.


I agree though I'd have to think a bit more about the exact way for apps to speak to each other. My guess would be each app could have an api file with a bunch of functions. I wonder how that would work in a situation where you have 3 apps, each with a single model, and the dependencies are A -> B -> C. You could still run into situations where B isn't supposed to know anything about A yet nothing stops B from importing A's api.


It’s still possible to have circular dependencies in microservices. There is normally nothing stopping service A calling B and B calling A except common sense and discipline.

In fact I would say it’s MORE difficult to prevent circular microservice dependency programmatically than in code (where you can hook into module level imports).


The corollary being that the technical problem of a microservice architecture is easier to solve than the management problem of leading a single large team.

Probably some truth to that.


Well said. I think a microservices architecture exchanges an organizational expense for a technical expense. For certain organizations, it can be the right call.


Something to consider: when you then realize that the boundary isn’t the correct one or the environment evolved enough that it isn’t meaningful anymore, that’s really painful and expensive to change.


Microservices and monorepos are unrelated to each other.

You can have 100 micro services from a single repo, and the services are just different folders.

Microservices help people have bounded context when they are develop and they leak their abstractions less into the next one. Monoliths (admittedly I've only seen them in Python, Ruby and .Net) start reaching into each others pieces and adding to tech debt in subtle ways.

You are of course right, discipline is required in both architectures, but there are different emphasis in both.


1 year into my startup and I'm super happy to have kept everything a well-organized monolith. Every once in a while a thing pops up where some voice in my brain (remnants from previous jobs) says "just break it into its own service" but I've thus far resisted. Our deployments are trivial. Our local dev setup is trivial. Maybe some day we'll have a few small services broken off, but today is not that day.


I haven't found many articles from respected sources that will convince management that microservices are a bad idea. There's https://martinfowler.com/bliki/MicroservicePrerequisites.htm... , but this won't convince them.


Here is my non complete list:

- Who’s at the Helm? [0] - Microservices, hard as 1, 2, 3 - You can move it but complexity is still here [1] - Microservices — architecture nihilism in minimalism's clothes [2] - Do I Really Need Kubernetes? [3] - Your team might not need Kubernetes [4]

I doubt that any of those articles could convince higher management. I don't know what a "respected source" actually means. I general I believe that advice from consulting agency and cloud providers should be taken with a grain of salt.

If you drew the same conclusion from the "Who’s at the Helm?" article than I, there is only one way out which is a curated container catalog. VMware's Tanzu Application Catalog [5] is the only product in that space I'm aware of. I have quoted the price tag on HN, I think it was removed since that, so I won't repeat it here again. In any way it was magnitudes higher than the EKS base price which already seen as a showstopper by many small shops. Bottom line is: it is really hard and really expensive to operate containers in a secure manner. Probably too hard and too expensive that it would be a good idea to start with it before you actually need it.

0: https://dlorenc.medium.com/whos-at-the-helm-1101c37bf0f1 1: https://medium.com/@danielpetisme/microservices-hard-as-1-2-... 2: https://vlfig.me/posts/microservices 3: https://thenewstack.io/do-i-really-need-kubernetes/ 4: https://medium.com/faun/your-team-might-not-need-kubernetes-... 5: https://tanzu.vmware.com/application-catalog


What I really don't get about microservices is the performance aspect. Like cache << memory << disk << network in terms of time needed to read/write, and it seems to me that microservices are particularly prone to this by often using HTTP calls.

Sure, you can run them on the same box, but it's very easy not to, and I imagine that this will absolutely crater performance (at least in the DS/ML stuff I do).


Disk is often slower than the network now. You do need to keep an eye on this. However, using tools like gRPC instead of wasteful JSON REST calls makes a huge difference and brings other benefits too.

Typically speaking you're adding submillisecond latency per external call, or you've done something wrong.

Something wrong can include incorrectly organizing your system. Two separate services that constantly chat with one another should probably be considered as candidates to merge into one, for example.

Deployment matters a lot. I see people mocking systems like Kubernetes and tools like service meshes, probably because they don't understand this critical and hugely beneficial capability they bring which is to organize communication on the bases of (amongst other things) locality and system topology automatically.

In the end, most people seem to totally miss the real reason that "micro" services (the "micro" part should really be ignored today) are beneficial. Hint: it's not a technical problem being solved. It's a people problem. Service oriented architectures give very clear boundaries around which a development organization can structure in a way that allows parallel delivery.

The biggest problem any growing organization will face is: they don't pay attention is contention (yes like as in concurrency/locks/etc.) across the development team(s).

I have personally witnessed and worked with a 400+ person monolithic development organization. This was essentially a "unicorn" and an exception that proves the rule. The one and only reason this organization was successful was because they had top to bottom extreme coordination managed by a very small group that worked effectively together. Without that tight central orchestration it would've been a nightmare.

Most organizations can't orchestrate a 7 person team effectively, good luck getting 200 to share a monolith well. So you break up the org and the architecture follows, see: Conway's Law. It goes both ways, too. You break up the architecture and the org can follow.

With that in mind, it's extremely important to get your boundaries right and for the love of all that is sane in a daily job, don't break it up too much. So many organizations would triple or quadruple their productivity breaking a monolith into literally 2 parts. If you can do 3-5 that's cool, amazing! Just because you can see the dotted lines between 25 different potential services doesn't mean your organization will tolerate that.

Anyway that was a diatribe.

TL;DR: "micro" services solve PEOPLE problems, moreso than technical ones, and the price you pay is additional architectural and delivery challenges which you must balance well and plan for


As both an IC and Manager, I've now been on a lot of teams using monoliths and teams using microservices, and I would always choose microservices going forward.

Let's be straight, both have pros and cons, but I value the following pros of microservices a lot more than any of the cons:

- Enforces separations that are important for: - Maximum parallelization of work. - Scalability of each service. Never share database. - Reusability of the service from other services.

The trick is to learn how to cut microservices though. Establish an interface that it exposes, like GraphQL or RPC. Don't try to split too fine-grained, but find a balance that works for you.

I promise you, when it comes to scalability, you'll be glad you don't have to scale one massive database just because one part of your application is seeing more activity.

This also allows you to pivot quicker to a different database suddenly, or a different programming language, if requirements change. E.g. Node.js -> Rust if your service is found to be on the hotpath of everything, and you need correctness and performance.

I fundamentally view the difference the same way I would choose Rust over Clojure:

- Monoliths: Rely on discipline of the team to do it right (like Clojure won't catch your dynamic mistakes) - Microservices: The most important parts you need are built-in and enforced by the concept itself (like the Rust compiler guides you towards the correct approach).


While I agree with you on unreasonable rush for microservices in many cases, I do not think a company must be on the FAANG-level scaling to justify microservices infrastructure.

Modularisation is an efficient tool to address growing complexity of software. The code is organised in closed subsystems - modules - that work together via communication with each other over defined interfaces: functions, REST endpoints, messaging system.

Modules can be done in monolith by enforcing boundaries and policies: namespaces, packages, agreements on communicating only via "service" classes, isolating database entities, no foreign keys between entities belonging to different modules, etc. It is so called "modular monolith", and I believe it's an efficient way to progress for organisations before jumping on microservices train.

Microservices are the next step of modularisation and isolation. By introducing independent deployment of the modules and isolated runtime for each module (different machine, VM, containers) they enable independent CI/CD for the teams that helps to scale up the organisation and they help with isolating different pieces of software that have different technical requirements (traditional Web API with database calls, stream and batch data processing, etc.), different stability and change frequency.

While bringing a lot of additional concerns, requiring education and discipline, microservices are critical enablers for many companies to further grow their products and scale up the technical department.


A) Short-term mindset developer:

- Tech debt is for others to fix. While others cleanup after me, I will be completing my next task and paving my way to promotion.

- If it works, it's good enough. It does not matter if I can't explain why it works.

- Everything is an obstacle. Documentation? obstacle. Coding standards? obstacle. I just want to complete tasks.

- I do exactly what I am asked for. It does not matter if it is insecure, it does not scale, or if the entire system crashes. All of that will be someone else's problem later.

- Code does not have to make sense, as long as it runs it is good enough.

- I worship project management.

B) Sustainable mindset developer:

- The cost of tech debt compounds over time. I should fix tech debt before more code depends on it and becomes more expensive to fix.

- I need to understand how and why things work. Is this really working? Do I really understand what I am doing?

- Reviewing relevant documentation for the technologies I use is a good idea. Making sure I understand and follow coding standards is a good idea. Well crafted coding standards can save work.

- I understand that not all stakeholders are technical. I need to understand the technical implications of what I am asked to do, and push back if necessary.

- Code is written once, read many times. Readability makes everyone more productive.

- I acknowledge that project management documentation and tools are a form of documentation, and as such, it is not a source of truth. When it comes to things to do, the implementation is the source of truth, not tickets.

Bad companies promote A), good companies promote B).


There's also a C) Pseudo-sustainable developer, who thinks he's doing B but really over-investing in optimizing and overcomplicating (and over documenting) that which isn't critical and thus wasting a ton of time.


This sounds like a junior dev on the path to B). I think the desire to do the right thing says a lot and finding that right thing takes time.


I am less optimistic. A good developer documents the right parts because he has the understanding that this is helpful. An under-documenter doesn't get why that's helpful. An over-documenter doesn't get why it's helpful either, that's why he does it all over the place.


Good on them for puttin in the effort, at least. Some of us just show up for the paycheque


It takes more time and dedication to become effective at B). And at first, you may get things wrong, but the approach does pay off.

Someone that follows approach B learns more in a shorter amount of time. Because learning (advancing the frontier of your knowledge) happens when you encounter that limit, recognize it and try to move forward. Rather than dismissing it, which is what A) does.

B) always thinks: what is a better way of doing this? A) never does. Therefore, over time, B) becomes more effective than A).

In mind sports it's the same. You learn more in slow games when you take longer to think your moves. Every pro will recommend you to prefer slow games over fast games.


This doesn't really match my experience with people. You can be dogmatic about, for example, code comments - and dogmatics are not thinkers.

I used to work with a guy who was representative of this dogma. He'd give people a hard time if they didn't comment every line of code, and his own comments would be shit like:

  if (country == "Japan)     // If the country is Japan
    total_cost = price       // set total_cost equal to 
  price
  else
    total_cost = price + tax // Otherwise, set total_cost to price+tax
He thought this was great, and was upset when I'd tell him this is completely useless (comments don't provide any information over the code) and it's missing the one thing that would be good to actually document: why is Japan different?

People like this heard somewhere that comments are good, and thus good people write comments, and if I am a good person then by gosh I am writing comments. They don't think about utility at all.


Your code example doesn't represent approach B). You are conflating verbosity with readability.

A better example of B) in this context would be understanding that IEEE 754 floating point numbers cannot be used for currency, and perhaps adding a comment about that.

Meaning, really understanding what you are doing all the way, and understanding when it works and when it doesn't.


I've come around to commenting as if I was Statler and Waldorf.


Spot on. An experienced developer understands that incurring tech debt and paying it down is a balancing act.


Did anyone else notice that there is no software engineering management involved in the "ideal" picture? In this ideology, every developer acts autonomously, even by deploying to production. This removed all technical responsibilities from the well-payed management people. Conflicts? Resource management? Scheduling? Happens to other people. At the core of this ideology lies the principle that every software is just a bag of fully orthogonal features that can be implemented completely independent of each other and will play nicely together. If this assumption does not hold, it's the developer's fault (aka "ownership").

Dear developer colleagues, do not fall for this trap. Hold your managers responsible. Let them make the difficult decisions. Real leadership should lift some burden from the people being lead.


I'm presuming in this ideal land, the stories are already ordered in the most optimal way. So no scheduling conflict.

Resource and release conflicts don't appear if you have the right infrastructure.


Are you serious? The "right" infrastructure? Are you limiting your view to a single service, running on a big cluster? Because even then there are permanent conflicts in the persistence layer and resource conflicts express themselves in huge infrastructure bills.

But things get really messy when your software is distributed in heterogeneous environments. Say you ship a database yourselves. Or a multi-platform app, or a game, or the control software for medical devices, or...

Software development is not just running the next SaaS product.


Give me some concrete examples.

Higher infrastructure bills for spinning up new hardware is often cheaper than fixing resource conflicts


Ooph, just reading the highly effective vs low effective environment bullet points was triggering. I can think of environments I entered where a good chunk of it was highly effective and my most recent startup was plagued with the low effective one (and even then - it will still IPO). Worst part is - they really had no interest in improving it. To improve would require an entirely different management chain - one that didn't learn all their ideas in the 1990's from watching rocketships take off...

I sometimes wonder who these articles are written for. Am I supposed to share this with the CTO to show them how poorly the organization is being run? No, obviously not, that'd just fast track my firing. After all - if you're in an environment of psychological safety where you can air these issues then you're probably going to not have these issues very much.


>> I sometimes wonder who these articles are written for.

If you're a Leader (big L) in an org, this gives you a way to asses your organization. If you're a leader (small l) it may give you a way to concretize your thinking around problems so you can discuss them better.

>> Am I supposed to share this with the CTO to show them how poorly the organization is being run?

That would be the worst way?

Why don't you pick whichever problems on the list resonate the most, come up with a good case as to why it's a real impediment to your team, ideally quantify its impact, perhaps come up with some solutions, and then go have the conversation?

Ultimately, it's not important whether your org looks good or bad against some checklist, the "meat" is whether you have an opportunity to be more effective against your goals. If so, people generally want to have that conversation. Especially if you go in with a realistic understanding of why things are this way and the tradeoffs.

>> and even then - it will still IPO

Sounds like they know what they're doing then?


I find this hard to justify to myself. It’s literally my boss’s job to fix these impediments (and the boss’s boss, and their boss). Am I supposed to teach them how to do their jobs? In addition to doing my own?


While I get the employee/employer/manager relationship can be a bit shit a lot of the time.

This attitude is likely contributing to whatever problems you are encountering, and they are potentially trying to fix.


If you are a senior member of the team and problems are apparent to you, I would expect you to productivity call them out.

It sounds like you want to gripe about the problems on HN but not actually try to solve?


No, no. I’m talking about bringing the problem to some higher ups attention, but also needing to expend the effort to both come up with a plan for fixing it, as well as actually convince this 50 person team to do this thing.

And then still have nothing happen because the people above are mostly paying lip service to improvement, but not actually changing their ways.

It’s like needing to expend 200% effort because others are expending only 10%.

Edit: But yeah, I also gripe about it on HN because it’s carthartic ;)


It's a little hard to follow you but it still sounds like this to me: there's a problem I am seeing, others are "bad" for not fixing it but I am "justified" in doing nothing. So it's not surprising nothing gets done.

For what it's worth, my strategy has always been to grapple with these situations. Sometimes you get to have impact and learn in the process: eg how do I communicate about this problem in a way that gets traction? If you can do it, you will "level up" in your career.

Alternatively you may learn why things being as they are is a reasonable trade-off for the situation you are in. This may also level you up by helping you think on "what makes sense for the business" rathe than simply "I don't like it" level.

At worst, you'll have a story for your next interview about how you tried and failed to make a big improvement like this. I have a similar story and it resonates really well for senior roles.


Rocketships don’t need to know what they’re doing to succeed.


Is that so obvious?

I bet there are way more failed businesses that had followed meticulous software practices than there are 'rocketships' that took off despite(?) what may appear to be bad practices.


I don't think I agree with his feedback loop chart.

He claims 5-15 seconds to validate that a local code change works is highly effective.

But in practice, that amount of time makes for one of the most unfriendly environments to be in.

For example having to wait 5-7 seconds for an AWS SAM (Serverless) Lambda function to be built and locally invoked is so much worse than making a code change in a Flask, Rails, Django, Laravel, Node, Phoenix, etc. app and seeing the change as fast as it takes you to focus your browser and reload, or running something in a REPL.

When you do such a thing 100 times in a few hours those 5+ second pauses are deadly for motivation and productivity. It's a constant reminder at how crappy the development experience is and it's also not long enough to do something else while it completes so you're stuck wasting your life away on a tool.

Is anyone really happy when they need to wait 10 seconds to see a change when developing a web app?

I think the web context is important because you can relate to it from past experiences. In 2001 there was a near instant feedback loop with making a change to a PHP page and reloading the browser. Anything less than that 20 years later seems like a direct downgrade, especially considering computers are probably 1,000x faster today.


> Day in the life in a low effective environment

> notes that a previous feature has been approved by reviewers, she moves it into another branch that kicks off a long nightly E2E test suite that is almost always red, managed by a siloed QA team.

This is not a low effective environment. A low effective environment does not have a large e2e test suite. It does not have a siloed QA team, and people write little to no unit tests, never mind integration tests.

If you are in a "low effective environment", a "siloed QA team" is a godsend. At least someone is responsible solely for quality, as opposed to everyone churning out features as fast as they can. Disposing of the siloed QA team in such environments rarely helps matters.


It’s all relative.

Relative to the highly effective teams the article is describing, the points on the spectrum you discuss here are “low” and “lower”. Sure, low is better than lower. The article is making the case for “high” though.


Anyone who claims to teach paths by which to optimize Developer effectiveness ought to cite data to support those claims. Until data exists to contrast the results of Company A vs Company B, we'll never be short on speculative articles.


I know, that's why Aristotle is terrible... No citation or data... are we just suppose to blindly follow the scientific method? Preposterous!

Reading through Fowlers "Good Environment"/"Bad Environment" lists... I can use my own critical thinking to see why one is good, or bad. Fowler while he doesn't really get his hands dirty with these problems he is very good at rounding up the current zeitgeist and common sense in ThoughtWorks and publishing it to the world. He's rarely been flat out wrong. (I don't mean this as an appeal to authority, just how I look at this article, and Fowlers publications in general).


Sure, but if you're a consultancy, like Thought Works, you're under watertight NDAs on everything. I work for a consultancy that helps and assesses companies too, some of them selling for north of a billion. We can share general, anonymized, observations, and I would argue some of those are very valuable simply because of the breadth of situations we get to see. But we legally can't cite specific instances. That's the biz. :-/



This has been my primary focus for two decades, I litteraly can't work in slow environements:

1) I encourage people to use JavaSE on the server and hot-deploy to the servers directly with async. non-blocking: http://github.com/tinspin/rupy

2) For clientside: mostly C syntax (compiled with C++ compiler) is your best option and I recently made a in-app debugger for Windows: http://move.rupy.se/file/stack.html (this delivers something much worse than a Java stacktrace but as good as it gets without a VM).

I also hot-deploy the C/C++ app code with a .dll to my .exe and the debugger works for that hot-deployed .dll too!

On linux the .so hot-deploy works as well, the only reason I have not taken the addr2line source to port the in-app debugger is that I know which hardware I'm working on as I only plan to ship linux on ARM! Fight features where you can, in this case limit your porting and exposure to unknown unknowns!

Potentially I can hot-deploy the client .dll/.so over my hot-deployed server pipeline, making the platform a distributed real-time system where you can patch the native code in real-time remotely while users are using your app! Mostly usable for development I guess (¯\_(ツ)_/¯), but still really exiting!


Tools and stuff around the code will still not allow an effective environment.

Code design is a huge bottle-neck. I've been a part of 10+ engineering teams that have all the right tools and stuff, everything around the code is really frictionless and runs well (CI, tests etc.).

But the code is so badly written and designed that adding new features needs edits of dozens of files and sometimes even copy-pasting files (because of circular dependencies and singletons everywhere).

I guess existing developers can be effective in this kind of environment but getting new ones to contribute will still be a chore.

Although, even existing ones, when they forget what they did, start getting difficulties creating new functionality.

For example, the biggest issue I've seen is "dependency injection" mindset. It's a mindset that dependency injection library solves the dependency issue and you can just inject anything you need anywhere.

The entangled mess that this creates is horrible. Yeah, all of the objects initialize correctly but not thinking about separation of concern, proper encapsulation or what really needs the whole dependency will create a mess.


I am the first to ask for quick feedback loops, which makes the dev environment so much enjoyable. But I have also seen some detrimental long term effect of this.

I have seen developpers "coding to the test". By that I mean they are modifying a piece of code they do not know well, and assume that if test pass, that must be good. Without understanding that test will never cover all possible inputs/states of the system. This "coding to the test" can appear very fast with a quick feedback loop, making possible to "monkey code" something until it passes. If you do not have careful review, by people knowing the system, this will end badly, with race conditions only appearing in production, integration failing randomly.


I call this "silver bullet consulting".

The reality is that their customers have slapped a developer title on anyone willing to take their 80k/year job. And then they hire "product managers" who fill out schedules with arbitrary dates. Then when "dates are missed" meeting are added and developer training is added. That makes productivity go down even more. In come the consultants who will fix everything with a change to processes instead of being honest and saying that they need to fire everyone and start over from scratch.


How do you convince management to focus on improving workflows? There's often a huge focus on shipping features with that being the primary criteria for promotions.


There might an infinite amount of things to fix, but you have to first come up with one specific improvement to the workflow that is straightforward to implement, that can be clearly measured to improve rate of shipping features. Then convince your manager of that idea, usually by getting him/her to think that they came up with the idea on their own. Then keep on repeating this, and people will be able to accept more and more process improvements that have less and less direct ties to shipping more features, because they've seen the results of all the improvements along the way. And then eventually your manager gets promoted for doing such a great job on improving productivity in their team, and because they like and trust you, you get promoted into their vacated position.


send them this article


Q: Does anyone else find the body text hard to read?

Small and light grey text on a white background really doesn't work for my aging Mk I eyeballs.


I am on the precipice of starting a new project at an organization and it is a much larger and more diverse project than I have worked on before. This article provides some good food for thought, but is anyone fond of particular books or articles they've found particularly helpful?

I have worked in small teams of highly-effective teams generally. But I am about to embark on a much larger team consisting of a large range of skill levels. It would be nice to lay some effective groundwork from the start that isn't overly cumbersome.


Mythical Man Month by Freddie Brooks is pretty good but is not a quick read.


What I love about shit like this is they're walking around going "If you do X your productivity will increase."

Assuming this is true, you're still fucked, because nobody can force your executives to then force their employees to do X. If the executive doesn't tell people to do it, nobody will. So knowing what makes for good productivity is still worthless unless a handful of chuckleheads at the top actually make it happen. And they won't, for the same reason nobody below them is doing it.


> Very sensibly, most companies are on a journey towards achieving this environment

Yes, until they realize how much of a crock it all is, and go back to actually getting work done.

Backstage looks interesting though, so maybe I'll get something good out of the article.


So CI, automated tests, SCM, small patches, API docs, stand ups and the ability to take breaks.

This is not a “highly effective environment”, it’s the bare minimum to have a functional environment. Might have been good advice in the late 90s though.


IMHO the article has some good points. E.g. I would guess a lot of organizations use ancient build tools ( maven always doing a full rebuild) and try to mitigate slow build times by going to towards small Microservices.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: