I feel like some of the earliest advice and an early learning principle was:
- design for reasonable extensions in the future. Is your design modular, will it be easy to add features? Did you solve problems in a generic way that could be re-used for other other solutions? For example, instead of doing a graph traversal, did you solve the problem by first writing a graph traversal library?
I don't think it's quite as widespread, I would posit that this has changed to:
- (1) YAGNI, build the simplest solution you can for the problem at hand. That will make it as easy as possible to change when future requirements are presented. (2) Design your code to be replaced, not extended. Design your modules in a way that someone can rewrite one without having to rewrite the whole system.
Points (1) & (2) really contradicts the generalized solution approach. The generalized solution tends to be very abstract, hard to understand, and because there are layers - not very modular (which makes it hard to re-write)
So I was told this as bluntly as possible by someone 15 years ago - "you don't have to code all parts of your design".
Every line of code you write makes it harder to replace it. That the feature is an asset, the code is a liability. That the goal they had set out for me was to write as little as possible, but achieve the same result.
The only reason to rewrite a system out of tech debt, which is what I was doing then & getting lost in a sort of second system "this is the last rewrite" effort, is to write the next feature with less code - even if the next system needs you to rewrite bits you just wrote.
This might sound simplistic, but I've seen that work over and over again.
Plus, it was much easier to hand over the code to the next person when I wanted to do something new and interesting, if it was functionally complete and not full of half designed extension points all over.
> "you don't have to code all parts of your design".
That's an excellent articulation of what I think as "invisible seams" when I write code: they're soft points of extensions that don't need a separate interface/function/class _yet_. Sometimes I just mark them for myself with an extra newline within a function.
> So I was told this as bluntly as possible by someone 15 years ago - "you don't have to code all parts of your design".
I would say that's more false then true.
Why? Because the design you haven't coded is a fantasy design. I'll give you a counter-quite, and excuse the military context: "No plan survives contact with the enemy" (simplification of a longer quote by Helmuth von Moltke.)
I have yet to make any non-trivial design which does not require significant changes while it is being coded. Perhaps some design geniuses with much deeper insight can make this happen - but those are very few. Most people probably need several rounds of implementation and user interaction before reaching an acceptable design.
So those parts of your design which you haven't coded are likely partially irrelevant and un-implementable.
This is not to say that you should write huge amounts of code. But - try to avoid designing much beyond what you actually code.
You shouldn’t code all of your design precisely because “no plan survives contact with the enemy”. The bits you’ve designed but haven’t coded yet are much easier to change in response to future requirements.
I think it's important to keep in mind that there are many different types of "design": requirements design, interaction design, architecture design, code design... which inter-relate and are often worked on in parallel.
We've had a few people who like to do the first option. They would build a semi library inside the app to solve a problem and then when they move on to another company, no one really understands the design principals behind it and make changes that make it not so reusable and years later it never ends up getting reused for anything new anyway so we are just left with a very abstract, pseudo library with no documentation and years of tacked on changes.
Now we advise people to build the simplest and most obvious version for the problem you are looking at right now and if in the future you need it all over the place, replace it with a reusable one.
The exception seems to be React components, those generally seem to be very easy to identify as reusable UI and can be made generic from day one.
Good code should serve as documentation, in addition to serving a functional purpose. “Programs must be written for people to read, and only incidentally for machines to execute.”
Depending on the use case, this may not obviate the necessity of plain-text/wiki documentation; but I do note a tendency for many wikis, documentation web sites, etc, to be incomplete and/or woefully out of date. Half-assed docs can almost be worse than no docs; if it’s a priority, then it has to be folded into the entire SDLC, or have its own dedicated staff.
There's something in between those two sides that relieves the tension. You don't have to build a graph traversal library and you certainly don't have to publish it. My opinion is that while solving for the immediate problem, you slice off that bit about traversing graphs and handle that generic problem on its own. Now, you only have to think about that one narrow thing. This is also makes it much easier to modify or replace, no need to extend. If you don't do this, the overwhelming tendency is that the graph traversal algorithm snakes its way through your program and is a huge pain to change and reason about.
If you don't do this, the overwhelming tendency is that the graph traversal algorithm snakes its way through your program and is a huge pain to change and reason about.
Indeed. There is a balance to be found. Over-engineering and using the wrong abstractions hurts. Working with an “organic design” that has no useful separation of concerns, or forever reimplementing the basic data structures and algorithms of whatever domain you’re working in instead of doing creative work that is going to solve your real problem, also hurts. Too far towards either extreme and development progress slows to a crawl while quality plummets.
I wish we had a bit less emphasis in the industry on always using the latest shiny tools and a bit more on learning good fundamentals like data structures, algorithms and design patterns. Of course abstractions sometimes need to evolve as requirements do, but identifying which abstractions are helpful at present or when an existing abstraction isn’t helping any more and should be removed or replaced is absolutely a skill that can be developed with knowledge and experience, and it pays huge dividends to those who cultivate it.
I think both can be true. I practice and encourage “solving for a class of problem,” but typically where I see people generate technical debt is when attempting to apply that advice to domain/business logic that isn’t and shouldn’t be very abstract. Apply it to the things that make your app go, such as white-labeling or bulk inserts. Do not apply it to things like your new employee groups feature or payroll calculations. Applied to those you just add additional and arbitrary rules that have to be obeyed or else the abstraction begins to leak. And it always leaks.
At work I'm looking at exactly the simplest solution I could have written for the problem then at hand, and now that requirements have changed I'm going to have to rip it all out.
I'm curious, had you written a generic version, would you still be ripping the code out? (I'm guessing so, otherwise presumably you would now be making that code generic).
I still find it funny that the place to optimize is generally for 'rip-out-and-replace', rather than 'extend' (caveat emport, not always the case, but seems to be a better design to optimize for replaceable than it is to optimize for abstract & extensible)
We built to the wrong abstraction. We had a choice of two optimisation variables that were conjugates (i.e. you can convert from one to the other like frequency/time, momentum/position etc). We chose the one that was simpler to implement, it turns out the other one is actually more expressive and useful for our purposes.
The problem with the "generalized" approach is you often only have one data point to generalize across. If you try to focus on solving problems you don't yet have then you end up writing a lot of useless cruft.
If you write clean code that is easy to groc and easy to replace, then its also easy to extend once you know where you want to extend to.
"generalized" means the most simple code you can come up with. This simple code is then used thousands times from very different programs, and it never changes.
I think implicit in the point, most code does not go on to be used thousands of time (let alone dozens or even more than once). Even if some code is used more than once, making it the same and shared has significant costs as well.
Never said it is easy to build simple reusable code. Code itself must be simple but process behind devising it may not be. If the code is not simple it is more prone to error.
I see this kind of opinion a lot, and it's clearly resonating in the comments. I'm beginning to think most people are just very bad at writing reusable code. Despite all the negativity, it is actually possible to write highly-reusable code that is clean, understandable, and not "over-engineered". Why have you all given up on that?
My take is simply that the easiest code to repurpose is small code - because small programs are easy to read and easy to change.
Putting something behind an interface / abstract factory / whatever ironically often makes it harder to change because you have to read more lines of code before you can start modifying the program. Programs with a lot of lines of code are intimidating.
There are situations where you have clean module boundaries, which lets you separate your code into separate independent modules. (Eg POSIX separating user applications and libc / the kernel). But the downside is that once you've separated your code into modules (with documented APIs) the module boundary itself becomes harder to change. You're essentially betting that you have the right API. You lose flexibility at the boundary in exchange for flexibility in the code on either side of that API.
But all that said, I don't think there's any way to learn this stuff without experience. "The dao that can be spoken is not the eternal dao". There's no set of rules for this stuff that can be explained simply, and which will give you the ideal outcome in all situations.
Putting something behind an interface / abstract factory / whatever ironically often makes it harder* to change because you have to read more lines of code before you can start modifying the program.*
I think the “secret” is proportionality. A useful abstraction hides significantly more complexity than it introduces. That could be a single function, if that gives a name to a simple but common algorithm on a particular data structure like map or reduce. Or it could be a 100 function API with 100,000 more lines behind it, if that provides a correct, efficient implementation of some entire field of programming that your application rests on. But it’s probably not a one-line function that needs a longer name to describe what it does in words than its whole implementation, and it’s probably not a library with a 100 function API that only obfuscates a few standard data structures and custom types that are locked away in the implementation for little benefit.
> it’s probably not a library with a 100 function API that only obfuscates a few standard data structures and custom types that are locked away in the implementation for little benefit.
Thanks for expressing this - I think this is a fantastic insight. I love C as a language, and I've been really enjoying Rust lately too. Both languages make it easy to design around simple, raw structs. And a simple struct with 3 public fields is often a much easier interface to work with than 100 getter & setter methods hiding the same struct which has private fields.
I have more success writing reusable libraries that do very, very fundamental stuff or add ergonomics to existing features. For example, in Go when generics landed I immediately wrote a few small libraries using them. They're in use all over the system I work on.
When I try to build similar abstractions over business processes, they typically don't survive the next wave of changing requirements.
I have more success writing reusable libraries that do very, very fundamental stuff
I concur. If we look at abstractions that have stood the test of time, they are often relatively simple to describe, but they express powerful ideas that are widely applicable. A few examples come to mind:
• Structured programming
• Iterators and generators
• Promises
• Atomic transactions on databases
• Model-view-whatever patterns in UI code
• Regular expressions
• Pipelines
• Continuations
• Haskell’s standard typeclasses (particularly Functor, Applicative, Monad, Monoid, Foldable and Traversable)
Many of these have proved so useful that popular languages now directly incorporate them or perhaps, for the most abstract ones, incorporate features that are more specific instances of the general idea.
I think the last example is particularly telling. The patterns Haskell developers work with are in one sense very abstract: each is ultimately just a short list of mathematical properties that some type can have. This turns out to be quite a high barrier to entry and understanding their true nature is something of a rite of passage that many fledgling Haskell developers never complete. It also took some very smart people many years of thought and discussion before the community eventually reached the list above.
However, once you do recognise the patterns, you see them everywhere. They create some very clear seams in your software design that allow separation of concerns and composability to a degree that less powerful abstractions only dream of. And because they’re rooted in relatively simple properties, they’re about as stable and future-proof as anything in programming can be.
Maybe one day they’ll be supplanted by a more effective abstraction with better language support. Effect systems, perhaps? But for now, you can express ideas in a few lines of Haskell that would take 10x that in many other programming languages, because you can compose tools from a vast toolbox of data structures, algorithms and design patterns in almost arbitrary ways. But you have to learn some abstractions that aren’t widely known and have silly names first. :-)
As an old coworker used to say: code reuse in the small (libraries) and in the large (databases) are solved problems, but code reuse in the medium is Hard. I agree, though I find it difficult to understand or explain why.
p.s. I also wrote that same library when go generics landed.
The team manager at work likes to bring up the list of "time investments which we will benefit from later" which we never benefited from later. It's a good point that usually its better to wait until the benefits are obvious rather than predicting at some point there will be some benefit later.
My general strategy - usually applied on a much smaller scale - was the first time I needed to do something similar (or the same) I'd copy the code. The second time, I'd consider abstracting it. By that time I had three applications and could better identify what was common and what was specific to a particular usage. I wonder if something like that can be applied to larger libraries.
> it is actually possible to write highly-reusable code that is clean, understandable, and not "over-engineered"
You might be able to do that. That code (most likely) still needs to solve a complex issue and needs documentation to survive next to it. Also, there is a good chance that the next guy coming to fix it is not going to investigate the extra time to fully understand the code if he can fix that one crash by throwing in a hotfix at the specific line accessing a nullpointer. Lastly, at some point, if you need to solve a difficult problem, the code to do so is likely going to be more difficult to read, too.
It's not impossible to do this, but having the time to do a clean solution, having the foresight to be right about the necessary complexity and also have the environment for it to stay clean is simply rare.
> it is actually possible to write highly-reusable code that is clean, understandable, and not "over-engineered"
This is true for systems where you already have working code and you have multiple concrete examples of code that would benefit from refactoring or rewriting a reusable module.
I think the skepticism is more toward the idea that you should design for reuse up front before you have working code. There are two failure modes. One is that you spend time making something reusable that you only end up using in one place. The other is that you bake assumptions into the code that later turn out to be false. In both cases, the resources spent on reusability could have a net negative impact on the project.
Part of the problem is discoverability. As the library gets larger, without adequate time invested into docs or standardised naming to enable faster code-based search, people can end up writing their own functions when one already exist. Often these are not discovered unless PRs are reviewed by someone that either wrote or has experience with a reusable function.
People where I work tend to act like their code is going to never be changed later. Like yo, I could rewrite my thing 5 times in the amount of time it took you to build the "generalized" solution that'll probably break anyway. My entire job the past 4 years has been keeping some overengineered thing on life support until its author leaves the team and I nuke/rewrite it, and one of my teammates is like me.
Where I work you decide whether to produce over-engineered or under-engineered crap prior to writing it and people cannot waste time on a rewrite just because someone left.
Where I work everyone decides to rewrite everything they inherit and nothing ever gets delivered because the business loses patience and reassigns priorities as the rewrite drags on.
2. It is inter-dependent with code you can't rewrite 5 times
3. You're probably no longer there anyway.
So I'll have the generalized solution, thank you. Of course, it's even better if you can use an already well-established FOSS solution, since that's likely to see improvements over time which may be applicable to your code.
> 2. It is inter-dependent with code you can't rewrite 5 times
We have a CSV parser library in our codebase. It’s probably from 2005 or so. I’m sure it was a good idea at the time but at this point it has resisted replacement because of dozens of legacy reports that rely on bugs or quirks in it (mostly around quoting).
In 20 years, it's gone, the business requirements are unpredictably different, probably the language I wrote it in is gone too, maybe the entire company as well. I'm writing application code, not a kernel or something, but even kernel pieces have API boundaries and can be replaced. An API can live 20 years, so I take that far more seriously.
Hah. For a startup I wrote some code using interfaces and some (what I though to be) reasonable abstractions to future proof us to an extent. Some of it remained quite low level where appropriate.
All the other dev leaders hated it. They said it was too complex and they couldn't iterate on it quickly enough. It took one of them 2 hours to change a fundamental bit of it!
They then (after I left that position to work on the cryptography) proceeded to rewrite the entire thing using domain oriented design. It was a beautiful set of interfaces. It was about 100 times less efficient (running on mobile phones). It took them 6 months to rewrite it.
I figure that everyone just has their preferred style. The right amount of abstraction is whatever you wrote!
A lot of the people who complain about the bad code of legacy systems don't understand that their code has a good chance to be legacy in 20 years. Makes me wonder how well the "microservice/k8s/many third party packages" apps will age. I suspect not well.
I am always shocked when I learn the some code I wrote as throwaway-tp-be-done-right-later 10 years ago is still in use.
All those stupid front end libraries will be the new mainframe.
>Guys I’ve been stuck with this application that says its a web app doesn’t render in the browser engine. It has its own version of a browser engine that’s from thirty years ago and it needs 200 dependencies from a repository that doesn’t exist anymore. How am I supposed to maintain this?
I've generally found pair programming to be more useful than code reviews. It's too hard to drop in at the end of someones weeks of work and try to meaningfully review it. And a lot of suggestions you might have would have been useful at the start of the work but aren't worth going back to fix.
If you actually sit along and see the whole process you can actually understand the whole piece of work as well as give suggestions when they are useful and much cheaper to action.
All OCD and preferences should be encoded in the linter/formatter or a wiki page. It's obviously a waste of time to argue menial preferences in review.
Some of the documented preferences are annoying too, but at least you can adhere to them without going back and forth with a reviewer. What builds friendships is when you review each other's code in a way that ignores the parts of the style guide you both consider silly.
The xkcd cartoon hits on an orthogonal dimension, almost universally ignored, which is why I liked it:
It's an economic endeavor, not an artistic one. I've been in some choruses, and it's really worth it to make every single thing as good as you can possibly make it. The audience for the concert appreciates the totality of what you do.
A coding project is not art like that. It's not worth making most things perfect, and moreover, you can't tell now which ones are going to be worth it.
Every hour you spend on polishing something that's going to be thrown away in two years is an hour you can't be doing something more valuable. The Net Present Value of extra work five years from now is almost zero.
The goal of a code review should be to establish shared ownership. If that is valuable to your engineering org or team, then code reviews are valuable. Often teams don't have the time or discipline to treat reviews that way, in which case they are not valuable (or negatively valuable because they waste engineering bandwidth). A practice that I've seen work well is for the engineer writing the code or the team in general to have enough situational awareness to realize when something really would benefit from having immediate shared ownership vs when something doesn't and the responsibility falling on the code author or otherwise team to determine which pieces of work should be more thoroughly reviewed.
One of the goals of a code review should be to establish shared ownership (or at least shared-semi-competence).
But code reviews have at least one other way of contributing value. They help find (moderately) obvious bugs, unsafe code, and unclear code. There is value in keeping blatant stupidity out of your codebase.
I said you get the second order benefits by establishing shared ownership. Of course the qualities of the code are conferred to callers… that’s obvious.
I mean, the thing about a quick hack job is the person building the code has a very well defined problem and knows exactly what will fix it. And so that's almost a perfect requirements to solution match.
Code is liability, guys. Functionality is value. And total value is the integral of the instantaneous value, and total cost is the integral of the instantaneous cost.
Engineers frequently have trouble with this. They place the code on the wrong side of the balance sheet. They place it on the assets side. It's on the liabilities side.
I hacked together a deployment system in Perl when I first started at reddit. It was the first deployment system reddit had (previously they would just scp the files to the servers by hand).
I heard it wasn't retired until 10 years after I wrote it (which was about 6.5 years after I left).
I wrote an amazing feature for a client's website. Saw how awesome it was going to be, wrote it so we could use it on other websites. Even pitched it to my boss on how to pitch it to another client. That code shipped and was never used again.
Generally, the more (over-)engineered something is, the more of a pain it is to reuse/maintain/etc. Simple code is more likely to be grok'd easily by readers, and therefore reused as many places as possible.
Working in an organization where there's a lot of overengineering going on constantly and it definitely feels like a significant portion is wasted effort for the time horizon such projects survive for.
Randall Monroe observations for this kind of stuff are always really spot on. I wonder what his creative process is like. I should probably read XKCD more often...
In my opinion if you have something that you can afford to spend tons of time over engineering before it’s delivered, then it’s not important to the business.
If you’re being screamed at to deliver asap and having to take shortcuts, that’s when your short term MVP proof of concept gets embedded unless you have someone really good watching over you
I wish I could convince my org that the easier to delete items are far preferable to the over engineered ones.
It is fun, as we "know" that "build one to throw away" is wise advice. We still find every way possible to make it hard to throw what we are working on away.
This may ring true in part due to survivorship bias. The quick hacky approach can potentially produce a lot more code than the perfected implementation. If both are actually equally likely to be used in the future there are a lot more survivors from the quick hacky stuff.
I don't think it's quite as widespread, I would posit that this has changed to: - (1) YAGNI, build the simplest solution you can for the problem at hand. That will make it as easy as possible to change when future requirements are presented. (2) Design your code to be replaced, not extended. Design your modules in a way that someone can rewrite one without having to rewrite the whole system.
Points (1) & (2) really contradicts the generalized solution approach. The generalized solution tends to be very abstract, hard to understand, and because there are layers - not very modular (which makes it hard to re-write)