
Solve Less General Problems - kiyanwang
https://hacksoflife.blogspot.com/2018/08/solve-less-general-problems.html
======
elvinyung
The interesting thing is that the programmer's tendency to abstract and
future-proof their code mirrors the late 19th and early 20th centuries' high
modernism movements, as James C. Scott analyzes in _Seeing Like A State_ and
Michel Foucault discusses in depth in _Discipline and Punish_ and others.

"Modernism" is basically characterized by attempts to predict and control the
future, to rationalize and systematize chaos. In the past this was done with
things like top-down centrally planned economies and from-scratch redesigns of
entire cities (e.g. Le Corbusier's Radiant City), even rearchitectures of time
itself (the French Revolution decimal time). The same kinds of "grand plans"
are repeated in today's ambitious software engineering abstractions, which
almost never survive contact with reality.

~~~
twic
> The same kinds of "grand plans" are repeated in today's ambitious software
> engineering abstractions, which almost never survive contact with reality.

Yonks ago, i was reading an introduction to metaphysics at the same time as an
introduction to Smalltalk, and it struck me that the metaphysicians' pursuit
of ontology was quite similar to the Smalltalker's construction of their class
hierarchy. The crucial difference, it seemed to me, was that the
metaphysicians thought they were pursuing a truth that existed beyond them,
whereas the Smalltalkers were aware that they were creating something from
within them.

It's very likely that my understanding of one or both sides was wrong. But
ever since then, i've always seen the process of abstraction in software as
_creating something useful_ rather than _discovering something true_. A
consequence of that is being much more willing to throw away abstractions that
aren't working, but also to accept that it's okay for an abstraction to be
imperfect, if it's still useful.

I, probably arrogantly, speculate that metaphysicians would benefit from
adopting the Smalltalkers' worldview.

I, perhaps incorrectly, think that the ontologists' delusion is endemic in the
academic functional programming [1] community.

[1] Haskell

~~~
derefr
> But ever since then, i've always seen the process of abstraction in software
> as _creating something useful_ rather than _discovering something true_.

Software, in my personal experience, is closest to the study of mathematics:
there is an arbitrary part (the choice of axioms)—but, once that part is in
place, you must _obey_ those axioms and discover what facts are true _within_
them.

If you don't obey your own chosen axioms, the system you will create will be
_incoherent_. In math, this means it just fails to prove anything. In
software, this means that it might still be useful, but it fails to obey the
(stronger) Principle of Least Surprise.

The regular PoLS is just about familiarity, effectively.

The _strong_ PoLS is more interesting. It goes something like: "you should be
able to predict what the entire system will look like by learning any
arbitrary subset of it."

The nice thing that obeying the strong PoLS gets you, is that anyone can come
in, learn the guiding axioms of the system from exposure, and then, when they
add new components to the system, those components will _also_ fit naturally
into the system, because they'll be relying on the same axioms.

~~~
bonoboTP
> there is an arbitrary part (the choice of axioms)—but, once that part is in
> place, you must obey those axioms and discover what facts are true within
> them.

However, that's almost never the way math is originally developed. As a
student one gets this impression, but that is usually on a topic that has been
distilled and iterated over again and again, with people spending a lot of
time on how to line out the "storyline" of a subfield.

More commonly, some special case is first encountered and then someone tries
to isolate the most difficult core problem involved, stripping down the
irrelevant distractions. The axioms don't come out of the blue. If a certain
set of axioms don't pan out as expected (don't produce the desired behavior
that you want to model with them, but for example "collapse" to a trivial and
boring theory), then the axioms are tweaked. Indeed, most math was first
developed without having (or feeling the need for) very precise axioms,
including geometry, calculus and linear algebra.

I don't address this to you specifically, but I see that similar views of math
in education make people believe it's some mechanistic rule-following and a
very rigid activity. I think it would help if more of this "design" aspect of
math was also shown.

Even when mathematicians feel they are discovering something, they rarely feel
like discovering axioms, more like types of "complexity" or interesting
behavior of abstract systems, where this complexity still has to be
deliberately expressed as formal axioms and theory, but I'd say that's more
like design or engineering or the exact word choice for a writer vs. the plot
or the overall story.

------
tabtab
From a lifetime of coding I indeed came to a similar conclusion: "Micro-
abstractions" are more useful in the longer run than large-scale abstractions.
Nobody can predict the future accurately: I've seen many Grand Abstractions
byte the dust face first. BUT, with micro-abstractions you can often re-use a
decent portion of them when the future doesn't go as planned. Unlike large
abstractions, you don't have to marry micro-abstractions: you just date them.
If any one doesn't work out, you can ignore and replace it without much
penalty: no messy divorce. I've built up a library of "handy" functions and
mini-classes that I can borrow as needed per project. I've translated the
commonly re-used ones into different programming languages.

Further, one can experiment with different micro-abstractions without
investing too much time. They are micro-experiments. If one fails to provide
re-use or flexibility, so be it: just stop using it. It's like picking stocks:
some stocks will wilt, but if you buy enough, you'll roughly approximate the
general market at about 10% annual growth.

I do find dynamic languages make such mixing and matching easier, though,
because they are less hard-wired to a specific type or iteration interface:
less converting needed to re-glue. I've yet to get as much re-use in type-
heavy languages, but not ruling it out. I just haven't cracked that nut yet.
Maybe smarter people can.

~~~
wwweston
Interesting.

Could we say the much-debated ORM is (usually) a large-scale abstraction?

~~~
munchbunny
I think that's fair.

On the more general point, I think that the larger scale the abstraction is,
the larger the number of actually encountered use cases needs to be. You
shouldn't write an ORM because you feel like existing ones don't serve your
specific needs. You should write SQL, and then replace it with an ORM once
it's clear you will benefit from the extra abstraction.

There seems to be a fairly consistent trade-off between the scale of the
abstraction, the degree of over-engineering, and the leakiness of the
abstraction. So if you're going for a large scale abstraction, then I'd want
lots of proof that it's not a leaky one.

~~~
tabtab
I'm kind of bothered by the choice of direct SQL versus ORM. I'd prefer
something in between that assists with SQL but doesn't try to completely hide
you from the RDBMS: helper API's. I know many developers "hate" the RDBMS
(perhaps because it's unfamiliar), but shifting the burden from potentially
tricky RDBMS's to potentially tricky ORM's is not problem solving, just
problem shifting. But maybe the ORM debate is another topic in itself.

~~~
munchbunny
I think I phrased my opinion badly. I'm generally in favor of using ORM's
because they simplify 99% of the stuff you need to do in the database. There
will always be stuff that doesn't fit your choice of abstraction well...
that's just a fact of life.

I'm mostly wailing against _writing your own ORM_ (and also other forms of
"not invented here" syndrome), which is something I've seen done way, way too
many times, because you have to reinvent a ton of things you probably don't
have much experience with, will probably get wrong in ways you don't
anticipate, and will miss scaling requirements you don't know about yet.
Because of this, I think there should be a heavy burden of experience on
writing "core" libraries.

~~~
tabtab
ORM's "simplify" things until something goes wrong, then they can be a pain to
troubleshoot. A "light" set of utilities can also simplify a good portion of
the database interaction. However, they don't give you a false sense of power
such that you tie yourself into a knot that bites you 3 years down the road.
But, it depends on the shop: If your shop always has a couple of good ORM
experts around, then they can deal with and/or avoid such snafus as they
happen. But, it's hard to "casually" use ORM's right.

~~~
munchbunny
I suspect our experiences differ, or maybe it's our philosophies. I'm of the
opinion that if your self-tied knot bites you 3 years down the road, that's
better than the status quo. :P

That's situational, of course. My current projects are tech giant things, so
some choices actually do have a >3 year window, but then there's a tech giant
sized stack of use cases to design against.

When my projects were startup problems though, I'd be happy if that component
still worked well after 3 years of growing and pivoting. In that kind of
environment, getting more for free is by itself valuable for your ability to
pivot quickly.

~~~
tabtab
The environment does indeed matter. With startups, the short term is critical:
if you don't grow fast enough, there will be NO "3 years from now" to fix. A
big bank will care about glitches 3 years from now.

I'm not sure what you mean in the middle paragraph, per "stack of use cases to
design against". You mean getting it done "now" overrides problems 3 years
from now? Your org making that trade-off is fine as long as it's clear to
everyone and perhaps documented.

------
dahart
Definitely agree. In ~25 years of professional coding, I've witnessed more
money wasted by over-engineering, over-generalizing, and solving unnecessarily
hard problems, than I have of not being general enough.

I've seen that often enough that I'm personally now starting to make the
mistake of not being general enough regularly. :) But it seems like a good
thing if I'm 50/50 between not general enough and too general.

Running a startup was good practice, you just can't afford to work on problems
you don't really have. If multiple customers aren't screaming for it right
now, it can probably wait.

Ironically, I think I learned this lesson more thoroughly with writing than
with computer science. Using abstract language, using categorical and abstract
words rather than specific examples, makes for _super_ boring reading.
Communicating a pattern is even sometimes even more accurate when you list
three specific examples than when you choose the most accurate abstract word;
the reader will see the pattern.

Just googling for examples, this one seems decent. Lots of writing advice
online leans in favor of being specific.
[https://www.unl.edu/gradstudies/current/news/use-definite-
sp...](https://www.unl.edu/gradstudies/current/news/use-definite-specific-
concrete-language)

"In proportion as the manners, customs, and amusements of a nation are cruel
and barbarous, the regulations of its penal code will be severe."

vs "In proportion as men delight in battles, bullfights, and combats of
gladiators, will they punish by hanging, burning, and the rack."

------
chaoticmass
I see the author's point, and it is valid, but I'd like to give a counter
example.

I was asked if I could make a program that could 'put a photo on top of
another one'\-- the idea being they wanted a program where you could load a
template image, maybe one that looks like a birthday card, and it would have a
spot where someone's picture could be put on top of the birthday card image.
Simple enough, but I decided to build a generalized templating system with a
built in editor so new templates could be created. This way it could make
birthday cards for one person, or make a "Good Job team!" cards for multiple
people. It was completely agnostic to what kind of templates you could make
with it. It supported layers, conditions, database integration.

Sure enough, over the course of months, I was asked to make the program do
more things than it was originally asked to do, and I was able, with few
exceptions, to accommodate these requests without needing to modify the source
code. Even when I did need to modify source code, it was to extend
functionality, not change the basic template abstraction model. It found its
way into other uses as well (digital signage, ads, etc).

If I had only solved the least general problem first time, I would have been
back at the drawing board rewriting most of the app every time something new
was requested.

Maybe this is a rare counter example, but sometimes pursuing the general
solution pays off.

~~~
commandlinefan
I actually agree with both of you, _as long as it's up to the implementer_. I
see requirements specifications that deliberately try to remain vague in the
hopes that the developer will produce a "general" abstraction that can be
reused - and always end up shooting themselves in the foot by never saying
exactly what they want done. Or the "disruptive scrappy startup founders" who
want you to build "legos".

------
alexeiz
I've seen the other side of the coin, where the desire to solve a concrete
problem forces a solution to be unnecessarily brittle and actually harder to
implement than a more generic solution.

I'll give an example: "we're never gonna have more than three devices, so this
array should have a hardcoded size of 3." It sounds pretty extreme, but
believe me, some variation of this theme comes up more often than I would
like, especially from people who are not software engineers. It seems like
some people think that every abstraction (and potential genericity) is
expensive, so they tend to put constraints in places where none need to exist.
Very soon you'd need to make a small modification to your code, or extend it
slightly, and those unnecessary constraints are going to make things
difficult.

Personally, I'd like to approach a problem from the bottom-up. Start with
solving a concrete case, and see if any patterns emerge in the process. And if
they do, then I generalize those patterns. Frequent refactoring is very
helpful during this process. Sure, mistakes and over-generalizations can still
occur. But they are usually not very expensive if noticed quickly.

~~~
moring
I think this conflates requirements and implementation choices. "we're never
gonna have more than three devices" is a statement about requirements, and
implies the absence of a requirement to support more than 3 devices. It does
not imply a requirement to fail to support more than 3 devices though, and
leaves an implementation choice how many devices to support, as long as it's
at least 3.

On the implementation side, the "0, 1, n" rules already mentioned by muxator
dictates that this specific aspect of the implementation should not treat 3
different from 5, and implement support for n devices. (By "This specific
aspect" I mean that other factors will surely prevent supporting n = 1
trillion, but we don't have to care about that since it doesn't contradict the
requirements).

------
maltalex
This is one if those pesky differences between “computer science” as it’s
taught in college, and actual software engineering.

I believe that bigger parts of our education should revolve around reading and
analyzing professionally written code as opposed to solving completely
artificial problems.

~~~
wilsonnb3
Agreed. What really needs to happen is the splitting of software development
and computer science into two related but distinct education tracks.

~~~
k_sh
One could say that bootcamps were the first to pave the software development
track.

~~~
gmfawcett
Hardly... applied programming education has been alive and well in community
colleges (and some higher ed. institutions) for decades. Not every program,
everywhere, is theory-heavy!

------
ilovecaching
I highly recommend following a data oriented approach to software
organization. OO loves to abstract data, but why? Data is the main character
of the story. I also hate the take on Knuth's "premature optimization". Don't
sweat the small stuff, but do architect for performance, it is a feature. If
you focus on performance and not hiding your data, the code will fall into the
right level of abstraction.

Solving general problems happens when you start thinking about data you don't
have, but might, and throw performance away for premature organization.

------
alkonaut
I'd formulate the "don't overgeneralize too early" as "make concious decisions
about where you solve specific and general problems". It's fine to solve a
specific problem knowingly, and while considering the cost of solving a
general problem.

Saying "let's cross that bridge when we come to it" is the idea (rather than
"whoa we never saw that bridge we should/shouldn't have crossed")

There is also a human perspective to this. Solving general problems is FUN,
(at least I tend to think so). The generalization is the fun part of the
problem solving. Some people think seeing the working product is cool, no
matter how elegant or general. I don't mind if everything is half-finished
forever so long as the half-finished product is elegant and general
(exaggerating but you get my point about the different personalities). This is
also why you need different perspectives (and probably people) on a team.

------
adiusmus
Build one. Build another. Now you have built two. Hopefully you learnt
something and now you know where things are headed. Now you can write an
abstraction. Now you can rebuild the first two using the abstraction along
with the next. If you’ve done the abstraction right all three now share and
use a commonality that helps clear up and simplify.

~~~
chrchang523
[https://en.wikipedia.org/wiki/Rule_of_three_(computer_progra...](https://en.wikipedia.org/wiki/Rule_of_three_\(computer_programming\))
lines up better with my experience: postpone writing the abstraction until you
are actually implementing the third thing. With just two cases, there will
generally be a bunch of incidental commonalities that are better left out of
the abstraction, and it's frequently not obvious which ones they are.

~~~
adiusmus
Nice to see it codified. Sometimes you don’t realise you need or decide you
want to abstract until you’ve hit 5 or more.

------
pankajdoharey
The true meaning of a General solution does not mean providing support for
every kind of scenario your data may encounter, the true meaning from a
functional perspective is to build composable structures/mechanisms to produce
the correct shape of data the problem requires. So for a near sighted
programmer YAGNI would work, but for a software or machine to be really
durable, it needs to be composable over a period of time. An example of this
is emacs, it has kept with the passage of time because of its composable
nature. Vim on the other hand is undergoing a massive rewrite project known as
Neovim. This is the difference between a true general solution and a specific
YAGNI based solution. The only differentiator between the two is Durability,
and this durability comes at no extra cost, just the wise decision of writing
software in a much more composable language solves half the durability
problems.

------
bovermyer
This is my favorite (and least useful) quote from the article:

> I made the right decision, after trying all of the other ones first - very
> American.

It's not my favorite because it's snarky, or because it makes a generalization
I think is accurate. It's my favorite because it has an element of the person
behind the technical article.

Too often, people writing about technical subjects get lost in the topic, and
forget that the act of writing for an audience is a human conversation.

------
j7ake
Richard Hamming or John Tukey said "It is better to have an approximate
solution to the correct problem than the perfect solution to a slightly wrong
problem."

It might apply to this case here.

------
childintime
Jonathan Blow discusses harmful abstractions in several youtube videos, while
discussing the Jai Programming language. That language isn't out yet, but an
overview is available at:
[https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md](https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md)

------
swsieber
Hmm... it's not quite addressed here (they address generality as the size of
the scope of the solution), but I have a slightly different take:

I love generalizing things. If I need to remove a couple of prefixes from a
list of strings, I don't hard code those strings in a function. I make a
general string prefix removal and pass those prefixes in. I find that I write
more correct and stable code by switching from special cased code to
generalized code.

One can take this to extremes of course. I think the article is against
generalizing solutions that shoehorn things together that normally don't go
together. E.g. hal for handling disparate hardware abstractions. Its a forced
abstraction to gain certain things.

I guess it's abstract vs generalize. Generalize is good, if it doesn't
introduce abstractions... that's the thought process I just had anyway.

~~~
adrianratnapala
Suppose I already had two function functions like:

    
    
       def strip_prefix(prefix, str):
         ...
    
    
       def name_from_fooid(fooid):
          return strip_prefix("foo", id)
    
    

Now is it better to call `name_from_fooid` or to call `strip_prefix` directly?
It depends on the high-level semantics we are trying to express. But assuming
we really are extracting a Name from a FooId, then the former is probably
better.

But then if you want a name_from_fooid function, it's just an implementation
detail that you happen to implement it on top of a strip_prefix utility.

------
gilbetron
Great abstractions exist, that doesn't mean every abstraction is great. Most
are bad.

------
zakum1
This is a fantastic description. As smart, analytic thinkers it is so easy to
generalise. My experience is that we should generalise when we are creating
conceptual designs, but be very lean and specific in the implementation.

------
iamleppert
A good abstraction leverages the shared knowledge of other previously
unconnected pieces of information. It has more to do honestly with psychology
than computer science; I don’t need to learn anything about tree or graph data
structures when I can use HTML in the same way as I am used to organizing
information in bulleteted lists.

------
ken
I'm going to risk being contrarian and say that I don't see the HAL as the
problem. The problem I see is "the product was canceled, I was moved to a
different group, and the HAL was never used again". When the product is
cancelled, it doesn't matter what you built, or how good or bad its
architecture was. It's dead.

Our industry has an organizational structure for shipping software today that
makes every line of code have zero value until it's in a consumer product that
ships, and maybe even then it'll be cancelled next month and its value drops
to zero anyway. We've long known that most software projects fail. What's
wrong with this picture? Why do we continue to work in a system where the most
likely result of any day's contribution is literally a useless waste of time?

It seems as though the problem was short-sightedness, and the proposed cure is
to be even more short-sighted. (You're looking down as you walk, and you
occasionally glance up. While looking up once, you tripped over a stone. The
solution is to focus even harder at the 6 inches in front of your shoes --
then you'll never run into trouble!) Is it any wonder that most software
projects fail?

We joke about "resume-driven development" but economically that's the only
certain gain from any new software project. I'd be foolish to optimize for the
lottery ticket of "successful product", instead of the solid long-term
investment of "increased personal knowledge".

The Gossamer Albatross succeeded where others had failed when they realized
that the actual problem was fixing the process first [1]. They won big by
solving the more general problem. What if we fixed the software development
process so that everything we built could be useful?

[1]: [http://www.azarask.in/blog/post/the-wrong-
problem/](http://www.azarask.in/blog/post/the-wrong-problem/)

Sure, some code is just bad, and the trash can is a useful tool, but a DV HAL
sounds like a useful tool, too, and I don't think anyone is better off that it
got built and then discarded. Could there have been a secondary market for
commercial libraries whose products were cancelled? Could it have been open-
sourced? Could it have been built as part of an alliance, so other teams could
have helped drive the design, and reap the benefit even if one team dropped
out? I don't have The Answer but it seems awfully unlikely to me that
optimizing one's architecture for the next day or week is an optimal process
on any larger timescale. That's basically admitting that we're still at the
hunter-gatherer stage of software development. You can't plan for next year's
harvest, if you haven't invented agriculture, and don't know where you're
going to be in 6 months.

Bonus: "At the time templates tended to crash the compiler, so going fully
templated was really expensive." Yes, we need to reduce our scope, because
this other program we're using reduced theirs! The limit of this function is a
catastrophe. If I were to get a batch of bad steel from my supplier, I
wouldn't try to compensate by using twice as many bolts.

------
gcb0
playing devil's advocate: if the author's manager had communicated the tech
improvements correctcly ("we can now support any hardware"), maybe others
teams knowing about that would have moved the company in a different direction
instead of "cancelling everything after firewire support". maybe they even got
to that decision because one new input source took so much time in the first
place.

------
RyanShook
Can someone better explain the leaky abstraction concept? I take it to mean an
abstraction that really isn’t.

~~~
ben509
A leaky abstraction is one that _mostly_ works, but details and assumptions
leak through.

For example, many languages attempt to provide a cross-platform API for
filesystems; they give you an algebra that lets you construct and manipulate
paths.

So if you have a Path, the `/` abstract operator might join path parts, and
"basename" and "dirname" are functions that extract parts of the pathname.

Typically, the API designer makes some intentional (and some unintentional)
design decisions to keep the API comprehensible but at the expense of being
inconsistent with the underlying reality being abstracted.

As an example, some filesystems are case-insensitive, others aren't, so to
know if two paths refer to the same file, e.g. if you want to use paths as
keys, you need to determine what volume each is on. This gets complicated in
Unix due to symlinks (which can be any component of a path) and mounts that
can be anywhere in the tree. You could resolve a path to an inode, but then
other filesystems might not provide an inode.

~~~
tenaciousDaniel
By this logic though, aren't all abstractions leaky in the end?

edit: not arguing, just curious. I haven't thought much about abstraction but
it seems like it could be a truism that every conceivable abstraction could
succumb to this problem in some way.

~~~
gameswithgo
>By this logic though, aren't all abstractions leaky in the end?

Almost all. the question is just one of degree. The fact that they are almost
all leaky is why you should be hesitant to abstract away in the first place.
The effort you think you are saving may be taken up later when the leaky
details bite you. "I'll use this game engine instead of raw vulkan" -> "oh no
I can't render all my widgets fast enough on mobile" or "I'll use this easy UI
library" -> "oh no I can't actually do a list of text boxes that is scrollable
with this"

------
hyperbole
that's a whole lot of blathering to say pre-optimization is the root of all
evil... @example: the Intel meltdown &Spectre bug.

