
The Complexity Trap: Think Before You Leap - chedine
https://danielwestheide.com/blog/2018/12/07/the-complexity-trap.html
======
afpx
As one who frequently deals with Scala code, I have to say that this guy is
right: the Scala community in particular seems to relish in complexity.

Scala is a wonderful language and well-designed, especially in contrast to
Java. The learning curve requires a good deal of effort, but with experience,
it becomes incredibly fun to write Scala code. Once I got past the learning
curve, Scala became my go-to language.

That said, there’s an underlying arrogance of Scala developers who, in my
experience, tend to often abuse the language to show off their cleverness at
the expense of users and maintainers. I don’t know why this is. I suspect it’s
because Odersky set the bar so high with his obvious brilliance that some feel
insecure.

One take away quote for me:

“Unfortunately, we don’t like boring. We like to take the language and the
compiler to their limits. We also like to adopt the latest fad. But the
software systems we work on in our jobs should not be a playground or a test
bed for the latest fad.”

Scala developers aren’t doing anything new here. People have abused fads in
programming for decades. “Design Patterns” and “Test Driven Development” and
“RESTful” come to mind. However, it’s unfortunate that Scala developers in
particular do this. Because, Scala seems to be a language specifically
designed to avoid past mistakes.

~~~
zie1ony
I worked professionally for a year with Scala. For me the biggest problem were
functional fanatics that design solutions only top 20% of devs can understand.
The rest comes from the object oriented realm and can't deal with monad
transformers etc. That might work in small teams, but when you want to scale a
team to 100 people you can't ignore this effect. And functional fanatics are
strong in ignorance.

~~~
reificator
> _functional fanatics that design solutions only top 20% of devs can
> understand_

Not a part of that group, but I think this is a false narrative. 20% might be
accurate, or it might be a bit high even. But it has nothing to do with `top`
or `bottom` performers. It's purely to do with exposure to functional styles.

Learning Haskell first doesn't make you a top 20% developer, much as I'm sure
HN in particular will want to argue with me on that point.

All I'm trying to say is: there's not an intelligence spectrum of `imperative`
to `functional` with `object oriented` in the middle. They're different, and
many who learn one first struggle with the others, but that doesn't mean
they're lower performers in what they do know than the ones who can talk about
currying vs partial application.

~~~
xamuel
>many who learn one first

Everyone learns imperative first. "Wake up!" or "Eat your vegetables!" are
imperative. No-one says to their kids, "map(vegetables, eat)"

~~~
reificator
I mean, those are much closer to functional than imperative tbh. At best it's
simply too abstracted to discuss either way. Recipes would be a better
example.

If you want to use that example it'd be more like this snippet that no one on
earth would tell their children:

> _Starting from n = 0, take the nth vegetable from your plate, insert your
> fork into that vegetable, raise your fork to your mouth, bite that
> vegetable, remove your fork from your mouth, masticate, add one to n, and
> repeat._

------
titzer
“Everyone knows that debugging is twice as hard as writing a program in the
first place. So if you're as clever as you can be when you write it, how will
you ever debug it?”

\--Brian Kernighan

How many times have I bitten myself this way? Dear past me: stop being so
smart.

~~~
hnuser355
This is an astounding quotation

------
Verdex_3
I really like it when people become concerned about cognitive complexity. I
think it's one of the significantly ignored concepts in our industry and if we
ever find a way to address the issue (either with some sort of cognitive
complexity solution or some sort of big oh like math framework that describes
how things are complex) then the future generations of software engineers will
look at what we're currently doing now the same way we look at how people used
to do alchemy for our lack of being able to handle cognitive complexity.

Personally, I want a mathematical structure of "problems" such that we can
track when we're actually making things more complex or less complex. In the
article, removing boilerplate was a problem because internal domain models
were being exposed publicly. But what I really want to know is "what are we
actually gaining by removing the boiler plate and what are we losing by
exposing domain models". In like a general mathematical sense such that if
everyone about programming changes (maybe suddenly we switch to APL or
something), the lessons we learned are still valid.

So far the best I've managed is [1]. I've done a bunch of refinement in the
last year and a half, but the general structure is about the same. I'm not
suggesting that what I have is complete or even necessarily the right
direction to go for this sort of thing. But what I am saying is that in order
to make definite statements about how the problem and/or code you're dealing
with is complex (that don't rely on appeals to authority) then you probably
need to have done at least as much "work" as I have (if that makes any sense).

[1] - [https://www.sep.com/sep-blog/2017/04/25/objective-code-
quali...](https://www.sep.com/sep-blog/2017/04/25/objective-code-quality-blog-
series/)

~~~
skohan
I like the alchemy analogy. I think a huge trap we tend to fall into as a
profession is presuming to know what software engineering practices are good
or bad when we actually have next to zero empirical data on the matter. Most
of what the accepted dogma is comes from either authority, or from what
techniques lend themselves to the strongest arguments.

------
kreetx
Has somewhat similar intro to the Rich Hickey's famous "Simple made easy"
talk. A good read nevertheless.

~~~
goldrake
...and that's IMO a good thing.

~~~
kreetx
Well, it feels borderline weird -- since it's so obviously similar, it could
use a reference.

------
jondubois
>> Hence, we try to isolate business logic from effects and implement the
former as pure functions.

This is a valuable insight. Especially if your main programming language
supports both functional and procedural and/or OOP approaches; This statement
offers a good guideline as to where the boundary should be.

I find that functional programming is great for business logic but not as
great for more concrete logic where performance is important. For example,
when dealing with low level concrete objects like sockets.

------
divan
It's a refreshing read even for non FP/Scala development world.

It's not hard to see that modern software engineering is cherishing accidental
complexity, and as it comes from the "tools we use", the fair share of problem
is in the languages, frameworks and tools we use. Unfortunately, there are
very little of them that really take a tough stance on fighting complexity,
instead of adding more features. Actually I can think only of one such
language – Go, and, unsurprisingly, it's critizised heavily in many circles
exactly for the lack of accidental complexity.

Tony Hoare wrote this in 1974: “But the pursuit of simplicity is one of the
most difficult and challenging activities of the human mind. Progress is
likely to be extremely slow, where each complexity eliminated must be hailed
as a breakthrough. We need not only brilliance of intellect but breadth of
experience, nicety of judgement, excellence of taste, and even more than our
fair share of good luck.“

I was quite impressed after reading this essay (it's from "The software
design: a parable" [1]) – written almost half a century ago, and there seem to
be a little progress, if not negative. So books and talks are unlikely to
change anything – we, people will still use the path of least resistance and
always will be subjected to congitive biases – so probably only tooling can
make a change. Go, for example, simply doesn't allow you to do many overly
complex things (on purpose) and it results in extremely readable codebase –
chances that you open random piece of Go code on internet and will not be able
to understand it are almost zero.

I find it a terrific achievement, and wish other (future) languages could
embrace the importance of it.

[1]
[https://people.dsv.su.se/~jpalme/s1/hoare.pdf](https://people.dsv.su.se/~jpalme/s1/hoare.pdf)

------
Ericson2314
Scala is an especially messed up situation. Everything is either
overengineered or simplistic; there is no good middle ground because the
language is a mess.

This is a little-acknowledged industry-wide problem that extends beyond hot-
shot overengineering to reems of banal code too, so I would rather it be
written from a less expected perspective.

------
lmm
> Doing that certainly means that you scrap quite a bit of boilerplate and
> reduce the complexity in your code. However, if you look at the bigger
> picture, this is the opposite of removing complexity. Exposing your domain
> model like this almost always leads to strongly coupled systems that are
> very difficult to evolve. If you ask me, this is one of the most promising
> ways to create a distributed monolith. It really amazes me how many people
> are totally okay with simply making their internal model their public API,
> especially since functional programmers are usually crazy about
> abstractions.

Good functional programmers are happy not to introduce an extra layer of
indirection yet, because they know that their code is easy to refactor. The
article itself seems to come to that point later, yet it gets it completely
backwards here.

Generate JSON formats the easy, zero-boilerplate way for now. _As and when_
you have a requirement for a JSON representation that looks different from
your domain representation - perhaps because you want to evolve your internal
domain model while keeping the JSON representation the same - you can
introduce manual JSON codecs (or DTOs, or whatever you prefer) at that time.

> Object-oriented programmers call this technique stubbing, and they get
> mocked for doing it, mostly by people like us.

They get mocked for doing it because it breaks all the rules of the language.
It usually relies on reflection, bytecode manipulation, or other such horrors.
It's often thread-unsafe. Having tests that you can't refactor the way you
normally refactor code is a huge, mostly-hidden cost.

> But you have no clue if your production interpreter, which is talking to an
> external system, is working correctly at all. So abstracting over the effect
> monad and using the Id monad in tests doesn’t even help you to write
> meaningful tests.

Tests need to be decomposed like any other code. There's a huge value in
testing your business logic separately from testing your interactions with
external systems. Functional core, imperative shell achieves that, but at a
higher cost: you can't interleave your interactions with external systems and
your logic in the natural way in code, you have to explicitly, manually push
all the interactions to the edge instead.

> What if we did server-side rendering of HTML instead? There wouldn’t be any
> need to design an evolvable JSON API, which, unlike simply exposing your
> domain model, is not trivial, and there would definitely be no need for
> defining any JSON codecs. And that’s just what you get on the server side.
> On the frontend side, you would have one piece of software less to maintain,
> you’d get faster load times and you could get rid of duplicated business
> logic.

You'd get slower load times as you had to do page transitions, and more
duplicate logic as things like form validation had to be done on the frontend
and backend with separate code.

> At some point, we realize that we have an architectural problem at the macro
> level. Maybe, if we continue to ask why long enough, we will come to the
> conclusion that all these services we need to communicate with exist in
> exactly this way because of how the company is organized.

And which is easier to fix? At some point you decide that the entirety of
western civilization is fundamentally flawed, and you can either start ranting
about how we need to rebuild society from the ground up, or you can put a bit
of complexity in your software and make something actually useful.

> I was recently talking to someone about Rust, and that person said, “A
> language without higher-kinded types is completely useless!”. Yes, certain
> abstractions are not possible without them, but I have never seen a project
> fail due to a lack of higher-kinded types.

That may have been me, and I absolutely have seen projects fail due to lack of
higher-kinded types. Not in an obvious way; rather it felt like a general
sludginess to development, a need to repeat the same tasks over again, until
the project eventually dies from a thousand cuts. (Hell, for all I know the
company I left several years ago may still be trying to make that project
work).

> Projects fail because the problem domain is not well understood, or because
> the architecture is not a good fit for the requirements. They also fail
> because of how the organisation is structured and because of politics, be it
> intra- or inter-organizational. Projects fail because of people, and because
> of people not talking to each other.

This is facile. Projects fail because of people talking to each other too
much. Projects fail because people tried to fix their organisational politics
instead of implementing technical workarounds. Projects fail because the
technology choices were simply bad.

Reading this article was weird, because I found myself nodding along with most
of the woolly principles but then massively disagreeing with every single
example. It makes me think the article is arguing at the wrong level of
abstraction somehow: no-one thinks you should use a complex solution when a
simple one will do. Very few people will claim that the problem with today's
software is that programmers think things through too much and should rush to
write code instead (though for the record I think that's actually the case).
But those consensus principles don't actually help us at all when it comes to
discuss specific decisions.

~~~
marcosdumay
> Very few people will claim that the problem with today's software is that
> programmers think things through too much and should rush to write code
> instead (though for the record I think that's actually the case).

People coming from easy to refactor languages have widely different rule books
from the ones coming from boileplate-filled ones. It expected that the same
person thinks people should rush to write code earlier and gets to understand
how a project can fail because of lack of higher-kinded types. The hard part
is explaining to people how those weird languages make one more productive,
because their thought follows completely opposite rules.

------
monksy
I was pleasantly surprised to read an article like this. Less surprised that
it came from Daniel Westheide. He know's his stuff.

What I was expecting: I was expecting the fad pushing an article about how too
many people push components together and then they're trying to sell some kind
of microservices nonsense.

------
bg4
See also:
[http://www.lihaoyi.com/post/StrategicScalaStylePrincipleofLe...](http://www.lihaoyi.com/post/StrategicScalaStylePrincipleofLeastPower.html)

------
rue
Scala is for people who weren’t around for C++ template metaprogramming.

------
leowoo91
tl;dr
[https://en.m.wikipedia.org/wiki/Chunking_(psychology)](https://en.m.wikipedia.org/wiki/Chunking_\(psychology\))

