
In favor of small modules and plumbing - NetOpWibby
https://blog.webb.page/2020/small-modules-and-plumbing
======
mhd
I get an allergic reaction to any rationale that includes the tired phrase
"easier to reason about". That's on my bingo card right next to "promote
synergy"...

Now, I'm not inherently against having small modules for small tasks, but I'd
much prefer if those are used like sprinkles on a more solid foundation. So a
small Perl module on CPAN? Sure. Your whole web app consisting of small
modules, turtles all the way down? No, sir, no.

It also means that you've got plenty of build systems, it's incredibly hard to
change the output of all your dependencies to not-ES5, it's way more frequent
that one developer introduces new dependencies, possibly duplicates etc.

Sadly there's no decent way out. JS is getting a new syntactic feature every
second Monday, but no big std lib movement is out there that I'm aware of.
Sure, you probably wouldn't want to fit the Go/Python/Java core lib into a
browser, but for the backend it would be quite nice, and would make tooling
for the frontend less fragile. (And to be honest, even frontends are
approaching mid-tier desktop UI complexity; otherwise we could've and
should've stayed with jquery)

I don't need to be able to reason about single modules independent from each
other often. I need to trust the whole system enough to not lose sleep over
it.

~~~
lmm
> It also means that you've got plenty of build systems, it's incredibly hard
> to change the output of all your dependencies to not-ES5,

On any realistic sized project you already need tooling for handling
dependencies in bulk. If you want to build everything for ES6, that should be
a problem you solve once in your build/dependency manager, not per-depednency.

> it's way more frequent that one developer introduces new dependencies,
> possibly duplicates etc.

I actually disagree with this: if you have large frameworks then you're more
likely to have duplication, because you'll have frameworks that overlap but
don't completely cover each other. E.g. C++ projects tend to end up with 5
different string classes, because every framework you use will bring its own
(because C++ relies on big heavy frameworks because it has a very high per-
library overhead due to limited tooling). Small focused dependencies make that
less likely.

~~~
mhd
I don't quite get the latter. First of all, if the foundations are sound and
the core language provides a good enough module already, neither large
frameworks nor small modules need to duplicate that.

And if it _isn 't_ included, why would 16 small subsystems have fewer
duplicates than 2 large frameworks?

By the way, C++ is exactly the trajectory I see JS moving towards. A plethora
of different sub-languages and library sets. Only with the difference that
it's both easier from a tooling and culture perspective to bring in new tiny
modules.

~~~
lmm
> First of all, if the foundations are sound and the core language provides a
> good enough module already, neither large frameworks nor small modules need
> to duplicate that.

A language standard library is just another framework - Python's one is
famously "where modules go to die", but we see much the same problems in e.g.
Boost. A "good enough" solution in the standard library can be worse than
nothing, as it means better alternatives struggle to gain enough momentum, and
this gets worse as time goes by: functionality that's in the standard library
has to evolve slowly.

> And if it isn't included, why would 16 small subsystems have fewer
> duplicates than 2 large frameworks?

Because the common things can be in their own small pieces. If a database-
access library, a web templating library, and a config parsing library all
need to do string manipulation, then if they all implement it themselves we'll
see duplication, whereas if they all use an external library then we won't
(this is after all the point of libraries). Maybe the ecosystem will have two
different string manipulation libraries on the go with one gradually
displacing the other, but that's likely to only happen when there's a genuine
improvement being made. Whereas if everything's a big framework then
eventually everything implements its own copy of the common utilities that
just about everything ends up needing - collections, datastructure
manipulation, all the really generic stuff.

------
stareatgoats
FWIW, I've taken the exact opposite route. Much as I respect sindresorhus, my
projects nowadays have very few dependencies; I treat every dependency as an
elevated "point of failure"-risk, and I would not recommend anyone having
potentially thousands of "left-pads" sprinkled across a business-critical
system.

~~~
eropple
You shouldn't be relying upon NPM or any other public package registry for a
"business-critical system" in the first place.

When the left-pad guy had his snit, we didn't notice at all because we had a
caching NPM mirror.

------
wereHamster
Small packages are fine, if the maintainers keep them up to date! You can't
publish packages and never touch them again. This leads to situations like the
one with merge-deep that I recently discovered:

[http://npm.anvaka.com/#/view/2d/merge-
deep](http://npm.anvaka.com/#/view/2d/merge-deep)

All but one of those dependencies are from the same maintainer. Yet I see two
major versions of kind-of, for-in, and lazy-cache. What's the point of
creating reusable functions if you're not going to reuse them?

This is a huge waste. First on the filesystem, no wonder our node_modules
folders are hundreds of megabytes big. Second it wastes bytes in memory and in
CPU caches since they don't share byte code and each function has to be
parsed, compiled, optimised separately.

I seem to be in the minority thinking this situation is insane…

~~~
unlinked_dll
It's insanity all the way down to the copper on the board frankly. People only
care about making things more sane when you can prove there's a dollar to be
made or saved by doing so.

Like we all know nodjs is slow(er than other things). The question is not how
to make it faster by doing things like keeping dependencies up to date but how
much there is to gain by doing so. I suspect very little.

------
KaiserPro
All of this seems to be justifying a lack of a decent standard library.
However its appears to be written from the point of view of someone who's
never had the joy/pain of a decent standard library.

The problem with lots of tiny modules is consistency and power.

Sure its "open source" but each one of these modules are in various states of
decay. better to lump them together in sets of functions, so that more eyes
can be concentrated in the same place.

------
asterisk_
This doesn't seem too controversial of an opinion, haven't small modules
already been at the core of a Unix philosophy? Once usage patterns emerge,
these smaller libraries are composed into frameworks which ultimately build
end-user applications.

The entirety of engineering is premised upon leveraging powerful abstractions;
cf. the reason we're not still coding via electric charge.

Edit: To expand on the analogy: "A two-person startup already uses twenty-
eight other tools" [1].

[1]
[https://news.ycombinator.com/item?id=22449314](https://news.ycombinator.com/item?id=22449314)

~~~
jasode
_> , haven't small modules already been at the core of a Unix philosophy? _

I assume you're talking about userland programs such as "ls", "cut", "grep",
etc. ?[1]

But C Language is also part of UNIX and programmers typically link against a
large more comprehensive "standard library" such as glibc:

    
    
      libc.a, libc.so  <-- includes printf
    

... instead linking of a hundred little object files specified explicitly like
this:

    
    
      printf.o
      strcat.o
      strncat.o
      malloc.o
      leftpad.o
      ... <hundred or thousand more .o files>
    

What happened with Javascript is the language didn't have a comprehensive
"standard library" like C, C++, Java, C#, Python "batteries included", Go,
etc. Therefore, NPM _became_ an adhoc "standard library" via bottom-up
crowdsourcing. There are both positives and negatives to that approach. One
negative is that leftpad() isn't produced by a canonical entity like Netscape
or Mozilla but a random programmer. When he wanted to take his ball and go
home, he broke everyone's build that depended on it. That type of event didn't
happen with "printf()" in the C Language world.

[1]
[https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_a...](https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_and_Do_It_Well)

------
cousin_it
I just looked at the source code of left-pad for the first time, and it's
strange. They noticed that concatenating a bunch of single-space strings is
O(n^2), and optimized it to O(n log n) by using repeated doubling. So in the
end they take longer than O(n) to fill an array with a constant value. Why?

[https://github.com/left-pad/left-
pad/blob/master/index.js](https://github.com/left-pad/left-
pad/blob/master/index.js)

~~~
rhardih
Why write code to left pad your strings, when you can get an O(n log n)
implementation just like that. You only need a "minimal" amount of bundle
tooling and a few hundred MB of node_modules. Count your blessings. This is
the way.

------
z3t4
Small focused modules dont need much maintenance or braking changes, so you
basically only have to download them once, and treat them as if they where
your own code.

------
osrec
The article highlights a huge pitfall of having lots of small dependencies,
provides no real remediation, and says small dependencies are easier to reason
about, so they're better anyway.

I disagree. I personally find that with large projects, small dependencies
actually become much harder to reason about - they sort of get lost in the
codebase. Not to mention the duplication of small packages on npm doing
similar things - with a large team, it's easy to add both to your codebase and
then we're in a new sort of duplicative mess.

------
pubby
Small modules are only useful when they're self-contained. Leftpad was mocked
because it brought in three dependencies on top of itself. The more
dependencies one has, the more breakage hell one must deal with. So please,
keep your code self-contained when sharing with the world.

------
jscholes
> Regarding Jobs' quote, here's another way to look at it: the most beautiful
> hotel in the world wouldn't feel like such if all the toilets overflowed
> when flushed.

But this would be a functional problem, irritating at best and a potential
health hazard at worst. Using plywood for the back of a chest of drawers is no
such thing.

