
Layered Programming (2013) - sktrdie
http://akkartik.name/post/wart-layers
======
_bxg1
Similar to Spring dependency-injection:
[https://www.journaldev.com/2410/spring-dependency-
injection](https://www.journaldev.com/2410/spring-dependency-injection)

It seems too powerful, though. I could see it quickly becoming intractable to
find where the code lives that is causing a certain outcome.

The root problem it's trying to solve is the fact that things often don't fit
neatly in a single bucket, but our traditional idea of modules as mutually-
exclusive and hierarchical categories forces you to choose just one place to
"put" something. An idea I've had is to get rid of hierarchical modules
altogether. Really they're a relic of directories and files. Instead, one
could "tag" a function/class/constant as belonging to _multiple_ modules, and
one's editor could aggregate all members of a cross-cutting module as needed.
Organization within actual files on disk becomes an implementation detail.

~~~
akkartik
> Instead, one could "tag" a function/class/constant as belonging to multiple
> modules, and one's editor could aggregate all members of a cross-cutting
> module as needed. Organization within actual files on disk becomes an
> implementation detail.

Yes, this is interesting. Are you thinking of something like Ruby's modules
and `include` directives? ([https://ruby-
doc.com/docs/ProgrammingRuby/html/tut_modules.h...](https://ruby-
doc.com/docs/ProgrammingRuby/html/tut_modules.html)) Is there a reason the
editor has to do this rather than the language?

~~~
_bxg1
> Are you thinking of something like Ruby's modules and `include` directives?

Unfortunately I'm not very familiar with Ruby, so I'm not sure

> Is there a reason the editor has to do this rather than the language?

The language would definitely have to support it (though maybe you could
accomplish something similar with annotations and/or special comments in an
existing language), but without special editor support you wouldn't gain much
from it. Your constructs would still be scattered across files in ways that
make compromises. Whereas editor support could pretend the files don't exist
at all.

------
hacker_9
Well this is interesting, I recently spent a weekend creating a 'splicing
compiler', which could take a typescript project split up into features, where
a feature was written in just one single typescript file. Each section of code
could then specify new 'code hooks', for future code to insert itself at
during compilation.

My original inspiration was that English language is spoken sequentially, but
ends up in a graph database (i.e. the brain's neural network). So you'd write
code like a thesis, and the hooks you specify would allow the splicer to form
the code graph.

In theory it seemed like it would be useful, so I built Pacman with it and,
well, it didn't really work in practice! Things ended up being more abstract
in the end, and I'd be mentally juggling where hooks were created and what the
final code would look like. Even showing the code output to the right wasn't
that helpful, as the way the code was spliced together put things in not the
most readable order.

Additionally how to split up the code wasn't the most obvious, i.e. when
should a hook be added? It often felt I'd only know after writing the code in
full. At which point it just felt like extra work. I'd often find myself to be
inserting code all over the place to get something working, and so the
refactoring needed at the end became overwhelming. Ultimately these reasons
lead me to abandon the idea.

~~~
akkartik
I've written 35k LoC with this tool, and it does require some taste but not
intractably so. A couple of rules of thumb:

\- Don't use layers in too fine-grained a manner. Often if I need to change
two lines in a function but they're separated by 6 lines I err on the side of
replacing the entire paragraph or even the entire function. Each layer should
be readable in a self-contained manner as much as possible.

\- I never invalidate tests in an earlier layer. You never want to be in a
situation where a layer lies to you about what desired behavior is.

------
c3534l
It seems like a lot of programming ideas are just ways to manage spaghetti
rather than reduce the spaghetti. I worry that a lot of programming is a sort
of local minima, where people dig themselves into a hole that they can't
iteratively dig themselves out of. Something like this, which allows you to
write code full of cross-cutting concerns and lets the programmer fool
themselves into thinking they wrote clean code because they swept everything
under the rug - that kind of tool is just going to dig deeper holes.

~~~
akkartik
> It seems like a lot of programming ideas are just ways to manage spaghetti
> rather than reduce the spaghetti.

Agreed! This is why I built this approach in a ~1000 lines of code. The
ability for tools to create new problems was very much on my mind.

My current project using this approach is a bootstrapped computing stack from
scratch:
[https://github.com/akkartik/mu#readme](https://github.com/akkartik/mu#readme).
It's at ~40k LoC (of machine code) and self-hosted, and it can create bootable
images (when packaged with an OS kernel). I expect to get to approximately C's
level of expressivity in another ~20k LoC or so. So I care very much about
what I call the "thrust to weight ratio" of a codebase. The less code you
have, the less spaghetti you can create.

My implementation of layers will keel over and fail terribly for large
projects with many team members. I consider that constraint to be a _good_
thing.

------
zwieback
Sometimes I think this is a good idea but generally it seems like refactoring
in a way that the code base always makes sense, even to a newcomer, is the
correct goal. History can be kept in source control, if it's necessary to
follow the history of code development it's probably a sign that the
abstractions are not chosen correctly.

~~~
akkartik
(Author here.)

The analogy to `git log` is just that, an analogy. Here's a concrete thing
that you get with layers that you don't get with `git log` or conventional
abstractions and refactoring: alternatives. Often we see comments in codebases
that some complex, hairy function is really a much simpler function with
certain optimizations applied. There's been lots of work done to try to
separate out functional spec from optimization so that the two can be managed
separately. But this is all open research. Layers let you get the value of
this separation today, with very little code (just a lot of reorientation in
mindset). If you adopt layers you can use the simple version in an earlier
layer, and then replace it in a later layer. Your CI system will exercise
both, and flag if either stops passing tests. What used to be a code comment
now stays up to date.

~~~
zwieback
that's a good point about continuing testing reference and optimized solutions
- I could see a case being made for standardizing that kind of thing

------
veddox
So you‘re trying to remove complexity by making it invisible? Sounds great
while things work, but doesn‘t that turn debugging into a nightmarish
exercise? Because I never get to see the code that‘s actually executing, the
pieces of a function might be scattered over a dozen different layers. Strikes
me as the ultimate recipe for self-modifying spaghetti code... Or am I missing
something here?

~~~
akkartik
I'm not sure what you mean by 'invisible'. Notice that I make zero claims of
reducing complexity. Layers are only a way to manage and stage complexity.

I responded to tooling questions at
[https://news.ycombinator.com/item?id=21766557#21767393](https://news.ycombinator.com/item?id=21766557#21767393)

~~~
magicalhippo
Well studying 010vm.cc, I wouldn't know that 014literal_string.cc alters the
to_string function. That information is hidden from me, until I stumble upon
it in 014literal_string.cc.

~~~
akkartik
Most of the time you wouldn't care. The presence of a waypoint is a hint that
_something_ is inserting code there. But as long as tests in 010vm.cc continue
to pass, it shouldn't matter to it what future layers do.

When debugging things, however, you often want to see the big picture of how
the layers come together. In those situations I look at the generated C++.
It's designed to be clean and easy to read, and it has #line directives so
that you can tell what layer each line came from.

~~~
magicalhippo
But there's no "waypoint" there, it uses the function signature as a marker to
insert the code.

Guess I'd have to use it for a bit to get a feel for how it is in practice. My
initial reaction is that it just moves the complexity around a bit, without
really helping much.

~~~
akkartik
Ah, I see what you're saying. (Links for any others reading:
[https://akkartik.github.io/mu1/html/010vm.cc.html#L606](https://akkartik.github.io/mu1/html/010vm.cc.html#L606)
and
[https://akkartik.github.io/mu1/html/014literal_string.cc.htm...](https://akkartik.github.io/mu1/html/014literal_string.cc.html#L121))

I should have been more hygienic in this old prototype. But honestly this idea
of a waypoint being a hint is something I just thought up. Mostly my mindset
is that layers should be oblivious to later layers. Ditto for readers; the
goal is for you to not have to worry about later layers when you're reading a
simple version.

> My initial reaction is that it just moves the complexity around a bit,
> without really helping much.

From OP: _" We aren't just reordering bits here. There's a new constraint that
has no counterpart in current programming practice — remove a feature, and
everything before it should build and pass its tests."_

If this doesn't seem valuable, I probably can't persuade you. In general I've
grown a lot less concerned about how code looks and a lot more concerned about
having lots of different _runnable versions_ of a single program that a reader
can play around with to build up an instinct for it.

Each unit test in a program is conceptually a distinct runnable version.
Layers push further in that direction.

~~~
magicalhippo
When I read your rationale, it feels a lot like literal programming, as I
experienced it when reading PBR[1].

But without the documentation around it, I'm not sure I feel the same way
about it. It feels harder to reason about what where why.

Also, I'm assuming bugs should be fixed without using this layering. After
all, without that layer you got broken code. But often I find there's not a
clear-cut distinction between feature and bug fix. What then?

Anyway, interesting idea, would have to try it to get a feel like I said.

[1]: For example [http://www.pbr-
book.org/3ed-2018/Sampling_and_Reconstruction...](http://www.pbr-
book.org/3ed-2018/Sampling_and_Reconstruction/Sampling_Interface.html#BasicSamplerInterface)

~~~
akkartik
It's definitely inspired by Literate Programming. My code instructions in OP
refer to a directory called wart/literate.

> But without the documentation around it, I'm not sure I feel the same way
> about it. It feels harder to reason about what where why.

I don't understand this. Are you saying my approach doesn't permit
documentation? I don't see why. Check out the start of the first layer:
[http://akkartik.github.io/mu/html/000organization.cc.html](http://akkartik.github.io/mu/html/000organization.cc.html)

Do you mean that the documentation doesn't look as nice as Literate systems? I
think that's a _good_ thing: [http://akkartik.name/post/literate-
programming](http://akkartik.name/post/literate-programming). Giving important
concepts in your program a good central place is the essential need of
documentation. Layers give features a single consolidated place (a
[https://en.wikipedia.org/wiki/Focal_point_(game_theory)](https://en.wikipedia.org/wiki/Focal_point_\(game_theory\))
) where you can include documentation and be confident that someone interested
in the feature will find it. Next to this attribute, I think typography is
almost meaningless. I think it's far more important that documentation be
accessible from within my programming environment, than that it be typeset
well.

> I'm assuming bugs should be fixed without using this layering.

Yup! From the final paragraph of OP:

 _" Codebases have three kinds of changes: bugfixes, new features and
reorganizations. In my new organization bugfixes modify a single layer.."_

------
SkyBelow
I think the core problem is that one of the root causes of the patchwork hard
to understand code is lack of spending the effort to keep the code
understandable. Refactor into new models, move around code so it makes more
sense given changes in requirements, etc.

If that level of maintenance isn't being performed (which is almost a given,
assuming we are looking at a problem with such a patchwork problem), then this
new tool won't be fully followed and the end result will be a patchwork code
base that reaches nightmarish levels where some features are added in this
layered approach, but other features are added directly to the code, and even
other features are added directly to prior layers.

If the tooling force people to use this in the recommended way then it may
have interesting payoffs that are worthwhile. But lacking such an enforced
requirement, I don't see applying being beneficial except in the code bases
that least benefit from it.

~~~
akkartik
I find this approach to be easier than conventional approaches, and my
motivation and effort levels seem constant on both sides. You're right that
there's a social component here. This tool tries to make things easier for
authors _under the assumption that people try to use it tastefully_.

------
mdewing
There seems to be some confusion between layers and AOP. Layers are a static
mechanism - all the layers are composed to a final program before compilation.
The layers have no effect at runtime (unlike AOP). If the all the layers are
composed (tangled) without the '#line' directives, the code should look (and
be) the same as a normally-written program. Modifying the program most likely
will require adding a new layer. (If "layer" == "git commit" then this is only
option under existing programming practice.) Programming with layers means the
programmer has the additional option of modifying previous layers. Whether
this can scale to large programs is an open (and interesting) question.

~~~
akkartik
Thank you for the clear articulation!

------
jillesvangurp
Reminds me of Layom, which is something my phd supervisor worked on in the
nineties. Later aspect oriented programming (which is similar) became a thing
with languages like AspectJ around 1999 or so.

These days you see a weaker form of this in the form of annotations in e.g.
Python or Java or macros in Rust. Spring actually uses an AOP library that has
its roots in AspectJ.

I actually prefer a more recent trend in e.g. Kotlin to use internal DSLs
instead of injected magic. The advantage is that it is simpler to debug and
part of normal type checks, autocomplete, etc. since technically all you are
doing is working with the language instead of augmenting it via generated
code.

------
cryptonector
Once you get to codebase sizes like... an OS, a programming language's tools,
a database... this sort of thing begins to look awfully simplistic.

Instead you have to properly clean tech debt from time to time.

~~~
akkartik
Or you have to keep the codebase small. I'm not convinced an OS or programming
language or database needs millions of lines of code.

I'm not alone in this scepticism. Check out Alan Kay's STEPS project from a
few years ago. The goal was to create a complete computing stack in 20k LoC.

My current project has a similar goal:
[https://github.com/akkartik/mu#readme](https://github.com/akkartik/mu#readme)

You mentioned tech debt. I think I have a more aggressive definition of tech
debt than most: any dependency you rely on that you don't understand the
internals of is tech debt. It's fine to take on in the short term, but you
should be planning to pay it off in the long term. The approach modern
software takes, of continually adding dependencies (each of which is itself
growing monotonically complex) faster than anyone can internalize them, this
approach feels utterly insane. Under this definition, my tiny tool is the
least of someone's problems if say they depend on 60 Ruby Gems.

Look at my definition of x86 instructions:
[http://akkartik.github.io/mu/html/013direct_addressing.cc.ht...](http://akkartik.github.io/mu/html/013direct_addressing.cc.html)
If this is tech debt, I'll stay in debtors' prison, because it's safer inside
than outside.

~~~
cryptonector
Once you ship it gets very difficult to keep the codebase small because your
customers will want you to stay backwards-compatible.

This is a problem you want to have, of course, because you want to ship
because you want to have revenue because you want to have earnings -- or just
because you want to ship open source and keep a user community happy even if
you make no money from it.

~~~
akkartik
Making breaking changes and pushing them to your customers without warning is
bad, no question.

Staying compatible with everything forever is equally bad, but we only realize
that over relatively long timescales.

I think there's a middle-ground between these two extremes that hasn't been
explored: when you make an incompatible change, fork. Patch only the worst
bugfixes and security fixes on old forks (but continue these indefinitely).
Point new users at the latest fork in each point in time. Support switching
forks on your users' schedule.

This approach adds some overhead for authors and maintainers. But I think it's
not as much as we fear. And it can be automated over time.

There's a selfish reason this never happens, but we don't talk about it in
polite company: if you start forking yourself you make it more acceptable for
others to fork you. I think that's a good thing. We should all stop playing
shallow politics and encourage being forked by others.

~~~
cryptonector
Sure, you tell your customers that in so much time you'll be removing some
feature. If you've got a large enough and popular enough OS, say, even so your
codebase will get very large.

~~~
akkartik
No, you didn't understand what I meant. There is no single codebase serving
all users. Some users are on v1, and they use its codebase. Some users are on
v2, and they use _its_ codebase. And on and on. There is no notion of removing
features. All forks live forever. They just get updated less and less
frequently over time. No single fork grows larger and larger over time.

~~~
cryptonector
No, that doesn't work.

~~~
akkartik
It may well be a bad idea. But I don't think you can be sure until we try it.

------
canadaduane
Some other, related ideas & paradigms:

Context-Oriented Programming:
[https://arxiv.org/pdf/1105.0069.pdf](https://arxiv.org/pdf/1105.0069.pdf)

Feature-Oriented Software Development: [https://en.wikipedia.org/wiki/Feature-
oriented_programming](https://en.wikipedia.org/wiki/Feature-
oriented_programming)

~~~
akkartik
Yes, I was aware of both as well as AOP when I built this. Some comments on OP
discuss them as well.

The big difference between my approach and these is the weight-to-thrust
ratio. These academic approaches tend to assume people don't want to change
the pieces, just combine them. And they create huge tools to make that
possible. Why can't people just change the pieces? I've never understood that.
I don't need an algebra of program transformations intermediating between me
and my code. I'd much rather hold the code directly in my mind. And that keeps
the tooling needed really, really small. Just a hundred or so lines of C.

Tooling is a double-edged thing. Adding tooling often encourages increases in
codebase scale. Keeping tooling rudimentary can be a good thing by keeping
codebases small.

------
jimbo1qaz
Fascinating idea. I've definitely written and encountered programs where
functions were more complicated than they could've been, in order to support
features separate from the "main idea" of the program. Like FPS calculation,
Ctrl+C handling, passing intermediate audio data to extra buffers for
visualization...

Could "splicing in" code it be implemented using aspect-oriented programming
frameworks? [https://en.wikipedia.org/wiki/Aspect-
oriented_programming](https://en.wikipedia.org/wiki/Aspect-
oriented_programming) I've never used them though.

> There's a new constraint that has no counterpart in current programming
> practice — remove a feature, and everything before it should build and pass
> its tests: > build_and_test_until

Looks like added tooling is needed, but seems worthwhile.

------
daralthus
Interesting, although the features might leak into other codebases and
services too, at which point compiler directives won't be enough.

I believe this is a lot more tractable with Unision's[0] content addressable
hashing. It's kinda made for that although more with an eye towards updates to
distributed systems.

[0] [https://github.com/unisonweb/unison](https://github.com/unisonweb/unison)

------
spoondan
About 15 years ago, I built a system like this for a web application. I
defined a series of core extension points such as the request router and
authentication and authorization providers. Modules provided extensions that
connected to the core extension points, as well as define their own extension
points that other modules could connect to. Dependency injection was used
throughout so that implementations could be easily swapped (mostly used for
the tests). It took quite a bit of time to develop this framework, but, as a
result, everything was modularized, contained, and composed.

It was a complete nightmare to reason about and work with, even for me, and
especially for newcomers. A natural way of trying to understand a system is to
look at its entry point. With a system like this, it looks like a skeleton:

    
    
        let di := DependencyContainer.make()
        let moduleSystem := ModuleSystem.with_container(di)
    
        di.register_singleton[IModuleSystem](moduleSystem)
    
        moduleSystem.find_and_register_modules()
    

Where do we go from here? The next logical step is into
`find_and_register_modules` or into the documentation for the module system
(just kidding: I was a "senior engineer," I didn't write docs).

We are left with a lot of questions. How does anything happen? It's somewhere
inside the modules, but which one? Does the order that modules are loaded
matter? The answer is maybe. Clearly, there are places where order of
operations matter. For example, the main navigation should be ordered. How do
we accomplish that? (The answer for my system was to have the interface for
navigation extensions specify there was a `readonly weight: Float64`
attribute. But then to actually understand why things are in the order they're
in, and to get things ordered correctly, you need to look up the values for
other navigation items. It's secret coupling.)

More recently, I saw a team at my previous company build a system like this.
There were dozens of interfaces and implementation classes and code for
composing all of this, and the end result was you couldn't just go somewhere
and see what was happening. What we really want to see is:

    
    
        match maybe_user
            Some(&user) => nav.add(UserProfileNavItem.for_user(user))
            None => nav.add("Login", "/login")
    
        nav.add("Browse", "/browse")
        nav.add("Search", "/search")
        nav.add("Help", "/help")
    

Do you have to make a change here whenever something gets added? Yes. You do.
But the idea that your program needs infinite flexibility in all areas is
simply wrong. And, in fact, everything ends up secretly coupled and
inscrutable.

For this reason, my advice has long been to limit extensibility to specific,
narrow use cases (for example, filters in an image editing program). Do not
build entire systems around extensibility. You are not building an abstract
system, so don't waste your time with unnecessary and obscuring abstractions.

~~~
akkartik
Author here. I read your comment and just think I'm not a very good writer.
Layers are absolutely not about adding extension points, and layers have
nothing to do with modules with fixed interfaces. Often when I want to extend
behavior in my layered programs, I just modify the line in place.

I mentioned a couple of rules of thumb for when I create new layers here:
[https://news.ycombinator.com/item?id=21766557#21767499](https://news.ycombinator.com/item?id=21766557#21767499)

Here's the entry point for my current project which uses layers:
[http://akkartik.github.io/mu/html/000organization.cc.html](http://akkartik.github.io/mu/html/000organization.cc.html)

You can find a list of layers here:
[https://akkartik.github.io/mu1](https://akkartik.github.io/mu1) (URL is
slightly different; it's a previous prototype. But should suffice for this
thread.)

------
crazypython
This is just AOP.

~~~
canadaduane
It's related, but I don't think AOP has a concept of an immutable past?

~~~
akkartik
Hmm, layers don't have an immutable past. Can you elaborate?

~~~
canadaduane
Ah, I misunderstood. I thought the layer stack was immutable.

------
bruth
Posted in 2013

