
Application Layering – A Pattern for Extensible Elixir Application Design - aaronrenner
https://aaronrenner.io/2019/09/18/application-layering-a-pattern-for-extensible-elixir-application-design.html
======
jerf
This is good stuff, not just for Elixir. I've been increasingly heading this
way in my Go programs as well, and anything else that I've had to write of any
significant size.

One of the interesting side effects is that the whole microservices thing just
kinda vaporizes as a consideration when you can just take a chunk of your
program and run it on another server or something if you need to. I'm still
not doing this as fluidly as I'd like, but I've definitely ripped bits out and
also merged some bits before without too much hassle.

One of the major problems I've hit with this is just that the mechanisms for
composing these pieces together is all over the board. Do you want to plug in
a handler for some chunk of the URL space? That works entirely different from
composing in two network services on two different ports, which works entirely
differently from creating a new protocol that has plugable components, which
works entirely different from composing together two services that both end up
sharing the same database connection pool.

The other problem I've had is the availability of code that can be mixed and
matched like this. One example I recently fixed up in my code base is that we
previously had a module for managing metrics for the entire project, which
declared each metric value. It was non-trivial work to take everything that
module did and made it so each sub-application could register its own metrics
independently. It wasn't _hard_ ; the design that leaps to your mind is
probably basically what I implemented. But it also added yet another way in
which composing these things together is a challenge. Composable configuration
is also annoyingly difficult.

There's no one big thing that makes this hard, it's a ton of little things
everywhere that afford you tying your program together more tightly than it
really has to be. I think that if languages and/or libraries supported this
better, it wouldn't be _that_ much more difficult to write solid code that
worked this way all the time, but right now there's just impediments
everywhere.

~~~
hinkley
> different from composing in two network services on two different ports

I don't think this should be a thing anymore either. Container orchestration
is slowly fixing this by default but really, it shouldn't be my server's
business to track and figure out where all the other services are. That should
be in one place. In production it should be owned by Ops. In local it should
be optionally controlled by the dev.

The dev should only have to worry about server names when they're
collaborating on a bug fix with the owner of a particular service. The rest of
the time they should send it all to a single proxy on a single port, with
everybody namespaced. Let the proxy talk to Consul or what have you to sort
out the rest.

This is also one of the tricks you use to break up a monolith, but it has
other uses as well.

~~~
jerf
"I don't think this should be a thing anymore either."

I routinely set my primary web services on one port, and the access to
debugging and profiling information is available on a web server running on an
entirely different port.

In principle, this is not necessary, because everybody and everything should
be able to route by URLs, and of course, everybody is using these tools and is
intimately familiar with them. In practice, if I depend on that, those things
tend to "fail open" and mean that if I don't explicitly block my debug routes
from being exposed, it will tend to end up set up so just anybody will be able
to reach them. By having by debug & monitoring on a separate port entirely, it
makes it so that functionality tends to fail closed; if I don't open that port
to people explicitly, it just stays entirely inaccessible, which is the
failure state I'd rather have.

Also, there's a whole world of people who aren't working in an environment
where there's a dozen people dedicated to maintaining a kubernetes setup for
them, and are still building services that aren't built in that paradigm. Plus
it seems to me like having really amazingly separable services, even ones not
initially thought of as being separate, is a plus for Kubernetes rather a
problem.

~~~
hinkley
I'm talking about having to configure separate fully qualified URLs for every
service in your company, with server name and port, versus talking to a proxy
which knows all these mappings. When your apps are >1 the cost of maintaining
these lists everywhere becomes painful.

I brought up Kubernetes because Kubernetes makes you solve this out of the
gate.

------
dvdhnt
<rant>

As an aside, I find it interesting that we are still contained by the
limitations of our surfaces, specifically paper and screens. Hell, one could
throw stairs and ladders into that.

The result seems to be a mindset that the holy grail of software excellence is
distilling our work down to a tree-based structure that will result in us
experiencing some Thanos-like post-snap rest.

Yet, at the risk of mass downvotes, I dare say it doesn't particularly matter
in 99% of our work. Unless you are working on software that could take/give
life or increase/decrease suffering.

Does that mean we should just throw our hands up and build slop? No, it just
means that perhaps we would all be better off if we embraced the ideal that
our software is more like an organism than a space shuttle. Birth it, mold it,
shape it, feed it. Let it grow into a robust, complex thing. Eventually, it
will grow old and you can let it die. It will have been both beautiful and
disgusting depending on whom you ask.

The point was to say software is at the very least three dimensional. Reducing
it to anything less and applying a handful of labels that all end in "ability"
will not have made your life worth living.

</rant>

Go outside today.

~~~
hinkley
How would you rate yourself on spatial intelligence?

I'm sure there was a time I could say exactly what you did but I am not
'normal' with respect to being able to reason about objects. If I help a
friend move I am forever re-packing their car to reduce the number of car-
loads to one less than they thought they needed.

They try to debunk 'learning styles' routinely but the fact of the matter is
that certain media are much more _comfortable_ for many people, and 3-D is bad
for certain modes of reasoning. But perhaps as importantly, it's bad for some
forms of tooling. For example, "show me the difference between the model now
and a year ago when we first discussed this." We use tools like this as a kind
of 'isomorphic projection' of something that has way more than 3 dimensions so
we can figure out the 'shape' of things.

~~~
dvdhnt
I would rate myself high on spatial intelligence based on my love of solving
puzzles, and anecdotally, my perfect score on that portion of military
testing.

However, I will admit that as I age, my interest in arbitrary puzzles has
declined. I much more enjoy putting together puzzles of scenes or artwork I
find attractive.

Funny enough, I would say I used to do just as you describe - rearrange
carloads of things or otherwise challenge myself for whatever reason to come
in "under budget" on space and time. It's probably from sheer exhaustion that
my drive for doing so diminished.

Your closing thoughts inspired me to search for 'isomorphic projection'; that
led me to find 'graph isomorphism' [1]. I have not dived deep into that
particular page, but the image of Graph H is very interesting as it
illustrates what I see in my head when trying to distill software.

1\.
[https://en.wikipedia.org/wiki/Graph_isomorphism](https://en.wikipedia.org/wiki/Graph_isomorphism)

------
benologist
I have been extracting these layers of boilerplate like accounts you need to
launch a web app into a parallel web application that accompanies your web app
instead of being bundled with it. I'll be launching this in October.

[https://userdashboard.github.io/home](https://userdashboard.github.io/home)

The software provides UIs for users and administrators and APIs and tests and
documentation for accounts, organizations, Stripe subscriptions and Stripe
Connect platforms all added through modules. It proxies your application
server, and you either serve your application within a template or occupy the
entire page, how you write it and what you use is left to you. My software
tells you who is using your server, and is fully API-driven in case you want
more info.

It's ready to use now, and the core software is ready to go but Connect
requires client side JavaScript for France, and subscriptions requires the new
payment SCA flows which are in-place but doesn't have tests yet, and
localization needs to be finished up and that's all probably another week or
so.

------
AlchemistCamp
This is interesting. So far, I haven't encountered any Elixir apps that have
grown beyond the limits of Phoenix contexts. I've encountered quite a few
umbrella apps in the wild, but they were generally written before contexts
were introduced or by people unfamiliar with them.

I'd _love_ to work on an Elixir codebase the size of the one it sounds like
the author is!

~~~
borski
Check out Tinfoil. We use a ton of Elixir and our newest product (API Scanner)
is all Elixir. :)

Shoot me an email at borski@tinfoilsecurity.com if you’re interested. :)

~~~
loceng
Just wanted to say I enjoy your company's name. :)

Edit: Someone downvoted this? There really are some miserable people on HN.

------
roylez
This is good stuff and makes me rethink about one of my personal projects.

However, I have reserved opinion about the idea that a module should only
access its direct children. More than often, I find modules interact with each
other forming a loop, either by direct function calls or by message passing.
If having to organize all modules into tree structure, I sometimes find it is
hard to decide which modules should act as parents without introducing
additional layers of abstraction. Also, what if I would like to pass a message
from children to their parent? As according to the article, it is better to
make children swappable and do not rely on parents' code, either direct
calling parental functions or passing message to parent's pid looks bad, all I
can think of is to use an event bus, which is extra complexity. Not to mention
after so many abstractions module names could be too long to be clear.

Personally now I am good with parent, children, and siblings talking to each
other, beyond these is probably too far. Not every module needs to be
swappable, and too much abstraction makes stuff harder to understand and not
always easier to maintain. Erlang thrived probably even before TDD was there
with its "let it crash" philosophy. I do not mind not having 100% test
coverage, as long as every crash scenario has been fixed along the way and
possibly test guarded.

------
bdibs
I would love to see a real world use for this, maybe I’m underestimating the
scale of an application that would need this.

I think Phoenix contexts are plenty and I’m not sure adding more layers of
abstraction is a great idea.

------
rodmena
This article is a step into the right direction. Useful. Thanks Aaron.

