I do a lot of stuff with my house. As a result I have on hand, at various points in time:
* contact cement
* superglue
* mastic
* wood glue
* gorilla glue
* liquid nails
* various caulks
Why so much glue? Because the right glue to use depends on the materials being joined, and the context in which the joint will need to survive (moisture, temperature, duration).
As with construction and art, so with software. The Unix pipe is powerful because it enforced "the only thing you can connect together is line-oriented byte streams". But when you are building something where that's not the substrates in question, it's the wrong tool, and you need a different kind of glue.
Expand that to all the possible different things you may wish/need to "glue" together in software, and it becomes fairly apparent why there's a huge diversity in the glue being used, and why dreaming of a world in which you get by with just the software analog of gorilla glue is the wrong thing to do.
> Why so much glue? Because the right glue to use depends on the materials being joined, and the context in which the joint will need to survive (moisture, temperature, duration).
On that topic, I am obligated to link to a wonderful no-nonsense guide choosing on the best glue for your particular materials:
Unix pipes and filters were not presented as "the" solution, but as an example for the properties we want from our glue, i.e. we should be able to glue our pieces together without having to write much additional code.
However, for most programming tasks, we have one kind of glue (with minor sub-variants): the procedure call. And so instead of having something that snaps together with a bit of glue, we have to write enormous amounts of boiler-plate in order to make the pieces fit. (The analogy only goes so far...)
I'm been a compiled-language programmer for 35 or so years. I don't view procedure calls as glue, I perceive them as a fundamental mechanism for organizing code. At the level of code that I normally operate at, if you don't write everything in-line, then you've got procedure calls. Not glue, at least, not primarily.
Glue is a way to move data (and perhaps other stuff too) across code boundaries that are imposed by design, language or other constraints. So, sure, stuffing a bunch of stuff on the stack, jumping to a new address and doing something with whatever is on the stack is, in some sense, a sort of glue.
But your TFA seems to be talking about a much higher level of glue than this, I think. The sort of glue that's represented by things like (random list): protobufs, RPC, serialized JSON, even ye olde CORBA/OLE models. Questions like "I've got this customer data, I need to give it to some code that will munge it in some way... how do I do that?"
In that sense, procedure calls are low level mechanism that will support a higher level solution. Again, if it's not all just inline execution of machine code, then at some point, there's going to be a procedure call no matter what abstraction level you're working at.
> I'm been a compiled-language programmer for 35 or so years.
40 or so for me. :-)
> I perceive [procedure calls] as a fundamental mechanism for organizing code.
Yes, I've been arguing this for some time: that the call/return architectural style has been so incredible dominant for so long that it is not really seen as a specific architectural style, but instead as simply all there is. It is the mechanism for organising code.
A paradigm in the true sense.
I call this The Gentle Tyranny of Call/Return[1]. "Gentle" because as architectural style go, it's probably one of the better ones to have, particularly if you only get one.
Anyway, if you study software architecture, you will find that procedure calls[2] are just one of many ways to organise software. However, they are given special status, because they have special language support. So if you use procedure calls to organise your code, to glue your components together, you get to write code naturally. If not, it starts to look messy. And it turns out that procedure calls not always ideal, because...
> Glue is a way to move data (and perhaps other stuff too) across code boundaries that are imposed
And a lot of what we do today is moving data around. Computers don't really compute all that much[3].
>But your TFA seems to be talking about a much higher level of glue than this, I think.. protobufs, RPC, serialized JSON, even ye olde CORBA/OLE models
Yes, these are also glue. But so are procedure calls.
> In that sense, procedure calls are low level mechanism that will support a higher level solution.
One might even say they are the "Assembly Language" for connecting: [2]
Also, note that the use of a pipe from a Unix shell is very different from the use of a pipe in most (all?) languages. So the property you're interested in is not really the fundamental mechanism (which is quite a complicated object, with error conditions and required logic to handle i/o), but the abstraction presented by the shell.
Correct, no single interface does what the UNIX pipe did for command-line text-based IO-stream processing programs. In FOAM, we have about 20 standard interfaces, which correspond to your different glue types. Things like DAO (Data Access Object), Model, View, Factory, Parser, Adapter, Authenticator, Outputter, Validator, ... and as long as everything that does a certain thing, implements the corresponding interface, then everything can be piped together. Again, see: https://www.youtube.com/watch?v=S4LbUv5FsGQ or the longer more complete version: https://www.youtube.com/watch?v=PsFLlgrzn2E
To continue your analogy, if 95% of your house were made of glue, and 85% of the cost of your house was to buy glue, then that would be some cause for concern. That is pretty close to the truth for software.
Interesting observations; although by definition glue code is amorphous and hard to pin down, I would like to see more concrete examples of the problem discussed here.
Is this a general problem related to how we build interfaces for our implementations? Or is this more a problem with particular kinds of module systems? (Or both)
It's all about compartmentalized code... Glue happens whenever compartments interact with one another. Examples, thinking of an app I used to work on:
a) App frontend (UI logic) is compartmentalized from the app backend. Communication is handled by an async messaging system, requiring bundling up messages in both directions.
b) The app backend talks to a few different servers, each with their on expected message set. glue glue glue.
c) The app backend talks to the app database, inserting/altering info, and extracting database rows into objects which can be passed around.
d) (We also had peer-to-peer communication over local wifi, which required its own massive pile of communication code...)
e) Even different objects/methods in the backend code end up requiring small amounts of glue code to arrange+prepare argument sets to call one another.
So, every time we create a division between two pieces of code, we introduce boundaries, which introduces glue.
---
One way to mitigate the complexity is by making the passed messages first class. In our app, there was some 'AppThing' object which represented an instance of the fundamental thing the app was about. You then stuff that object with every piece of info about AppThings you could ever want, and pass it around freely between components. Invariably it gets kinda bloated and fractures, when someone doesn't want to bother building up the WHOLE thing with the necessary DB and server calls to ensure the data is all accurately filled in, when you just want to pass one or two fields to a neighboring UI element. And then you find yourself with lots of logic checking that certain fields are present amongst the million fields in your universal object...
Another approach is to aggressively prune the graph of possible component interactions. Functionality should be well-encapsulated, with absolutely minimal APIs for external interactions. Actually draw the 'cell membranes' between different parts of the app, map out the interactions between cells, and find ways to cut it back.
I think it’s interfaces and protocols at a granular level.
If every interface is bespoke while also being explicitly described top down, then we end up with this quadratic complexity.
Uniform, extensible interfaces and generic data structures are the solutions to this, as well as shifting complexity to data and away from code. Several modern languages, techniques and protocols address this in varying degrees and from different angles. An old language that does this very well is SQL.
I get frustrated with the layers between relational data and data structures you pass around in code.
So for example, if you want a build a simple REST api for your data model, you have glue code to read/write the data and transfer it to/from DAO's (Data access objects). Then you have glue code to transfer the data into to/from object model you expose via the API. This feels like 90% of server side CRUD app programming these days.
It's not quite boilerplate because it's a mix of ORM optimizations for your data model and data massaging to make it more digestable to end users of your API.
It feels very applicable to the functional vs imperative programming debate, and I say that as someone who's mostly on the imperative side. The top row of the "perimeter" is very "functional".
It's interesting to note that people from the same school of thought as those who built Unix (or even the same actual people) ended up creating Golang, which is anything but a functional language.
You have two systems, so you need to write a layer of glue code to make them work together.
The glue code turns out inelegant, spaghetti-like, hard to work with. So you spend a little bit of time cleaning it up, abstracting reusable things out, and generally systematizing things.
Now you have three systems, so you need to write two layers of glue code to make them work together…
I wrote a non-programmer summary to some of my friends, because my mind is blown. Here it is (warning: it's very informal).
This is some next level 80/20 stuff, I'd argue it's even 99/1 (consultant speech for "being efficient" by seeing if you can exploit Pareto's principle [0]).
A guy on YT [1] simulates in 7 minutes how the two biggest operating systems were built in the sixties (Multics by IBM and Unix by 2 guys in a garage). Nowadays we only have descendants of Unix, this simulation shows IMO two things:
1. How a chain of command screws up productivity (I doubt that anyone at IBM had the power/autonomy to change complete structure of the product. This was needed, but such change management is insane).
2. How the pipe command in Unix changed the mathematical nature of developing an operating system (in fact, it's even better than 99/1, when you think about it). In normal human language, the difference is: IBM programmed modular blocks. Unix programmed modular blocks. IBM then programmed how those modular blocks should interact. Unix created a pipe command and allowed computer users themselves to implement how these modular blocks should interact. At the time, computer users were all savvy enough like that.
Result: when IBM created a feature, it needed to integrate it with all the other features that came before. When those 2 guys in a garage (Unix) created a feature, all they needed to do was create that feature.
So mathematically you could say (where each term is the amount of work required for one new feature + integrating it with the operating system):
IBM/Multics: (0 features_before + 1 new_feature) + (1 features_before + 1 new_feature) + (2 features_before + 1 new_feature) + (3 features_before + 1 new_feature) + (4 features_before + 1 new_feature) + (5 features_before + 1 new_feature) + (6 features_before + 1 new_feature) ... + (n_features_before + 1 new_feature) --> simplifies to my favorite formula which is: n(n−1)/2 --> you can also see it as --> 0.5n^2 - 0.5n --> So, roughly this is n^2.
Just count the amount of numbers, it comes down to: 28 units at the 7th term/timestep.
open standards. there must be one of those game-theory economic analyses.
but frankly i don't see how its _ever_ a good idea to structure yourself around closed worlds unless the seller really has something that novel and indispensable.
I'm not advocating a closed system. Just like UNIX doesn't lock you in to any hardware or CPU vendor. FOAM is open source and most of the functionality is provided by existing languages, libraries and databases.A universal adapter, not a universal provider. https://www.youtube.com/watch?v=PsFLlgrzn2E
https://www.youtube.com/watch?v=S4LbUv5FsGQ covers this topic.
As I say in the video, no point solution (ie. some new API) can solve the problem, when the need to integrate (glue) between so many point solutions, is the problem.
I think this the most important problem in software development to be given the least amount of attention, or even awareness. Another new MVC framework, build tool, or even programming language, isn't going to solve this problem. First we need to acknowledge the problem.
As with construction and art, so with software. The Unix pipe is powerful because it enforced "the only thing you can connect together is line-oriented byte streams". But when you are building something where that's not the substrates in question, it's the wrong tool, and you need a different kind of glue.
Expand that to all the possible different things you may wish/need to "glue" together in software, and it becomes fairly apparent why there's a huge diversity in the glue being used, and why dreaming of a world in which you get by with just the software analog of gorilla glue is the wrong thing to do.