"Modernism" is basically characterized by attempts to predict and control the future, to rationalize and systematize chaos. In the past this was done with things like top-down centrally planned economies and from-scratch redesigns of entire cities (e.g. Le Corbusier's Radiant City), even rearchitectures of time itself (the French Revolution decimal time). The same kinds of "grand plans" are repeated in today's ambitious software engineering abstractions, which almost never survive contact with reality.
Yonks ago, i was reading an introduction to metaphysics at the same time as an introduction to Smalltalk, and it struck me that the metaphysicians' pursuit of ontology was quite similar to the Smalltalker's construction of their class hierarchy. The crucial difference, it seemed to me, was that the metaphysicians thought they were pursuing a truth that existed beyond them, whereas the Smalltalkers were aware that they were creating something from within them.
It's very likely that my understanding of one or both sides was wrong. But ever since then, i've always seen the process of abstraction in software as creating something useful rather than discovering something true. A consequence of that is being much more willing to throw away abstractions that aren't working, but also to accept that it's okay for an abstraction to be imperfect, if it's still useful.
I, probably arrogantly, speculate that metaphysicians would benefit from adopting the Smalltalkers' worldview.
I, perhaps incorrectly, think that the ontologists' delusion is endemic in the academic functional programming  community.
Software, in my personal experience, is closest to the study of mathematics: there is an arbitrary part (the choice of axioms)—but, once that part is in place, you must obey those axioms and discover what facts are true within them.
If you don't obey your own chosen axioms, the system you will create will be incoherent. In math, this means it just fails to prove anything. In software, this means that it might still be useful, but it fails to obey the (stronger) Principle of Least Surprise.
The regular PoLS is just about familiarity, effectively.
The strong PoLS is more interesting. It goes something like: "you should be able to predict what the entire system will look like by learning any arbitrary subset of it."
The nice thing that obeying the strong PoLS gets you, is that anyone can come in, learn the guiding axioms of the system from exposure, and then, when they add new components to the system, those components will also fit naturally into the system, because they'll be relying on the same axioms.
However, that's almost never the way math is originally developed. As a student one gets this impression, but that is usually on a topic that has been distilled and iterated over again and again, with people spending a lot of time on how to line out the "storyline" of a subfield.
More commonly, some special case is first encountered and then someone tries to isolate the most difficult core problem involved, stripping down the irrelevant distractions. The axioms don't come out of the blue. If a certain set of axioms don't pan out as expected (don't produce the desired behavior that you want to model with them, but for example "collapse" to a trivial and boring theory), then the axioms are tweaked. Indeed, most math was first developed without having (or feeling the need for) very precise axioms, including geometry, calculus and linear algebra.
I don't address this to you specifically, but I see that similar views of math in education make people believe it's some mechanistic rule-following and a very rigid activity. I think it would help if more of this "design" aspect of math was also shown.
Even when mathematicians feel they are discovering something, they rarely feel like discovering axioms, more like types of "complexity" or interesting behavior of abstract systems, where this complexity still has to be deliberately expressed as formal axioms and theory, but I'd say that's more like design or engineering or the exact word choice for a writer vs. the plot or the overall story.
Seeing Like a State describes a similar situation, where people had to give themselves last names so they could be taxed reliably. I wonder how many people have had to enter their Facebook name differently for it to be considered a "real name"? Software that "sees" the world in a particular way is a lot like a state bureaucracy that "sees" the world in a particular way, especially when this software is given power.
The French were 99% successful with their decimals everywhere (everything except time) in 99% of all countries (everywhere except the US and some freak dictatorships).
> The same kinds of "grand plans" are repeated in today's ambitious software engineering abstractions, which almost never survive contact with reality.
Unless these "grand plans" just standardize and compromise the right things. Then they conquer the world (PC, USB, TCP, HTML, ...)
> The state could insist on the exclusive use of its units in the courts, in the state school system, and in such documents as property deeds, legal contracts, and tax codes. Outside these official spheres, the metric system made its way only very slowly. In spite of a decree for confiscating toise sticks in shops and replacing them with meter sticks, the populace continued to use the older system, often marking their meter sticks with the old measures. Even as late as 1828 the new measures were more a part of le pays legal than of le pays reel. As Chateaubriand remarked, "Whenever you meet a fellow who, instead of talking arpents, toises, and pieds, refers to hectares, meters, and centimeters, rest assured, the man is a prefect."
I would imagine likewise for the other standards that you mentioned. And in a lot of ways they succeeded exactly because they weren't (thinking specifically about how HTTP/HTML succeeded because it was less comprehensive than, say, Xanadu).
But calling the "decimal time" a failure is just unfair, given that they rest of the program was a huge success getting rid of what we would call today "legacy" (of course in hindsight. When else?)
Given how long it took to transition even in France, it sounds pretty expensive. Given that the rest of Europe wouldn't have bothered with it if the French hadn't gone on a crazy conquering spree, it sounds pretty expensive.
Given that the imperial system actual works pretty fine for those who use it, and given that we are stuck with lots of discordant units of measure anyway, and new ones are being invented every day, it doesn't look like the gain was with the cost.
I'm glad we try to understand the world and bend it to our will.
Further, one can experiment with different micro-abstractions without investing too much time. They are micro-experiments. If one fails to provide re-use or flexibility, so be it: just stop using it. It's like picking stocks: some stocks will wilt, but if you buy enough, you'll roughly approximate the general market at about 10% annual growth.
I do find dynamic languages make such mixing and matching easier, though, because they are less hard-wired to a specific type or iteration interface: less converting needed to re-glue. I've yet to get as much re-use in type-heavy languages, but not ruling it out. I just haven't cracked that nut yet. Maybe smarter people can.
Abstractions usually come earlier in the design cycle than optimization for me and the wrong one makes that later stage harder.
> I do find dynamic languages make such mixing and matching easier, though, because they are less hard-wired to a specific type or iteration interface: less converting needed to re-glue. I've yet to get as much re-use in type-heavy languages, but not ruling it out.
I go back and forth. I mostly develop in a mix of Python and Java. Python may give me a lot of room to explore ideas, but I find refactoring a fragile nightmare. Java's type system is fairly simple and lets me quickly pull up all use sites and trivaly catches misspellings.
It lets me back out of and refector the wrong abstraction much more simply.
Could we say the much-debated ORM is (usually) a large-scale abstraction?
I've built up a series of "SQL helper" functions to do a lot of the SQL syntactic and Prepared Statement grunt-work to avoid ORM's. They are NOT intended to entirely wrap or hide the RDBMS, but rather help with the app's interaction with the RDBMS. Helpers rather than wrappers. If the helpers don't work in a particular case, you can ignore them with minimal code rework. If you depend heavily on an ORM, such is often not the case.
On the more general point, I think that the larger scale the abstraction is, the larger the number of actually encountered use cases needs to be. You shouldn't write an ORM because you feel like existing ones don't serve your specific needs. You should write SQL, and then replace it with an ORM once it's clear you will benefit from the extra abstraction.
There seems to be a fairly consistent trade-off between the scale of the abstraction, the degree of over-engineering, and the leakiness of the abstraction. So if you're going for a large scale abstraction, then I'd want lots of proof that it's not a leaky one.
I'm mostly wailing against writing your own ORM (and also other forms of "not invented here" syndrome), which is something I've seen done way, way too many times, because you have to reinvent a ton of things you probably don't have much experience with, will probably get wrong in ways you don't anticipate, and will miss scaling requirements you don't know about yet. Because of this, I think there should be a heavy burden of experience on writing "core" libraries.
That's situational, of course. My current projects are tech giant things, so some choices actually do have a >3 year window, but then there's a tech giant sized stack of use cases to design against.
When my projects were startup problems though, I'd be happy if that component still worked well after 3 years of growing and pivoting. In that kind of environment, getting more for free is by itself valuable for your ability to pivot quickly.
I'm not sure what you mean in the middle paragraph, per "stack of use cases to design against". You mean getting it done "now" overrides problems 3 years from now? Your org making that trade-off is fine as long as it's clear to everyone and perhaps documented.
ORMs are not done yet :-)
Yes! I think all of the people I've met who advocate the article's position were using static languages that make it harder to re-use what they've written. This article sounds like it's about a C++ project, which doesn't surprise me.
Where are all the articles about this happening to dynamic data-driven projects? In contrast, people seem to go to incredible lengths to keep old Lisp code running. A lot of people (even non-Lispers!) would apparently rather write a new Lisp interpreter than port their Lisp code to another language -- because the Lisp code is largely a declarative description of the problem and solution.
Data itself is already a Grand Abstraction that has survived the test of time. I don't need a library of mini-classes.
Lisp does make "swapping" between data and behavior easier, but many say Lisp code is just hard to read in practice. I've been in Yuuuge debates about Lisp, but my summary conclusion is that many just find existing Lisp code hard to read. Perhaps its too much abstraction. Short of eugenics, we are stuck with that problem. Lisp has had 50+ years to try mainstreaming; time to give up.
I've seen that often enough that I'm personally now starting to make the mistake of not being general enough regularly. :) But it seems like a good thing if I'm 50/50 between not general enough and too general.
Running a startup was good practice, you just can't afford to work on problems you don't really have. If multiple customers aren't screaming for it right now, it can probably wait.
Ironically, I think I learned this lesson more thoroughly with writing than with computer science. Using abstract language, using categorical and abstract words rather than specific examples, makes for super boring reading. Communicating a pattern is even sometimes even more accurate when you list three specific examples than when you choose the most accurate abstract word; the reader will see the pattern.
Just googling for examples, this one seems decent. Lots of writing advice online leans in favor of being specific. https://www.unl.edu/gradstudies/current/news/use-definite-sp...
"In proportion as the manners, customs, and amusements of a nation are cruel and barbarous, the regulations of its penal code will be severe."
vs "In proportion as men delight in battles, bullfights, and combats of gladiators, will they punish by hanging, burning, and the rack."
I was asked if I could make a program that could 'put a photo on top of another one'-- the idea being they wanted a program where you could load a template image, maybe one that looks like a birthday card, and it would have a spot where someone's picture could be put on top of the birthday card image. Simple enough, but I decided to build a generalized templating system with a built in editor so new templates could be created. This way it could make birthday cards for one person, or make a "Good Job team!" cards for multiple people. It was completely agnostic to what kind of templates you could make with it. It supported layers, conditions, database integration.
Sure enough, over the course of months, I was asked to make the program do more things than it was originally asked to do, and I was able, with few exceptions, to accommodate these requests without needing to modify the source code. Even when I did need to modify source code, it was to extend functionality, not change the basic template abstraction model. It found its way into other uses as well (digital signage, ads, etc).
If I had only solved the least general problem first time, I would have been back at the drawing board rewriting most of the app every time something new was requested.
Maybe this is a rare counter example, but sometimes pursuing the general solution pays off.
I'll give an example: "we're never gonna have more than three devices, so this array should have a hardcoded size of 3." It sounds pretty extreme, but believe me, some variation of this theme comes up more often than I would like, especially from people who are not software engineers. It seems like some people think that every abstraction (and potential genericity) is expensive, so they tend to put constraints in places where none need to exist. Very soon you'd need to make a small modification to your code, or extend it slightly, and those unnecessary constraints are going to make things difficult.
Personally, I'd like to approach a problem from the bottom-up. Start with solving a concrete case, and see if any patterns emerge in the process. And if they do, then I generalize those patterns. Frequent refactoring is very helpful during this process. Sure, mistakes and over-generalizations can still occur. But they are usually not very expensive if noticed quickly.
On the implementation side, the "0, 1, n" rules already mentioned by muxator dictates that this specific aspect of the implementation should not treat 3 different from 5, and implement support for n devices. (By "This specific aspect" I mean that other factors will surely prevent supporting n = 1 trillion, but we don't have to care about that since it doesn't contradict the requirements).
Isn't the "0, 1, n" rule a thing?
I believe that bigger parts of our education should revolve around reading and analyzing professionally written code as opposed to solving completely artificial problems.
I graduated ~6 years ago and there was no opportunity for specifically software engineering classes; there were only computer science classes.
I believe that what we need is something in the middle - a solid CS education combined not with in-depth dives into specific technologies, but with a wide range of samples of solid engineering in various real world projects.
Solving general problems happens when you start thinking about data you don't have, but might, and throw performance away for premature organization.
Saying "let's cross that bridge when we come to it" is the idea (rather than "whoa we never saw that bridge we should/shouldn't have crossed")
There is also a human perspective to this. Solving general problems is FUN, (at least I tend to think so). The generalization is the fun part of the problem solving. Some people think seeing the working product is cool, no matter how elegant or general. I don't mind if everything is half-finished forever so long as the half-finished product is elegant and general (exaggerating but you get my point about the different personalities). This is also why you need different perspectives (and probably people) on a team.
> I made the right decision, after trying all of the other ones first - very American.
It's not my favorite because it's snarky, or because it makes a generalization I think is accurate. It's my favorite because it has an element of the person behind the technical article.
Too often, people writing about technical subjects get lost in the topic, and forget that the act of writing for an audience is a human conversation.
It might apply to this case here.
I love generalizing things. If I need to remove a couple of prefixes from a list of strings, I don't hard code those strings in a function. I make a general string prefix removal and pass those prefixes in. I find that I write more correct and stable code by switching from special cased code to generalized code.
One can take this to extremes of course. I think the article is against generalizing solutions that shoehorn things together that normally don't go together. E.g. hal for handling disparate hardware abstractions. Its a forced abstraction to gain certain things.
I guess it's abstract vs generalize. Generalize is good, if it doesn't introduce abstractions... that's the thought process I just had anyway.
def strip_prefix(prefix, str):
return strip_prefix("foo", id)
But then if you want a name_from_fooid function, it's just an implementation detail that you happen to implement it on top of a strip_prefix utility.
Our industry has an organizational structure for shipping software today that makes every line of code have zero value until it's in a consumer product that ships, and maybe even then it'll be cancelled next month and its value drops to zero anyway. We've long known that most software projects fail. What's wrong with this picture? Why do we continue to work in a system where the most likely result of any day's contribution is literally a useless waste of time?
It seems as though the problem was short-sightedness, and the proposed cure is to be even more short-sighted. (You're looking down as you walk, and you occasionally glance up. While looking up once, you tripped over a stone. The solution is to focus even harder at the 6 inches in front of your shoes -- then you'll never run into trouble!) Is it any wonder that most software projects fail?
We joke about "resume-driven development" but economically that's the only certain gain from any new software project. I'd be foolish to optimize for the lottery ticket of "successful product", instead of the solid long-term investment of "increased personal knowledge".
The Gossamer Albatross succeeded where others had failed when they realized that the actual problem was fixing the process first . They won big by solving the more general problem. What if we fixed the software development process so that everything we built could be useful?
Sure, some code is just bad, and the trash can is a useful tool, but a DV HAL sounds like a useful tool, too, and I don't think anyone is better off that it got built and then discarded. Could there have been a secondary market for commercial libraries whose products were cancelled? Could it have been open-sourced? Could it have been built as part of an alliance, so other teams could have helped drive the design, and reap the benefit even if one team dropped out? I don't have The Answer but it seems awfully unlikely to me that optimizing one's architecture for the next day or week is an optimal process on any larger timescale. That's basically admitting that we're still at the hunter-gatherer stage of software development. You can't plan for next year's harvest, if you haven't invented agriculture, and don't know where you're going to be in 6 months.
Bonus: "At the time templates tended to crash the compiler, so going fully templated was really expensive." Yes, we need to reduce our scope, because this other program we're using reduced theirs! The limit of this function is a catastrophe. If I were to get a batch of bad steel from my supplier, I wouldn't try to compensate by using twice as many bolts.
For example, many languages attempt to provide a cross-platform API for filesystems; they give you an algebra that lets you construct and manipulate paths.
So if you have a Path, the `/` abstract operator might join path parts, and "basename" and "dirname" are functions that extract parts of the pathname.
Typically, the API designer makes some intentional (and some unintentional) design decisions to keep the API comprehensible but at the expense of being inconsistent with the underlying reality being abstracted.
As an example, some filesystems are case-insensitive, others aren't, so to know if two paths refer to the same file, e.g. if you want to use paths as keys, you need to determine what volume each is on. This gets complicated in Unix due to symlinks (which can be any component of a path) and mounts that can be anywhere in the tree. You could resolve a path to an inode, but then other filesystems might not provide an inode.
edit: not arguing, just curious. I haven't thought much about abstraction but it seems like it could be a truism that every conceivable abstraction could succumb to this problem in some way.
Almost all. the question is just one of degree. The fact that they are almost all leaky is why you should be hesitant to abstract away in the first place. The effort you think you are saving may be taken up later when the leaky details bite you. "I'll use this game engine instead of raw vulkan" -> "oh no I can't render all my widgets fast enough on mobile" or "I'll use this easy UI library" -> "oh no I can't actually do a list of text boxes that is scrollable with this"
1) Humans make mistakes and are often ambiguous.
2) Perfection is often not economical. 98% good enough may require 1/10 the code of 99.9% perfect and customers don't want to pay for 10x the coding. See: https://en.wikipedia.org/wiki/Worse_is_better
3) New standards or technology comes along that challenge the original assumptions made. (I gave an example of OOP GUI libraries versus Web elsewhere.)
4) Performance: heavy-handed abstractions are often too slow because they have more conditionals and converters/fixers to adjust the translations or interfaces.
A good abstraction is not leaky for most use cases. A bad abstraction is leaky for many use cases. The main issue is that you can't foresee the use cases, so premature abstractions usually turn out to be leaky.
However, runtime may be dramatically different. One may take a second, and the other one days.
That's because SQL is an imperfect abstraction. It isolates you from the underlying details on how to compute the results of a query, but only to some extent. On many occasions you need to actually understand how the declarative query is translated into an execution plan.
For instance, when the author created an HAL instead of a specific problem specific driver, the HAL meant that he had to solve 2 problems instead of 1
The end product had to work, and had to be compatible with the HAL as implemented. The abstraction leaked.
Just making the driver would have yielded a solution to the problem that would"t have required as much refactoring of old code. It's a way of saying he made more work for himself than was necessary.
A very YMMV observation.
See also: https://www.sandimetz.com/blog/2016/1/20/the-wrong-abstracti...
Your ivory tower parser/data format/framework is great until you need to fit it in 1/4 the memory or MIPS. That's when you start pulling out the abstractions to get back to where the product actually needs to be.