It feels like one of those low-sci-fi settings where we thought it'd be funny and quaint to have post-collapse scavengers endlessly repairing retro tech, but now it's actually happening and it's not funny anymore.
Like for example: My dad bought a hulking integrated Akai amp / cassette / turntable combo in the early 80s. Every user-facing component was brushed aluminum, the volume pot was the size of a Dallas church, and it probably would have killed any living organism you dropped it on.
My dad died over a decade ago and I guarantee that amp is still sitting in someone's living room heating the place up. I'm just mad it's not in mine.
In fact the Gnorts would not have "a long list of rules to memorize" with "no underlying principles".
They would instead have a history and culture (or many histories and many cultures) to learn in order to contextualize words and symbols and find their actual meaning, because meaning doesn't really exist without context.
Imagine that the 2028 Democratic presidential candidate for some reason consistently uses the term "colored people".
We all know what's going to happen. The underlying history and culture would change within the span of 24 hours, and suddenly "colored people" would loose it's racist connotations.
Awareness of history and culture won't help you understand language rules. Instead, to avoid saying something racist, one must be keenly aware of political expediency.
"Sugar: The Bitter Truth" (https://robertlustig.com/sugar-the-bitter-truth/) is a pretty long watch, but it's incredibly eye-opening in terms of explaining in detail exactly why our current dietary sugar intake is so damaging to our bodies.
Note: Robert Lustig is a professor of pediatric endocrinology at UCSF, I promise I wouldn't ask you (the reader) to watch a long-ass youtube video unless it contained extremely relevant science about how you (the biological machine) work.
When you make accusations like this why not offer a specific illustrative example to avoid suggestions of mere name calling. Scientific consensus?. Actually nutrition & medical science is replete with the abandonment of 'received opinion'. How about applauding researchers who have novel ideas outside the consensus while at the same time insisting they demonstrate their evidence or in problematic areas, convincing reasons for pursuing their path?
This is one of those areas where scientific consensus needs to catch up with the alarming facts that have been discovered. Scientific consensus was against handwashing for doctors and plate tectonics, and it still is in favor of "clearing amyloid plaques will totally fix Alzheimer's", but the facts just keep on being what they be.
This isn't true. He has books and lectures explaining the metabolic processes when eating refined sugar, and why it's bad for you. Nowadays it's pretty mainstream stuff. Nevertheless, can you give an example of one of his opinions which is "well outside scientific consensus".
> Nevertheless, can you give an example of one of his opinions which is "well outside scientific consensus".
Lustig specifically claims that sugar is addictive; that fiber somehow mitigates the absorption of fructose; that calorie restriction does not cause weight loss; that in fact, weight loss is somehow a function of insulin, not calories; that fructose is uniquely bad relative to other sugars; that fructose causes inflammation; that recent decades' increase in obesity is caused by increased sugar consumption; that statins are essentially useless; that some kinds of LDL cholesterol are good for longevity; that non-nutritive sweeteners have the same impact on fat/weight gain as sugar; etc, etc, etc, etc.
A few of these claims are wholly unsubstantiated by research; the rest have some research and the research does not support Lustig's claims.
>>The evidence supports the hypothesis that under certain circumstances rats can become sugar dependent. This may translate to some human conditions as suggested by the literature on eating disorders and obesity.
>that fiber somehow mitigates the absorption of fructose
>>Dietary fiber (DF), especially viscous DF, can contribute to a reduction in the glycemic response resulting from the consumption of carbohydrate-rich foods.
>that calorie restriction does not cause weight loss
>>Mechanisms smooth out the large day-to-day differences in energy consumption, decreasing the importance of the size of a meal. In the short term a reduction in energy intake is counteracted by mechanisms that reduce metabolic rate and increase calorie intake, ensuring the regaining of lost weight.
I'm not going to go on and on... UCSF, which is one of the most respected teaching hospitals in the country, isn't hiring cranks. He specialize in exactly this stuff. Yea, he's a bit more strident than would would expect from a scientist, yes, he deals with the extremes of childhood obesity, which isn't really relevant to most people's bodies, but christ, he's not a crank.
> In the short term a reduction in energy intake is counteracted by mechanisms that reduce metabolic rate and increase calorie intake, ensuring the regaining of lost weight.
If calorie intake increases, then it's no longer "calorie restriction".
If his actual claim was that calorie restriction does not cause weight loss, then that's wild despite your quote.
> I'm not going to go on and on...
Well you didn't address the other really egregious supposed claim, that "non-nutritive sweeteners have the same impact on fat/weight gain as sugar". If that's an accurate description of his stance, that's really bad.
>While people often choose “diet” or “light” products to lose weight, research studies suggest that artificial sweeteners may contribute to weight gain.
Again… things that are counterintuitive are exactly the purview of good science. The human body is an absurdly complex multi-variate system that is confounding even in the areas we pretend to understand.
The interaction of the neurologist of taste on biological processes may be affecting the hunger responses, thus weight gain.
This shit is not simple, and the simplistic models we use to explain these processes are exactly the type on ultimately wrong knowledge that Karl Popper rails against.
Again, I definitely think Lustig claims debatable things overconfidently, but he’s no crank.
"is a crank" is really far from "has some controversial or heterodox views"
I'm not arguing that everything Lustig says is correct, on the contrary, I've gone out of my way to criticize him. My point in this conversation is that saying a reputable scientist, at a reputable institution, doing reputable research is somehow a crank... is bananas.
If he actually said it's the same impact, that is much closer to crank than controversial without a few pieces of really strong evidence.
And the counterargument you had for calorie restriction described a situation that is not calorie restriction.
I'm not saying he's a crank for sure, but my evaluation of him is riding on whether loeg's description of his claims is accurate, not the evidence you brought, because that evidence really does not support the plausibility of those claims.
Directly following that, he goes on to totally misrepresent a 2011 study in which groups were assigned diet soda, water, milk, or full sugar soda; the study saw diet soda participants gain less weight than the water group and actually lost fat. (All other participants gained weight and fat.) That's https://pubmed.ncbi.nlm.nih.gov/22205311/ .
If he isn't a crank, he should stop going on podcasts and saying crank things.
And finally, he's a pediatric endocrinologist. Not a specialist in many of the subject areas he makes claims about (cardiology, gastroenterology, nutrition).
Use case #1: You have a problem table (e.g. a high-volume events table) that grows non-linearly as your business starts to scale up. A queue + columnar store package like Trench moves the problem table out to a system better equipped to deal with it and lets your DB server handle its relational business in relative peace and quiet.
Maybe I wasn't clear enough but my questions have been rhetorical. They were not for me. If one starts stating technologies, it is akin to describing the individual ingredients of a sandwich.
The question remains: Why choose Trench over just using Kafka and Clickhouse or any other message queue and columnar database / big data base?
If the goal of the post and the landing website is to entice people to use the tool, then answering these questions is important. If what is being discussed seems obvious, then who is the target demographic? Because they already know the space, use alternatives or have built their own.
Sites will for sure use the Web Integrity API(1) to blacklist "unsafe" user agents exactly the same way macOS blacklists apps from "unsafe" developers.
I've had to explain this to non-technical stakeholders many, many times over the years, and I always use the restaurant metaphor:
If you run a commercial kitchen and you only ever cook food, because selling cooked food is your business -- if you never clean the dishes, never scrape the grill, never organize the freezer -- the health inspector will shut your shit down pretty quickly.
Software, on the other hand, doesn't have health inspectors. It has kitchen staff who become more alarmed over time at the state of the kitchen they're working in every day, and if nothing is done about it, there will come a point where the kitchen starts failing to produce edible meals.
Generally, you can either convince decision makers that cleaning the kitchen is more profitable in the long run or you can dust off your resume and get out before it burns down.
Been thinking of explaining technical debt using a book library as an example...
Say you want to start a lending library, you hire one person and stock 25 books. The stock is small and one person can easily remember all of them so the employee just piles them up in a corner. If a customer wants fiction or literature or whatever, the employee could easily look up the pile and pull it out.
Over time the books grows from 25 to 50, the stocks still small and there's just one employee so they are all just added to the pile.
50 grows to 150, you hire one more. The old guy feels that since there are two of them one can lookup the first 75 while the other can search the next 75 and organizing would a waste of time and space.
When you hit 500 books the debt starts to kick in but again you try to solve it by hiring more people. Some of the new hires want to categorize the books into proper racks but that would mean shutting shop for few days and not adding any new books. This is unacceptable to a non technical manager so things continue to be the old way.
By the time you hit 1000 books some of the employees are fed up with the time consuming work and quit, the replacements have no clue what is where. Most of the customers were served purely based on the muscle memory of the old employees and now that they have gone the business starts to crumble.
I especially like this metaphor! A codebase doesn't exist in a vacuum; a team of developers work on it and also produce auxiliary artifacts like configuration and documentation. The software only "works" so long as the whole system "works". When too much knowledge is held only by people, and isn't recoverable from the codebase and other desiderata, the whole enterprise (lower-case e) falters once that knowledge is lost.
I just finished reading The Name of the Wind, a fantasy novel where the organizational scheme of the library at a magical University is a minor plot point. Every time the school gets a new Master Archivist they come in with bright and ambitious ideas about how the catalogue should be organized and abandon their predecessor's scheme, with the result that centuries later there are dozens of overlapping and even explicitly competing systems extant in the library and many books are basically impossible to find. This, too, reminds me of basically every large codebase I've ever seen.
Reading this I just clicked the audiobook. Sounds like something I could love working as a technical oriented data analyst after having studied literature.
At one particular location at one of our favorite customers it was a mess to get out from their site every day after work.
Why? Because they collected everyones id-cards/drivers licenses etc in the gate on their way in and then dropped them into a box - unsorted.
This meant for everyone that wanted to leave they had to sift through the box.
Now my colleague asked them at one point why they didn't sort the id cards as they received them and got a scolding to the tune of "can't you see how busy we are already? Now imagine if we had to sort the cards as well!".
The next time he was there however it had sunk in and leaving was as fast (or faster) than arriving.
This is a good parable, but a bad metaphor: no one would open a staffed lending library with a collection so small that the staff could keep everything in their head. The key to a good metaphor is that it should hit something familiar and normal, so that the listener can devote their mental cycles only to the relevant understanding.
There are really two properties which are essential for a metaphor to convey understanding (which is not to say no other aspects have an effect—they certainly do, but I'm just trying to narrow down the 'functional core' here):
1. It needs to be expressed in the terms of a subject already understood by the listener.
2. It should have a 'structure' which is similar to that of the new, not yet understood subject.
These two things allow an identification to be made between something already understood and something not yet understood, granting new understanding.
The idea that the library should be 'normal' is not essential; it just needs to be understandable. (granted if it became too unusual it could cause problems, but I'll just leave it up to the reader's judgement whether "a library with 25 books" is too large a stretch of the imagination)
What's really nice about the above metaphor is that its structure matches the situation with coding very closely: technical debt is about the up front time-cost of introducing new systems of organization to replace ad-hoc, unsystematic methods whose cost-effectiveness decreases as project complexity (to put it loosely) increases.
While this is a pretty abstract notion in itself, the library scenario captures the same key points and dynamics in a way that's comprehensible to anyone with an understanding that libraries use 'systems of organization' (even if they don't know an abstract term for it)—this still works even if the particular set up for the 'library' in the example would not be found in the real world.
We’ll have to agree to disagree. I think a metaphor that has nothing to do with reality causes people to think too much about it. You end up replacing one abstract thing with another abstract thing (and the reason to go metaphoric in the first place is to make the concept less abstract). The metaphor makes more sense you because you’re already deeply familiar with the concept of technical debt so you’re able to easily evaluate (2) — you’re not actually using it for understanding. If I told this metaphor to my spouse (for whom I’m often using metaphors to explain things), the reaction would be: “What are you talking about? Library with 25 books? Huh?”
> (and the reason to go metaphoric in the first place is to make the concept less abstract)
I completely agree with you there. But I disagree that the unusual library example is more abstract.
As a point of contrast, I gave an abstract account of technical debt for the purpose of comparison to the more concrete library metaphor (to show that the abstract account would be more difficult for many):
> technical debt is about the up front time-cost of introducing new systems of organization to replace ad-hoc, unsystematic methods whose cost-effectiveness decreases as project complexity (to put it loosely) increases.
A library with 25 books is not more abstract, though it is a hypothetical concrete thing [1]. If someone has difficulty with hypothetical concrete things, that could certainly make using metaphors with them more difficult—and I can see why you would add additional constraints onto what makes a good metaphor. In my personal experience, saying, "imagine a library, but with only 25 books, which is run by a single person," would be not asking too much of an audience. But I see your point that some may struggle with it.
[1] There is a handy way of considering this distinction: becoming more abstract involves making some 'parameter,' which was fixed, free instead: so for instance a more abstract notion than any particular library is: a manager of a collection of objects which are loaned out to people for limited durations of time. In that example, we free the parameter 'book' so that it isn't set to anything specific. If we instead say, "a library but for CDs instead of books," it's equally concrete since we haven't freed any parameters, we just replaced one concrete value for another. That's what happened in the "library with 25 books" example, which is why I say it is not more abstract.
>2. It should have a 'structure' which is similar to that of the new, not yet understood subject.
If you'll forgive my pedantry, at this point you're not discussing metaphors and have veered into analogies. Maybe we can meet in the middle and call these metaphorical analogies? ;)
Metaphors are when you say something is something it is not ("my wife's smile is the bright sun I wake up to"). There need not be anything unfamiliar with a metaphor or any explanatory power behind a metaphor. Analogies are when you explain or frame a concept using an already understood concept ("the car is out of gas, like when you're out of energy and don't want to run around.")
As an aside, I'll mention that the idea that "analogy as the core of cognition"[0] has been kicking around in academia for years, and comes from Douglas Hoffstader (author of Gödel, Escher, Bach).
> There need not be anything unfamiliar with a metaphor or any explanatory power behind a metaphor
The purpose is still explanatory, it's just possible for the explanation to be in terms of perceptions rather than structure: in your example with wife/bright sun the purpose is still to make an identification between two domains, one familiar, one not ("bright sun" is familiar to the reader; author's wife is not), but rather than the two being "structurally similar" it's that they evoke similar experiences in the author, and the author is able to explain their experience by sharing this identification.
That said, I agree with you that what I was discussing is closer to definitions of analogy in certain areas of cognitive science (e.g. Hofstadter's work, like you've pointed out—or John Sowa's: http://www.jfsowa.com/logic/theories.htm)
But the original example with the library would've been better termed an analogy to begin with, rather than a metaphor—I just adopted the previous commenter's terminology :) (though admittedly I'm more inclined to interchange the two fluidly because I don't find the differences essential: I would just call the kind of metaphor you used a perceptual analogy. Maybe I am missing something important about metaphor though—I'd be curious to hear if so!)
I tend to agree with him. It takes too long to tell that story. In a boardroom meeting type environment people don't have time or the attention span to settle in for story time. You need to be able to illustrate your point in 3 sentences or less.
Whenever I had to explain it to management I tell them something along the lines of:
When was the last time you clean your office cupboard? If anyone else from management needs to find something in there without your help how long would it take compared to you?
It's the same with our scripts. They need cleaning up and documenting in case one of us decided to call it a day.
Edit: When the situation allows(tbh I rarely miss a chance) I follow up with: You have to thank our sales guys making unrealistic promises for our lack of documentation.
that is pointless. It isn't about message, it is all about messenger. If they don't want to listen to you, they wouldn't. My friend who has made his way pretty high here in the SV told me recently how the same CXX/"boardroom" people who wouldn't listen even to his 1 sentence, now have all the time and attention in the world to listen to him all day long.
This is true. You can always try to convince people but if you're not string puller, you're in mercy of listeners. If they fix on something (e.g.profit, own promotion), your story should be aligned to that(their KPI). Then only you can have hope.
When shit breaks, no one wants to take blame and people start to forget 'selectively' your earlier warnings of shit.
Interesting. Like a month ago I came up with a similar metaphor. Only, instead of a health inspector, I focused on the fact that dirty dishes get piled up.
You put them in the sink. The sink gets full. You put them on the table, on the stove, on the counter-top, on the floor. You can hardly walk. You're working on a little corner on the table because the rest is full of stuff. How are you supposed to continue to work efficiently with all that clutter?
The manager cares only of how fast you can get food out the door. To them, washing dishes is a waste of time and money. Customers aren't paying you to wash dishes. They're paying for the food.
The problem is seeing how the buildup of clutter affects the speed of preparation of food.
Of course, quality is also affected. For example, you need a sieve. It's dirty; you used it for flour. Now, you want to use it for powdered sugar. Well, that's similar enough, right? There's no time for washing; the customer is waiting.
Totally. Another part of the metaphor I use is that messes are easier to clean right after you make them. Leave the pans for a week and cleaning things up is a real pain. Cleaning after the meal is done is easier. And easier still is to clean as much as possible as you cook.
And Anthony Bourdain has a nice related bit on how working clean reduces error and increases efficiency. From Kitchen Confidential:
Mise-en-place is the religion of all good line cooks. Do not fuck with a line cook’s ‘meez’ — meaning his setup, his carefully arranged supplies of sea salt, rough-cracked pepper, softened butter, cooking oil, wine, backups, and so on.
As a cook, your station, and its condition, its state of readiness, is an extension of your nervous system...
The universe is in order when your station is set up the way you like it: you know where to find everything with your eyes closed, everything you need during the course of the shift is at the ready at arm’s reach, your defenses are deployed.
If you let your mise-en-place run down, get dirty and disorganized, you’ll quickly find yourself spinning in place and calling for backup. I worked with a chef who used to step behind the line to a dirty cook’s station in the middle of a rush to explain why the offending cook was falling behind. He’d press his palm down on the cutting board, which was littered with peppercorns, spattered sauce, bits of parsley, bread crumbs and the usual flotsam and jetsam that accumulates quickly on a station if not constantly wiped away with a moist side towel. “You see this?” he’d inquire, raising his palm so that the cook could see the bits of dirt and scraps sticking to his chef’s palm. “That’s what the inside of your head looks like now.”
That is a very close analogy to coding for me. If I keep the code clean, the inside of my head can be clear, well-ordered, precise. And if it's a mess, I become a mess.
Unrelated to technical debt (or maybe it is and someone can find a way to make the connection) but this quote from Bourdain resonates well with me and very much aligns with why when I used to entertain guests (before covid obviously) I would not let guests try to help clean. It’s an appreciated effort but I really am that particular about how I like my space and kitchen maintained, much prefer if they nearly place a dish in the sink if they would like to (otherwise I have no issue “bussing”) and resume enjoying themselves with the rest of the guests.
Made for an annoying moment once when one guest would not relent on being allowed to clean and made the situation of cleaning my kitchen about them and their life hangups in a weird and inordinate amount. They haven’t been invited back. Bourdain is right: do NOT mess with the cook’s meez.
(Also Kitchen Confidential is such a stupidly marvelous book in general)
I think the metaphor still holds, if we consider manual processes to be disposable dishes. Each time you use one, it incurs a cost, and the costs increase linearly with how many of them you use. However, with disposable dishes / manual processes, you don't have to think about logistics so much, and in certain situations it genuinely does make sense to go with the disposable solution. You just need to know which situation is which.
On a good night I will be cleaning while I cook. I don’t want to serve dinner with a pile of every dish that I touched during prep piled dirty in the sink. In an ideal scenario, after the last course is placed on a serving dish, I have exactly two things left to clean; the pan that cooked that last thing, and the utensil used to transfer it.
On a truly good night, the kitchen ends up cleaner by the time dinner is cooked than when I started.
Often the first thing I do before cooking a meal is some cleanup.
OK, I’ve abused this metaphor enough now. Thank you for that one!
And yet the people managing engineers are supposed to be those "smart business people" that "know things" that those nerdy engineers don't understand about business. Somehow they completely cannot understand tech debt no matter how many times it's explained to them.
Tech debt is a bad metaphor. Having lots of debt can be a smart business decision. The problem is usually engineers do a terrible job telling the story around why having lots of tech debt will cause problems. Usually if you've got non technical managers the best case is they've lived though it before and know.
Debt is a good metaphor. Carrying technical debt can be a smart business decision. Constantly taking out more loans and never making payments leads to bankruptcy.
In my experience the problem is usually it's hard to quantify the costs. It's easy for managers to tell themselves engineers are just perfectionists. Someone's bonus might depend on not listening and they probably won't be held accountable when something goes wrong.
Actual debt has a few major differences with tech debt though. First it's easy to quantify the cost of. Second and more importantly where it's good and bad are almost the opposite of where tech debt is good and bad. You want to use debt for financing when you have a relatively stable mature business where as when things are still uncertain you want to use equity.
Conversely both have the same basic effect: they magnify the consequences of decisions. Fiscal debt is essentially a multiplier on your sensitivity to revenue changes. Tech debt is a multiplier on delivery time for new features, or your sensitivity to changes in the platform's assumptions.
As someone who struggles with contrived metaphors for everything, I think this is a pretty excellent one :)
Although I'm not sure dishes are the best. You're not really going to make it a day without cleaning the dishes.
Deep cleans are a good metaphor, but maybe also just general maintenance of kitchen equipment? E.g. having a stove where half the burners don't work half the time, an oven with wildly variable temperature, duller and duller knives, etc. seem much more akin to technical debt as they make it continually harder to produce quality results and work efficiently. However, you can ignore them for quite a long time, and you don't need to pay attention to problems every single day. (It's fine to lose an oven for a while, but you'd better fix it quickly.)
The Capability Trap. The core idea is that you have pressure to deliver a product (get "real" work done), but also maintain or improve your ability to do the work. If you ignore that maintenance portion, you end up being able to produce less over time. And the cost to recover the capability increases over time due to neglect.
In the kitchen example, cleaning and maintaining the fryers every (period of time) means that you can get years, if not decades, out of them. But failing to do so may force you to turn them off (produce less food for customers at a time). Then you have to either replace or pay for expensive maintenance and repair work, which is usually more costly than just having someone come in and drain the system, clean it, and give it a once over every (period of time).
EDIT:
I thought that paper had been submitted and commented on more recently, but the last commented submission was from January 2015. So it's now submitted here:
> The Capability Trap. The core idea is that you have pressure to deliver a product (get "real" work done), but also maintain or improve your ability to do the work.
This nudged a decade-old memory of how non-profits (and to an extent, the public sector) find it much easier to get funding for specific outcomes, but incredibly hard to get funding for what they usually call "capacity building". The medium-term solution seems to be outsourcing to a vendor that can both deliver an outcome and amortize capacity building across many customers in your sector, but over the long term this leads to a form of organizational learned helplessness, and vendors that eventually switch to rent-extraction rather than delivering outcomes or building capacity (or rather, the capacity they are still building is largely the ability to secure contracts).
Ironically, the longer the vendor takes to switch to a rent-extraction strategy (and the more slowly they boil the frog), the more entrenched they become as a defacto standard, and the harder it becomes to eventually displace them (for a competitor) or replace them (for a customer) as a vendor.
> You're not really going to make it a day without cleaning the dishes.
I always do a review right before I'm ready to commit my code, to see if there are any obvious refactorings, need for comments, or style issues to be addressed.
I find this constant tidying up, like cleaning the dishes after every meal, prevents having to do a massive clean up at the worst possible time down the line.
Deep cleans seem to corner the experience too much.
To the article's institutional knowledge idea, specialties, allergies, or chemical compatibilities might be a good catch all.
The more features you add with more customers and developers from different teams, the harder it is to guarantee any dish won't affect nut allergies. Mixing orange juice and milk WILL happen, and you'll only find out about it when the wrong customer experiences it.
Software development is like a city planner with unlimited space.
The developers are also the construction workers that use cranes, trucks, and tools to build the buildings. But the problem is that they don't have time and leave a lot of equipment behind.
Now there are finished buildings with cranes still assembled, cement trucks parked in the middle of the road, piles of dirt on the ground, and scaffolding everywhere.
Luckily space is unlimited so the developers build cat walks, helicopter pads, and bridges over obstructions. If city planners have a lot on their plate without random construction equipment laying around, imagine how hard it would be if they worked knowing construction a lot gets left behind.
I like this analogy because it also hints at some of the problems with overzealous tech debt work.
Sometimes especially new devs push for removing or replacing a crane even though it's working fine just because they don't understand how to use it or because a new model is available.
Cities would never get built if we replaced all the cranes and scaffolding every few months.
My favorite metaphor: bridge and road maintenance. Every now and then you have to fix potholes and repaint the bridge. Sometimes in order to do this you have to close some lanes to traffic. If you don't do this, you will accrue more and more potholes, and the bridge will start to rust. The longer this goes on, the harder it becomes to fix. If you ignore the problem long enough, the bridge will collapse one day with very little warning.
I think that metaphor is not very apt, because it doesn't capture the drag on future work. Bridges and roads are stationary.
This is why I don't like this kind of metaphor, and I don't like the Technical Debt metaphor. Because it doesn't convey the right concept. It conceals, obfuscates actual truths.
- Bad code
- Bad decisions
- Bad design
- Lack of maintenance
- Lack of proper business concept understanding
Those things mean something, you can make them more precise, but they could be actionable. Technical debt always needs a follow up question: so what is actually going on?
I use the example of publics works projects in San Francisco. There's a road called Van Ness that is going to have a bus rapid transit lane running down the middle of it. Should be really easy to do. But it's been about a decade and they're just hitting the last steps now. Why? Well, they needed to also upgrade the utilities below. But when they dug they found layer after layer of abandoned utilities, undocumented utilities, etc. So what should have been a simple operation dragged out for years because each action revealed something new.
The city now wants to add bike lanes to Market Street and is anticipating the same issues.
> I think that metaphor is not very apt, because it doesn't capture the drag on future work. Bridges and roads are stationary.
Poorly maintained roads and bridges ARE obviously a drag on future work, since nearly all other work depends on them for transportation, it drives up costs (eg. potholes start to drive up maintenance costs for vehicles and increase commute times), and reduces tax revenues (whether it is through depressing real estate prices or other mechanisms).
To say nothing of the fact that it is much more likely to fail when it is under load, so it's not just a question of replacing the bridge, but also the likely loss of vehicles and possibly lives.
Additionally, proper maintenance (or its analog to other disciplines) will usually identify when it's time to do a full replacement. Which is much better than finding out by way of catastrophe.
I wonder if this is an area where, if you can get the initial agreement, a special tax district or some kind of cooperative with dues makes sense. That way the bridge stays maintained.
The I-35W bridge collapse in Minneapolis is the perfect example of this. A contributing factor in the collapse was the weight of the maintenance equipment they had on the bridge at the time.
> or you can dust off your resume and get out before it burns down
Last time I left a job, this was a big part of it. The company simply refused to learn from their mistakes.
Now I find myself in a similar boat again. The only thing that seems to matter is the day's emergency, and everything else is put off indefinitely. This, obviously, leads to more emergencies in the long run. And yet for some reason the company seems to be coping and is growing rapidly. Perhaps it is time to dust off the ole CV again, or perhaps this is a transient thing. Time will tell?
IME (22+ year career so far) there's a balance to be struck. If you wait to ship till you're happy w the code, you probably waited too long. Successful businesses are often built on surprisingly shoddy software, and time to market matters. OTOH, only ever running in naive startup mode -- "ship it yesterday, add features nownownow, any means necessary, shortcuts be praised" -- is a recipe for disaster. Given your strong feelings on the matter, maybe you could find a way to help surface the costs and help your org to start fixing them. Because some amount of it is inevitable, and things within a given company only get better when someone cares enough to do something about it. If it works, you're a hero and things improve. If not, you can still jump ship, but with a much better narrative: not "I got tired of the intolerable levels of tech debt", but "I did (X) in an effort to address [same], but ultimately couldn't overcome the institutionalized patterns that led to it, so I'm looking to work for a company that cares about software quality".
One way to separate the two is took look for a steady stream of removed features. A startup that pivots from trading magic the gathering cards to selling bitcoins can obviously remove a lot of dead code. That’s essentially what it means to be an agile company.
Hoarders however only try and add features which always results in a drop in quality as the UI becomes unmanageable. Long term it’s an unavailable dumpster fire in the making and a clear sign to jump ship.
In an ideal world, this is how things would work. The problem is, those features were created for customers. They will not take kindly to those features being removed. And no-one wants to lose customers.
This is why the second law of thermodynamics applies also to software engineering. Entropy only increases.
That is an interesting image. "surfacing the costs". It evokes the idea that most technical debt is below the surface (hidden by the UI ?) and so is rarely noticed by anyone other than the programmers.
The trick is, how do you let the internal problems and inconsistencies become visible to the end-users so the decision makers will decide to put time and effort into resolving them without being fired because the program appears to be falling apart ?
Thanks; and yes - I do believe, strongly, that business owners / stakeholders generally radically under-appreciate the costs of unintentional tech debt, which is mostly invisible to them.
Regarding your proposed (tongue-in-cheek?) tactic of somehow making it apparent to end users as a way to get the attention of said stakeholders:
user-perceived performance is actually, seriously, the bridge you're looking for.
Companies are increasingly aware that performance (in the sense of site speed / WPO / UX latency) matters ^1. Suboptimal performance follows directly from failure to address tech debt. Efforts to improve performance are always stymied by tech debt. Thus, execs who learn to care about performance have become your allies.
It looked filthy to him when he first started working there, but then he learned what was really needed for a "clean" bakery and what was just cosmetic.
(Been a long time since I read the rest of the article, so not going to vouch for it.)
So which is the case at your company? Is the code just cosmetically ugly, or is there something more fundamentally wrong? If they are able to continue delivering features and growing the company, maybe it's more cosmetic, at least for now?
If not, yeah, you should update your CV sooner rather than later.
Kitchen staff use a kitchen and surface clean the kitchen, but rarely are responsible for building the kitchen, performing real maintenance on the equipment in the kitchen, or the periodic deep cleaning of the equipment. If a piece of equipment is on the fritz, they cope with it as best they can; if it slows down how quickly they can get an order out, or it results in a lot of wasted product, or it forces items to be taken off the menu, then so be it. Sure, the kitchen staff may be frustrated with how crappy their kitchen is, but they've still got to get orders out and make the best of it that they can.
Customers don't see that, they only see the plated dish. The manager in charge of the front of house knows nothing at all about kitchens, but knows they're pushing out the orders coming in and not getting any complaints from customers. So from his perspective, whatever the state of the kitchen is acceptable.
The back of house manager (or head Chef) is responsible for that. If business has outgrown the original kitchen's capacity, he needs to make sure the head boss understands the need for investing in new equipment or remodeling, and the risk of not being able to keep up with orders if that doesn't happen. If equipment is fritzy, he needs to make sure the head boss is aware of the potential consequences of using it in it's fritzy state, as well as the risk involved if that equipment completely goes out.
But even if that occurs and the risks are successfully conveyed, the head boss may decide to leave things as-is. If the kitchen is currently keeping up with demand with things as-is but the waiters aren't, the "emergency of the day" for him is hiring more waiters. If the business doesn't have the capital to do both at once, he'll spend the money on more waiters first and punt the kitchen's needs. Or if the equipment only impacts a few low volume items, he may decide the risk of it going out is acceptable vs. the cost to address the issue. Or if the whole kitchen needs renovated to ensure it's robust enough to keep up with the growing demand, but the kitchen is already operating at capacity, there's just no good time to renovate it. You'll either be operating at reduced capacity while it's happening, or completely shutting down for a remodel. Both of which carry all kinds of risks - the business could lose market momentum, order quality could go down while the kitchen staff break in the new equipment. If the kitchen staff is already stretched to capacity, disrupting the kitchen is likely to cause customer-facing issues. So even if it means the kitchen staff will be dealing with a constant stream of "emergencies of the day", as the costs/impact of addressing the kitchen's issues may genuinely be outweighed by the risks of not doing so. At least in the short term.
--
Sometimes a company can end up in the situation you described because of incompetence or biases in upper management. But other times it can be competent and unbiased management fully comprehending the situation and deliberately deciding what to prioritize. It's also not limited to tech - every functional area of a company has to deal with their own equivalent of "technical debt", and are subject to having their needs prioritized or marginalized based on what upper management deems the most critical. As an employee in a situation where your team's needs are being marginalized, the best you can do is determine how transient or permanent the marginalization is, and whether what you find out is acceptable for your situation or if it's time to look elsewhere.
Similar with middle management - the best you can do is ensure you successfully articulate to upper management the current state of what's under your purview, the potential risks that exist for not investing more in what's under your purview, and the potential benefits that exist from doing so. Based on the results you get from doing that, you can get a read on how permanent or transient your situation is, and what the likely long term aspects are for what you're in charge with, and decide what to do from there.
You left out the part where the kitchen staff collectively quit and go work for the competitor across the street, with the brand new kitchen (VC funded).
The VC funded kitchen doesn't need you, they replaced you with a robot[1].
That aside, you're absolutely right that higher turnover is a distinct and real possibility. Which is one of the risks that should be articulated to the powers that be, and taken into account when they make decisions.
Since we're all naming our favorite metaphors, I guess I'll name mine: It's the ol' Dull Axe analogy. There are a lot of variations and even little fables/stories around it, but the gist of it is, don't be the guy who spends all day chopping wood with a dull axe, and who doesn't have time to sharpen the axe because chopping wood takes all day.
I've seen this play at my current work for the last 2 years, but it plays out in such slow motion that a lot of people just don't realize it. Or, they do, but they've got a deployment to do so they get on with that and they'll come back to thinking about it when the deployment is done.
Nothing made me understand how awful and damaging this dynamic truly is than playing factorio. I got to see it play out in real time in the microcosm of my factory. Running on the treadmill of maintaining resources to produce ammo to kill biters and fix things they broke never making any forward progress until a dramatic shift in strategy that freed up a lot of cycles.
What's interesting is that tech debt is already a metaphor. You can take out a loan to be able to move faster and earn more, but you will need to pay interest, and at some point that will drag you down if you do not repay it.
Not a sufficient one though, because debt has to be paid off, whereas technical debt is much easier to ditch by doing a rewrite, suffering unmeasured business outcomes, etc.
No, debt does not need to be paid off, it can be continually rolled forward, as long as someone is willing to provide you credit.
When a government or a corporation has bonds due, it normally pays off the principal by selling new bonds.
The analogy with technical debt is honestly pretty good. As your creditworthiness declines, the cost renewing debt increases as creditors demand higher and higher interest rates.
It’s not (only) decision makers that need an explanation, it’s developers. Too many confuse technical debt with technology.
Maybe your kitchen is clean but not so optimized with classic tools. Instead of cooking each meal manually, why not build advanced machines for each category of food, on different sites (microservices and components)?
At first it’s very efficient, but surprise it’s not McDonald’s and the menu needs a change. The first version of the kitchen can adapt relatively fast, but in the second it’s less trivial. It can only accommodate small changes (parameters), otherwise the machines have to be completely reworked. The difficulty is compounded by the dependence of the machines to each other on different sites.
By definition the second version has more technical debt despite having the newer “tech”.
Still good thing about kitchen is fixed menu (either small or big). Sadly today's agile software-world, nobody works hard to know customers and fix deliveries. It is like, you'r running italian restarunt. But if someone wants indian curries, you will run to market, will buy ingredients, learn recipes and deliver before customer leaves.
So Final result -
- you delivered food which looks like Indian cuisine but doesnt taste like one.
- now your customer is unhappy
- so your boss is unhappy
- you're tired after doing long circus but you feel good as you learn little bit about Indin cuisine.
- Now let kitchen die (anyway your boss is not happy) and start applying Indian restarunts saying you can also cook Italian
- Mean while in next day, you will do same circus for Japaneses cuisine
- Now you will start applying Japanese restarunts also, saying you can cook Indian and Italian
A slight twist would be to argue that a customer will get food poisoning. This avoids the metaphor needing an external force (health inspector) and instead correlates well with production breaking from poorly tested/rushed code.
Disagree - I think the point is that for a restaurant, the health inspector will shut it down first. For the code, you won't have anybody to shut it down before your customers start getting food poisoning.
It seems like you're arguing for my point. Code doesn't typically have external orgs playing roles like health inspectors so health inspector could be removed from the metaphor.
I've used the document drawer metaphor in the past.
When you start getting paper documents, it's okay to put them in their own drawer and you're done with it. But they keep piling up, and after a while, if you do nothing, you will keep searching through a big pile until you find the one you need. You need to organize them into categories and folders. You need to rethink the way you store them.
The thing is that if you talk to any professional chef, keeping the kitchen in clean, organized, and well prepared is absolutely paramount. Its the one thing they drill in to you in chef school. Its core to their culture. Kitchens are hard places to work because everyone is expected to stay in line, and keep their workstations 100% perfect. Discipline is required. If asked how long it takes to deliver meals, no one would dream of not including preparation and cleanup time, because its core to the culture.
Software has the opposite culture. People glorify "hacking together", always look for easy short cuts being new languages, frameworks, or straight off copying from stackoverflow. No one wants to do hard work, learn anything properly, or architect anything for the long term. Everyone just wants to go to a weekend hackathon, put something together with tape and rubberbands and somehow make it big. It never works. Our culture is destroying so much of our work.
Depends on the billing model, right? Hourly contracts create the worst micromanagement situations. As an employee I can just decide that something needs doing & spend the afternoon on it, and my boss trusts my judgement. An arms-length customer isn’t going to be easily talked into paying for that time.
I prefer to say that code is like a Rube Goldberg machine.
While it technically does the job, it is over complicated making it hard to add new features and easy to introduce bugs.
If we rush to add a new feature, it will be like adding another crazy contraption making it even harder to make changes to in the future.
But if we take a little time to clean up the code, it will be like a well oiled machine giving us a solid foundation to build new functionality on top of.
Another interesting point is that although Rube Goldberg machines are complicated, they are extremely decoupled from start to finish... I've never thought of decoupling code in a negative way before...
In my experience the fields which are seen as critical and involve software, are quite regulated. For example finance. There are lots of various standards that are enforced, for example PCI-DSS for credit card processing and so on.
Yes, programming is not regulated, but many business areas which are basically result of programming, are.
Finance eh? Experian, Capital One, probably a few other data breaches I'm forgetting. All of them were PCI compliant. But hey, at least we got away from TLS 1.1...
I don't like this metaphor at all. What are the features? The meals? Every day, the same meals? With clear recipes and easy estimating when they're done. Hundreds of features everyday and doing the dishes pretty much resets everything to a perfect clean slate for the next iteration of hundreds of features. Everything else needed for the features, stove, fridge, utensils, deteriorates on a totally different timescale, years and are always completely replaced.
I think the embedded video raises a very good argument that metaphors is dangerous and can/does shape thinking. So I don't want this to catch on and get people arguing with their stakeholders about "we need to do the dishes" when there are heavy refactorings that need to be done. :/
This requires incentives that promote long term thinking. It's been my experience that in most organizations the buck stops with a product owner/manager. I can't think of a worse person to be deciding if we have time in the sprint to refactor. I advise teams to go up as high as you have to in the organization until you find someone incentivized to deliver value over a longer time horizon, someone who will feel the consequences of too much tech debt. Often no such person exists. At this point it should be clear to everyone what the future will look like and they can make their own personal career choices.
THANK YOU for this apt analogy! It's about as close to perfect as I can imagine. (I've been doing software for a living since 1998 and somehow never encountered this one before.)
> Generally, you can either convince decision makers that cleaning the kitchen is more profitable in the long run or you can dust off your resume and get out before it burns down.
A better commercial kitchen analogy might be mise en place, since it's literally about productivity and is completely uncontroversial.
Another one. Say you want a jet that can fly much rather than the previous one from a few decades ago.
So you start to design a new jet. But the executive has no time for that. So he says to just slap some bigger engines on the old one and add some software to compensate for how much it wants to stall.
For many startups the business model is to sell the company before severe technical debt becomes a problem, or, before you really have to understand your code.
Software engineering is different from many other fields. You're evolving a product over a long period of time, each step has uncertainty and requires creativity and exploration, and you can change not just you process but the materials and tools you work with.
It's tempting to find a simple metaphor to try and explain why technical debt matters, but if you ignore the things that make software engineering unique then you're probably not helping your case. Also, equating yourself to fast food workers is not a good way earn respect.
I more and more think we should challenge that difference. IMO software engineering is bad in many ways because we lack experience. And we do lack experience because we do not repeat what we do. Most developers work on legacy code that is older than their own stint at the company. Yet, the constantly ship new features. That means they are constantly working as maintainers and producers in parallel.
My suggestion would be to have dedicated producers and dedicated maintainers. And software should have a planned shelf life. You should buy a piece of software the way you buy a house. Huge up front investment followed by a long period of negligence interrupted by frantic maintenance efforts.
We also insist that we're different instead of examining the lessons learned from related disciplines. Perhaps the closest to software engineering is systems engineering. They have a huge body of work that closely parallels what we've done in software engineering, along with many ideas on how to model and understand both our systems and our approaches to development and organization.
But as an industry, we seem to want to make our own path, rather than learn from others.
Many other field are constrained by the limitations of the physical world. To build a house you have to source the materials, transport them, assemble them, and so on. That's the bulk of the work.
In software things of that nature can be automated. The only things left are the things that are custom to a project.
When managing a team you also have to deal with human psychology and I've noticed that many get bored doing maintenance and want to leave their mark designing and building new things.
We thus rotate maintenance/new and difficult and fun work based on the energy levels of the team.
Oh hey, I can answer this -- I work out to ease ADHD symptoms and live in a shitty wood-framed SF apartment. Using a kettlebell or any heavy object you can hold in your hands, you can do the following routine in near-silence first thing in the morning:
Goblet squat, deadlift, bent-over row, pushups, overhead press. It's reasonably comprehensive, takes less than 10 minutes, and involves enough exercise to keep me moderately sane all morning.
Like for example: My dad bought a hulking integrated Akai amp / cassette / turntable combo in the early 80s. Every user-facing component was brushed aluminum, the volume pot was the size of a Dallas church, and it probably would have killed any living organism you dropped it on.
My dad died over a decade ago and I guarantee that amp is still sitting in someone's living room heating the place up. I'm just mad it's not in mine.
reply