That's an interesting set of questions, and it feels frustrating that we'll probably never get an answer for most of them.
My most important question regarding software engineering would probably be something like: how well are resources allocated and could we do a better job? Consider how many brilliant and outstanding people go work on some useless software at a big adtech company rather than helping to improve the world. The greatest tragedy of our time.
I think the question is applicable to more than just engineers. Our economy is a paperclip maximizer of sorts - in the sense that its maximization target is far removed from the target of building the best possible world that we can imagine building. And the costs and risks are running away from us. How far removed is it? Seeing how we're "slowly" but surely killing the biosphere I think paperclip maximizer is not a bad analogy.
> I think the question is applicable to more than just engineers. Our economy is a paperclip maximizer of sorts - in the sense that its maximization target is far removed from the target of building the best possible world that we can imagine building.
My impression was that the maximization target is the building of the best possible world that can be imagined _for the owners of capital, in proportion_.
A farm in any time period is built to maximize the success, convenience, and happiness of the humans who own and operate the farm, right? Not the happiness or convenience of the chickens or pigs. If making the livestock happier and better fed helps the farm's goals, it will be done, and if not, not.
This frame of reference helps explain a lot more about the world than imagining that anyone with any relevance actually cares about the world being good for most of the people in it. Most of us are just livestock, a means to an end for the proprietors.
Irrelevant. Many who have read Marx (and peers like Ricardo, Smith) would agree his description of the problem was a lot better than his guess at a possible solution, and that's okay. Considering Marx was describing a very new phenomenon, it's not surprising the first attempt to guess at a solution turned out hare-brained. The detailed description of the problem still is very valuable and shockingly apt today. And now we have a lot more data.
Don't comment on Marx if all you know is "lol communism didn't work in russia". That's a borderline non sequitur that just reveals you don't know what Marx said at all. It's like British people meeting Mormons and saying "Oh you're a real mormon? What like from South Park?" You don't have to read Marx in the original, just spend half an hour on a Wikipedia or cliffsnotes summary at least beforehand. Maybe do the same for Smith and Ricardo, too, so you can have a good overview.
A nitpick about country names, really? These are really low-effort comments. And you keep trying to derail the conversation back to communism even though you're the only one fascinated by it here.
Do you also refuse to learn basic physics because Isaac Newton was imperfect and despite coming up with some great physical laws, also believed some nonsense about alchemy that later was shown to be nonsense?
Do you bring up the craziness of alchemy every time newtonian mechanics comes up, and refuse to believe in momentum, mass, and kinetic energy?
Having to argue about this crap every time Marx comes up is tiring and intellectually ineffecient.
At this point I wish we could blame just the smart things Marx said on someone else like say, Mark Twain, just so the adults in the room can discuss the workings, benefits, and problems of capitalism without constantly getting interrupted by irrelevant low-effort arguing.
I would normally agree with you but I'm bored with both of these takes. Communism bad, no communism actually not bad. Whatever. Can we move past this already?
What I want is a good faith argument for something that works as a solution for Marx stated problem. What can we propose? What can we learn?
>> Communism has a worse track record with environmentalism.
That doesn't have any relevance to a critique of capitalism. They both have problems. Also, even if we compare them on "environmentalism", that's only one criteria.
> That doesn't have any relevance to a critique of capitalism.
Pedantically, you are correct. But pragmatically, the people who write "capitalism is bad because of X" are usually interested in replacing it with leftism.
> only one criteria
Your post would be more interesting if you pointed out one criteria where Marxism is better.
These alleged people aren't here though. This is the definition of strawman arguing. Sorry, but you need to engage with the actual positions of the people who you are actually responding to, not the positions of some other people you once met somewhere else but that remind you of us. That kind of discussion is a total waste of time.
>people who write "capitalism is bad because of X" are usually interested in replacing it with leftism
This sounds like looking for a fight that has been fought thousands of times before rather than looking to learn something interesting though.
To me a more interesting question would be, if we steel man the public conception of the problems of capitalism, and we accept that the public conception of communism is problematic to implement to say the least, what remains? How can we move forward without endlessly getting stuck in the same tired argument about communism? How can intelligent people discuss solutions to these issues? What are our alternatives?
What is your point?
A Marxian analysis doesn't inherently mean the person you're replying to is endorsing communism.
They're just saying that Marx correctly understood this particular issue with capitalism.
The alternative doesn’t need to be central planning
You could even make an argument that some of the current US problems are caused by central planning, for example in the form of tax code based incentives or major spending packages
Another separate (but related in some ways) argument is that many companies and individuals can externalize their costs, and this distorts markets
Perhaps the disastrous outcomes were not a result of central planning per se, but the skewed incentives that central planning mandated. Not sure if central planning can work differently though, probably it's first goal is self preservation.
The problem is inherent to central planning, because central planning is unable to deal with the local complexities of a problem.
For example, during the gas crisis around 1980, the Dept of Energy was empowered to determine the gas allocation for every station in the country. The allocation was based on the previous year's markets. This was well-intentioned, but it did not take into account the fact that markets change constantly. The result was stations with gluts and stations with shortages.
Market forces created an economy which made everyone rich including the government which could then afford to spend a very large amount of money on one project, which also wouldn't have worked without the skills people acquired and the products that were created in the market economy.
Market forces also brought us things like slavery, child labor, and troves of other human rights abuses....
I don't think we can pin either system as being more virtuous or not, we just seem to have more hisotry of centralized corrupt power at this point since thats been the global state of affairs and humanity just started experimenting with alternatives relatively recently that the verdict is still out on. Both approaches have their ups and downs and both approaches need governance to keep their power in check. Which has the higher corruption rate? Who knows.
Centralized systems seem to have a quicker escalation from start to corruption but they also tend to destablize quickly. Meanwhile, decentralized systems seem to still reach a sort of steady state of corruption at some point and seem to be more difficult to destablize (fix) because it's easier to argue the state of the system evolved "fairly" in some way that we can't just fix.
Free market forces ended slavery (as slavery could not compete with free labor).
Child labor used to be necessary for subsistence farming as otherwise everyone would starve. Improving productivity due to free markets ended that, and made it possible for children to not need to work.
Have you ever thought about children in your workplace? In most I'm aware of, they'd be useless and even counterproductive. Child labor is only for the most menial of tasks, and those have long been automated.
Not for a second. Ubuntu is a repackaged Debian and both Debian and Wikipedia are volunteer-driven.
They are not for sale and therefore unrelated to "market forces".
Besides, tax-funded government-managed research gave us telephone networks, semiconductors, computers, GSM, GPS, fiber optic, satellites, Internet, airplanes. Is that enough?
This whole thread is silly, the outcomes we've had in the US are a result of our particular mix of central planning and market forces. And lots of central planning in implemented in the form of adding incentives to push market forces in the direction we want. And there's a whole lot of corporate lobbying going on in our "central" planning.
Attributing particular achievements to one or the other is impossible.
Volunteer projects are certainly part of the market and are subject to market forces.
> tax-funded government-managed research
Um, read up on the history of the 1903 Wright Flyer. It was not government managed research. The government project, Langley's airplane, fell into the Potomac like a sack of wet cement (and cost 20 times as much).
Computer networks grew out of telegraphy networks, which were privately funded.
Semiconductors came from Bell Labs, a private company.
Societal value does not equal central planning. I understand what you were implicitly referring to (Soviet-style "socialism"), and I agree it can serve as a negative example at best.
I think the real dichotomy is societal (long-term) value vs. individual (short-term) value. The latter has led to an industry destroying the planet, "social media" actively promoting hatred etc.
How to implement socially valuable objectives instead of those destroying the basis of human life and atomizing society is a difficult political question. Central planning is probably not the answer, but that does not mean we should just give up.
How well central planning could work with all the computational power we now have is still an open question. Too bad there doesn't seem to be a good way to explore it without also falling to panopticon levels of social control even faster than we currently are.
The smartest people I met in my career were at Google, and their main concern, institutionally, ultimately, was increasing ad click thru rates. (their main concern, individually, was doing whatever the institution required for getting promoted)
There's a lot of people on HN who can relate to this experience. That's where our intuition that there's a lot of people working on stuff that is useless to 99% of people, but will help a small elite get richer, comes from. It's literally what we do for a living, or used to.
Look at the top companies on the S&P 500 and where most of their revenue comes from. How much are these activities really improving the world?
If only the internet had had a built in mechanism for automatically micropaying for content as part of your internet bill (like flip phone ringtones used to work in the aughts, or how Minitel supposedly worked in France in the 1980s) adtech would never have happened and we would have a totally different internet. Instead we have this tragedy of the commons where the greatest minds money could buy are figuring out how to get people to waste more of their precious minutes on earth clicking on more cat videos so candy companies will pay for the server bill.
>Facebook was created to connect people and ended up facilitating genocide.
Throwing out the baby with the bath water here. How many lost friendships were recovered through FB? In my network of immigrants and refugees: countless.
>Breakthrough in private health-improving tech can bring unfathomed gap between rich and poor.
This is absurd. Why is the gap more important than the absolute level of the poor? All medical tech began as expensive and only for the rich.
"Throwing out the baby with the bathwater" is an expression that's never really made sense to me. The bath isn't a permanent position, babies and bathwater aren't fundamentally inseparable, why do we want to throw out the bathwater in the first place, is it even safe for the baby?
In the case of Facebook, the bath is toxic, we should remove the baby from the bathwater, toss out the bathwater, and then figure out what's going on (maybe we can just refresh the bath, maybe we need to figure out a regulatory framework for designing baths).
history of civilization is a history of people making other people miserable using whatever tools available. this _maybe_ changed a bit in the second half of XX century... looks like we're back to usual programming nowadays.
> Consider how many brilliant and outstanding people go work on some useless software at a big adtech company rather than helping to improve the world.
I've had conversations with young people nearing graduation about their career choices. Often it revolved around compensation levels at various tech companies. I'd ask them if they really wanted to dedicate their life to finding better ways to serve advertisements, or do they want to do something great?
There are many outstanding athletes who spend their whole career on the bench. Your resource allocation concern is a valid one, but it isn't one that is going to be addressed by big companies anytime soon, at some point individuals needs to weigh the 'pay vs helping world' thing.
You've extrapolated quite a bit. The truth is that I don't have a resolution, it is a very difficult problem that has many moving parts. The salaries that these 'highly talented' developers working on 'useless technology' command is not something that can easily brushed off.
Right, we agree on that. The salaries are a symptom of a system built on a number of perverse incentives that have ended up putting the majority of humanity's most productive minds to work thinking about stuff other than how to mitigate the catastrophe of the current mass extinction event
These all stem from the complete lack of empiricism and scientific method in this discipline. I'm pretty sure we all have opinions on most of that stuff. None of which are backed by any evidence whatsoever, we are basically always going with our gut.
"23. Has anyone ever compared how long it takes to reach a workable level of understanding of a software system with and without UML diagrams or other graphical notations? More generally, is there any correlation between the amount or quality of different kinds of developer-oriented documentation and time-to-understanding, and if so, which kinds of documentation fare best?"
This is such an important question and it's just the tip of the iceberg of a very deep problem that is rotting our software systems. We are absolutely pathetic at dealing with complexity and we actually enjoy complexity. We don't tackle questions such as 23. ANYWHERE near as seriously as we should.
Developers overestimate their mental bandwidth which leads them to pompously build over-complicated tech stacks despite only having archaic tools to mitigate and navigate their complexity.
Companies don't need to hire more devs to deal with their complex software systems, they need better tools to navigate their software systems. But because companies don't truly value their money and devs don't truly value their time, we end up in the situation we are in now.
We should have hundreds of companies investing on initiatives akin to Moldable Development[1], instead they play the following bingo:
1) let's just hire more devs and hope to land on a 10xer
2) let's build our own framework
Additionally, we overvalue specialization. By overloading developer brains with complex tech stacks, we encourage a culture of specialized profiles who find solace in trivia. Doing so, we limit cross-pollination and stifle true innovation. This attitude is actively killing-off thousands of valuable ideas.
Every second, there's a coder out there who thinks of something wild, which requires very specific tools from different fields and finds out that the people who built such tools couldn't be bothered making them accessible under a sensible time-budget to people outside of their niche/ivory tower. So the dev either drops the idea or gets sucked up into a niche.
This is tragic, but hey look! We have a new (totally not low-hanging fruit that could be predicted 10 years ago) Generative Model, WOW! "What a time to be alive"!
> Developers overestimate their mental bandwidth which leads them to pompously build over-complicated tech stacks despite only having archaic tools to mitigate and navigate their complexity.
Spot on. Most engineers don't (and can't) demand a working understanding of the systems they build, and are perfectly happy reverse engineering even their own creations (although we rarely admit so bluntly that this is the case). So, we're primarily a trial and error species, not architects. Alchemists, perhaps?
Tests, which is the only way any moderately sized software holds together, is just a way of automating the trial-and-error process from development. It even kinda works, but is limited by (among other things) the imagination of the test author, which frequently happens to be the business logic author.
Definitely agree in that we need better tools, but i'd argue that we just need better reasoning to be able formulate the right questions. Asking the right question can lead to surprising and sometimes trivial solutions.
that's because creating software is all about making human artifacts, not studying the natural world and that's an inherently subjective task. It's like asking "what's the best set of tools for a blacksmith?" or "what's the best way to brew beer?". Well, depends very heavily on the blacksmith or the brewery. Of course that doesn't mean all tools are equally good, but all the interesting cases are going to be personal.
There is no scientifically correct answer to what often are not objective questions, you will quickly find out that almost everyone will weigh the questions that the article asks very differently, for often legitimate reasons.
> that's because creating software is all about making human artifacts, not studying the natural world and that's an inherently subjective task
Electronic circuits are also human artifacts, but there objectively valid principles and practices for developing those. Of course there are also opinions and subjectivity on some characteristics that generally don't matter to the actual functionality, but that seems far from where software development currently is.
This may be, but it is useful to point out that, in many cases, engineering practices in more tangible scientific areas are also driven by some personal preferences / opinion.
For example, in which situations should you use Redlich-Kwong equations of state over a simple ideal gas or VdW? There will always be one model which gets closer to the measured values in a specific condition, but the entire domain may be difficult to determine which model is most appropriate.
Even in these fields the decision falls to opinion - some models may be more precise and accurate, but require more processing/ development time. Which is just software engineering again.
My previous comment came out harsher than I intended -- I don't mean to imply that opinion and experience don't play a factor in other disciplines, we are more prone to see issues with software engineering because we are more familiar with software engineering.
However I still think the situation is a bit worse. The tradeoffs in the models you mention are well understood. The equivalent in SE would be something like "should we use python or go", but it's very unclear that we can definitively state which use cases are better for which. There is simply no evidence, the plural of "anecdote" isn't "data".
I'd love to see (or do myself if I ever get the chance) some research into some of the hot-button software architecture, tooling and project structure topics that come up a lot here and in every team I've worked on. They always boil down to a religious debate with strong opinions on all sides, and everyone wanting to copy something they've read about how their favourite MAGMA company does it, but it feels like some of these should have empirically right answers:
1. Under what circumstances is a monorepo beneficial as opposed to multiple smaller repos and what are the determining factors? (Team size, codebase size, tooling, software coupling, project velocity, .... etc?)
2. Under what circumstances are different system architectures better, and what are all the different factors that influence this? (eg microservices, 3-tier, one big blob of code, ... etc)
3. Can we empirically measure or determine whether a language, framework, library etc is the right choice for a given situation and how might we do that? Is it possible to formulate rules to inform good decisions here or is it always going to be a matter of judgement?
4. How do different styles of working (Pair programming, scrum, TDD etc) affect team productivity, code quality, developer happiness, project velocity etc? What are the factors that make one preferable over another in a given situation?
5. What's the right team size for a given project?
6. What's the best way to discover and communicate software requirements?
...
I could go on. But in other engineering disciplines, a lot of the analogous things are solved problems rather than being topics for heated debate.
Oh, that's a nice one - definitely way better than FAANG!
That being said, I think that Meta is evil and should not be promoted in any way - so we might go for a version without it, for example MAGA! Oh, wait... :P
> my impression is that we’ve gone from packaging or building an installer taking 10% of effort to cloud deployment infrastructure being 25-30% of effort, but that’s just one data point
The old bad days of shrink-wrapped software involved installing in environments we could not control. Tbe direct costs of writing an installer and testing (the larger effort) install, upgrade, uninstall, recovering from partial installs, etc. were nothing compared to the costs of dealing with all the weirdness that could exist on a customer's machine.
We're doing things like blue/green deploys and CD - things not remotely possible in that other world. And we're hooking up all kinds of monitoring and observability tooling that . . . even if we could have shipped that to customers on physical media, we'd be unable to harvest data from in those old times.
I think we're spending a lot less engineering time and getting a bigger bang for the effort.
I worry, though, that because software engineering has the shared professional memory of a goldfish that can always swim to an unvisited side of its bowl, we'll somehow try to repeat that pattern we've overcome.
I would offer the view, that for teams flush with money, everything you said is great. But if you're a solo dev or in a small team, deployments and finding the correct balance is actually tricky to manage. Developers created a solution to the old deployment problem by abandoning it and doing something else. That problem exists and you'd think it'd just be solved now so rolling out a desktop app wasn't such a crazy pain.
I worry that software engineering is building a lot of tech stacks that need active maintenance and can't really go into a classical long term support style use.
> Developers created a solution to the old deployment problem by abandoning it and doing something else
Also by shipping a standard webview and js app, if you want to go that route. But there were plenty of cross platform solutions before that, most notably java.
Of course you know all that, guess I'm just trying to remind us that desktop deployment wasn't that unsolved and still isn't.
Steve McConnell's Code Complete published by Microsoft is a classic text in this area, one of the only ones I'm aware of that actually uses academic sources for its claims. However it was published in the 90s and only updated once in 2004, so I wonder if there's a newer version or new book which updates this?
*At what point is it more economical to throw away a module and write a replacement instead of refactoring or extending the module to meet new needs?*
If you have rotation in the team it is a lot easier to get new people write new code than learn old ways - especially when original authors are all gone.
If you have stable team, as long as knowledge is there and module fits style of the team I think rewrite will not be needed unless someone wants to shove some not fitting requirements there, which would be better handled by just making separate module.
For the first question, putting the documentation into the source code has a major problem, there is no good place to put an overview of the whole system. Block diagrams are difficult to draw with just ASCII, images can't be included, formulas, and more. Also many things in software connect to many other things, and if the documentation is in the source code you can likely only describe one side of these connections (or you have to describe both fully on each side of the connection, thus duplicating a lot of the documentation.) Comments are immensely useful, but I don't think they work for general purpose documentation of a system. They can work well for libraries where each API needs to be described separately, but even then I wouldn't consider them complete documentation of a library (how do you combine graphics functions to build a rendering pipeline for example, do you have to described how to build pipelines for every single function that might be used in a pipeline?)
Some of these could be tested among college students or high-school students. Let students form groups and participate in a tournament. Make the task complex enough that the tournament requires sustained work (e.g., 12 weeks). Then randomly assign coding practices to the participating groups and evaluate:
> Do doctest-style tests (i.e., tests embedded directly in the code being tested) have any impact long-term usability or maintainability compared to putting tests in separate files?
> Has anyone analyzed videos of coding clubs for children or teens to see if girls are treated differently than boys by instructors and by their peers?
> Has anyone ever studied students from the first year to the final year of their program to see what tools they actually start using when. In particular, when (if ever) do they start to use more advanced features of their IDE (e.g., “rename variable in scope”)?
This is an awesome list of questions and it's incredibly thoughtful that you documented and shared it with us.
It reminds me of Hitchhikers Guide where they know the answer but need the question. So many answers are out there floating around that we are often trying to force into certain frames for various motivations.
Having questions that get us thinking, recognizing how unclear things are in current state, this
feels like such a powerful thing to share.
Apologies for the diatribe, very much appreciated.
This is an amazing list, and really highlights how much we take on trust in this industry: do X for good/better/best outcomes.
Taking a step back I wonder: if there did emerge clear, incontrovertible evidence that doing X led to better outcomes, how many teams would actually adopt it?
And how many would dismiss those findings regardless?
In these modern times I suspect we'd see a far higher rejection rate than 10-years-ago me would've anticipated.
I would love to hear what people do for 25? I’m struggling with this myself - with static site generators you can’t always put files in the same directory as the markdown
My most important question regarding software engineering would probably be something like: how well are resources allocated and could we do a better job? Consider how many brilliant and outstanding people go work on some useless software at a big adtech company rather than helping to improve the world. The greatest tragedy of our time.