Programming is specialized work (I'd call it a discipline, but then we'd have to argue whether it's more an engineering discipline or art discipline). You have to gain knowledge and experience to understand it.
A bridge blueprint is not understandable for all. It is understandable for all who took time and effort to gain education necessary to understand it.
(The motivation between this comment is to counter the increasingly noticeable sentiment in our industry that everything needs to be dumbed down to the lowest common denominator; it's an understandable sentiment if you're selling something and want the biggest market, but it goes against what's needed to build great things.)
The thing is, people have been building bridges for thousands of years. What a bridge is and what it does changes very slowly. Almost all the specialised knowledge that bridge builders have is about _how_ to build a good bridge.
Software is not like that. We may be experts in programming languages, algorithms and data structures, but the things we create and the problems we solve keep changing all the time.
We're not usually experts in these problem domains and in some cases there are no such experts at all. So our code needs to be a lot more descriptive and written for readers that may not be very familiar with the problem at hand.
We are more like lawyers supporting law makers or even like law makers themselves.
We think bridges are a very well explored domain. Then something like the Tacoma Narrows incident happens, and we realize how much of a fallacy that really is.
Bridges are unique to their locations and their scale: what works over a creek will not work over a ravine. What works for a pedestrian will not work for a car. What works for a car will not work for a train. What works for a train will not work for a marching army.
The Tacoma bridge collapse has a wikipedia entry exactly because bridge building is a very well explored problem and bridges don't usually collapse.
I don't doubt that each bridge comes with unique challenges. But the purpose, user interface and constraints of bridges have remained stable enough for long enough to allow specialisation. That is not the case in many areas of software development.
>everything needs to be dumbed down to the lowest common denominator
That kind of ought to be the default for most code. However, it's the natural tendency of code to become abstruse and unnecessarily complicated.
Since market forces will often tend to exploit rather than reward the work done by programmers to "dumb their code down" and inadvertently reward "clever magic understood by a limited number of experts" there's more of an incentive to amplify this effect than to work against it - especially where money is involved.
I'm a bit conflicted because on the one hand I don't want to help developer compensation get ground down by billionaires or other business owners with an entitlement complex who consider developers to be "spoiled brats", but I do prefer working with clean, straightforward code.
There are times when you simply have hard problems.
For example, take code generation (I actually have in mind a SQL query generator, but a compiler back end would do just as well). The problem is already abstract - the input is code, you're naturally writing code that manipulates code, like you would with macros or reflection. The problem is complex: you can generate simple code, but it won't perform well. There's irreducible complexity here that cannot be simplified away.
You'd also like the code generation itself to be fast. That puts a limit on how much you can break it up into parts that can be understood individually and in isolation. Optimization works in opposition to abstraction because the optimal often requires steps that span multiple abstraction layers, and often requires multiple instances of the specific over few instances of the general.
Personally, I think the two biggest reasons code becomes unnecessarily complicated these days are (a) testing and (b) local modifications. Unit testing in particular encourages over-parametrization so that dependencies can be replaced for the purposes of testing; while normal software maintenance under commercial pressure leads to local modification because nobody has time to understand the whole. People instead make conservative local changes by adding parameters, extra if-statements, local lookup maps, etc.
I've found elegant solutions are often on the other side of a hill from over-engineering. You write specific solutions, then you climb the a hill of abstraction as you add layers, indirections, parameters etc., until you reach a summit, where you can see the whole, and can then start boiling things back down again, only retaining abstraction where it's actually necessary, or perhaps replacing multiple abstractions with a single more powerful abstraction (I've found this to happen a lot with monads; another one is converting control flow into data flow).
>There are times when you simply have hard problems.
>
>For example, take code generation (I actually have in mind a SQL query generator, but a compiler back end would do just as well). The problem is already abstract - the input is code, you're naturally writing code that manipulates code, like you would with macros or reflection. The problem is complex: you can generate simple code, but it won't perform well. There's irreducible complexity here that cannot be simplified away.
Yeah sometimes you do, and that is exactly the kind of problem that is irreducibly complex, but I think new kinds of problems like this don't tend to crop up in the wild very often and when they do they tend to show up in subtle and non-obvious ways.
The problem you've described is far from a new problem - it's the same problem space that is covered by ORMs. Furthermore, if I were working on a team where a developer has uttered the words "I've created my own ORM" (or something to that effect), my face has probably already landed in my palms.
The rest of what you wrote I'm in vigorous agreement with though- especially the parts about unit testing, local modifications and "the other side of the hill". Seen all of that.
> The problem you've described is far from a new problem - it's the same problem space that is covered by ORMs.
This is digression.
I have in mind something I wrote, the most complex piece of code I've written in the past couple of years. It isn't actually well covered by ORMs. ORMs are usually tuned for (a) static schemas, and (b) graph navigation in OO-style. Give them a problem like "here's a filter in the form of a syntax tree, please give me the top 100 results from this 10 million line table" - where the table schema is determined at runtime - well, most ORMs can't even answer this question because schemas are assumed to be static. And if you want to control the join order using nested subtable joins, with predicates that don't need joins pulled out of the filter expression and pushed down, because MySQL's optimizer observably doesn't reliably do the right thing, ORMs don't give you that control.
It wasn't an ORM kind of problem; think more something like a read-only Excel spreadsheet, but with typed columns, and a rich autofilter, on the web, scaling to millions of rows. The output of the database query is a page of tuples, not objects in any behavioural sense.
In some ways a database seems like the wrong solution, but morphologically it's exactly right: the user data is rectangular, relational, has foreign keys to other tables, and needs to be sorted, filtered and joined with other user-defined schemas. Modelling the user schema as database columns performs better than any other database solution, and database solutions are preferred because shared state, transactions, etc. Because the product is closer to being an actual database than a program using a database, ORMs aren't tuned for it.
Ok so when you said "SQL query generator" I interpreted "a generic tool for SQL query generation". i.e. ORM. Metaprogamming makes sense there.
Your rather unique use case certainly falls outside of the remit of what ORMs provide (cutting down on generic SQL boilerplate) but I'm unconvinced that what you built is in need of especially powerful language features to build it.
> I'm not sure what hard problems have to do with code readability.
Code can be hard to read because it's badly written, or it can be hard to read and understand because it deals with a hard problem.
> I've seen hard to read code that solves simple problems and I've seen easy to read code that solves hard problems.
I've seen easy to read code that solves simple problems that seem hard because of things like combinatorics (e.g. Sudoku solvers and the like). Actually hard problems don't have simple solutions; there's a complexity that doesn't go away no matter how you express the solution. This is doubly true when there are constraints on the solution in execution time and space, because such constraints limit how much you can break the problem down.
> Why are you advocating for the hard to read code?
Exactly the kind of over-engineering I'm talking about.
The objective is to solve business problems, not write tests. Warping design to introduce unnecessary abstractions and indirections for testing is what leads to Java-itis, with factory factories.
This is exactly why I only ever test at the edges of a project, using mock systems that are as realistic as possible given time and resource constraints. The industry has an unhealthy obsession with ultrafast, ultra-unrealistic, low level unit tests that tightly couple to implementation details rather than behavior.
I don't think this is over-engineering though. Even if unit tests didn't warp designs they'd be a bad idea. I think it's just bad engineering based upon stupid dogma spread by the likes of J.D. Rainsberger and Uncle Bob.
The problem is, the converse is not always true. Problems that are very easy to formulate ("find the k shortest paths from my house to my office") might require very sophisticated, non-obvious, non-simple code to be solved in a non-naive way.
We all know that sometimes problems are easy to state and hard to solve. To my mind the problem you describe sounds quite possibly impossible to do better than brute force, so any code at all that solved it would be to a certain extent insightful (but the simpler, the better - and truly great code for that problem would be code that let me understand how an easier solution was possible).
I highly disagree with this, though I like your statement on principal.
Greatness is highly dependent on context. I think your statement on greatness would apply to code that helps you learn and think.
This is, however, NOT the kind of code I'd want to see at work. Great code in a business setting is simple, easy to understand, and has absolutely no subtlety.
If I was looking for a literary example, great code in a business setting would be like the writing style you'd see in a newspaper: written for easy consumption by the greatest number of people possible.
