Those at the top are, for lack of a better term, at one with their profession, and programming is certainly no different. A good programmer can feel, in a very visceral way, the difference between good and bad code, in the same way that the average person can feel the difference between Michelangelo and street graffiti. I consider myself to be only a very mediocre coder, and yet every now and then, I see a program that is just... beautiful. Aesthetically pleasing. And I couldn't tell you why.
Things like that can't be diagramed or taught; it's up to the individual to grow that seed of what Pirsig calls 'Quality' within themselves.
My approach is to consciously absorb as much of the problem at hand as possible, then let my subconscious crunch on it for a while, the length of time depending on the complexity of the problem. Coming up with the big concept for a large design, both building or software, can take a few days if its revolutionary, or a few minutes if its not. The key is to have a process that helps you leverage the power of your subconscious. Trying to force a solution in your conscious mind on a harder problem can be both painful and a waste of time.
You hit on a good insight in the essay: we don't know how we program. But I think you incorrectly assumed that this is unique to software development. Instead, I think it's an inherent part of any job that requires problem solving.
Even architects find it hard to plan for growth it seems :)
The McMansions and office blocks aren't that different from 95% of people using Windows or Mac. It's just more painful in architecture because it's so visible.
driving a car, flying a plane:
These are extremely poorly understood tasks, but fortunately parts of the average human brain are pretty good at things like that.
running a company:
Smart people fail at this all time.
teaching a child:
Why are so many people unable to remember the math they learned in grade school? Why are we engaged in a public debate about the value of teaching to standardized tests instead of some less well-defined but apparently better approach?
curing a disease:
Come on, new methods and approaches are constantly being invented in medicine! We have no general way of doing this.
You mean like 2+2 or 9x5? Most people forget the advanced math that they learn in high-school, but the basic stuff is pretty well ingrained into the average subject, accounting for variations, of course.
I'm not against hacking per se. I'm doing it right now. But to take it as the de facto way the software industry functions is wrong. Hacking has certain problems, like unanticipated problems tend to stall the process (My own recent encounter had me blocked for three days, trying to think of an adequate solution to the problem of how authentication should work in an environment where HTTP servers and clients are essentially "decoupled", that is a solution to the problem of where presentation and application data are coming from two different sources, how does the data provider verify the presentation logic is trustworthy? It's a hard problem to which I have only a partially satisfactory answer... but I digress.)
The process for writing code tends to be unwritten, simply because in the best case it is so trivial that writing it is an insult to the reader. I guess the industry's collective problem is that the best case happens so seldom.
On the contrary, you touch the heart of the matter. You say what you are doing is "hacking", with the implication that there exists some other process by which your problem could be solved. Can you tell me what this process is, and why you are not following it?
AKA, I need a function called add that adds each element in an array and returns the result.
Now what data type is in that array? What should you return if the array is empty or null? etc.
So, you can't make a design doc detailed enough to make the programmer make zero decisions, or you could just compile that design doc.
Perhaps not all schools teach it this way, but my computer science education included a semester of lectures that started us off in the process of Stepwise Refinement.
In S.R. you start with a goal and progressively, or step-wise, refine it until you are at a level in which it is obvious what code should be written, whether it be an assignment, expression, loop, functional call etc.
I believe S.R. dates from Dijkstra and the Algol-W period.
That's the fun part.
I disagree. There is a process, but it is so basic we may not realize it:
1. identify the big problem
2. break it down into smaller problems
3. solve the smaller problems
4. repeat as needed
Your process is so vague, it's like saying the process to get from Seattle to Hawaii is:
1. Leave Seattle.
2. Arrive in Hawaii
Consider writing a connection counter for linked in. How many connections do you have, and how many 2nd level connections do you have through those people. If you decide to break this down into smaller problems, you are going about it the wrong way in all likelyhood. An optimal solution will involve a dynamic programming algorithm, which solves the entire problem in a tricky way (a clever hack).
* Make a list of use cases
* Grab a stack of 3x5 cards and start writing nouns on them
* Check to see if someone else has partly or completely solved the same problem already
* Write down the basic data structures associated with the problem
* Identify a minimal command-line program you could write to investigate the hardest problem(s)
* from __future__ import solution (or: M-x solve-problem)
Step one: Look medusa in the face.