I've talked to Knuth. He claims he could not have implemented MMIX without literate programming. Literate programming is really valuable but you only understand that once you really try it. I gave a talk on this subject at the WriteTheDocs conference:
You can write a literate program in any language, for instance, in HTML:
There are some "gold standard" literate programs:
"Physically Based Rendering" by Pharr and Humphreys won an academy award. "Lisp in Small Pieces" contains a complete lisp implementation including the interpreter and compiler. The book "Implementing Elliptic Curve Cryptography" is another example.
Suppose your business depends on a program. Suppose your team leaves (they all do eventually). Suppose you need to change it... THAT's why you need literate programming. Nobody is going be around to remember that the strange block of code is there to handle Palm Pilots.
Companies should hire language majors, make them Editor-in-Chief, and put them on every programming team. Nobody checks in code until there is at least a paragraph that explains WHY this code was written. Anybody can figure out WHAT it does but reverse-engineering WHY it does it can be hard.
Imagine a physics textbook that was "just the equations" without any surrounding text. That's the way we write code today. It is true that the equations are the essence but without the surrounding text they are quite opaque.
Imagine how easy it would be to hire someone. You give them the current version of the book, send them to Hawaii for two weeks, and when they return they can maintain and modify the system as well as the rest of the team.
Do yourself a favor, buy a copy of Physically Based Rendering. Consider it to be the standard of excellence that you expect from a professional programmer. Then decide to be a professional and hold yourself to that standard.
Whilst I haven't read Physically Based Rendering, I had the same illuminating experience reading C Interfaces and Implementations: Techniques for Creating Reusable Software by Hanson which was also written in the literate programming style.
Aside from that, i tend to agree with you, and i aspire to the same. Unfortunately laziness and time constraints often get the better of me, though.
Unless someone puts a comment above it?
Literate programming isn't the only method of documenting code.
The thing is, as I wrote in the other comment, that "modern"(like Lisp, as someone noted ;-)) languages already allow you to structure the code as you see fit. You don't have to define (or even declare) functions before their first use in the file, you can easily extract any part of the code into a function or method and you can place it anywhere you want. There are many mechanisms available for threading the context through functions, which makes direct, textual inclusion simply not needed.
In short: for LP to work you need to organize your code a way you'd like to read it. Many languages are capable enough that they don't need third party tools for these.
And I assumed that you do it anyway, because there's no real downside for this in many languages. If you don't do it in a language which supports it - shame on you. If you do, then "enough comments" is the only thing you need for your code to begin being literate.
Basically, if there are any parts of the program that you would describe differently in conversation, then it isn't the same thing.
Now, I fully grant I am getting close to a scotsman argument. I also grant that this is not necessary to write bug free software. However, I do see them as different things.
And those companies will get their lunches eaten by folks who know that you can do well enough simply by throwing bodies at the problem, unfortunately.