I actually agree with your main point — in fact I would go further and say that in an ideal LP system, the (or at least a) source format would just simply be valid files in the target language, with any LP-related markup or whatever in the comments. The reason is so that LP programs can get contributions from "mainstream" programmers. (It's ok if the LP users have an alternative format they can write in, as long as edits to the source file can be incorporated back.)
So the following correction is not intended as disagreement but simply to share some history / facts:
> Consider this file. Those who are expected to interact with it in this form […] You aren't expected to read a literate program like this. In an ideal world, where we had computing better figured out, Knuth wouldn't interact with it like this, either.
Actually, even in the world we live in, no one is expected to read a literate program like this (Knuth has comments to that effect somewhere), and everyone is "supposed" to read the typeset program (today that would be tex.pdf or whatever). Knuth reads it off the printed book. He navigates the program using the indexes in the book, and as recently as this year he raved about how good his system was (https://tug.org/TUGboat/tb42-1/tb130knuth-tuneup21.pdf):
> While I was preparing this round of updates, I was overjoyed to see how well the philosophy of literate programming has facilitated everything. This multifaceted program was written 40 years ago, yet I could still get back into TeX’s darkest corners without trouble, just by rereading [B] and using its index and mini-indexes! I can’t help but ascribe most of TeX’s success to the fact that it has enabled literate programming.
This works for him, because he started programming in the days when program "listings"—printouts of source code—were the most common way to read programs, and it's also how he wrote TeX (https://news.ycombinator.com/item?id=10172924: “Knuth wrote the entirety of the first version of TeX on yellow legal note pads, and then typed it all in, and then started debugging” — he worked full-time on the program, writing it with pencil on paper, over several months, before typing a single word of it into a computer). And of course, he's a professional book-reader and book-writer (and book-tweaker, as his books keep getting newer editions and updates), so it comes naturally to him.
> Actually, even in the world we live in, no one is expected to read a literate program like this (Knuth has comments to that effect somewhere), and everyone is "supposed" to read the typeset program (today that would be tex.pdf or whatever).
Sorry if it wasn't clear, but that was the point that I was making. The file that susam has linked to here is nominally "plain" "text", but no reader is expected to consume it in this form, or really interact with it in this form at all. This form serves exactly one set of humans: Knuth, et al, and only while they are actively editing it.
To reiterate my earlier claim: a more perfect LP system would be designed in such a way that the plain text input format would be a suitable way to read programs written in that system. (And when I say "suitable", I don't merely mean "possible". I mean designed expressly for that purpose, like how Markdown was _intended_ to be used before the GitHub contingency got their hands on it.)
> LP-related markup or whatever in the comments
That's sort of the opposite of what I'm after! Please do check the Raskin article I linked to.
Ah I see, thanks for clarifying. I had missed the subtle distinction in your comment between "interact with" and "read": I treated them as synonymous, when in fact you were referring to the interaction that is confined to actively editing (i.e., for Knuth, "typing up" the edits he has already made on paper).
About the more perfect LP system, I agree it would be nice if the input format was designed to be the suitable way to read programs: in other words, there would be no "weave" step. [Aside: should this input format be plain text? Maybe notebooks like Wolfram's or "nbdev" count?]
The idea of having the markup in the comments is instead related to eliminating the "tangle" step. (I think there are some LP-related tools that do this; one I know is Harold Thimbleby's "warp": http://www.harold.thimbleby.net/cv/files/warp.pdf — but the structured comments are in (ugh) XML. See also Visual Studio's XML comments…. He also has a later "relit" that does "reverse" LP, extracting source code out of the intended-for-publication format.)
(These goals are indeed sort of opposites, but can be reconciled: I guess we can agree that the ultimate would be a format that eliminated both the tangle and weave steps!)
(My reason for wishing eliminating the "tangle" step (and/or enabling going back from the tangled document to the LP input format) is simply an acknowledgement of network effects: LP is unlikely to ever catch on enough to be the majority, so there needs to be a way for a random programmer using their preferred IDE/editor to edit a "literate" program without getting too annoyed.)
I read Raskin's article and it has good complaints about IDEs, and (although I don't see a close connection) I think I understand what you're suggesting as the ideal language/programming environment. But for me, because of the "LP will never be mainstream" belief, I'm still thinking of targeting mainstream languages, with "code" and "comments". :-) Anyway, this part from Raskin's article is particularly inspiring and quotable:
> …the result was very readable and maintainable. To achieve this, we had regular code-reading sessions in which one programmer read and commented on each piece of code written by another programmer. In addition, an expert documenter/writer worked alongside the programmers. If he did not understand a piece of code, he would interview the programmers until he could write a cogent explanation.
> Incidentally, many bugs and conceptual errors were discovered during this process, which more than paid for itself in decreased debugging time. Some programmers balked at the procedures at first, but all came to be enthusiastic when the project did not slow or founder as completion neared. The project was on time, on budget, and bugless (meaning that the software was released commercially to tens of thousands of users and produced no bug reports).
So the following correction is not intended as disagreement but simply to share some history / facts:
> Consider this file. Those who are expected to interact with it in this form […] You aren't expected to read a literate program like this. In an ideal world, where we had computing better figured out, Knuth wouldn't interact with it like this, either.
Actually, even in the world we live in, no one is expected to read a literate program like this (Knuth has comments to that effect somewhere), and everyone is "supposed" to read the typeset program (today that would be tex.pdf or whatever). Knuth reads it off the printed book. He navigates the program using the indexes in the book, and as recently as this year he raved about how good his system was (https://tug.org/TUGboat/tb42-1/tb130knuth-tuneup21.pdf):
> While I was preparing this round of updates, I was overjoyed to see how well the philosophy of literate programming has facilitated everything. This multifaceted program was written 40 years ago, yet I could still get back into TeX’s darkest corners without trouble, just by rereading [B] and using its index and mini-indexes! I can’t help but ascribe most of TeX’s success to the fact that it has enabled literate programming.
This works for him, because he started programming in the days when program "listings"—printouts of source code—were the most common way to read programs, and it's also how he wrote TeX (https://news.ycombinator.com/item?id=10172924: “Knuth wrote the entirety of the first version of TeX on yellow legal note pads, and then typed it all in, and then started debugging” — he worked full-time on the program, writing it with pencil on paper, over several months, before typing a single word of it into a computer). And of course, he's a professional book-reader and book-writer (and book-tweaker, as his books keep getting newer editions and updates), so it comes naturally to him.