One thing really stood out for me from the book "Programmers at Work".
Many of the programmers repeatedly emphasized the importance of data structures to both understanding programs as well to solving the problem. And after noticing how I work, I find that to be true.
English documentation of the core data structures is more important than verbose description of the steps in the algorithm. This is the key thing that is different among progarms. By and large, we use the same algorithms again and again. And I can trivially find English documentation of those to understand rapidly. I don't need it in the code also.
And by documentation I mean something that explains at a programming language level with the identifiers etc. For git, it would be insufficient to say "trees are collections of blobs". I would need to know the struct names and an explanation of each of the members. And whether the "collection" is a array/list/hash etc.
Having looked into literate programming, it seems to emphasize documenting the algorithms more than it does the data structures. Hence it misses the "big picture" which is what is really needed.
Look at the gb_graph module of Knuth's "Stanford GraphBase" (the Stanford GraphBase is available for download at his web-site). His explanation of his graph data structures, written in CWEB, are the clearest documentation of a complex data structure that I have ever seen.
I use literate programming, and Knuth's CWEB in particular, and I find it fits for me perfectly. Honestly, I think the biggest problem is that most programmers have never heard of it or think it's something like JavaDoc.
I find it astonishing that so few people are willing to try something developed and strongly advocated by the man who single-handedly programmed TeX, MetaFont, and the Stanford GraphBase, and wrote the most respected texts in computer science.
I recently started using Leo for a combination of outlining, literate programming, and general project management.
I don't go as far as Knuth and try to formulate a complete narrative of my program, but Leo's outlining features let me clone and regroup blocks of code on a task-by-task basis and document: "In these blocks, I'm implementing feature x," or "fixing bug y." It's very useful for getting the big picture and making sense of problems in the codebase. When using only a code editor, you waste a lot of time jumping through the project and searching through files, and I don't have that problem anymore.
A fun reminder that where most programmers (I suspect) are of the iterative sort -- especially where the language is concise and testable -- Knuth is an inveterate planner. If you've trained yourself to strategize mathematically and execute long streams of pseudocode in your head, a word processor is your friend. :)
That said, I've long wanted more expressive commentary in the trickier parts of the codebase. i.e. some kind of comment xml that IDEs make into a rich text area, for the diagrams and maths and whatnot.
Why not do the entire codebase as some sort of nested structure, designed only to be presented inside an IDE? So not only comments, but types, namespaces, every bit of programming metadata there is.
Many programmers feel insecure abandoming their text editor -- because this is something they're accustomed to and which they understand conceptually. Some sort of nested structure expressed in a markup language like XML would be just as understandable conceptually. Many of the incompatibilities between languages would disappear if there was a widely used standard for such a code representation.
As a user of Smalltalk for almost a decade, let me assure you that walking away from source files is not folly! In fact, I get something which is just as nimble and reliable, and yet even more powerful. I can write snippets of code that act like queries into my codebase. With a short script I can rewrite broad swaths of my code with syntactic precision, and with a little work be assured that I have done a perfect refactoring. I can undo/redo all changes. I can even do dangerous things that even bring the VM to a screeching halt, and not lose any code. All of this is enabled by giving up source files and going to a "change log" which is like a transactional log of my changes.
You can think of this as something like going from ext2 to ext3, then onto ZFS.
i haven't looked into this myself, but i think that Mathematica 'notebooks' tries to reach this ideal of "just leave everything in mathematica, and we'll take care of code, tests, visualizations, documentation, etc."
One other point I think is critical to understanding its lack of lift is that it solves some problems most modern languages don't have with code organization. Earlier languages were much more restrictive about how you could organize code, and giving the programmer the ability to rearrange code in a more flexible manner could be a genuine advantage. Nowadays our problems lie more in unwise use of our extensive ability to organize and reorganize code, rather than our inability to do so at all, and adding in "conventional" LP only gives us another dimension to screw up on.
I do not believe this is the entire story, but it's an important part of it.
Where can I get this penelope system? It sounds interesting and I am an Ada programmer. Anything that can make software more reliable is a good thing, imho.
Incidentally I've used cweb before and can second their comments about it. It's a fantastic idea, that needs a bit of a refresh. Particularly getting away from simple commenting of how things are done to the why/data structure/higher level arena.
Many of the programmers repeatedly emphasized the importance of data structures to both understanding programs as well to solving the problem. And after noticing how I work, I find that to be true.
English documentation of the core data structures is more important than verbose description of the steps in the algorithm. This is the key thing that is different among progarms. By and large, we use the same algorithms again and again. And I can trivially find English documentation of those to understand rapidly. I don't need it in the code also.
And by documentation I mean something that explains at a programming language level with the identifiers etc. For git, it would be insufficient to say "trees are collections of blobs". I would need to know the struct names and an explanation of each of the members. And whether the "collection" is a array/list/hash etc.
Having looked into literate programming, it seems to emphasize documenting the algorithms more than it does the data structures. Hence it misses the "big picture" which is what is really needed.
I'm not convinced that LP is useful.