
Show HN: Lit – a modern literate programming tool - cdosborn
https://github.com/cdosborn/lit
======
porker
I don't know if this counts as literate programming, but one of my favourite
outputs for learning about code is Docco:
[http://jashkenas.github.io/docco/](http://jashkenas.github.io/docco/)

Sample output:
[http://backbonejs.org/docs/backbone.html](http://backbonejs.org/docs/backbone.html)

~~~
desireco42
Yep, he pretty much applied literate programming and first made it practical

------
cdosborn
I think the greatest motivation for literate programming is providing context.
After five minutes of reading, you can have a very good high level
understanding of a complex program. This just isn't achievable in the same
amount of time otherwise.

~~~
jostylr
In that vein, you should consider the ability to coordinate a project using
literate programming. It is hard enough to understand a code-base in a single
file, but when a project has several different directories with a variety of
files in each, having a starting document that describes all this structure is
extremely important. And one can write the compiler to use that starting
document itself to generate that complex structure.

------
ppereira
I love literate programming, however my main issue with most tools like this
one is that they do not properly support compiler/debugger error messages.
This is a huge, often unstated caveat. It is very important to be able to read
error messages and quickly locate the offending line in the literate source.

For languages that support file/line pragmas like C, literate programming
works very well. Alternately, if the language supports goto and can sensibly
unravel these statements, one can cobble a weave/tangle script to still have a
1-1 mapping between program and source line numbers.

Otherwise, I find that I need to program in a very functional style in order
to force my tangled program to have the exact same line numbers as the
literate source. This is possible, but it also negates the advantages of
having natural language macro names, since they are essentially equivalent to
the function names. In this case, tangle becomes an identity mapping.

~~~
cdosborn
This is a feature I'm looking to add. It was designed to be included. When I
parse the lit file, I record the line no. which will be represented in
comments in the generated source file. Hopefully, this will mitigate some of
the extra workflow.

The granularity will be for every macro defn, so there will be some ambiguity.

------
padator
The main issue I had with LP in the past is that sometimes you want to modify
the code directly, not the literate document and synchronize them. I made such
a tool: [https://github.com/aryx/syncweb](https://github.com/aryx/syncweb)

~~~
jostylr
Neat. Do you find that you use that a lot? I have thought about it
implementing it in the literate-programming tool I wrote, but so far I have
found a split screen editor (vim for me) with search to be fairly effective.

On the opposite side of things, I rather enjoy mucking about with the compiled
code to diagnose it, and then recompiling to get rid of the diagnostic debris.

~~~
padator
I use syncweb all the time. I can't go back to just using noweb. Many of the
subprojects in pfff have a literate document that is synchronized with the
code using syncweb (e.g
[https://github.com/facebook/pfff/blob/master/h_visualization...](https://github.com/facebook/pfff/blob/master/h_visualization/Treemap.tex.nw)
). When I have errors in my code, or when I debug my code, I do it on the
generated code, not the tex document, so of course if there is a fix it's
easier to fix the code directly and far later to synchronize.

------
lindig
Telling from the number of recent implementations, Literate Programming enjoys
a bigger following than I had expected. Here is my implementation of a similar
tool in OCaml:
[https://github.com/lindig/lipsum](https://github.com/lindig/lipsum). I have
some projects using it on GitHub like [https://github.com/lindig/ocaml-
hyphenate](https://github.com/lindig/ocaml-hyphenate), which implements
Knuth's algorithm for hyphenation. The README is basically the literate
program of the project.

Question for the author: Are the spaces around `<<`, `>>`, and `>>=`
mandatory?

~~~
cdosborn
No they are not

------
thristian
From the description in the README I feared that the entire source-code of the
program would appear twice in the resulting document; first under the
definition of the " _" macro and again wherever each code-fragment was
defined. Looking at the contents of the "examples" directory, however, I can
see that the "_" macro works more like a table-of-contents.

That's reasonable, in a minimalist kind of way, but it's a bit unfortunate
that Lit syntax winds up unmodified in the output document; I'd wind up having
to put a paragraph at the top of each document explaining what Lit was and why
all the "<< >>" tokens throughout the code weren't actually part of the code.

Also, the resulting HTML doesn't actually validate:
[http://validator.nu/?doc=https%3A%2F%2Fraw.githubusercontent...](http://validator.nu/?doc=https%3A%2F%2Fraw.githubusercontent.com%2Fcdosborn%2Flit%2Fmaster%2Fexamples%2FhelloWorld.hs.html&schema=http%3A%2F%2Fs.validator.nu%2Fhtml5.rnc+http%3A%2F%2Fs.validator.nu%2Fhtml5%2Fassertions.sch+http%3A%2F%2Fc.validator.nu%2Fall%2F&parser=html5&laxtype=yes)

~~~
cdosborn
The project is still early, and the html generation is rudimentary. Will be
working soon to validate.

Not sure what you mean by appearing twice. I debated removing the "<< >>"
syntax in the generated html, but I don't think it's bad. It's essentially the
syntax for macros, which aid in providing context. It's useful in html,
because you can quickly refer to the defn of a macro by following the anchor.

~~~
thristian
The readme says:

 _lit only has two valid constructs: A macro definition: << ... >>= and a
macro reference: << ... >>_

...from my experience with other macro systems, I assumed that a "macro
reference" would be replaced with the content of the macro definition, leading
the code-block to appear at the top (under the star macro) and also in the
macro definition. I'm pleased to see that's not the case.

I'd rather not have the "<< >>" syntax in the resulting HTML, because it's the
syntax for _Lit_ macros... if I'm writing a document in human language A to
explain concept B in programming language C, that's already a lot of context,
and requiring the reader to also be familiar with literate-programming-tool D
is a drawback.

Linking to macro definitions is definitely a useful feature, but I'd rather
those links were distinguished with a CSS class so that I could define their
location and appearance in the stylesheet, rather than giving them specific
text and markup.

~~~
cdosborn
There should be an HTML class. In the resulting docs I still need a way to
separate them from the underlying language (that could be anything). Initially
I thought `<<>>` looked archaic, but its rather unique and probably the syntax
which clashes with the least number of programming langs.

------
taeric
I really really want to like this. Org mode for emacs has really expanded what
I could want from a literate programming tool, though. Specifically, being
able to execute small segments and have the output immediately usable is
rather nice. That you can also feed tables of data directly into the language
is also nice.

Granted, I do feel I have a fair bit to learn in how to structure code for
others to read. Myself included. I am learning the rather obvious point that
keeping a narrative to the code is not easy. And, of course, in the couple of
attempts I've done recently, I end with a fairly large dump of "and here is
the boring stuff" at the end.[1]

Also, I can't recommend reading straight from Knuth's site heavily enough. His
programs are rather interesting by modern aesthetics, but they are all still
runnable.

[1]
[http://taeric.github.io/DancingLinks.html](http://taeric.github.io/DancingLinks.html)

~~~
jostylr
My literate-programming tool, [https://github.com/jostylr/literate-
programming](https://github.com/jostylr/literate-programming), has this
feature. It really is nice to be able to pipe bits of code into various other
functions for compiling (or evaling).

The notion I am working towards is actually more of a literate-project.
Something that can do all the grunt tasks, such as linting and testing code,
importing data, etc. and weave together bits from multiple literate documents.

~~~
taeric
Do check out org-mode, then. It is surprising how far that tool has been
extended.

Many of these features are actually under the label "reproducible research"
nowdays. There was a really good talk given on this at a pypy convention a
couple of years ago.[1]

[1]
[https://www.youtube.com/watch?v=1-dUkyn_fZA](https://www.youtube.com/watch?v=1-dUkyn_fZA)

------
edmack
Nice! What do people use literate programming for? I've yet to, but I'm sure
I'm missing some cool benefits.

~~~
taeric
It really is a great way to introduce a narrative into the code. Too much of
reading programs feels like taking a look at either just a plain convoluted
mess, or an index that someone put together for another larger work. (This is
especially true for pieces where someone makes a lot of single use functions.
Sure, the pieces may be "self documenting," but it is almost akin to just
seeing a bag of screws. You know what they do individually, but you don't know
why they are there.)

If you have the time, take a look at any of Knuth's programs[1]. Obviously,
they are not all immediately approachable, but render them to pdf and give
them a try. You'll hopefully be surprised just how much you do pick up.

The only downside, to me, is that I become less concerned with modern trends
of long variable names and abstractions that "self document." These are still
great, of course, but they don't go nearly as far as knowing the narrative of
why a piece of code was written.

[1] [http://www-cs-faculty.stanford.edu/~uno/programs.html](http://www-cs-
faculty.stanford.edu/~uno/programs.html)

------
tunesmith
Wouldn't literate programming encourage a very linear style of programming
with long methods? It seems like it would encourage the opposite of single-
responsibility-principle, since it's really hard to read a narrative that
jumps all over the place.

~~~
taeric
I think the catch is the the majority of programs are more linear than the
index of methods would give credit to. Sure, some folks are making utility
data structures that don't really have much of a narrative to them, I suppose.

I would think for most programs there is a very direct sense of linearity in
getting a task done. Sure, you will often have a lot of support methods. But
seeing them just directly listed can be as much of a distraction as anything
else.

Note, also, for nontrivial applications, you will likely have a collection of
literate programs. Not just one giant literate program.

------
mck-
Literate Coffeescript comes out of the box with Coffeescript. I kind of like
it, especially for complicated algorithms where the ammount of comments exceed
the lines of code. Here's an example [1] of a kd-tree implementation in
literate coffee. It reads beautifully on Github as well, since it's Markdown!

[1] [https://github.com/axiomzen/Look-
Alike/blob/master/coffee/kd...](https://github.com/axiomzen/Look-
Alike/blob/master/coffee/kdtree.coffee.md)

~~~
cdosborn
Literate coffescript falls under one one of the pseudo literate tools, i.e.
just nice comments. Not to discredit that approach, its just a compromise for
most of the benefit. Many of those tools avoid the extra step of having to
expand macros and such, before compilation/being interpreted. I ended up
borrowing the watch feature, which aids the workflow

------
thyrsus
Erlang uses <<stuff>> to express binary strings. In other literate programming
environments, e.g., org mode and noweb, I have to break up the string like

    
    
        <<"word"
        >>
    

...which is ugly. I've tried other workarounds like defining a macro

    
    
        -define (bs(X), <<X
        >>
    

and then use

    
    
        bs("word")
    

...which is still ugly. Any better suggestions?

~~~
cdosborn
My program has only the two constructs `<<>>` and `<<>>=` which you could
rather easily alter. src/Processing.he and src/Parse.hs

------
SomeCallMeTim
My gut reaction is that I don't want to be creating a table of contents list
of sections by hand.

I feel it's a design error to require the sections be listed at the top _and_
named below. If I want to change a name below, then I have to remember to make
the identical change above.

Unless there's something obvious I'm missing about how literate programming
works, it seems like a "generate table of contents" macro would be vastly
superior to having to maintain it by hand.

~~~
daveloyall
NB: I've only read the README.md.

My first thought was to use emacs org-mode to manage these .lit files.

My second thought was to consider: do I want name.lang.org.lit or
name.lang.lit.org? ... I am leaning towards the latter.

~~~
macmac
If you are using org-mode you should consider org-babel which does all you
need for litrate programming and more.

~~~
daveloyall
I've used org-babel, but I'm still interested in Lit. The 'watch directory'
feature interests me, as does the 'proper and complete' aura of Haskell.

I'd previously played around with getting org to do stuff "on commit", via git
hooks, but I didn't make much progress (and given the environment I was
committing from, I was sure that I was going about it backwards anyway).

Org still doesn't have a universal character escape mechanism!

------
cdosborn
Here is an updated example, because a few people were asking for a more
complex example. This is the literate version of the root of the program.
*Note that the markdown doesn't include links (in the macro names) like the
html, it's just the most convenient to share.

[https://github.com/cdosborn/lit/blob/master/examples/lit.hs....](https://github.com/cdosborn/lit/blob/master/examples/lit.hs.md)

------
gohrt
What does Lit offer beyond the standard tools?

[http://www.haskell.org/haskellwiki/Literate_programming](http://www.haskell.org/haskellwiki/Literate_programming)
[http://www.haskell.org/haskellwiki/Haddock](http://www.haskell.org/haskellwiki/Haddock)

In the README, the example shows a "literate" comment above a Haskell comment
-- what's the point of having two kinds of comment?

~~~
cdosborn
I really need a more mature example to show, which will be coming soon.
However, many tools exist which are just eesentially auto documentation (java
docs, literate Haskell, docco). Mine is the simplest iteration of a true
literate programming tool which allows for code to be restructured in the way
best fit for explanation. If you look at cweb, funnel-web, noweb, mine
attempts to be a modern version that still implements macro functionality in a
modern context (markdown, HTML, easy to build). Literate programming is more
than just comments.

------
jostylr
Great to see more tools like this out there. I wrote
[https://github.com/jostylr/literate-
programming](https://github.com/jostylr/literate-programming) which is
similarly based on markdown. I use _"section name" to insert code from the
named section and use markdown headings to delineate the sections.

------
lifeisstillgood
I'm interested - but literate programming like this, as far as I understand
from README is not a lot more than having good docstrings in each function.

So maybe I am missing something but sphinx / python seems as good a
replacement as can be - although the easy mixing of real code and docs is nice

------
sytelus
Let's say you broke down your code in good small functions with comments on
its purpose, params etc. You also commented blocks of the code with good
comments.

I know people often say LP is not just "good commenting". But, really, is
above so different than LP?

~~~
cdosborn
This is my opinion
[https://news.ycombinator.com/item?id=8185142](https://news.ycombinator.com/item?id=8185142)

In some cases LP won't provide a benefit over commenting. But the idea is that
you step outside of the rules of the language (or more appropriately the
interpreter/compiler) and write a story where the order is defined by the ease
of explanation, not the order mandated by the machine.

~~~
eru
Though in Haskell's case, the compiler leaves enough freedom in most cases
anyway.

------
elwell
Would like to see some more complex examples. I like the philosophy behind it
though.

~~~
cdosborn
I need to read more of Knuth on literate programming. The plan is to finish
the source in literate style.

~~~
jostylr
You definitely should. It is fantastic stuff. His primes example program is
good to do.

You should also consider writing up some easy programming example such as
FizzBuzz.

Bootstrapping your code into your own literate programming environment is a
must to sit at the lit table :)

------
CogDisco
Not bad. Not sure it'd come close to replacing org-habits for me.

------
lucio
Litescript is literate from the start.
[https://github.com/luciotato/LiteScript](https://github.com/luciotato/LiteScript)

------
palhimanshu
I like it but if you can give some more detailed examples on complex issues,
it would be much better. :)

~~~
cdosborn
If you follow the project, I'm currently writing up the src in literate style
:).

