
Literate Programming: Empower Your Writing with Emacs Org-Mode - jworthe
https://www.offerzen.com/blog/literate-programming-empower-your-writing-with-emacs-org-mode
======
antt
>On the writing side, the main issue is that literate programming tends to tie
your writing into the tools that support your literate programming. This can
make collaboration on a document difficult if the people you’re collaborating
with are not as sold on the tools as you are. The moment you need to work with
a business person who prefers to use Google Docs to share a document, or a
university department that insists on receiving drafts as Microsoft Word
documents, you start to face the pain of exporting to those proprietary
formats. Like many issues in software development, this is really a social
issue: for literate programming to work, all of the writers need to agree on
the tools being used.

I've used org-mode at work and this was actually the strongest selling point.
Using pandoc and massaging the org-mode reader to map better onto the pandoc
markdown ast I could export to pdf, html and docx (!!!). I had production code
+ prose sent to the CTO as a word attachment in an email and she loved it.

The tangle comments also meant I have a workflow to share the code as regular
code with the rest of the team and use git to see which code sections they
changed. There wasn't enough traffic to justify writing an un-tangler that put
their code back into the original document but it seems like a very
interesting problem.

~~~
swiley
It’s strange how few people use pandoc for word processing. Even if your goal
is just to produce a docx file in the end it’s so much more pleasant to write
it in markdown in a distraction free editor you’re comfortable with than to
deal with word or writer.

~~~
chrissoundz
Along the same lines, I've used Pandoc to convert a org-mode file into a
revealjs presentation.

~~~
lionyo
I've been using revealjs too! I've been using this:
[https://github.com/yjwen/org-reveal](https://github.com/yjwen/org-reveal)

------
jpfr
Jupyter notebooks seem to be the modern form for literate programming.

People are writing entire books in the literate style:

[http://nbviewer.jupyter.org/github/rlabbe/Kalman-and-
Bayesia...](http://nbviewer.jupyter.org/github/rlabbe/Kalman-and-Bayesian-
Filters-in-Python/blob/master/table_of_contents.ipynb)

The style is similar to literate classics like "Structure and Interpretation
of Computer Programs" and "Structure and Interpretation of Classical
Mechanics".

The only downside is version control. Jupyter uses a custom file format. It is
difficult to work with colleagues on the same notebook, since we cannot use
the usual code-versioning tools (git/diff/etc.) to manually merge together
concurrent changes.

Anybody knows a good solution for this?

~~~
sasvari
You can use a filter in your gitconfig to strip away the outputs, and thus
make it play nicer with git [0].

gitconfig:

    
    
      [filter "nbstrip_full"]
          clean = "jq --indent 1 \
                  '(.cells[] | select(has(\"outputs\")) | .outputs) = []  \
                  | (.cells[] | select(has(\"execution_count\")) | .execution_count) = null  \
                  | .metadata = {\"language_info\": {\"name\": \"python\", \"pygments_lexer\": \"ipython3\"}} \
                  | .cells[].metadata = {} \
                  '"
          smudge = cat
          required = true
    
    

gitattributes:

    
    
      *.ipynb filter=nbstrip_full
    
    

[0] [http://timstaley.co.uk/posts/making-git-and-jupyter-
notebook...](http://timstaley.co.uk/posts/making-git-and-jupyter-notebooks-
play-nice/)

------
startupdiscuss
The issue with all the ideas to make programs easier to understand is that
they require time and work.

If I had more time and energy there are plenty of things I could do: refactor,
make comments, change variable names or try literate programming.

But given the limited time and energy which is best?

Put another way, how can you beat thoughtful variable or function names and
judicious comments?

~~~
ljm
Literate programming might be my least favourite approach when writing normal
code, because you're going to read the prose only once or twice and then find
that it gets in the way of the implementation you're concerned about.
Maintaining something done with literate programming must be an absolute
nightmare if you've got any kind of refactoring, because you basically have to
rewrite at least a few paragraphs so the prose continues to make sense. The
code is more important than the prose.

But when it comes to writing a tutorial that you can _also_ execute to get the
final result, or interact with along the way, then there is nothing better
because, really, you're more interested in the prose that lends explanation to
the code you're showcasing.

Although this article's about emacs and I've taken the literate approach to my
own config too. That's less for the benefit of adding paragraphs of
explanation to my config, but because org mode offers a fantastic way to
organise code in one file (which is more often than not what an emacs config
tends to be).

In either case I don't think you can do better than being thoughtful when
writing code, thinking about the bigger picture and not just the individual
functions and variables in isolation. For others and also yourself. This is
also true for emacs itself, considering the idiosyncrasies it presents when
most of us learn elisp by copying someone else, rather than reading the
language docs.

~~~
Retra
The main problem I have with literate programming for tutorials is that it
forces your writing order to match the code order. You get stuff stuff like
this:

"Ignore these for now" ... import statements and setup

"Now lets talk about X" ... some code

"Remember those import statements? Lets talk about them, but scroll up to see
the code because I can't repeat it." ...

"Oh yeah, and we could have done this on X, but again, can't show any code."
...

~~~
bennofs
With the noweb syntax, I think you can just write:

<<imports>>

code you want to show

and then later define what `imports` should be.

~~~
andreareina
Indeed. You can also define several `imports` blocks, and the contents of each
block will be concatenated during expansion. So your literate source can have
the imports located near where they're used, but in the tangled code they'll
be up top where most people expect them to be.

------
heinrichhartman
Another selling point for me is the relative ease of writing custom exporters
from org to other formats [1,2].

My latest use cases are: \- Extracting a list of issue descriptions (for Jira)
from a Getting Started guide, that included "todo" comments. \- Extracting a
list of contacts from notes taken at during a conference. \- Drafting Jekyll
blog posts in org-mode.

For todo and contact information, org properties [3] are used to specify
semantic fields, that can in turn used by the exporter.

To be fair, the work on these exporters in not completed. I am still getting
my lisp up to speed. But from what I can see this way easier than writing
pandoc backends or jupyter notebook exporters.

[1] Exporter Documentation: [https://orgmode.org/worg/dev/org-export-
reference.html](https://orgmode.org/worg/dev/org-export-reference.html) [2]
Example Exporter (.md): [http://repo.or.cz/org-mode.git/blob/HEAD:/lisp/ox-
md.el](http://repo.or.cz/org-mode.git/blob/HEAD:/lisp/ox-md.el) [3]
[https://orgmode.org/manual/Property-syntax.html#Property-
syn...](https://orgmode.org/manual/Property-syntax.html#Property-syntax)

~~~
agumonkey
Reminds me of college prof. doing requirements parsing to extract code out of
desires. I'd love to write in commentscript.

------
IshKebab
Literate programming is fine for done things - e.g. the sort of things you
might put in a Jupyter notebook - maths, algorithms etc.

For large normal programs though? I don't think you need that much prose. A
program with 100k lines of code would become enormous.

Still, I do wish programming languages had better support for rich comments -
why do no IDEs render comment blocks as markdown? Why can't I put diagrams in
comments? I have to resort to shitty ASCII art like a commoner.

~~~
taeric
Oddly, large programs that won't change heavily benefit from the literate
style. Almost as a forcing function to keep you from changing them. Not
surprising, given Knuth's liking of stability.

~~~
jf
> large programs that won't change heavily benefit from the literate style.

This has been my experience. Returning to code I wrote as a literate document
is a joy.

------
jf
This is a fantastic article. I've been wanting to write an article like this
and Justin Worthe did a better job at describing org-mode and Literate
Programming than I could have.

For my part, I've written at least 3 "non trivial" literate programs using
org-babel and describe them here:
[https://gist.github.com/jpf/d71453f535065a0d9281672152541386](https://gist.github.com/jpf/d71453f535065a0d9281672152541386)

I should mention that one of my "dirty secrets" of writing literate programs
is that I always start with an "illiterate" program, tweaking, changing and
updating as I go along. Then, once it's something I'm ready to "chisel in
stone" I start converting the code into a literate document.

I do this by checking all of my code into Git, then re-create the code inside
of org-mode. Every time I "tangle" from org-babel, I do a "git diff" and make
sure that I haven't changed the code by documenting it.

I do this because, while it's easy to make changes to a literate program, but
it's harder to do a major refactor.

------
smallnamespace
The only thing holding me back from experimenting with org-mode is its tight
coupling with emacs.

It's a non-starter on a team where people aren't willing to switch over to it
(at least to view and interact with org files).

It's difficult for anyone who has a heavy reliance on IDE features; I've spent
literally probably close to a hundred hours trying to get emacs + IDE-like
integrations working for a variety of languages, with only mixed success.

Yes, I understand that going with emacs is a 'different way' of doing things,
and perhaps you won't need your IDE if you just do things the emacs way, but
the barrier to entry is very high, between building in new muscle memory for
shortcuts, the highly non-standard UX compared to every other editor today,
and the hours and hours required to configure everything by hand, even with a
starter config like spacemacs.

~~~
stevesimmons
> The only thing holding me back from experimenting with org-mode is its tight
> coupling with emacs.

As a non-emacs user, my successful route to orgmode was with Spacemacs and
evil for Vim keybindings. After several false starts, what worked for me was:

1) realizing I didn't need to go full emacs: I stick with Vim for plain text
editing, Visual Studio for coding at work and PyCharm/Visual Studio Code for
coding at home.

2) consciously learning one new org-mode feature at a time, and adding their
shortcuts to my own cheat sheet.

------
segmondy
Literate programming fits in fine for very dense languages. I was writing
J(APL) a while ago and it was nice to employ literate programming using
Jupyter Notebook.

Most folks say J/APL is unreadable, but I find that I can always come back and
understand the code as fast as I can read the notebook.

------
stevekemp
I've used org-mode with inline code a few times now, and it is terribly
expressive and useful when you want to show reports with "live" results in
them, etc.

That said when it came to formatting my Emacs init file, with descriptions,
and justifications, I chose to use markdown:

[https://github.com/skx/dotfiles/blob/master/.emacs.d/init.md](https://github.com/skx/dotfiles/blob/master/.emacs.d/init.md)

Markdown allows text, and code, to be mixed, and while there is no inline
expansion support or other neat features it is minimal enough that it was
painless to write and process.

~~~
celeritascelery
Curious why you choose markdown over org, especially for an emacs init file?

~~~
stevekemp
I spend a lot of my time writing markdown, much more than I do interacting
with org-mode, so it felt more natural.

Org-mode I tend to only really use for dynamic things. (i.e. "Reports" or
tutorials which contain ebedded code rendered output.) Markdown seems like it
is easier to demonstrate to other people - and doesn't really require
introducing orgmode (which is worth learning, and which does have excellent
documentation of its own).

------
auvrw
first, i like emacs alright, and the article is awesome and full of good
recommendations..

.. _however_ , this article appears to be advocating a style of programming
that i'd be more apt to call Test Driven Development than Literate
Programming:

it demonstrates a way to use naming conventions and mocks/stubs to describe a
program, _not_ a way to use natural language, math notation, etc. to describe
a program.

for JS (the lang in the article), proxyquire+simple-mock is a non-emacs-
centric way to do this. toss in tape or some other testing library, and you've
got some amount of natural language documentation as well.

this is more what i'd consider literate programming in JS land

[https://marionettejs.com/annotated-
src/backbone.marionette.h...](https://marionettejs.com/annotated-
src/backbone.marionette.html)

... by way of disclaimer, i ought to say that, for all i know, i'm grossly
misinformed as to the meaning of TDD as compared to Literate Programming

------
gkya
The best thing with Org mode literate programming is that you can easily mix
and match code in multiple languages and make these communicate, and then
render the result as part of the document w/o any hassles.

------
JohnL4
It occurs to me that the emacs-averse can still use emacs as a backend
(emacsclient/(server-start)) so they can edit org-mode text in their favorite
editor and then process it with emacs magic.

Also, I bet this trick would work (maybe) for formatting comments or
introducing ASCII wireframes with tools mentioned at
[https://news.ycombinator.com/item?id=2651745](https://news.ycombinator.com/item?id=2651745).
(For the ASCII art, maybe an extra hop through emacs isn't necessary.)

------
your-nanny
Literate programming sounds nice... but then I read all those people outlining
the horrors of too many comments. I have trouble reconciling these approaches.

~~~
taeric
The comments of a literate program are not just comments scattered throughout
the program. Not are they just an index of what is in a program. Rather, they
build a narrative. It is literally explaining the program. In full.

~~~
jf
> It is literally explaining the program. In full.

Precisely. Here are real world examples of this:

A program that implements on-the-fly encoding of images into SSTV audio files
(3620 words): [https://github.com/jpf/dial-a-cat#1-855-meow-jam-sending-
cat...](https://github.com/jpf/dial-a-cat#1-855-meow-jam-sending-cat-pictures-
over-the-phone-via-space-age-technology)

An example SCIM server (7966 words): [https://github.com/joelfranusic-
okta/okta-scim-beta#welcome-...](https://github.com/joelfranusic-okta/okta-
scim-beta#welcome-to-the-okta-scim-beta)

An example OIDC "RP" implementation (7731 words):
[https://github.com/joelfranusic-okta/okta-scim-
beta#welcome-...](https://github.com/joelfranusic-okta/okta-scim-beta#welcome-
to-the-okta-scim-beta)

~~~
your-nanny
I'm sorry, but I think I don't get it. all those examples consist of regular
old sparsely commented code files with really thouroughly written READMEs,
written in org. how is that special or any different than Readme driven
development?

~~~
jf
The README and the code always stay in sync. An update in once place changes
both the code and the documentation.

------
TheCapeGreek
OfferZen? On MY HackerNews? It's more likely than you think!

I think they've made quite the impact on the South African dev market to get
to the front page here.

~~~
pixelpoet
There are plenty of saffers doing good work, just the vast majority of them
get out because the country is, well, you know...

Source: Saffer who got out, knows others from UCT etc.

------
mark_l_watson
I started using org mode at work and home a few months ago. Love it.

I want to try both Haskell’s built in literate programming support and try the
ideas in this article.

~~~
jmillikin
> _I want to try both Haskell’s built in literate programming support_

Haskell doesn't have literate programming support. The "lhs" format merely
changes the syntax of block comments, it provides no additional power over {-
text here -}.

------
pilmihilmipilmi
Instead of embedding code inside org-mode it should be the opposite: embed
org-mode inside comment sections. However this is not easily possible because
of ‘*’ gobbling up beyond the comment section. (At least for c)

~~~
kqr
I'm not sure that makes as much sense. At that point, why not just use regular
comments? Then add outshine-mode if you want a tree of headlines.

------
intrasight
A picture tell a thousand words - so unfortunately as much as I like emacs and
markdown, I end up using Visio.

------
avodonosov
Examples of large enough (real world) literate programs?

