
The Tools I Use to Write Books - misternugget
https://thorstenball.com/blog/2018/09/04/the-tools-i-use-to-write-books/
======
akavel
The most important quote here for me:

 _" What kept me from [writing my own tool] was a calm voice in my head
telling me that I’m here to write a book, not a preprocessor. [...] Now I have
written two books and zero tools, which I consider a success."_

~~~
scrollaway
It's a crazy important quote and everyone should really keep it in mind when
they set out to do something.

Many people have problems focusing on the task they set out to do. It's a type
of procrastination. Insert that famous gif from Malcolm in the Middle.

This is relevant to startups as well (and business in general): Being able to
decide whether something is in scope of a project or not. You are encountering
a roadblock, how do you react? Do you set out to solve it perfectly or do you
work around it (or even, do you give up)?

The quote you pasted is the perfect way to put it into words. Another common
theme I hear about is people asking: "How do I become a successful [profession
here]?" (eg. "streamer", "youtuber", "musician", "programmer", "writer", ...).
The answer is always the same: "Start".

~~~
enriquto
> It's a crazy important quote and everyone should really keep it in mind when
> they set out to do something.

I don't agree at all. It seems to me that the most interesting advances of
mankind have happened precisely when people were trying to do something else,
and had to invent new tools. The tools turn out to be more important than the
final result because they can be used by other people.

~~~
misternugget
> when people were trying to do something else, and had to invent new tools.

(Author here.) My point was not that you shouldn't invent tools if you need
them. By all means, go ahead. My heart beats for creating tools (especially
tools to create other tools...).

What I wanted to express was this: if you want to write _a book_, first write
the book and only then worry about the tools. If you then notice that you need
a custom toolchain to get the book in exactly the shape you need it to be, do
it.

I've had people tell me that they would love to write a book, but, well, the
existing tools are not sufficient enough. And I think that's the wrong
approach to writing a book: write something and you'll quickly realize that
the tooling is not your biggest hurdle.

~~~
bocklund
> I've had people tell me that they would love to write a book, but, well, the
> existing tools are not sufficient enough.

That’s hilarious. Humans were hand writing books on paper and giving them to
other humans to hand write copies. A good craftsman never blames his tools.

~~~
dragonwriter
> Humans were hand writing books on paper and giving them to other humans to
> hand write copies.

Not very many were, and a big reason for that was the available tooling.

> A good craftsman never blames his tools

Someone who writes a book by blames the available tooling support for the low
quality of its content is a craftsman (but, per the saying, not a good one)
blaming his tools.

A person who chooses not to write a book at all because the anticipated
opportunity cost given the existing tooling support and that person’s personal
utility function is not.

------
grecy
I settled on something similar for my needs, and I'm really happy with it.

1\. Book is entirely written in LaTeX, every chapter in it's own file, master
doc with everything combined. I really like having different gutters for left
and right pages, ultimate control over formatting, fancy things like Lettrine,
header layout and design etc. When I want to do something across the entire
doc I write a new command and can use it anywhere.

I Create perfectly formatted print-ready pdf to go to printers directly from
this.

2\. Run LaTeX file through pandoc to create an ePub.

A little known trick that is now supported by pandoc is conditionals in the
LaTeX doc, so when formatting gets hairy I declare commands like this in the
master LaTeX document:

    
    
      \iftoggle{ebook}{
        \newcommand{\textbreak}{\newline\hrule\newline}
      }{
        \newcommand{\textbreak}{\begin{center}\LARGE{$\Psi\quad\Psi\quad\Psi$}\end{center}}
      }
    

So I can use \textbreak{} anywhere I want, and I know it will be formatted
perfectly in both output formats.

3\. Un-pack the ePub and tune some of the html/css to my personal taste, then
pack it back together. I run it through kindlegen to make sure it's valid, and
I have an epub/mobi ready to upload to stores.

I like this workflow so much I'm using it again for my next book

~~~
sehugg
I've never been able to get a good LaTeX -> epub pipeline working. There are
too many issues with things like tables and splitting code listings across
pages -- epubs are just optimized for fiction. I find it's better just to ship
a PDF.

~~~
grecy
I agree epub are painful when you have complicated layouts like tables and
code listings.. it's just how the format is.

My book was a non-fiction novel, so that made life much easier, I only had
some tricky image layouts to deal with.

------
davedx
I'm writing a sci-fi book at the moment. Somehow I've come much further than
any of my other book writing attempts, including one that I actually
(self)-published.

I use Sublime Text, sometimes in distraction free mode, but usually not
because I often want to Google sciencey things.

I have a single file called "story.txt" that has all the important notes - the
names, ages and visual descriptions of the main characters, a detailed
(historical) timeline, and chapter outlines I wrote before I started when I
knew roughly what the overall plot was going to be. The chapter numbering has
changed drastically but the overall outline has stayed pretty true to the
original idea.

I really like this system. Keep it simple and focus on writing!

------
nickjj
If anyone is curious this open source tool will convert markdown (with support
for images) into pdf, epub, mobi and html:

[https://github.com/softcover/softcover](https://github.com/softcover/softcover)

It was written by the same guy who created the original Rails tutorial. You'll
be able to generate books that look and feel like this
[https://www.railstutorial.org/book](https://www.railstutorial.org/book).

I've released a couple of PDFs using it, and it's pretty decent to work with.
There's room for improvement but I haven't come across anything better.

~~~
mhartl
Softcover was used for all the Learn Enough books as well. In fact, _Learn
Enough Ruby_ just came out today:
[https://news.ycombinator.com/item?id=17919998](https://news.ycombinator.com/item?id=17919998)

~~~
nickjj
Are you going to update softcover to include that theme by default?

------
miles_matthias
I've written a couple of technical books, and then I've helped my wife grow
her business as a USA Today best selling romance author.

The biggest challenge here, to me, isn't writing the book or figuring out what
tools to use, but it's marketing. Email lists, FB ads, etc. are really what
drive the success or failure of your book. Nathan Barry's writing on this in
Authority is my bible for that.

Also, if we're talking tools, shout out to Leanpub. Love that platform,
especially for developer books!

EDIT: Nathan Barry doesn't specifically talk about FB Ads, so in terms of a
marketing bible, I guess courses and private FB groups have helped us the
most. This guy is actually the main goto for fiction authors:
[https://selfpublishingformula.com/](https://selfpublishingformula.com/)

~~~
_Nocturnal
> The biggest challenge here, to me, isn't writing the book or figuring out
> what tools to use, but it's marketing. Email lists, FB ads, etc. are really
> what drive the success or failure of your book.

I've looked into creating a mailing list with MailChimp and/or its
competitors, since I think it'd work great for my target niche. The thing that
stopped me is that you need to provide your real name and location. Though I
understand the reasoning, I'm publishing under a pseudonymous identity and do
not wish to tie my real identity to it. Bummer.

~~~
kukabynd
Have you checked out the self-hosted [https://sendy.co](https://sendy.co)?

------
Jeremysr
For project-based tutorials/books, which are centred around building a program
step-by-step by following a series of human-readable diffs, I think a tool is
very much needed.

For
[https://viewsourcecode.org/snaptoken/kilo/02.enteringRawMode...](https://viewsourcecode.org/snaptoken/kilo/02.enteringRawMode.html)
, the code is kept in a git repo where each commit is a step in the tutorial.
This way steps can be modified, reordered, or split into more steps using git
interactive rebase (git rebase -i).

I'm working on a tool which abstracts away git rebase, and gives you a nicer
interface for manipulating the series of steps in your tutorial. For example,
you can run `leg 3` to checkout step 3, make edits to it, and then run `leg
amend` to apply the changes, resolving conflicts as the changes make their way
through the rest of the tutorial's steps.

The tool is still in very early development, but here's a quick tutorial to
get a better idea of how it works:
[https://github.com/snaptoken/leg/blob/master/TUTORIAL.md](https://github.com/snaptoken/leg/blob/master/TUTORIAL.md)

Also, I'm playing around with a "literate diff" file format that might be of
interest: [https://github.com/snaptoken/tgc-
tutorial/blob/master/doc/02...](https://github.com/snaptoken/tgc-
tutorial/blob/master/doc/02.oneFellSweep.litdiff)

~~~
misternugget
> I'm working on a tool which abstracts away git rebase

This is 99% of what I need! In the best case it's just files/folders I can
edit manually or with the help of a CLI.

> The tool is still in very early development, but here's a quick tutorial to
> get a better idea of how it works:
> [https://github.com/snaptoken/leg/blob/master/TUTORIAL.md](https://github.com/snaptoken/leg/blob/master/TUTORIAL.md)

Now, if those diffs are rendered nicely and the splitting up of a step into
multiples (or: explain separate parts of one step) works, I think that really
is 99% of what I need. The 1% being the grouping of steps into a chapter and
some other small things probably

------
leephillips
From the article:

"But, most importantly, the code is also duplicated: one version lives in a
Markdown file and one (or more) lives in the code folder that comes with the
book. If I want to update a snippet of code presented in the book, I have to
manually update every copy of it. Yes, cumbersome."

I had a similar problem in writing a book about gnuplot: keeping code samples,
code in the text, and the generated plots synchronized. I decided doing it
manually was too tedious and error-prone, so I wrote some Pandoc filters to do
it for me: [https://lee-phillips.org/panflute-gnuplot/](https://lee-
phillips.org/panflute-gnuplot/)

~~~
xrd
Nicely done.

------
funkaster
I'm pretty sure that orgmode + Babel would get him a lot closer to what he
wants: you could have chunks of code that edit using the natural editor mode
and can join them together in an external file (tangle) in any way you want.
You can even post-process right from the editor when tangling.

Orgmode Babel: [https://orgmode.org/worg/org-
contrib/babel/](https://orgmode.org/worg/org-contrib/babel/)

~~~
jeromenerf
In my experience, Org mode and babel shine at

\- "literate programming" where the output is actually commented code and the
eventual documentation

\- "reproducible science" articles, where small independent snippets are
executed, even piped into each other, to produce "results" in the form of
tables, charts, ...

The author seems to need to highlight differences, which may not fit without
hacking its own features, which he appeared to be reluctant to.

------
bovermyer
I use pandoc-flavored Markdown (it's a specific superset), sprinkled
occasionally with LaTeX to do things like force page breaks in specific
places.

I should write a blog post like this. My use case is different, since I write
role-playing game books.

------
logfromblammo
I use a plaintext editor to write xhtml+css, and 7Zip to go directly to epub.

I made a blank epub template with 50 empty chapters in it, so when starting a
new book, I copy that and write new IDs and strings to the metadata, and if I
finish it, I delete the unused chapters and remove their entries from the
metadata files. It's easier to delete empties than add new chapters, and doing
the metadata by hand ensures there's no auto-generated cruft in my file.

For illustrations I use Paint or GIMP or Inkscape, but the template doesn't
have more than just a cover image in it.

After testing various ebook readers, there's no way I would recommend
xhtml+css zipped to epub for anything technical. The only reader that actually
renders text they way I expect, right out of the box, is Edge. Epub is for
novels, not textbooks, papers, or manuals.

------
ryneandal
I use Scrivener on Mac for putting together various thoughts for game design
docs. Adding references for the various things I've come across to assist with
any hobby game dev is awesome, particularly when attempting historical
accuracy.

------
dorfsmay
I'm surprised not to see any mention of AsciiDoc, I know that it is what some
publishers use.

[https://en.wikipedia.org/wiki/AsciiDoc](https://en.wikipedia.org/wiki/AsciiDoc)

~~~
xrd
Yes, O'Reilly. I have another comment in the thread about using asciidoc with
a solution they call snippets that more or less solves the OP problem.

------
xrd
I wrote a book for O'Reilly and they have something called O'Reilly Snippets
which allows you to embed snippets from another file inside your asciidoc
files. It sort-of solves the problem noted in the OP.

The problem I found with it was that I would add the snippet, then later in
the chapter, I would refactor the code to better fit the story, and that
worked, as long as you only needed one copy of the file. But, it didn't work
if you introduced a basic version of the code and then later elaborated and
expanded that code and wanted it to reside in the same file. Additionally, you
have to keep the code in the same repository, which I didn't want to do since
I published companion repositories (which couldn't contain the sources for the
book, obviously) and I didn't want to maintain and synchronize two copies of a
file.

So I wrote a little preprocessor that ingested a subtle variation on the
snippets O'Reilly pioneered. Normal snippets work. But also, you could specify
a file and a SHA hash, and if it saw that, it would pull the file at the
commit (just using git show). Then I could update the file inside the
repository as the code progressed, and still have all the sample code. And my
repository always had working code, no matter what branch or point in time I
had checked out.

It's here: [https://github.com/xrd/oreilly-
snippets/blob/master/README.m...](https://github.com/xrd/oreilly-
snippets/blob/master/README.md#special-additions-to-snippets)

------
_Nocturnal
I wrote a free e-book tutorial for the Kitura web framework called Kitura
Until Dawn [0] using an almost identical workflow: per-chapter Markdown files,
pandoc, and kindlegen. Pandoc is an amazing tool. In addition to tweaking some
Latex stuff to get the output I wanted, I also use a simple Lua script to
rewrite internal links so that they don't break. I export to several different
downloadable formats, and host an HTML version via GitBook[1] - though I may
self-host at some point.

I have the entire book in a GitHub repository [2], including the export script
[3], which may be of interest to those wishing to write a book a similar way.

(Pardon the new account, but my "normal" HN account is more closely tied to my
IRL identity than my Nocturnal identity.)

[0]: [https://learnkitura.com](https://learnkitura.com)

[1]: [https://nocturnal.gitbook.io/kitura-until-
dawn/](https://nocturnal.gitbook.io/kitura-until-dawn/)

[2]:
[https://github.com/NocturnalSolutions/KituraBook](https://github.com/NocturnalSolutions/KituraBook)

[3]:
[https://github.com/NocturnalSolutions/KituraBook/blob/master...](https://github.com/NocturnalSolutions/KituraBook/blob/master/export.sh)

------
runemadsen
There's also this!
[https://github.com/magicbookproject/magicbook](https://github.com/magicbookproject/magicbook)

------
Koshkin
I find TeXmacs to be one of the few most amazing and useful pieces of software
in existence.

------
delinka
I've recently been researching this for my own needs, and this author's md ->
pandoc -> etc chain is pretty much where I settled. Reading this article, I
was hoping to find a way to point code blocks at a git repo (local, remote,
w/e) so that as code changes, the book follows (I do mean during the
formatting phase - not that an ebook would follow a remote repo to get code.)
I did not find that solution in this article.

Has anyone come up with a method for doing what I want? Maybe the ```lang code
blocks in md could take parameters like repo, file, line range; then an update
to pandoc would fix it all right up. Thoughts?

~~~
misternugget
You should take a look at embedmd, which allows you to include files (or just
portions of a file) in a Markdown file:
[https://github.com/campoy/embedmd](https://github.com/campoy/embedmd)

As I wrote in the post, I didn't use it because it doesn't allow me to include
a portion of something and then comment out a part of that, but today I got
thinking: if it could include a portion of a function and also the relevant
(semantic?) context, it would be 99% of what I need. But the "relevant
context" is the tricky part, I guess. I like what git diff does, when it shows
the signature of a changed function (does it simply use the first of the
previous lines that are less intended?), but I guess it's hard to get really
right...

edit: GitChapter is also a really interesting project. If it didn't use a real
git repository, but something like folders that resemble single commits, I
think I'd use it.
[https://github.com/chrissound/GitChapter/](https://github.com/chrissound/GitChapter/)

------
aikah
Interesting, I tried to use [https://github.com/jostylr/litpro-
commonmark](https://github.com/jostylr/litpro-commonmark) to do literate
programming when writing a tutorial but was completely unable to go from their
specific markdown flavor to a simple markdown file void of all extra macros
and annotations that I could just transform to HTML. i didn't know about "pp"
which looks like what I need to use.

------
codazoda
I used Libre Office to write a few short eBooks. I found the "flow" of writing
in a word processor better for me and it avoided the thought of writing my own
tools for the job. I've published step-by-step instructions for the document
setup that worked best for me.

[https://www.amazon.com/dp/B00A4112ZM](https://www.amazon.com/dp/B00A4112ZM)

------
munificent
The fact that Thorsten manually synchronizes the snippets in the text and
external source files seems like crazy talk to me. But the fact that he's
cranked out two really good books on languages while I'm still slowly
lumbering through my one [1] is proof that his system works for him.

Personally, I'm too error-prone to pull that off. I change the code snippets
frequently after chapters are done (often in response to bug reports from
readers) and having to manually update the snippets in the text would drive me
bonkers.

Also, I want to rigorously, automatically validate that the code in the book
is correct and contains everything you need.

To do that, I wrote a Python script [2] that weaves the code and the Markdown
together. In the Markdown files, you use a marker like (here in the chapter
"Strings"):

    
    
        ^code memory-include-object (2 before, 2 after)
    

This means "insert the code snippet labeled "memory-include-object" here.
Include 2 lines of contextual code (which is shown grayed out) before, and 2
after. In the code, the marker looks like:

    
    
        //> Strings memory-include-object
        #include "object.h"
    
        //< Strings memory-include-object
    

The "//>" begins the snippet and the "//<" ends it.

The build script reads the "^code" markers and finds the snippet of code in
the sources that correspond to it, as well as any desired surrounding lines.
But it can also do a lot of other useful things with the snippet markers:

* If some lines of code are marked as being part of a chapter, but that chapter doesn't have a "^code" marker to include them, it reports an error. This way I can't accidentally drop code on the floor.

* It can generate a separate copy of the entire codebase that only includes the snippets that are in a given chapter (and the preceding ones). In other words it can output "what does all the code look like by the time the reader reaches the end of chapter ___" for every chapter.

I output the code for each chapter and then compile that and run all of the
tests. This way, I can ensure that by the end of each chapter, the program
supports what it's supposed to support by then. I've found countless bugs
doing this -- places where I inadvertently put code in a later chapter that is
actually needed earlier.

However, this build script is _not_ a generic "build a book" system. It's a
bespoke program that only supports this one specific book. That helps keep the
complexity down and makes it much easier to put little hacks in when I run
into weird edge cases my specific book has. I think that's a reasonable trade-
off to keep me producing a high-quality book without too much yak shaving.

One of the days, I'll write a full blog post going through this in more
detail.

Of course, none of this is a criticism of Thorsten's system. The simple most
important skill in writing a book is figuring how to get those pages done. It
doesn't matter if that means waking up at three in the morning and writing on
parchment with a red pencil. Whatever works for you, works.

[1]:
[http://www.craftinginterpreters.com/](http://www.craftinginterpreters.com/)

[2]:
[https://github.com/munificent/craftinginterpreters/tree/mast...](https://github.com/munificent/craftinginterpreters/tree/master/util)

------
amrrs
Recently came across this article where it shows how easy it's to write a book
using `bookdown` (an R package that uses R + markdown) and publishing on
Amazong [https://blog.datascienceheroes.com/how-to-self-publish-a-
boo...](https://blog.datascienceheroes.com/how-to-self-publish-a-book/)

------
pjmlp
Interesting that the language is called Monkey.

Garmin's own programming language for device apps is called Monkey C.

[https://developer.garmin.com/connect-iq/programmers-
guide/mo...](https://developer.garmin.com/connect-iq/programmers-
guide/monkey-c/)

~~~
misternugget
There's also Monkey X, a dialect of BASIC:
[https://en.wikipedia.org/wiki/Monkey_X](https://en.wikipedia.org/wiki/Monkey_X)

It was previously called Monkey but then changed to Monkey X, which allowed me
to use Monkey. Fun fact: I originally wanted to call it Tiger, because they're
my favorite animals, but not two days after I wrote that name down, I opened
Appel's Modern Compiler Implementation in ML and discovered that the language
implemented in the book is also called Tiger. Thus, a Tiger became a Monkey.

------
runningmike
I tried md, but go back to rst again. I tried many things, but all have some
disadvantages. Outline of tools tried and considered on my blog at
[https://nocomplexity.com/the-next-version/](https://nocomplexity.com/the-
next-version/)

------
gymshoes
> After nearly two years, I’m still a happy customer and the only two features
> I’d love to have are more payment methods and pricing per country, so I can
> set a lower price for readers in India, for example.

I think KDP allows setting different pricing for different countries.

------
netvarun
There was an Ask HN recently on what tools to use to write a textbook -
[https://news.ycombinator.com/item?id=17781569](https://news.ycombinator.com/item?id=17781569)

------
tmaly
I would love to see his Latex template for this as well as a few samples of
markdown.

I wonder how well this method would work for something like an illustrated
children’s book?

------
nwmcsween
Hmm why not asciidoc or basically any other doc markup language? Most provide
additional support for books.

~~~
phalangion
I love AsciiDoc and I use it wherever I can. I'd bet the reason for using
Markdown has more to do with mindshare than anything else. Plus, Pandoc is a
great tool for converting Markdown to anything else.

What I really like about AsciiDoc is that it's right near Markdown in terms of
simplicity, but with a bit more beef behind it to handle more complex things
like referencing figures. If you don't need them, you can stick with the basic
syntax, but the power is there when you want it.

------
jasode
_> , are written in GitHub Flavored Markdown (GFM) [...] code presented in the
books is contained in the Markdown files in the form of fenced code blocks._

I understand the appeal of markdown but I avoid it for writing coding
tutorials because it doesn't have an easy way to _annotate_ code with graphics
like arrows and floating callouts. Here are some visual examples of code
enhanced by arrows: [1][2][3]

Also, all types of text files besides programming syntax can also be explained
with graphical annotations such as configuration files or log output files.
E.g.[4]

Therefore, I use MS Word because it makes annotation easy. (LaTeX can also do
annotation but that has its own workflow issues such as the typical programmer
not knowing it which means the team can't easily update the documentation.)

IMO, annotation is important because if we performed eye-tracking and heatmap
tests with people reading code tutorials, we'd see that a significant amount
of time is wasted with the eyes constantly shifting back and forth between the
code and the subsequent paragraphs explaining that code that's located a few
inches down the page. I also think this is one reason why live coding
demonstrations can sometimes be more effective than reading a book because the
presenter is wiggling the mouse (or laser pointer) around the _exact_ spot of
the syntax while he's talking.

This can be somewhat mitigated with adding _inline code comments_ in the
example but often, it still doesn't point to the _exact_ point in the line
that needs to be studied. Code comments also don't easily illustrate (because
it has no spatial cues) how separate lines are related to each other.

One workaround of markdown's limitation would be to rasterize the code's text
into an image and then use Photoshop to annotate. However, with MS Word, that
cumbersome step isn't required.

[1]
[https://introcs.cs.princeton.edu/java/11cheatsheet/](https://introcs.cs.princeton.edu/java/11cheatsheet/)

[2] [https://www.oreilly.com/library/view/head-first-
python/97814...](https://www.oreilly.com/library/view/head-first-
python/9781491919521/figs/web/166fig02.png.jpg)

[3] [https://i.stack.imgur.com/D8JeF.png](https://i.stack.imgur.com/D8JeF.png)

[4] [https://csharpcorner-mindcrackerinc.netdna-
ssl.com/UploadFil...](https://csharpcorner-mindcrackerinc.netdna-
ssl.com/UploadFile/1492b1/restful-day-sharp6-request-logging-and-exception-
handingloggin/Images/dynamic%20log%20file%20name.jpg)

------
danso
How do people feel about Gumroad vs. Leanpub these days?

------
_emacsomancer_
Is he writing them in GitHub web interface?

------
purplezooey
screw writing an entire book in markdown. You'd have better luck writing it in
troff for chrissake.

------
mstolpm
Not much to see here despite some self promotion. It's mostly not about
writing but preparing for (self) publishing. In short: Markdown, pandoc and
kindlegen.

Missing is everything about the writing process, structuring content and
editing.

~~~
macmac
Did you not read the title?

------
6stringmerc
I take umbrage with calling these technical manuals / instructional texts
"books" due to the conflagration of the commonality of the term. Stephen King
writes books. There's no shame in calling a piece of text by its correct name.

~~~
travbrack
What about college textbooks? In your mind, is long form narrative writing a
requirement for something to be called a book?

