How I wrote and published my novel using only open source tools 391 points by ggambetta on Aug 18, 2016 | hide | past | web | favorite | 180 comments

 It seems odd to me that he would write and publish a novel using open source tool and then write and publish about that using a closed source platform. I wonder why he chose medium (I wonder why anyone chooses medium over their own site)
 Reading the post, it just seemed like an interesting and feasible challenge, as well as a learning opportunity — not an ideological purity test.
 Probably for simplicity of use, as always. Although it isn't that simpler than building your own blog, at least for a tech aficionado. Wordpress if you don't know web development, one of the hundreds of blog engines otherwise.I've used Jekyll on Github pages for quite a bit and it suited me perfectly. Being able to write a post in markdown and git push to deploy is all the simplicity I need.
 Simplicity of use, and distribution. In certain ways, Medium's social component makes it much easier for other people to find and read it vs. a personal blog (unless you already have a following.)
 Github is a closed platform too
 you don't need github pages to deploy jekyll. While source code of github pages itself is not available, since it supports a open standard, which can deployed to a webserver/CDN directly without major changes, I would consider it to be open.
 Just to follow this to its logical conclusion, the webserver may be running Apache or Nginx on Linux, but there are likely to be proprietary blobs in the drivers and the firmware is probably closed. Plus, much of the hosting company's networking hardware is probably running proprietary software.IANANE (I Am Not A Network Engineer) but I imagine with open firmware like Coreboot and networking distros like Cumulus and pfSense (BSD) and the whole Open Compute Project it might be possible to run a completely open-source hosting company. Well, no. Your UPS's and other auxiliary hardware are probably running closed OSes. Darn.
 Cumulus contains a large portion of closed-source code, including all the driver code provided by Broadcom, which is where much of the interesting stuff happens. (I assume their Mellanox products are the same, though I'm not familiar with them)I'm not aware of any efforts to write open source drivers for any high speed ethernet chips, though I haven't looked much. Intel's (formerly Fulcrum) switch chips have open data sheets (but not the reference code, simulators, etc) so it would at least be plausible to start there.
 someone is (or was?) trying: https://www.vikings.net/
 Perfect is the enemy of the good.
 Exactly. But that is not to say, that it is hassle free. I am using jekyll on my private page (till I eventually get the rewrite done). It is nice, to be able to write in .md and to then have two aliases in my bash_profile to produce a preview version on a subdomain and finally to deploy to live environment. But really it was a pain in the butt to get there. Also having images in my content makes things even harder.There were so many roadblocks, that I took on having a ton of fun - but nowadays I would love to just have an easy way to write text, without thinking on design, formatting, code embedding and so on...Sometimes I am scarred by myself when I find myself wishing I still had WP installed. shiver[Edit:]Also this way he might have some additional exposure of his novel by people stumbling upon him while on medium.
 It's using open standards. It's an open platform. Unless you consider every webhost to be a closed platform?
 I can recommend Ghost it's a light blogging cms
 I've recently switched to Ghost after using Medium and Wordpress for other things. Ghost is by far my favorite, it's really nice.
 I have a github pages blog, but I seem to find it psychologically easier to write directly into a browser window.
 I feel a similar question would be "I wonder why anyone chooses to write on Twitter over their own microblog." Medium's inherent social component makes it much easier to find an audience if you don't have one already.
 Medium is really easy to use and your articles look nice. Also, there's an audience already on Medium that you can tap. I guess if blogging isn't your thing and you just want to write and publish, Medium is "ok".I used Medium for a short while, for these reasons, before realizing that it's actually pretty bad. There are a few heavy hitters on the site and everyone else is busy pandering to one another with fake internet outrage and "listicle" style articles.
 I enjoy Medium, but I think of it nothing more (or less) than Twitter with more than 140 characters.That said, there has been some interesting journalism being done there. And it's interesting seeing the White House and Obama using it as a public platform to issue addresses and policy commentary.
 Wrote on this topic a bit at the beginning of the year.[1]I think for most people it really just comes down to a mix of Medium's beautiful design, a strong existing community there, and laziness. You'd be surprised how many times I've been told by friend's to cross-post my stuff to Medium for exposure, but so far I've done my best to hold off.
 >I wonder why he chose medium (I wonder why anyone chooses medium over their own site)http://blog.onyxbits.de/the-importance-of-running-your-own-w...Medium's business model is: you just started your own website and don't have enough content there to rank well with search engines/attract visitors? No problem! We got a shitton of content and are well indexed. Just write an article for us (add to the pile!) and we allow you to link back to your own project.Medium pretty much is on my ignore list. The only thing you find there are fluff pieces without substance that are meant to drive traffic elsewhere.And there really is not a lot of substance to telling the world that you can use Markdown to format a book (though I really wonder he didn't use LaTeX).
 Markdown is both faster to type and easier to read on source form. I actually wrote big parts of my dissertation in markdown and then transpiled to LaTex for integration and formatting.
 As far as I'm aware, none of iBooks, Google Play Books, Kindle Direct Publishing, nor CreateSpace are open source either for that matter.
 Nice writeup.I'm currently writing a (technical) book in markdown, and I know I am going to pay for it later, but I don't care because I'm able to write quickly without anything getting in the way. I'm able to use git to commit changes and a host of tools to preview the work.Also, using markdown now will force me to edit it again when putting a final version together for a real editor.For the skeptics: I don't see how using markdown is any worse than using a typewriter. Its an interim medium that allows for quick capture of structure and thought.
 If you use LeanPub, you can fully publish both ebook and paperback versions from the Markdown source. And for editing, if you have an editor who's willing to not use Word, you can use a tool like Authorea to track changes, and then incorporate those back into the markdown source.I really don't get the hate on Markdown; it's IMO the fastest way to get words from brain into text, more so than rST and other formats, because of its simplicity. And for slightly more involved formatting, with LeanPub, you can put down what you need, then fix formatting by previewing the published copy until it looks just right.
 I never understood this argument about rST. The syntax for the same MD features in rST is only /marginally/ different, certainly not harder. I use both, depending on the context (which forces me to use one or the other). Most of the time you don't even notice the difference, because both have a really good baseline.However MD falls short for writing technical documentation, which is why pretty much anyone has to dope the "standard" (GH flavored MD, and so on). The syntax for the extensions is not any better than what rST offers. rST generally allows a more human-readable writing style, whereas MD with extensions has a lot of very verbose inline formatting.In fact, rST CAN have a slight edge in my mind as a human readable document if you use the appropriate formatting, whereas MD with extensions is easier for a program to parse.But, I repeat myself, for the same set of MD features, rST is hardly different. If you don't believe this, use pandoc and try to transform a vanilla MD document into rST.Now pick any rST page from sphinx documentation, and move back to MD. Hardly any better. sphinx is often criticized for being "hard to write" or illegible, transferring the blame to rST, but for the same set of formatting MD is not better. sphinx allows a whole slew of extra features which require a lot of annotation, and MD doesn't save the day.What's really annoying if having to switch between the two. We have projects which have a good majority of text documents written in MD and then API references in rST. I hate that. There's constant mismatch.Just pick one :(
 Thanks for the tip! I had briefly seen LeanPub before when I was just starting out writing in the winter. I will give it a closer look, since I remember having a good impression.
 I think most of the skeptics are just people that need/expect to have "feature x" that they have a personal preference for.For others, simplicity is preferable.George R.R. Martin does all of his writing in WordStar 4 on a DOS machine, which is a good example of not needing the fanciest tools to be highly successful.
 Check our this guide on using ReStructured text with emacs. Great tutorial here.
 MD isn't much worse than a typewriter, but it's not a whole lot better either. There are all sorts of things one wants to do with technical documentation that either isn't possible or isn't very easy with markdown. For most prose it's fine, and there certainly isn't any downside to using it versus any other lightweight markup format for drafts of technical documentation.
 I agree completely about MD and don't see why choose that troff other than that it's the new hotness. Troff can be dumb and simple like MD which is fantastic for the writing bit, but has the ability to do rather complicated formatting if you need that. There are macro sets other than man and ms like mom that add features or make the syntax clearer. Most troff implementations are able to output or have tools to output to various formats like ps, pdf, text with control codes, plain text, etc. Plus, on most systems, either groff, heirloom troff, or Plan 9 troff is there by default. It makes a lot of sense to me because of its potential power and the portability of it.
 If you can't see why people prefer the first to the second, I don't know what else to tell youMarkdown: An h1 header ============ Paragraphs are separated by a blank line. 2nd paragraph. *Italic*, **bold**, and monospace. Itemized lists look like: * this one * that one * the other one  troff -ms: .NH An h1 header .LP Paragraphs are separated by a blank line. .PP 2nd paragraph. *\FIItalic\F[], \FBbold\F[], and \F(CRmonospace\F[]. Itemized lists look like: .IP \[bu] this one .IP \[bu] that one .IP \[bu] the other one
 "I don't see how using markdown is any worse than using a typewriter."...errrr...i think we have moved past the typewriter somewhat
 I use a similar flow for my own books, except Perl scripts instead of Pandoc. Starting with a markdown-style text format is so nice.One thing with LibreOffice is that you can generate documents, they are just XML zipped up. So my tools produce ODF files with proper formatting (ready for CreateSpace), automatically. The only thing I can't automate is producing the table of contents; for that I have to open the document once and update the ToC, then export the PDF.My tools are on github (https://github.com/booksbyus/mkbook) and there's an example book using them (https://github.com/booksbyus/scalable-c).Also to note:* gitbook works really well as a front-end to show Markdown books off a git repo and collect edits. See https://www.gitbook.com/book/hintjens/social-architecture/de....* If you write your own filters (text to whatever) you can do really interesting things. It's worth learning Perl just for this kind of work.* CreateSpace is excellent, once you have cracked how to produce perfect PDFs for the cover and interior. Highly recommended.* Amazon's author tools are overall good. A bit clunky sometimes (e.g. lots of separate logins), yet they work and can produce decent income for a self-published author.* LibreOffice produces excellent PDFs. Really top class, once you've figured out the details. The way I use it in the mkbook tool is by templating the format; i.e. design a book by hand, then slice out the various pieces of XML, and then generate the body of the book text using the template. This gives me 100% control over the results.* When you automate this, it can work really quickly, as in a few hours to to make a publishable draft, and perhaps 30 minutes to release a new version of the text to CreateSpace.* Get a professional cover designer. It isn't that expensive and makes a big difference.
 Recommendations for finding a professional cover designer? That's one of the things missing from the original article that I was interested in.
 The cover for my novel, which may be the most professional bit about my novel, was done by an artist I found through 99designs. I loved the process, I had over a hundred designs to choose from, tons of variants of these, etc. And I'm super happy with the result!
 Run a competition on 99designs.
 You should consider 4chan, those guys can be pretty unique in their art choices.
 I was lucky enough to get one via contacts. I'd expect though that you can find designers online and see their portfolios before making a serious commitment.
 > I use a similar flow for my own books, except Perl scripts instead of Pandoc.Same here, except I used Python :-) Great to hear about Pandoc though, I really need to check that out!
 Congratulations on a great accomplishment. For what it's worth, I found your post on how you reverse engineered Dan Brown [0] far more interesting than this one. Very cool!But it seems writing was the easy part; how do you plan to market the book? After four years of writing, surely you want to make sure it gets maximum exposure. Do you have any concrete plans here?Did you ever consider shopping the book around to publishers, or were you always set on publishing it yourself? Any comment on the fee structure of Amazon?
 > how do you plan to market the book?that's what he's doing here.
 You're not entirely wrong, but not for the reasons you think :) I hope this post raises the profile of the book a but, but I'm much, much more interested in getting the book into the hands of more people, than in selling more copies. In other words, I want more readers, not more customers.I can't just post a link to the book because of Amazon's terms, but as I offered at the bottom of the post, I'm giving free copies to anyone who sends me an email.
 I'm glad this all works well for the author, but still ...Why Markdown?I know that it is widely used, but that's only because GitHub pushed it so much.AsciiDoc is superior in almost every aspect. For larger texts I can strongly recommend it. The toolchain is mature, there are many modules for everything, even included UML from text description, if you need it.Moreover, in addition to HTML output, AsciiDoc has proper PDF and LaTeX output as well.Really, Markdown is only good within its niche, while AsciiDoc has a mature toolchain and has a proper extension system, so that new features are additional modules for the main tool, rather than incompatible forks of the Markdown processor.There is the original AsciiDoc tool, but for modern use I recommend Asciidoctor.
 Markdown was popular long before Github.The reason for using Markdown is simple: pandoc. Pandoc is just that damned good.
 Pandoc can also process reStructuredText [1], which looks similar to Markdown, but has more features and a single non-ambiguous specification. Me and my co-author rejected Markdown for our book about internet memes [2][3] after a short evaluation, mainly due to ambiguities [4] and lack of features.[2] The book can be downloaded for free: http://internetmeme.de[3] About the writing process: https://news.ycombinator.com/item?id=12311546
 I always like to go back to the figure from the pandoc homepage showing input and output formats. I think it gets taller every time I look at it:http://pandoc.org (scroll down past the text).Personally, I like that Markdown gives me so little control over formatting. It forces me to focus on the text instead, which is why I moved away from LaTeX. Formatting is something I'd rather do after the writing is done. If you're using LaTeX or HTML, you can embed formatting directly, and if you're using another intermediate format, Pandoc gives you a lot of control over how it's generated.
 I know. Personally, I prefer reST, but Markdown is more approachable than reST for a lot of people.I wouldn't say reST is all that similar looking to Markdown though: they have very different philosophies, and really only coincide on the fact that they're used for marking up plaintext documents.
 What do you mean with “more approachable” ? Markdown surely is more popular – but which constructs differ much in usability?
 Off the top of my head, the complaints I've gotten have referred to directives, the link syntax, roles, not being able to nest inline markup.None of this bothers me, and directives and roles the source of much of reST's extensibility and power, but apparently those are the things that make reST a bit intimidating compared to the likes of Markdown.
 Please elaborate on the complaints. What about the directives do users consider problematic?
 The two nicest things about directives once you get the point behind them are that they're consistent and flexible: once you know the syntax for one directive, you know most of the syntax for the rest.The problem with that is that the syntax is comparatively heavyweight. Part of this is due to reST's alignment rules and partly down to the generality of directives.Let's take 'image' as an example. The simple way to embed an image in reST is this: .. image:: foo.png  And in Markdown: ![](foo.png)  And with some alt text: .. image:: foo.png :alt: Foo!  And in Markdown: ![Foo!](foo.png)  I've had people struggle with image directives simply because it doesn't sink in that the :alt: attribute has to be aligned with the directive name, whereas the rough equivalent in Markdown has no such issues.OTOH, once you know how to embed an image in reST, you're 90% of the way to knowing how to embed a code block, whereas in Markdown you need to learn a little bit more syntax. The downside of that is that this: .. code:: print("Hello, world!")  Is more heavyweight than:  print("Hello, world!")   And people prefer the latter as it's less to type to the former, even if the former introduces no special-purpose syntax.This is where Markdown's ad hoc nature has benefits over reST's more structured nature: if you want to embed an image in Markdown, you learn how to embed an image; in reST, you learn the directive syntax so you can embed images. That gives Markdown and reST different learning curves. Markdown gives you lots of little easy-to-learn tools, but requires that you be constantly learning new tools as you go along, whereas reST gives you fewer, slightly more complex tools you have to learn up front, and all the new stuff you learn later is based off of those. Moreover, Markdown has less impact on the text due to its ad hoc markup being more compact, unlike reST, which takes the route of being more general at the cost of being more verbose.Somebody starting out confronted with either reST or Markdown will look at both and see that the latter looks more like plain text than the former, and thus there is less of an initial barrier to entry. That's why the likes of MkDocs exist in spite of the likes of Sphinx existing long before MkDocs and its ilk did.TL;DR: Markdown makes easy stuff super easy and hard stuff possible; reST makes the easy stuff a little harder than Markdown, but lets you lift mountains.Edit: s/codeblock/code/
 > Markdown was popular long before Github.I looked for the origins (not popularity, but a lower bound):Markdown 1.0.1 (18 KB) — 17 Dec 2004Github - October 2007 - First commitsource: https://github.com/about
 More accurate dates would be March 2004 (original release) for Markdown and April 2008 (public launch) for Github, which is a significantly larger amount of time.By the time Github launched, Markdown had eaten most of the lightweight markup language for lunch. If you want to compare its popularity, you first have to look at the relative popularity of the other lightweight text markup formats: https://en.wikipedia.org/wiki/Lightweight_markup_languageAlso, you have to keep in mind the amount of cachet Schwartz and Gruber had at the time, which helped no end in popularising it. Popular blogging platforms at the time of its release, such as Bloxsom and Moveable Type quickly got support for Markdown.Arguably, if you wanted to choose a site that's done a lot of popularise Markdown, Reddit, which launched in 2005, would be a much better example.
 Pandoc is damn good, but that's not my primary reason for using Markdown.I choose markdown primarily because it's readable.
 > Moreover, in addition to HTML output, AsciiDoc has proper PDF and LaTeX output as well.Have you tried pandoc? It allows interleaving Markdown with LaTex and outputs to every format under the sun, including PDF. IMO the best of both worlds, Markdown where you want, TeX when you need to.
 This is how I wrote my dissertation. Well over 100 pages, all in Markdown except for the floats, where I used LaTeX for fine-grained control, and the refs. Well, also the title page and the signature page, because you pretty much have to lay those out by hand. I was worried about citations, but pandoc-citeproc did great with my bibtex database.
 He only used the # and ## tags.Considering the other steps that he had to take between his text and final output, why not use Markdown?
 As I said, I'm happy that Markdown was good enough for the author.But I just would never recommend Markdown in general for text books, given the alternatives.
 Lots of stuff is more featureful than Markdown (rST is great, org-mode has a lot of possibilities). But the author didn't need a whole lot of formatting for the ePub. And he didn't use the original Markdown tool (or its relatives) for converting it, but Pandoc, to produce ePub and ODT.
 I looked at AsciiDoc and thought it was perfect for authoring. But the toolchain is one aspect where a pandoc-based workflow makes a lot of sense. When I saw AsciiDoc needs DocBook to render a PDF, I had to move on because I remembered how hard the tools are to use on Windows. Just now I looked up "docbook tools" and "install docbook". Lengthy, multi-page instructions to just get and set up the stylesheets. Lists of alternative tools. Where to start? What to pick? I think we had to use cygwin's docbook package to make it work back in the day.Compare to downloading a pandoc binary and miktex, which is a piece of cake.
 If I wrote a novel, I would probably do it in LaTeX, but Markdown is a good choice.Its philosophy is to prefer clean "plain as possible" source code over advanced markup. This can be annoying when documenting code or making complex reports with lots of tables and lists.But novels are just long blocks of loosely structured text.
 > I know that it [Markdown] is widely used, but that's only because GitHub pushed it so much.Markdown is a specification of the informal way people wrote plain text emails.
 I saw this a couple of weeks ago: https://www.youtube.com/watch?v=FtieBc3KptUI really like how he has no idea how to pronounce stuff, but he still managed to bend emacs after his will.
 This is great, and hilarious. A good place to get an idea of the video from is about 13 minutes in until 15 minutes.
 And actually quite inspiring. He got me to set up abbrev mode properly and actually start using emacs for _everything_ text related (even this comment, using "it's all text" for firefox).I set up it's all text with a bunch of different file endings depending on what language I use, so that abbrev mode works correctly. It's a godsend.
 Hey, OP! I admire the fact that you took all this on yourself, and also from reading the linked blog post[0] it seems like you took the hacker's approach to the planning and writing of it too.Honestly, I'm not sure it's a book I'd read - not a big Dan Brown fan - but I am deeply envious that you've completed such a huge undertaking :)
 ... then OP, could you share this structure? "Soon the structure underlying The Da Vinci Code and Angels and Demons and The Lost Symbol was laying bare before my eyes. I could see why the stories worked. I had reverse-engineered Dan Brown."
 Multiple mcguffins, drip-feed expository dialogue and prose, cliff-hanger in 30 minute reading time chunks. Always keep at least one good mystery open. Start in media res, slow the pace for the second act, which is where you consolidate to accelerate to a frantic conclusion where you draw seemingly unrelated threads together. Essentially, give the reader a reason to keep reading, and to keep coming back.It's not Brown's recipe, it's how pretty much all compelling fiction hangs together.
 Pretty much, plus mini-cliffhangers at the end of each chapter. And tons of chapters. My novel has around 70, so roughly 5 pages on average. It helps keep the pace, and because of the psychological need to read "just one more chapter", it makes it a "page turner" or "a book you just can't put down".
 Well I never knew about McGuffins before. Every day's a school day :)
 If you read more than two Dan Brown books, honestly it's hard to not see this structure. I read The Da Vinci Code and Angels and Demons, which were OK, but then reading Deception Point I could guess all the plot twists several chapters in advance. He's just very formulaic.
 I suspect OPs spreadsheet is lacking the metadata that exists in their mind to be interpreted usefully.
 Sure, sounds like a good idea for a follow-up post!
 Fair enough. I really didn't know whether I could pull off writing an entire novel at all, so for my first attempt I'm totally fine for it being formulaic. Even so, most of the people who have read it have enjoyed it - some read it in a single sitting, which is probably the best compliment they can make :)
 Agreed. After reading the snippet of the manuscript in the OP's screen-shot this sprung to mind:
 See my comment here - https://news.ycombinator.com/item?id=12314100Also, English is not my native language. In fact I wrote and published the novel in Spanish first (https://www.amazon.com/dp/B00I1EU1Q0). I didn't trust my English enough to translate it myself, so I hired a translator to do the job. The result is OK but I have the feeling I could have written it better! FWIW, I'm probably going to write whatever I write next in English.
 If you don't mind revealing this information, which version is doing better?
 They're both rounding errors, to be honest!
 Oh that is golden. "The critics said his writing was clumsy, ungrammatical, repetitive and repetitive." Apparently DB isn't an Oxford comma man?
 You can do it that way, but using something that's made for the job makes life a lot easier. Between Scrivener for writing and Vellum for compiling to ebook formats there's no reason to spend your time futzing around. If you want to format for print, either pay someone or learn InDesign because nice print layout is tougher than it appears.Then again, a lot of the self-pub strategy these days is specifically geared around getting as much out as quickly as possible, so that's not necessarily an invalid approach.In any case, I prefer to spend most of my time writing rather than screwing about with tooling.
 > learn InDesignAhem, or learn the open-source Tex/Latex, from which InDesign took inspiration. If you can format a beautiful scientific/mathematical textbook for print using Tex, you can format anything. And Pandoc is a great adjunct.But yes, it's not for people who wish to avoid screwing around with tooling. It's for perfectionists who want absolute control over layout/notation.
 > from which InDesign took inspirationDo you have a source? InDesign is much, much more than Latex, and certainly not just a derivative - though your post implies just that. Having spent many hobby-years in InDesign laying out newspaper spreads, magazines, covers, I'd argue that it has a fundamentally different, richer, use case, and is incomparable (unless you're just talking about technical documents).
 > Do you have a source?I don't have the source now, and I may be mistaken. I thought I remembered reading somewhere that InDesign's creators drew significant inspiration from Tex/LaTeX, but I can no longer find the original source. Perhaps I was thinking about QuarkXPress. I did not mean to imply it was a direct derivative. InDesign was openly created to compete with Quark, so its feature set was much more in line with Quark than with Tex.I completely agree that LaTeX is not ideal for designing magazines (although it's used by some academic journals), and its support for cover layout could be better (indeed, it took me longer than it should have to create a fairly simple book cover with Latex just last week).But as a typesetting system, it's great, and works great for books.
 LaTeX is purely for text/formula/equation-heavy academic publishing. It was created for that purpose by Lesley Lamport.ConTeXt - http://wiki.contextgarden.net/Main_Page (an open source derivating of TeX) developed and supported commercially by Pragama - http://www.pragma-ade.com/ is what you should use for typesetting magazines etc.,ConTeXt gives you complete artistic and layout control. While it is used heavily for non-technical publishing, people like Sanjoy Mahajan have used ConTeXt for publishing Math books like - "Street Fighting Mathematics" https://mitpress.mit.edu/books/street-fighting-mathematicsSee resulting PDF: https://mitpress.mit.edu/sites/default/files/titles/free_dow...
 Writing a novel in LaTeX is not the best approach if you're looking to write a lot of text and don't have to worry about notation. But Neil Stephenson famously wrote Cryptonomicon in Emacs and LaTeX, so hey, it works for some people.Edited to add that InDesign is much easier for back matter, front matter, laying out images, covers, etc. . . . And that's coming from someone who LOVES LaTeX. I set my old software company with some sweet templates and all of our proposals were typeset in LaTeX. It was awesome. But if you're writing a novel for print I really wouldn't recommend it.
 Respectfully, I disagree, but everyone has their favorite tools and procedures. I wrote a 350 page book [1] of mostly text with 500 references, initial quotations at the front of chapters, and illustrations. LaTeX was exactly what I needed [2], and the greatest pleasure of that experience was being able to choose my own tools instead of being mandated by some publisher to submit everything in Word. LaTeX did just fine with front and back matter; CTAN had examples of all this stuff [3] and it was well organized so required little searching. My work flow was basically emacs/auctex, some scripts to convert TeX to HTML and eventually XHTML, and Calibre and Sigil for the final touches. Working on a plain text format made for easy archiving, diffing, and compression. Can't imagine using something that requires and internet connection to work, like Google Docs (which I like, on the other hand, for other use cases).Happy to offer the epub or mobi to anyone who wants it, and the LaTeX source if you're curious how I made it work. Email my user name at gmail.com. [1] http://dictatorshandbook.net [2] http://therandymon.com/index.php?/categories/2-Tech/P2.html [3] http://ctan.org
 I agree that writing large volumes of text in LaTeX is cumbersome, and the same goes for text that includes many code samples. \begin{verbatim}...\end{verbatim} is just too verbose, and \verb|...| for inline code is also too verbose to my taste.So right now I'm writing my book in Markdown, and put it on leanpub for now (https://leanpub.com/deploy).For print, I'll either figure out a way to get a PDF formatted as I want it from leanpub, or use pandoc to create the markdown to LaTeX, and then write a custom template with all the customization.But this is something that I won't worry too much about until the manuscript is really finished.
 For code samples I prefer to have them in separate files and include them so it's just one command that imports them and formats them.I wrote both my Master's and Ph.D. theses in LaTeX and found it much better than the alternatives. With Markdown I'd be resorting to falling back on LaTeX anyway and many of the additional Markdown features weren't available when I wrote them.I had a predefined document class for my Master's but the university changed the guidelines by the time of my Ph.D. so I used the Memoir document class and that had a lot of flexibility and I didn't have to include many other packages to format things. I basically set up the format once at the beginning, checked against the guidelines once at the end and that was it.
 FrameMaker would be even better than InDesign for creating books--it has better support for managing book structure, and has decent word-processing facilities--it's a much better 'all-in-one' tool.
 I wrote my Ph.D. thesis in FrameMaker on a Mac. Then Adobe went and discontinued it on the Mac, which was much lamented. So now I write stuff in LaTeX instead.
 I don't think you are supposed to write in LaTex anymore than you are supposed to write in InDesign
 Spoken like someone who has never encountered auctex-mode
 ... anymore than you are supposed to write in HTML?
 If you have a particular layout in mind, then InDesign takes about 1/10 of the time to learn as it would take to achieve mastery of e.g. the memoir package in TeX.TeX's not the right tool if you need immediate, precise, flexible control over page layout.
 InDesign did take inspiration from TeX in the layout and hyphenation algorithms, but that doesn't make them totally equivalent.Although TeX does produce very good output InDesign is much better at eliminating rivers as it has an idea of the shape of the glyphs. I've seen stuff on doing this in TeX, but it was a feedback loop of producing the output, analysing it, and then tweaking the input to fix the problem.
 InDesign was the successor to PageMaker and a result of a strategy to outcompete Quark Xpress. I don't think it took much inspiration from TeX/LaTeX, given that it's a WYSIWYG tool with no markup language underneath.
 Make life easy on yourself. Learn InDesign.
 Why so condescending? Reading your comment, after reading both of the author's posts on his methodology, I take that his approach might not be everybody's, but I took a lot more from his writing, then from your writing.You tell the author, not explicitly but non the less, that his method sucks and he is an idiot for not using "the right" (TM) tools. Using your method is ok - for you. Using his method is ok - for him. His is not more right or less right. Both of you got your work done and your book(s) published.> no reason to spend your time futzing aroundWell for some it is also the process that counts. And the ability to get stuff done without proprietary software.On a final note: Maybe I just misread your tone (at least, it's the internet) - maybe you weren't so a stuck up. Or didn't mean to be.
 Well I was and I wasn't. I certainly don't begrudge a person for getting really involved with their tools. Depends on what your goals are. If it's productivity and a nice finished product then the tools he was using are likely not the best for the job, even if they are fun. No way to know for sure, but if I was handing out advice I would absolutely NOT suggest people use the tooling he used, even for developers.
 I'm doing this whole thing for pleasure, and for me f*cking around with the tooling was just as fun as writing the novel :D
 Ha! Then more power to you :)My point with this was from a workaday aspect. I both understand and respect getting nerdy about the tooling. And congrats, the first novel is the hardest.
 I use TROFF and friends for layout and Ghostscript for generating PDF. No screwing around with a stupid GUI. :)Once you have a collection of macros set up, it's a natural process and allows you to focus on writing entirely.YMMV, of course.
 I always thought TeX and friends had much prettier output than roff-family output and my professors always appreciated a TeX formatted paper.But for rapid documentation writing I found roff perfectly acceptable. It worked for Bell Labs, so why not? In fact for a while I actually published my resume in man format after slightly tweaking the an macros. Very few people got the joke and most just asked if I could e-mail a copy in Word format.If a sysadmin / engineer / devops dude showed up today at my door with his resume in -man format I'd think it was cool.The fact that Scrivener is now available on the iPad and Literature and Latte abandoned Linux makes an iPad Pro a tempting platform. But frankly there's nothing wrong with using a unix distribution on a Chromebook or droplet via ssh and roff to author a book from start to finish. And it would be far cheaper than an iPad Pro.
 Depending on what you're doing, LaTeX may be a better option. But if troff works for you, I'm not complaining.
 > InDesignA free software alternative(?) is Scribus. It doesn't nearly have as many features as InDesign but it is one of the few free software implementations of a desktop publishing programme.
 The last time I tried Scribus it seemed very limited--slow and unable to deal with large files. Is it any better now?
 And the only way to design large-scale posters on Linux without going insane. Incredible tool!
 If you ever need to produce more uncommon layouts, try http://www.laidout.org/
 Thanks, that is one truly unknown tool. Not very intuitive but I will manage.
 Well...tbh this seems far more just like a cheap way of promoting the novel than a way to actually provide useful information
 The OA specified the software used and the way that the tools were chained together, and he mentioned the somewhat kludgy workround for getting the chapter headings vertically centred on the page.I also found his other page linked from the OA about the actual writing process itself of interest. Specifically the way he 'reverse engineered' the plots of various popular novels by deciding on some common headings/categories and summarising each of the novels under those headings in a spreadsheet.What 'useful information' do you find to be missing?
 I agree, I found both articles very interesting in terms of both writing workflow and self-publishing workflow.
 I have a friend that writes porn using open source tools. Her stuff is on AMazon as well. She also told me how their numbers work. A book at this author's rank sells maybe 1 book a month. That's not good. Anyway...she uses openoffice to write her docs, then uploads to kindle. She authored some kinky shit like Control - A Billionaire Romance series that got high sales and turned on lots of folks. If you're into that sort of thing, like bondage and such.
 It's kind of impressive just how high sales you get with not even that good smut fiction. (coughs50 Shades.coughs)I've picked the wrong job.
 Yeah...me too. Wish I wrote her books. Her name is M.P. Lodi in Amazon.
 Let me recommend "working copy" for fantastic GitHub integration - basically it's a git client on iOS (and android?) and it has a good enough editor that I can write .rst and git push and I have published an article - this went up today - http://www.mikadosoftware.com/articles/alwaysgoodships
 Oooh, that sounds great. I'll give it a try. Thanks!Any idea whether it has good support for large files (e.g. a 300k character novel)? I've tried a ton of iPad editors but most of them can't handle large files, or don't have good scrolling support so it's a PITA to go to the end of the text.
 I have never tried anything so large. I suspect it's just pretty much iOS basic editor (whatever that is)Why not divide up the novel into chunks (chapters?) - I do this and stitch the chapters together before running through docutils - it's trivial and let's me divide and conquer (so far it's an unfinished manual for Dev Leads and managers)
 Yeah, that's a good point. I'll give it a try!
 I used Latex to write my own book, in case anyone cares: http://www.zerobugsandprogramfaster.net/If I had it to do again, I would consider using just tex and Makefiles.
 I'm a HUGE fan of using softcover.io to build documentation for runbooks and other technical documentation. You can write the text in Markdown (with some LaTeX and other goodies), and it automatically converts it to both HTML as well as epub, mobi, pdf. I like using it because you can make mobile documentation that can be pushed out to Mobile devices (and optionally managed by Apple's tools) so your on call resource has the ability to pull up the documentation about the alert that they are receiving on their mobile device so they know how to remediate/mitigate that issue.
 Since shutting down my own Internet presence for good, which also took down technical documents I had written years ago that - to my great surprise - I found were actually cited a lot according to Google Scholar - I'm thinking maybe linking to the archive.org link is better than linking to the actual URL? Any thoughts? My documents are still available under the orginal URL there, so if all those footnotes and citations pointing to one of my docs pointed there they would still work. It also has the added advantage of being able to link the concrete version as it was when you included the link.Example for the first link in the parent comment, pointing to the (right now randomly selected) 26 March, 2016 version of the webpage:https://web.archive.org/web/20160326165458/https://en.wikipe...TL;DR I recommend to always cite sources using their archive.org document version.
 A HTTP GET to a URL prefixed with http://web.archive.org/save/ archives the content behind that URL.I use the conkeror web browser [1] and have included the following code in my $HOME/.conkerorrc/config.js and execute “wbsave” for many documents (even my own) whenever I send the URL to someone else. define_webjump("wbsave", function(url) { return "http://web.archive.org/save/"+url; } );  [1] https://en.wikipedia.org/wiki/Conkeror  Probably best to use HEAD instead of GET to save downloading the result.  I added an updated version of the above text to my own website: http://news.dieweltistgarnichtso.net/notes/buch-schreiben.ht...  Most novels in the history of the world have been written using exclusively open-source tools.  I'd be careful with that assumption. Tthere aare over one million novels written every year now.The novel as a literary form dates only to the 1500s. Cervantes' Don Quixote is generally considered the first novel.I suspect a quite large share, possibly majority, ofnovels have been written since the word processor era.  A little more interesting was how he reverse engineered Dan Brown novels (see https://medium.com/@gabrielgambetta/how-i-wrote-my-first-nov...)I would love to see his spreadsheet on what makes a Dan Brown novel - it's like distilling literary evil.  "How I distilled literary evil" sounds like an amazing title for a follow-up article :D  I'd like to know what people are using for writing non-fiction. Mainly I'd like to know how people keep track of their research and cross referencing it from their editor.Here's what I have:* Many words typed up in One Note* Highlights and annotations in iBooks, Kindle, and Instapaper (although Instapaper allows export to MD)* Notes from PDF exported to Markdown using the Highlights app[0]* Images* Tons of shit to read in pinboard and other placesHere's the kicker, I travel a lot (every week) so I need most of my tools to work well offline/slow wifi and preferably on an iPad/iPhone.What do people use for organizing notes and research (literature of all sorts), cross referencing, organization, and first draft?  1. I use Devonthink Pro for notes / research: http://www.stevenberlinjohnson.com/movabletype/archives/0002....2. Many use Scrivener for writing, but I have never quite managed to agree with it properly, so I just use Word.3. I use Bookends: http://sonnysoftware.com/ for citations. No one it seems really likes their citation manager, but I like this one best.  2. Scrivener is great, wish it could do .md natively 3. Yes I hate Papers. Citation managers seem to only be useful for academic papers anyway.  I love DEVONthink Pro. I have the Pro Office version. I especially love the "See Also" feature as that will suggest other documents to look at when you're hunting through your references.  Scrivener is a one-stop shop for this sort of work. Ulysses might also be sufficient. The learning curve for Scrivener is fairly steep, though, by comparison.  I love Paperpile, which would accommodate some/many of these needs.  I also wrote a book called the Final Six Days. If you visit his Golden Legacy book page on Amazon you will see my book in the "Others Have Viewed" section. It's mostly due to me checking on his and other novels.I was planning a Medium article about my (similar) experience. I'm not sure why he'd chose markdown when epub is just structured HTML. If you're a software dev just stick with HTML. Also, to anyone planning a novel: get ready for annoying inconsistencies in display across the various Kindle + iOS + android devices. Fonts not displaying correctly, sizing is off, things not centering, some CSS selectors not being applied at all. It is IE/Netscape circa 1997 all over again.  > I'm not sure why he'd chose markdown when epub is just structured HTML.Markdown's a lot simpler/faster to read and write than HTML, and books rarely need the extra features HTML provides. When you do, you can always do inline HTML in Markdown.> Also, to anyone planning a novel: get ready for annoying inconsistencies in display across the various Kindle + iOS + android devices. Fonts not displaying correctly, sizing is off, things not centering, some CSS selectors not being applied at all.Oh yeah. It's even worse if you write a technical book.[1]  I fought getting fixed-width fonts to work for weeks across all these devices. Believe it or not, despite being a sci-fi novel it does have code snippets :)  What? Markdown is WAY better to write in than HTML, and there are a bajillion converters if worse came to worst  It's much simpler to just write your novel in Word, generate the HTML, write a small script to scrub out all excess HTML/CSS, then use Calibre to segment into HTML docs by chapter. As far as the HTML for novels is concerned, its very minimal, like a and bunch of tags.I'm not knocking markdown or anything, I just think it's a peculiar choice for writing a novel.  Can somebody show him how to take a screenshot on Linux please?  I know how to take screenshots on Linux :D Taking pictures of the screen felt more "authentic", in the same way that I took pictures of the printed book instead of pasting a PDF.  I have started to write a novel using almost exactly this setup. It's encouraging to hear it worked out for someone. So far I've been using plain markdown and just rendering it to HTML for a pretty view, but with plans to turn it into e-pub later. As mentioned in the article, the controlled sync is what makes this beat Google Docs. I set up Vim and Mercurial on my phone, so I can sync and edit the text anywhere. Fun tools to play with!  Go back to the early 2000s. I'm kind of obsessive about data portability, and the lifespan of a novel is closer to 30 years than the 3-5 months typical of business documents. Alas, I'm selling novels to Big Six publishers who like to chew on a Word file. So ...Writing tool of choice: vim. Markup format: Perl's POD (plain old document) format, because (a) I can mess around with the formatter if I need to (old Perl hand), (b) it can export to a variety of formats, (c) it has the minimum necessary set of directives -- headings, lists, emphasis -- that I need (novels are typographically simple unless you're playing Stupid Layout Tricks, hello Alfred Bester and Samuel Delany) and (d) Markdown didn't exist back then.Version control: a novel with a single author is piss-easy. I used rcs, because it was ubiquitous, easy to understand, and to the extent that I was keeping projects on multiple computers I was fine using rsync to keep a directory tree updated; remember, a novelist working alone has no need for multiple user IDs, no forking of multiple versions, virtually never any need to diff or merge -- just the ability to roll back to a previous point if I took a wrong turn in writing (in other words, undo checkpoints with names). No cloud services, of course. (Dropbox didn't come along for a few years.)Build: makefiles and some homebrew perl scripts FTW. Type "make", check out latest draft and generate HTML, RTF and (eventually, via an external toolchain) Word files I can ship to my editors.Back in those days, copy edits showed up as a bunch of paper print-outs with red ink on them and you mailed them back to production after you added your own chicken scratchings. (If you were smart you scanned/photocopied the pile first, for insurance: this saved my ass on two occasions when CEMs went missing in the post, thank you for nothing, Royal Mail). Page proofs ditto. So I was able to use my homebrew toolchain happily for several years during which I wrote "Singularity Sky", "Iron Sunrise", "Accelerando", "The Atrocity Archives", and the first three Merchant Princes novels using it.Then the publishers began moving to Microsoft Word tracked changes for processing copy-edits, and annotated PDFs for the page proofs, and it was all over bar the shouting; Word tracked changes suck, but trying to check the changes on a large document in a third party word processor like LibreOffice sucks even harder (one sleeper bug that nobody triggered before can cost you a couple of months work), forcing me into Word for the post-writing workflow. And then a Better Way came along for writing books in the shape of Scrivener, for which there is nothing remotely like an open source equivalent and which is well worth the price. Alas.Which is why my open source novel writing days are over. (But if I had to do it again these days? Markdown, Pandoc, and Git, probably with all the fancy Vim plugins I could lay my hands on.)  Interesting choice of words, "well worth the price." I took a peek and was shocked to find that I could pick up a copy for$40. I imagine if it's any good it is worth an order of magnitude more than the price.It's a little sobering what a good PC application sells for today, honestly.
 At some point I'd be really interested to read a writeup on Scrivener and your writing workflow.
 Sure: here you are!
 Awesome --- thanks!
 FWIW - Scrivener is a plausible novel-writing tool, mentioned a couple of times in these comments. It took a Long Time, but their iOS app was released a couple of weeks ago.Scrivener by Literature & Latte https://appsto.re/us/jqx95.i
 You probably don't want to give away specific numbers, but in generally, how is the profitability of selling on Kindle versus paperback? The price difference looks to me to be less than the cost difference, thereby making Kindle more profitable. Is that accurate?
 >You probably don't want to give away specific numbers, but in generally, how is the profitability of selling on Kindle versus paperback?Not OP, but I've sold thousands of ebooks, but only a few dozen paperbacks.Paperbacks are for people with distribution networks and warehouses. As a self-publisher, we are relying on electronic distribution to cut the overhead costs. Profit per unit is about the same between the two, but we can sell more ebooks because the cost to the customer is only a bout a 1/3 compared to print.Mostly print copies are for promotion. Handing them out as free prizes/gifts are a great way to build readers.
 There handful of proprietary software I actually pay for because I personally think they are by far the best tools of the trade. For creative writing I think I have found the best solution in form of Scrinver.It has been simply the best tool I have used to write my novels.
 This is pretty cool. But I would be interested to know how the author would have handled other requirements. Embedding images, other scripts, typesetting different column layouts fonts, spacings, etc.
 No idea. Another one of my projects is a Computer Graphics textbook (I love CG, see http://gabrielgambetta.com/tiny_raytracer.html) which includes source code and diagrams and images the manuscript is also in Markdown with embedded LaTeX for the equations, so I'll have to deal with something more complex soon!
 LibreOffice has a plug-in for generating ePub files. You can do the whole job in one step.Not only that, with LibreOffice, your book can have pictures!
 Markdown: ![arbitraary text](image-uri)
 I like this idea of this.Would be great to have a website site that shows entire open source workflows and the optimal software/ approach. Anything like this exit?
 Ultimately, any text processing toolchain that uses ReGex for parsing will inherently be difficult to extend. And slow. And perhaps even broken. The disconnect lies in the human-machine readability gap. The only way to do it right, I believe, is custom text formats, scanned via structured strings into FSM grammers.
 You do use a mac though...
 Interesting to see that open source DTP is still a challenge in 2016.
 "self published"
 ガンベッタさんは、良く頑張った、ね。
 vim, pandoc
 no git?
 I did this back in 1990. Using Emacs and TeX/Metafont. And so did many, many other people.
 Relevant, because https://medium.com/@gabrielgambetta/how-i-wrote-my-first-nov...
 "To create this file, I used Pandoc again, this time to convert the Markdown files to ODT (OpenDocument Text), which can be read by LibreOffice."Ouch. WYSIWYGs are terrible.
 I think he says he changed styles only, which is closer to the WYSIWYM ("what you see is what you mean") idea used by Lyx, for example.I agree that WYSIWYG type of editors are terrible, specially for big documents, but limiting them to the use of styles, is manageable (in my experience at least).
 ...but necessary as publishers deal with .doc, .docx, and .odt files almost exclusively, especially when it comes to fiction.
 Meh.I wrote two novels in emacs.(now I just need to find some time to finish the final proof read and actually PUBLISH the damned things).
 You managed to call one of your novels "recursion". Proof that the medium colours the work.
 Is it inappropriate to point out that most likely there were a lot of closed source binary blobs involved in making his computer actually work?
 Does pointing that out advance the discussion in a useful way? Does it help having a constructive debate? Does it make anyone's day or life better? Are you just posting in the hope of getting some upvotes for making a funny, snarky short comment? Is your comment actually relevant to the discussion? Does the fact that you ask that question indicate that you already know the answer?
 I did not write that from my Lemote, channeling Stallman, nor I have any general enmity towards closed source software (that I both use and write). I just don't like inaccurate proclamations of purity.
 Yes. It's not what he meant, and you know it, and being pedantic never helped anyone.Next you'll suggest that I start turning off nonfree javascript.
 I was surprised to read "Mac" in an article claiming ONLY open source tools.

Registration is open for Startup School 2019. Classes start July 22nd.

Search: