
Etsy’s experiment with immutable documentation - telotortium
https://codeascraft.com/2018/10/10/etsys-experiment-with-immutable-documentation/
======
habosa
At Firebase (and at Google) I have done a _ton_ of work at making our "how
docs" consistently correct with limited maintenance time.

Consider a page like this: [https://firebase.google.com/docs/firestore/manage-
data/add-d...](https://firebase.google.com/docs/firestore/manage-data/add-
data)

It has 12 "how" sections of code snippets, each with 12 language options. That
means 144 code snippets need to be correct for this page to be 100% correct!
And we don't auto-generate any of these snippets, since auto-generating
idiomatic code is a rabbit hole that's consumed many a good developer.

So we, like Etsy, have come up with a separation of concerns between How and
Why that works for us. The text on the page is maintained by a tech writer.
The code on the page is a series of blocks that look like this:

{% includecode snippet="android_firestore_write_doc" %}

We have a system that resolves these, at publish time, to match snippets on
GitHub. For example: [https://github.com/firebase/snippets-
android/blob/5f94890d7b...](https://github.com/firebase/snippets-
android/blob/5f94890d7ba329a8a4c25f0ec8b42d4b9d781bc0/firestore/app/src/main/java/com/google/example/firestore/DocSnippets.java#L318-L336)

These snippets can be maintained with all the wonderful tools that exist in
the world for maintaining code! We don't need to think of them as just docs.
So that "snippets-android" repo has continuous build, a robot that updates
dependencies, PRs/Issues for third-party contributions, etc.

What we've found is that this system allows us to maintain many thousands of
code snippets with only a handful of people watching over them.

~~~
dancek
I noticed the same system in the Google Cloud python samples repository. It's
actually quite useful to have the full code available and to easily see which
parts are displayed in docs.

Feature request, though: include a url in the source code that takes you to
the snippet's usages in documentation. Something like // [START
[https://codesamples.google/firebase/set_document](https://codesamples.google/firebase/set_document)
] or just the url as a separate comment.

~~~
habosa
We're working on making the system smart enough to work backwards, but I agree
that's a great idea!

------
swalsh
I think the part about this that is truly successful, is the use of slack
significantly lowers the transaction cost of adding the documentation. In a
wiki, where there is a certain level of quality expected, you might make the
decision to just move on with your life once you find the answer you need, as
you've already wasted enough time. With this, however, it's so cheap to leave
a breadcrumb it's just polite to do so.

~~~
forapurpose
> the use of slack significantly lowers the transaction cost of adding the
> documentation. In a wiki, where there is a certain level of quality expected
> ...

The wikis I've used also have slow editing interfaces because they have
separate read and edit modes. Does anyone know of a modeless wiki, which
probably means one that allows you to edit-in-place?

The problem with modes is that to edit something on a sizeable page you must,

1\. Load the page in read mode

2\. Read/search to locate the spot on the page where your edit goes, read the
context around it, and plan the edit.

3\. Load the page in edit mode

4\. Read/search to locate the spot on the page where your edit goes

5\. Type your edit

6\. Reload the page in preview mode (a form of read mode)

7\. Read/search to locate your edit and verify it

8-?. Repeat steps 5-7 until your edit is correct.

N. Reload the page in read mode.

Compare to a modeless wiki:

1\. Load the page in read mode

2\. Read/search to locate the spot on the page where your edit goes, read the
context around it, and plan the edit.

3\. Edit until satisfied

~~~
closeparen
Google Docs. Has comments, too.

~~~
forapurpose
How easy and efficient are other wiki functions, such as creating hyperlinks
between documents, creating new documents, and a simple, structured style
system? Last time I looked at it for these purposes, Google Docs failed in
those areas.

------
roblabla
Why isn't the how-docs tested? I don't work in ops or anything like that, but
it seems to me that it'd be the best way to ensure everything stays up to
date.

In Rust (and many other languages), when you document a function, you usually
add a code snippet that explains how to use this function. That code snippet
will be treated as a unit test and ran when testing the functionality. In this
way, documentation is always up to date (assuming the tests are ran).

Why can't tool documentation be the same? The tool documentation usually
contains a sample tool invocation. All we need to do is run the tool in a
controlled environment, and ensure that the invocation runs successfully. When
a tool gets updated, run the "documentation tests" to ensure all the commands
in the docs still run successfully. If they don't, then update the docs.
Surely, there are tools out there that already do this? Does anyone know?

~~~
pytester
This is kind of what I have been doing with this testing framework:
[https://github.com/hitchdev/hitchstory](https://github.com/hitchdev/hitchstory)

It's designed so that you can write tests that _are_ the basis of the "how to"
documentation - with a small script and some jinja2 you can generate readable
documentation in the form of markdown from the test:

Original test (behavioral example) :
[https://github.com/crdoconnor/strictyaml/blob/master/hitch/s...](https://github.com/crdoconnor/strictyaml/blob/master/hitch/story/email-
url.story)

Example docs generated from that test :
[https://hitchdev.com/strictyaml/using/alpha/scalar/email-
and...](https://hitchdev.com/strictyaml/using/alpha/scalar/email-and-url/)

It's also designed so that the testing framework can be programmed to rewrite
the test automatically for simple code changes (e.g. changing the wording of
that exception).

This makes building tests _much_ quicker and much, _much_ less tedious and
perfectly in sync docs _almost_ becomes a side effect.

Essentially "how to" documentation and example driven behavioral tests _are_
the same thing for all projects. It's just that most people don't realize and
don't treat them as such.

------
dguo
In the specific example given here of "dbconnect", I'd consider a different
approach: incorporating the "how" knowledge into the CLI itself. For example,
running the command without any arguments could trigger an interactive guide
that asks the user questions.

"What do you want to do?"

o Get a list of databases

o Connect to a database

"Do you want to connect to dev or prod?"

o dev

o prod

o I don't know (this option could link the user to documentation on the
different environments; maybe the user is a student on his or her first
internship and has never even heard of the concept)

"What's the shard number?"

"Which side?"

The CLI could output the equivalent, non-interactive command before performing
the requested action, making it a teaching opportunity as well. Maybe the "I
don't know" option or this approach in general is overkill, but I generally
think tools should be smarter. You don't need as much documentation if your
tools are inherently capable of guiding you to the right answer.

~~~
rabidrat
That's quite a bit more work than adding a reacmoji to 100 chars which were
typed into Slack anyway. We all want better tools with fantastically helpful
interfaces, but we also want functioning MVPs yesterday. I like the Etsy
approach because it fits into our existing JIT dev/doc workflow.

~~~
dguo
1\. I'd use a library or framework that handles the interactive parts, which
should make it fast to implement. We have tools to help make other tools.

2\. A tool's popularity/usage could be a way to judge when it's worth the
effort to add this sort of system.

3\. With the Etsy approach, I would also be wary of the cost of interruptions.
Getting constant Slack notifications is not conducive to getting work done.
You could silence them, but then what if someone can't move forward without
getting an answer to their question?

4\. There is a nice balance to be found between wasting time documenting
everything possible upfront and not documenting anything until someone asks a
question.

5\. Personally, when I find documentation that needs to be updated, it's
usually as simple as editing a wiki or editing Markdown and making a pull
request in GitHub's interface. To me, that's not much effort.

------
desc
This feels incredibly broken to me. The 'how' should not change often, because
it enforces rapid change on anything which depends upon it.

Any system inevitably acquires dependencies. Maybe it's fine to break those
dependencies if you're building some public website where you can always have
the latest version of the code running, but if you've got customers who depend
on having a stable build which was initially branched three years ago and you
still need to build, run and test that, you damn well better have a consistent
and stable mechanism for dealing with that codebase which spans multiple
years. Your ops and QA people will hate you otherwise.

Our own approach has been to pull out the coarse-grained operations and have
an 'environment' script which examines the version it's working with in order
to spit out relevant details. The 'porcelain' scripts are never allowed to
change their API in any breaking fashion, and their behaviour is adapted based
on the environment's output.

~~~
rhizome
DDD: Documentation-driven development. With apologies to Fred Brooks, write
documentation first, then write tests for the functionality and syntax
described in the documentation, then the code. I would speculate that doing
straight docs instead of something like Cucumber would be more productive.

------
kaycebasques
Cool!

One of the big picture value propositions of technical writers IMO is that
we're shepherds of organizational knowledge. For example, if someone sends an
email to me asking how to get a particular task done in Chrome DevTools (which
I write the docs for), I will ask their question on Stack Overflow, provide an
answer for it, and then send the link back to the asker. It's a similar amount
of work, but I'm making the knowledge public, rather than one-off.

What's really innovative about Etsy's idea IMO is that they have found a way
to make tribal knowledge shared on an ephemeral platform like Slack more
permanent. My big issue with real-time chat up to this point has been that all
the beautiful knowledge that you share on the platform isn't recorded in a
permanent way. I acknowledge that a lot of people prefer the real-time
messaging format, so it's really cool that Etsy has figured out a way to get
the best of both worlds.

------
munchbunny
This is a really interesting idea and I feel like it has the potential to make
the common usages much easier to ramp up on, but I'm skeptical because I run
into _conceptual completeness_ problems of the "how do I think about this?"
sort much more than "how do I do this?" problems.

Blog posts like this are always optimistic and celebratory. Sometimes it's
warranted, sometimes it isn't. I'd love to hear how people on the ground at
Etsy actually feel about it.

~~~
snarf21
I personally feel that the problem with most documentation is that it always
focuses on the how/what. That will always decay and be wrong. The code is the
only truth in the end.

The thing I suggest here is that if you are going to document the function, it
should only be the _WHY_. The how can change and the documentation doesn't
really decay. If you are making a new function you have to start with the
_why_ and if you can't then you shouldn't be adding a new interface. This also
helps build a better picture of how the pieces interface together.

~~~
mason55
This is my most common feedback when I review docs that people have written.

The other is when someone write “if you need to do X” with no information
about how you’d know if you need to do X

------
tchaffee
I'm surprised no one has yet mentioned how well automated functional tests can
document the how. I constantly go back to my automated tests to understand how
to do something I've since forgotten but that the code supports.

------
cat199
or alternately, treat the documentation as code and don't accept code changes
which are not properly documented..

~~~
snoman
This is it.

I would barely consider random snips about a thing/keyword, scattered over
time, to be documentation.

Certainly, a problem was likely solved with this FYI system... but it wasn't a
documentation problem.

~~~
GrinningFool
Discoverability is often a major documentation shortcoming. They solved task-
oriented discovery of 'how' snippets. Someone can search based on what they
want to do, and not based on a specific command.

I also prefer the 'documentation is code' approach, but it doesn't solve the
same problem.

------
gregknicholson
The article is both a why-doc and a how-doc.

I do agree that this is a useful distinction.

I'd have called how-docs and FYIs "howto", as I think that term would be
recognised more clearly, but that's a quibble.

~~~
ebiester
Internal documentation has a handful of patterns. There's the cookbook patern,
which corresponds to a "how-to." There's the "readme" pattern, which is meant
to conceptually get you started on a project. There's the "architecture"
pattern, which is a conceptual design and talks about the rejected
alternatives - this is not used everywhere but is a key deliverable in both
outsourced and waterfall applications. There is the "small chunk of
information that is hard to organize" pattern that has multiple solutions, be
it FAQs or Etsy's bot.

Immutable documentation, at its essence, isn't new. Wikis are in many ways
immutable documentation due to the diff. The interesting thing here is trying
to organize the "small bits of hard to organize" information around slack.
More often, however, the real problem is that we need to properly organize
this documentation at the point the organization becomes more evident.

For a tool, that should be obvious: --help should be obvious. It's not true in
all cases.

~~~
lazyasciiart
> Internal documentation has a handful of patterns.

This is a great way to articulate something I've been trying to talk to people
about for a while - is this something you've thought of for yourself, or do
you know of anywhere that talks about these patterns?

~~~
mr_tristan
I've noticed the concept of _information architecture_ is being used in
technical communication more frequently these days, borrowed from graphic
design. Which covers a bit more ground then patterns, including topics like
search, labeling, and taxonomy.

Sure, getting everyone to maintain docs with their features is useful. But
given enough time, you'll end up with so many "close but not really close"
concepts in documents it's very hard to stay on top of without someone putting
some organization into play.

~~~
ebiester
Old school tech departments used to have librarians for documentation. I still
think it would be valuable for departments, but it's one of those positions
for which it is hard to quantify until you get much larger as an organization.

------
TheOtherHobbes
It took me a while to work out why this doesn't feel like a full solution.

I can imagine it working for simple one-line commands and their switches.

I can also imagine it failing to work for any application that requires multi-
line code blocks and/or more detailed how-to examples.

For example - as a hypothetical, imagine you change your API from Oauth to
Oauth2, and you have to provide examples of new standard code, perhaps in
different languages.

That's very much a how-doc and not a why-doc problem, so supposedly it doesn't
belong in the Wiki.

But I really can't see the changes being explainable in a fyi.

Of course you can say "The new code is in repository X instead of X-1." But if
there are other questions and fyi's around Oauth2 - and there will be -
there's no guarantee that important fact is going to be at the top of the read
list.

For complex breaking changes that read list is going to get very long.

Simpler changes are going to have the same issues on a smaller scale.

So this isn't really immutable documentation - it's more of a database-hosted
command cheat sheet.

That's a useful thing in itself, but I don't think it's a complete solution
for documentation.

------
rootedbox
You should never break interface. You should expand interface.

If you do that.. then..

You never edit docs. You only add to docs.

~~~
mikekchar
Works _great_ for a year or two ;-)

Never refactoring your interfaces means that you can never improve the design
as you gain new information. It means you can never change to something more
appropriate if the underlying use cases change. It means that you can't easily
consolidate common functionality in a different place as new similar
interfaces are constructed. In other words, you are locking yourself in to a
brittle system that will degrade into a mess of spaghetti that nobody can
understand.

There are, of course, also downsides to refactoring. No simplistic rule of
thumb will tell you your best approach.

~~~
screaminghawk
You can have the best of both worlds by versioning your endpoints

~~~
nradov
Supporting and maintaining old endpoint versions is expensive. Especially if
the old endpoint shares a back end data store with the new endpoint. You can
get locked into a bad data model and unable to move forward.

If you can do it then great, but it's just not always technically feasible or
cost effective.

~~~
mikekchar
Yeah, you can sometimes build an adaptor to expose the old API on top of your
new API, but other times you have to actually maintain 2 parallel
implementations. It's always a hard choice. Or at least it should be if you
are thinking it through all the way :-)

A lot of times I find that developers lack experience of how legacy
applications get the way they do. Either they have only ever worked on green
field applications before (a surprisingly large contingent!) or they assume
that the crappy legacy code was created because the previous developers were
stupid. The reality is that most legacy systems were made with very reasonable
decisions at the time, but that the world shifted under them. The mistake was
locking themselves into their original decision and making it very expensive
to change. The decision to never deprecate old APIs is one of those decisions
that makes change expensive. Such decisions should always be made looking at
the trade offs in cost. How expensive would it be if we changed this API? How
expensive would it be if we _don 't_ change this API? Usually the second is
_far_ more expensive in the long run, but you have to take it on a case by
case basis.

------
siliconc0w
Maybe the docs are binary and code aware, so the doc system can pull up the
binary's code in source control and find all the possible doc strings.
Developers can't add/change flags without also updating these.

Maybe the docs are history/execution aware so they can list the top N ways the
binary is executed.

Maybe the docs are also team aware so they list the top N ways the binaries
have been executed by your team.

So you say, "Here at CoolTechCompany, we built a tool to help control our
widgets. Please see the `cli widget` command. And everything kinda works
automatically.

------
gtirloni
Reminded me of Infobot and the countless knowledgebase bots it inspired.

[https://en.wikipedia.org/wiki/Infobot](https://en.wikipedia.org/wiki/Infobot)

------
farazzz
I feel like searching for the solution may become time-consuming because you
have to search the right set of keywords to find the answer, which may or may
not be an intuitive set

------
BerislavLopac
There is also Stack Overflow for Enterprise:
[https://stackoverflow.com/enterprise](https://stackoverflow.com/enterprise)

~~~
TarpitCarnivore
This kind of falls into the same problem as wikis though. It requires people
to A. remember it's there and B. be actively monitoring it for questions. We
have one at my location and I constantly forget it's a thing.

With something like this tool it puts it right in a primary communication
tool.

------
jacobsimon
This doesn't seem like a solution to me but rather a symptom of a much deeper
problem in the way that we write and document code.

------
chatmasta
Why not just add a comment system to the doc page?

