
Research Debt - wwilson
http://distill.pub/2017/research-debt/
======
beambot
A lot of this seems to be a problem with academic publishing, peer review, and
the pedantry contained therein.

I tried (twice!) to publish a paper that was scientifically sound, but written
such that it could be understood by a lay audience. It was rejected; quoting a
reviewer: "This paper lacks math." That sentiment made me lose a lot of faith
in academia. Now, I simply refuse to contribute any more time to writing
papers or performing peer review (esp. in non-open publications). I'm sure I'm
not alone; I know of at least one "seminal" robotics paper that was rejected
from top venues multiple times for "simplicity" (lack of mathematics content)
that went on to become a foundational paper in the field after appearing in a
lower-tier venue years later.

The irony: it takes researchers a lot of time to make a paper dense & concise.
If they "showed all steps", it would probably improve researcher productivity
& make the material more approachable to newcomers. Instead publishers enforce
length restrictions... for which authors dedicate upto 25% to related work
(some of which is useful; much of which is pandering to likely peer reviewers
in small, niche fields). Length restrictions seem equally foolish in the age
of digital publishing. And again, inadvertent pedantry is the only explanation
I can imagine... but happy to be wrong.

~~~
PeterisP
This may be intentional - for most academic publishing venues, their target
audience is other researchers of that domain. It's counterproductive to
optimize an article for a lay audience if everybody in the actual audience has
(or is working on) a PhD in that domain and is generally expected to be (or
become) up to speed with the terminology and earlier results on that topic.
The journal or conference is generally not meant to communicate their results
to wider audience, it's generally a tool that some particular research
community made for themselves to help their research by an exchange of ideas
and results. An article in academic publishing is meant for someone who will
use that to further their own research in a related field - and the needs of a
person like that are very far from a lay audience, they will want to see
entirely different things in that paper - you want the paper to focus on the
novelty, on the "delta" between this result and what was bleeding edge a few
months ago; not spend two thirds of the paper describing what was already
known before.

There are publishing venues intended for a lay audience, but most academic
publishing is not, they have incompatible goals.

~~~
sdenton4
Unfortunately, the 'lay' audience is often not all that 'lay.' For example,
you might be working on a phd in an adjacent field to a given paper and still
have a hard time coping with the sub-field-specific jargon. This happens all
the time in math research, and contributes heavily to silo'ing of subfields.
This silo'ing is actually super counter productive, and can lead to the same
things being independently rediscovered, and important, useful discoveries
simply failing to be communicated across silo boundaries.

~~~
neutronicus
One thing I've learned is that someone in that position should be looking at
PhD theses before journal papers. They can be long enough that they function
much better as introductions.

~~~
fjdlwlv
It's a sad industry that has enshrined culture that the more
experienced/senior a person is, the lower quality their work is, because they
have to make it artificially inaccessible to make themselves appear more
elite.

------
btown
"What is the role of human scientists in an age when the frontiers of
scientific inquiry have moved beyond the comprehension of humans?"

The above quote is from Ted Chiang's short story "The Evolution of Human
Science," originally published in _Nature_ as "Catching crumbs from the table"
[0]. It's a brilliant depiction of this very problem: when new developments
contribute to an _increasing_ gap between those who can _make_ new
developments, and those attempting to understand the state of the art, the
entire process of scientific inquiry becomes less efficient. In fact, the
scenario depicted is one where the majority of researchers become
"distillers," to use the language of the original post.

While Chiang posits a science-fiction reason for the divide, "normal"
research/technical debt is insidious as well. Without incentives to reduce
debt, the knowledge gap widens until only a handful of experts can make
significant contributions. It's a problem that needs to be tackled head-on in
both research and engineering. I'd love to see more initiatives like Distill.

[0]
[http://www.nature.com/nature/journal/v405/n6786/full/405517a...](http://www.nature.com/nature/journal/v405/n6786/full/405517a0.html)
\- a highly recommended companion piece to the original post.

~~~
cs702
There's another subtle aspect to this: the same or very similar ideas,
methods, and tools show up or are reinvented again and again, under different
guises, in different disciplines and subfields that have their own jargon,
unnecessarily making human comprehension even harder.

Tibshirani's "glossary" of ML and Statistics terms is a canonical example:
[http://statweb.stanford.edu/~tibs/stat315a/glossary.pdf](http://statweb.stanford.edu/~tibs/stat315a/glossary.pdf)

~~~
dredmorbius
_If you look ahead to a future age, and consider the state of literature after
the printing press, which never rests, has filled huge buildings with books,
you will find again a twofold division of labor. Some will not do very much
reading, but will instead devote themselves to investigations which will be
new, or which they will believe to be new (for if we are even now ignorant of
a part of what is contained in so many volumes published in all sorts of
languages, they will know still less of what is contained in those same books,
augmented as they will be by a hundred—a thousand—times as many more). The
others, day laborers incapable of producing anything of their own, will be
busy night and day leafing through these books, taking out of them fragments
they consider worthy of being collected and preserved._

\-- Denis Diderot, 1755

[https://ello.co/dredmorbius/post/ayavb8clhm61natfm5lfca](https://ello.co/dredmorbius/post/ayavb8clhm61natfm5lfca)

[http://www.historyofinformation.com/expanded.php?id=2877](http://www.historyofinformation.com/expanded.php?id=2877)

(The History of Information is an absolute treasure of a website, BTW.)

~~~
btown
_" If the government participates in such an opus, it will not get done. It
should use its influence only to favor its execution. With a single word, a
monarch can make a palace arise from the grass; but a society of men of
letters is not like a herd of manual laborers. An encyclopedia cannot be
ordered up. It is a labor that must be doggedly pursued rather than launched
energetically... The more abstract the topic, the more we must strive to make
it accessible to all readers... Such are the principal thoughts that came to
my mind with regard to the project of a universal and analytical dictionary of
human knowledge; its possibility, its purpose, its materials, the general and
particular ordering of those materials, the style, method, references,
nomenclature, the manuscript, the authors, the censors, the editors, and the
typographer."_

The translated full text [0] of that Diderot quote is pretty incredible. Two
centuries prior to Wikipedia, he predicted many of the benefits and challenges
of such an endeavor. I can only imagine his reaction if he were to see
Wikipedia today; I imagine it would be something like [1].

[0] [http://quod.lib.umich.edu/cgi/t/text/text-
idx?c=did;view=tex...](http://quod.lib.umich.edu/cgi/t/text/text-
idx?c=did;view=text;rgn=main;idno=did2222.0000.004)

[1]
[https://www.youtube.com/watch?v=ubTJI_UphPk](https://www.youtube.com/watch?v=ubTJI_UphPk)

~~~
dredmorbius
Thanks for that. I'd only run across the fragment from which I'd quoted above,
and even that is tremendously informed,sightful, and illuminating.

I've been digging into the history of information and communication (and only
just scratched the surface of THoI website), including the explosion and
dynamics of information as it spread across Europe.

There's a paper which looks into the total number of _books_ (volumes in
total, not simply titles), from about 600 AD through 1800. That grows from
perhaps 30,000 volumes -- about 1 per million inhabitants -- to nearly 1
billion in 1800. The 15th century, with the invention of Gutenberg's press is
an absolute watershed, and the consequences, social, religious, political,
scientific, are immense.

I'm starting to wonder just what the hell it is we're unleashing with the
Internet and in particular the Internet in your Pocket, across the world.

Not all the previous developments have been postive.

------
cs702
Yes. Yes. Yes. A million times yes.

I can't count how many times I've invested meaningful time and effort to grok
the key ideas and intuition of a new AI/DL/ML paper, only to feel that those
ideas and intuitions could have been explained much better, less formally,
with a couple of napkin diagrams.

Alas, authors normally have no incentive (or time, for that matter!) to
publish nice concise explanations of their intuitions with easy-to-follow
diagrams and clear notation... so the mountain of research debt continues to
grow _to the detriment of everyone_.

I LOVE what Olah, Carter et al are trying to do here.

------
akyu
I really love this effort. Research papers are low bandwidth way to get
information into our brains. They take a lot of effort to read, even if the
ideas are not particularly complicated. Often when reading complicated
material, I have to come up with metaphors in my mind to make sense of it.
This is somewhat of a wasted effort, as the author who wrote the material
surely had metaphors for their own mind when writing, but too often they don't
share these metaphors, and stick to purely technical writing. I think this is
one reason why ideas like general relativity are so popular, even though the
material is actually quite complicated. The average educated person can give a
reasonable explanation of general relativity because the metaphors used to
explain it are so powerful, even though its very unlikely they understand any
of the math involved.

~~~
sdenton4
I've often thought it would be great for the Arxiv to make it easy to link to
a video of the 'talk' that generally goes with a paper. The talk is very often
the distillation of the ideas in the paper, as conducted by one of the
involved researchers. Indeed, one of the main points of conferences is to
allow us to trade these distilled versions of our research with one another
and place them in the context of everything else going on...

~~~
jessriedel
> I've often thought it would be great for the Arxiv to make it easy to link
> to a video of the 'talk' that generally goes with a paper.

This has been in the idea pipeline for a while, but the arXiv is slow-moving.
(They probably wouldn't host the actual video files for cost reasons, but
that's not a big deal.)

[http://blog.jessriedel.com/2015/09/02/additional-material-
on...](http://blog.jessriedel.com/2015/09/02/additional-material-on-the-
arxiv/)

------
pcrh
I like the ideas put forth in this article. I wonder, though, if
"distillation" is a re-casting of "scholarship" as considered by the
humanities.

People studying topics ranging from Biblical Studies to History to Literature
often do not create new source material, unlike in STEM. Yet there is a large
degree of effort taken to "distill" existing facts through new lenses,
producing novel concepts and interpretations. These efforts can transform our
understanding of many areas of human endeavour.

~~~
colah3
That's a nice connection -- it does have a similar flavor to that kind of
humanities scholarship. :)

------
adamsea
One thing I believe to be of great value which is not made explicit in this
article (which I think is an awesome article), is that research debt, as they
describe it, is basically _education_.

In other words, improve the educational resources for complex subjects.

In our age where we're blessed with cheap printing of books and the
possibility of creating complex interactive media, I think the question of
designing user-friendly, powerful, and beautiful educational resources is a
huge opportunity and pressing question.

Not just for people seeking to achieve a research-level understanding of a
complex subject, but for all subjects and all people.

Consider the social value of beautiful, well-designed and nontrivial
educational material for mathematics or basic science being widely available
for all classes of people at all ages.

I'd argue that when news organizations use infographics or interactive
journalism at its best, they are also performing this educational function.

Sorry for the long post, but to summarize, I think it's useful to recognize
research debt as a specific case of the art and practice of creating media for
education.

------
dluan
There are 'distillers' of large bodies of scientific research. Traditionally,
they are science communicators, and more specifically science journalists.

The goal of a practicing scientist is very much at odds with someone whose job
is to translate science into larger audiences. I've had very well-intentioned
rational research scientists tell me with a straight face that "my job is to
produce science results, not to communicate it. that's someone else's job",
usually with the attitude that it's less respected or somehow self-
aggrandizing. "The best science will be self-evident" attitude that all
researchers secretly aspire for, not realizing that 99% of impactful science
has had effort spent to promote, frame, or distribute it.

This weird stereotype is somehow beaten into scientists from the very
beginning, and I haven't been able to figure out where this comes from.
Obviously, yes, it's a lack of tools and accessibility into letting scientists
also become distillers themselves. But the motivations and incentives at the
center of the whole system is what's making this whole imbalance. I think
there are parts of our research system that actually say "No, you cannot and
should not distill your science".

Ultimately, for me, it gets back to funding. If review articles and outreach
weighed just as much as citation count in tenure and grant committees, then
maybe this could start to change. Yet, these committees still don't value open
access, and look how tough that battle has been.

Also - this solution is really great and commendable, but I don't see how this
works outside of ML/CS where research outputs are more like software
development - gists, snippets, prototypes that are immediately shared, pushed,
forked. More science fields like ecology, synthetic biology, anthropology,
will look like this, but it will take a few human generations.

~~~
bloaf
Science communicators/journalists can turn "Methodological observation of the
sociometrical behavior tendencies of prematurated isolates indicates that a
casual relationship exists between groundward tropism and lachrimatory, or
'crying,' behavior forms." into "Scientists find that falling down makes
babies cry" but they're less good at expressing concepts like " _math
equation_ forms the basis for all current modeling, papers A, B, and C each
use a different special case of the equation to reach their conclusions."

~~~
andai
Right. “Everything should be made as simple as possible, but no simpler.” The
best they can hope to do is extract the findings and de-obfuscate them.

------
tominous
Isaac Asimov anticipated the idea of a research distiller in "The Dead Past"
with the character of Nimmo, a professional science writer:

"Nimmo received his first assignment at the age of twenty-five, after he had
completed his apprenticeship and been out in the field for less than three
months. It came in the shape of a clotted manuscript whose language would
impart no glimmering of understanding to any reader, however qualified,
without careful study and some inspired guesswork. Nimmo took it apart and put
it together again (after five long and exasperating interviews with the
authors, who were biophysicists), making the language taut and meaningful and
smoothing the style to a pleasant gloss."

In the story Nimmo has less prestige than a "real researcher" but the role
pays well and he is in high demand.

------
cing
I don't see anything wrong with the "textbook" -> "review" -> "article"
strategy to climbing a mountain of debt. A good review paper should hit a
sweet spot in terms of exposition, digestion, abstraction, and noise
filtering. Transformative new ways of visual thinking, well, that's a
different story.

The problem is just that the pace of research in the authors field is too fast
at the moment. What's the hurry? Over time, the citation graph will reveal the
most significant work, and the community will naturally distill that research
for maximum effect. The danger is that beautiful distillation of an extremely
"niche topic" will not change the fact that it has limited scope and may even
limit abstraction. Of course, I say danger with tounge-in-cheek...

~~~
colah3
You're right that reviews and textbooks do valuable distillation, but I'd
claim we can do much much better.

I think we can distill things massively better -- create massively better
explanations and ways of thinking about ideas -- if only we invested more in
it. It isn't just _when_ the distillation happens, it's the quality.

That's a really bold claim and all I can do is point to the occasional example
where someone put effort into distilling and did something wonderful. My
intuition is that these aren't one-off miracles, but are the expected result
when the community takes this really seriously.

The problem is that distillation is requires focused energy. Reviews and
textbooks give us a bit, but it's diffuse. I think we can do much better, and
I hope we'll be able to demonstrate that with Distill.

~~~
mncharity
> distillation [] requires focused energy

As well as an underappreciated magnitude of expertise. You might like my
"Scientific expertise is not broadly distributed - an underappreciated
obstacle to creating better content"
[http://www.clarifyscience.info/part/MHjx6](http://www.clarifyscience.info/part/MHjx6)
.

> create massively better explanations and ways of thinking about ideas

Agreed.

My hobby project this month is: _assuming_ it's possible to teach size down to
atoms in early primary school [1], could we then leverage that to do a multi-
scale interdisciplinary "nucleons-up" introduction to atoms and materials.
Current content is _so_ notably incoherent, and leaves both students and
teachers so very steeped in misconceptions, that there seems an opportunity
for supplemental content, if it can be shaped to work.

[1]
[http://www.clarifyscience.info/part/RBigE](http://www.clarifyscience.info/part/RBigE)
"Remembering Sizes 2015" \- the approach seemed promising... the videos
didn't.

------
PaulHoule
There is no research distillation because scientists don't get grants to do
it.

~~~
colah3
And because it doesn't lead to jobs or lots of citations.

Fixing that is basically the goal of Distill: build an ecosystem where this
kind of work is supported and rewarded.

~~~
kd0amg
I worry that these will be citations (and publications) that "don't really
count" in heavily career-impacting decisions. Just getting citations to appear
won't make funding agencies, tenure committees, etc. value distillation itself
-- they will definitely know the difference between a survey paper and a novel
result.

~~~
colah3
Yep. That's why a big part of Distill, behind the scenes, is a campaign of
talking to senior faculty, scientific funders, and leaders of industry
research groups, to build consensus that this is important.

I'm not sure how successful we'll be, but we're taking it seriously.

------
eddotman
Pretty compelling arguments.

I do think that the ML / CS / etc. community is actually more open than other
academic fields, and so this is definitely the right subfield to start in.
Putting open access preprints online is not common practice in all
disciplines, although it really should be.

I wonder if it makes sense for Distill to also publish on fields outside of
pure ML - e.g. as applied to specific problems in other domains. I work in
materials informatics, and I suspect that research in such fields (ML +
applied sciences) might benefit quite a bit from having key results
'distilled' in this format.

~~~
colah3
In the very long run, I'd like to see Distill or Distill-like journals cover
all of math/cs/science.

But I think the right approach is to start with a narrow topic and do really
well there. I guess startup people would say that we're focusing on a single
vertical. :)

We haven't done very good line drawing for journal scope yet, and I'm not sure
how we'll handle cross-disciplinary work.

------
CogitoCogito
Article is really good, but it's reference to the pi vs tau debate is kind of
silly. It really isn't a big deal. When I did my math phd I wrote 2 pi all the
time, but this didn't matter. The tiny convenience gained by changing to tau
is totally trivial and doesn't even deserve a mention compared to the rest of
the things in the article.

~~~
colah3
2 pi doesn't seem like a big deal to you because you've internalized the
notation. You paid the cost a long time ago (and perhaps it was a small one
for you).

I think it can be hard to empathize with what it's like to be a beginner. I
learned about the pi definition debate a few years after I learned
trigonometry -- there were a bunch of essays prior to the tauday one -- and it
seemed tremendously better. I was also still in high school at that point, and
I saw students struggling daily with these ideas and it seemed like part of
the pi definition was the stumbling block.

So, of course the pi vs 2 pi thing looks trivial to a mature mathematician: we
aren't the people paying the cost.

~~~
CogitoCogito
That argument _may_ apply to very young learners. I.e. students in high school
or younger. It certainly does _not_ apply to the people the original article
is talking about (i.e. researchers). Anyone who can't handle the circumference
of a circle being 2 pi has no chance at understanding modern mathematical
research no matter how simplified it becomes. It's not even on the radar.

By the way I simply am not convinced by this 2 pi vs tau debate. Now the area
of a unit circle would become tau / 2\. The original problem doesn't go away
it just changes. The argument can be that you write write tau more than you'd
write tau / 2, but the fundamental difficulty remains. Notation in mathematics
is _extremely_ important, but pi vs tau is so far down the rabbit hole of
diminishing returns, that I just see the debate as a total waste of time (and
that's before I consider the fact that it won't ever change due to
momentum...).

~~~
jessriedel
The fact that you write down factors of 2 less is a weak signal of the
improved notation, but it's not the point. (If you can't see that tau is
_conceptually_ a more fundamental quantity than pi, then you actually don't
understand the point!)

The reason to use an elementary example like pi/tau is that it's accessible to
a wide audience. Of course it's nearly trivial for experts. It wouldn't make
sense for the author to pick a high-level stumbling block that's only
important for a tiny group of experts in one subfield.

~~~
CogitoCogito
> (If you can't see that tau is conceptually a more fundamental quantity than
> pi, then you actually don't understand the point!)

Well then I guess I don't understand the point. That claim is essentially
equivalent saying that circumference is more fundamental than area. You may
accept such a strong claim, but I certainly do not.

~~~
jessriedel
It's not equivalent, but it's connected: circumference is "more fundamental"
than area simply in the sense that 1 dimension is more fundamental than 2. In
particular, when we build things up in higher dimensions, it's much more
clearly done using a 1D primative than a 2D one, e.g., for the Fourier
transform, there are powers of 2pi=Tau in any dimensions.

~~~
nercht12
>> it's much more clearly done using a 1D primative than a 2D one

Or you could just tell people that pi is the edge length of half of a circle.
Pi and Tau are both 1-dimensional numbers that happen to be related to these
things, so the argument really is silly.

The reality is, we're going to have to pay some kind of intellectual debt in
learning _anything_. So we keep teaching Pi. Suppose everyone switched to Tau.
That would make all kinds of older documents inaccessible and harder to grasp,
and we would be paying intellectual debt that way.

~~~
jessriedel
> Or you could just tell people that pi is the edge length of half of a
> circle.

You can certainly tell them that, and they may even find it easier to
remember, but it won't actually be the reason that one convention is better
than another.

> The reality is, we're going to have to pay some kind of...

Whether pi or Tau is preferable a priori is logically distinct from whether it
makes sense to change conventions once one is established.

~~~
nercht12
> You can certainly tell them that, and they may even find it easier to
> remember, but it won't actually be the reason that one convention is better
> than another.

Indeed. But it was arguing against the logic of the parent post, which argued
that it _was_ a reason for one convention being better than the other. Not to
be rude, but please read the parent post before responding to a response. It
tends to make more sense.

> Whether pi or Tau is preferable a priori is logically distinct from whether
> it makes sense to change conventions once one is established.

Agreed. But that's not what I was arguing. I was arguing that we have to pay
some kind of intellectual debt regardless. I could have used a different
example, such as cases where it makes more sense to use pi than tau, and in
such cases, we would be paying some intellectual debt. But please, please try
to see how the idea applies beyond the initial example. Don't just take the
argument at face value.

------
mballantyne
When I've talked to senior researchers about these problems, they say that
they have no problem finding distilled information about new results; they get
it from in-person conversations at conferences that they attend frequently.
The publishing of distilled research would most benefit low-status, newer
researchers (like Ph.D. students), but it needs to be valued by senior
researchers to make it into the incentive systems of hiring and grant funding.
It seems like a tricky problem to fix the incentives here.

------
amelius
I think one of the biggest things missing is a means to communicate openly
about published research. What I'd like to see is a forum for every paper, and
this forum should be properly moderated (perhaps by a peer-review system).

Such a forum could make it much easier to decipher published work, and to fill
in details which were missing. Also, errors in publications become clear more
quickly.

~~~
unboxed_type
Sounds interesting! I would definitely join that forum. This sounds like a new
kind of sci resource with well-defined purpose.

------
haddr
Brilliant thing! As a person pursuing a PhD I'd something that is in my
opinion the best way to avoid research debt: have a good tutor or a good group
(peers). This way you can learn new things (or ask for something you don't
understand) and really get to the peak sooner that doing everything alone.

------
killjoywashere
3blue1brown's Essence of Linear Algebra series on YouTube should get some sort
of honorary inclusion in Distill. There have been many "visualization of
algorithms" posts on HN over the years. A collection of those would be good as
well.

------
erikb
What I think is funny is that so many people complain about how bad science
has become and how the papers and funding institutions enforce a very
ineffective way of doing it. Why is nobody attempting or proposing new ways to
make money with science? E.g. have people tried to do science in a
subscription based model (like artists)? Or using free pappers+consulting on
how to replicate the experiments or apply them in real projects (like open
source)?

------
RangerScience
Ooooh. I've actually been working on a blog post where I try reading a science
paper (knowing only a little about the science) and learn / explain as I go.

This seems like the better way to do it (my way is taking way to many words) -
something like better science reporting.

My concern is the act of writing - Medium has a real nice web editor. What can
I use to write an article using this HTML/CSS/JS without literally writing the
HTML?

~~~
sacheendra
Have you checked out Adrian Colyer's Morning Paper. You might get some insight
from how he does it.

~~~
tedmiston
Looks like an awesome resource

[https://blog.acolyer.org](https://blog.acolyer.org)

------
Drup
As an (aspirant) programming language theorist, we have a really great
advantage in this field: advances in the field of prog lang have a natural
distillation process: getting into a "real" programming language.

And it's great, it means I can toy with features and consider "what could I do
if it was in a real language ?". Also, we can observe idioms and usage that
gets developed when some advanced feature start being used by "normal"
programmers.

Computer science, in general, does have the advantage that the distance to
applications is quite often much shorter than math, which forces part of this
distillation process to proceed a bit quicker. On the other hand, the
formalization is highly non-uniform, due to how young it is as a science.

------
fmap
In my experience this is already being practiced. Maybe it's different in
machine learning, but in type theory simpler explanations are well worth
publishing and are published all the time even in high impact venues.

As far as I can tell, progress towards eliminating "research debt" takes two
forms: on the one hand there is a place for good exposition, typically in the
form of textbooks, and on the other hand concepts get better understood over
time and people come up with simpler explanations. In both cases the results
can already be published...

Is the situation in machine learning really so bad that nobody is going to
publish simpler explanations of known results?

------
intrasight
How about we just abstract to "Intellectual Debt"? A subclass that I've
recently encountered is "Learning Debt". I think all these types of debt can
equally apply to learning - as in formal education. Came to mind while reading
"Undigested Ideas" because I recently had a conversation with my daughter
(sophomore at Yale) that a bad test grade means that you may have Intellectual
Debt that needs to be retired by taking time after the test to make sure you
understand what you did wrong.

------
euske
When I hear things like this, I always think of Richard Feynman. He was simply
a distilling genius to me.

~~~
zevyoura
One of the greats; luckily a lot of it is available on Youtube:
[https://youtu.be/j3mhkYbznBk?list=PLLzGzdSNup63lMYeOpU9Hax6M...](https://youtu.be/j3mhkYbznBk?list=PLLzGzdSNup63lMYeOpU9Hax6MBsTjdDas&t=359)

------
nnonynous
This is nonsense.

Here is a (well-publicized) post which only serves to hype up and stake a
claim on what is already valued and practiced, the exposition of research. The
message seeks to capture an ignorant readership and practitioners with short-
term memory, and have them walk away with the thought "This is the place where
good research expositions will be."

The machine learning learning community has already benefited from the myriad
great expositions provided online for free, in addition to the locations where
even source code is given alongside research findings. I will not link to
these sites, in hopes to not seem a salesman, but if you've taken an interest
in the field and taken some time to search on a topic (for example neural
nets), you will have likely already found one of several free, helpful
resources. This includes online courses, online books, blogs, videos, &c.

These individuals give examples of already well-written expositions going back
years (associating these well-received expositions with their own endeavour),
yet the theme is one of "newness", helped by the usage of terms like
"distillers" and "research debt". In Silicon Valley, it seems that if
something's been given enough press and sounds new, you should at least hop on
the bandwagon for a while, lest you risk missing out on being an "early
adopter".

Publishers are not a new conception either. They're a funnel which selects
what you see. In an era where the larger population is beginning to see how
much power publishers have over what they think, new efforts should strengthen
decentralization. Playing to the tune of publishers has gotten us into the
mess we're in.

Peer review is a mainstay of science, and I am in complete support of it, but
peer review and publishers do not need to coexist. I and others will happily
use a decentralized system with peer review, additionally and crucially
providing transparency.

Good exposition should occur. Good exposition does occur. Indeed, people are
learning: Ask yourself when you last learned from someone's writing. Audiences
get their information from many sources, and if they can't understand those
sources, they don't go to them.

Addendum. I will not discuss at length the repercussions of teaching the
population how to create intelligent systems, but that is a dangerous road for
all of us, and not one easily traversed. Companies and other powerful persons
have a strong interest in guaranteeing they have a large pool of subordinates
who have skills that they can profit greatly from, so it's obvious why the
push for software engineering and artificial intelligence teaching is so
strong (the promise and hype has been strong). But this is heavy-handed and
short-sighted. Businesses have functioned with this approach in the past
however, so I wager they assume past performance can be used for prediction in
this case as well. If anyone's doing any thinking at all.

~~~
neuromantik8086
A normal person might use the term "teacher" or "science communicator" instead
of "distiller". This is, however, from Silicon Valley, so naturally uninformed
perspectives from individuals who don't seem to have spent substantial time
deeply embedded in academia (adjuncting as a side job doesn't count) should be
more valuable than insiders with actual expertise.

------
tomkat0789
My scientific writing was transformed by Rafeal Luna's "Art of Scientific
Storytelling". I'm on my phone, so I can't do it justice here, but I think
that style will be crucial for the distillation TFA is discussing.

------
stuartaxelowen
One take away I got from this is the immense value of great content marketing.

Especially for software, the best content marketing is an immense value to the
community, helping it understand core ideas and adopt new technology more
easily.

------
therealdrag0
In terms of clear communication, I highly recommend "The Sense of Style: The
Thinking Person's Guide to Writing in the 21st Century" by Steven Pinker

------
safeandsound
Perhaps distillers should be the reviewers of top publishers so they can have
more power.

------
jeyoor
Kudos to this team for dogfooding their platform. That choice makes me far
more likely to evaluate and eventually use it.

------
PeterStuer
Unfortunately arcane notations are an effective tool to obfuscate the
shallowness of some ideas.

------
gburt
This is a really awesome perspective.

------
mfsch
I agree 100% that it is way too hard to climb the mountains of knowledge, that
it could be much easier, and that there is little incentive and cultural
support in academia to work on these topics. Time and time again I see
examples of where outcomes are not limited by the existing knowledge, but by
the accessibility of said knowledge. I would gladly devote most of my time to
distilling knowledge if I saw a viable career path for it.

Over the last years, I gave a lot of thought to the topic of conveying
information. I see research as the process in which knowledge is created. This
knowledge then has to be “encoded” in a format that allows for the
transmission to other people. This encoding can be optimized in different
ways. Research articles have the advantage that they are close to “lossless”
in that they are supposed to contain all information necessary to build up
that knowledge. This makes them well suited for archival, especially as they
can be stored as a stack of paper.

However, research articles are often not optimized for building up that
knowledge in an efficient way. I believe that the “encoding” optimized for
learning & understanding should be more like a progressive image codec, in
that it provides a comprehensive view as soon as possible, filling in further
details along the way. This also makes it possible to stop whenever you have
reached the level of detail that is relevant to you. The challenge in creating
these encodings is to extract the information that provides the most clear &
useful picture as soon as possible. I like the word “to distill” for that
process, as it is really about extracting the essence of a body of
information.

Doing this work for research articles is how I understand the goal of
distill.pub, which seems extremely valuable to me. However, I think this is
just the first step. What is the most useful distillation of all of deep
neural networks? Of all of machine learning? Of all of computer science? As
others mentioned, there are some forms of publications (review articles,
textbooks) that do part of this distillation process, but they only cover part
of the spectrum. Preciously few textbook contain a well thought-out summary of
their contents and not just an introduction. In my experience, often the least
amount of thought is given to the highest level of abstraction (e.g. what is
the essence of mathematics?), even though they are the most fundamental ones.

It would be great if there was more focus on extracting useful understanding
from the ocean of knowledge we already have (useful both in the sense of being
applicable on its own as well as being a solid foundation to build more
knowledge on). It looks like distill.pub is a step in that direction, and I
really hope it will bring more attention and recognition to this kind of work.

------
RSchaeffer
Chris, will Distill support community editing of articles like Wikipedia does?

~~~
colah3
Each Distill article is a github repository, so people can make pull requests,
but we mostly encourage people to use that to make corrections.

If it's technically feasible to do community editing of articles, why don't we
encourage that? It's actually a pretty deep question about how you think about
research distillation.

It's tempting to see distillation as putting a layer of polish on ideas --
something that people can just jump into and do. But I think it's much more
about deeply internally digesting ideas, perhaps with a few collaborators, and
putting it together into a framework of thinking.

For that, you want a small number of people with a strong vision and a deep
investment in the article. If other people have a different vision, they can
write a different article.

(My impression is that wikipedia works very well for getting lots of facts
onto a page, but does a much more mediocre job of distilling ideas.)

~~~
bfirsh
Wikipedia seems to be the closest thing we have to broad distillation right
now.

It works pretty well if you are researching a subject in depth. The technical
articles tend to be written by experts for experts and they have lots of links
to primary sources.

But, as you hint at, they are poor quality because they are being written by a
committee.

I would love to see a collection of community written distillation articles,
as you describe. Perhaps with some kind of taxonomy like Wikipedia's.

------
debtor123
This abstraction of debt doesn't really work for me...

~~~
treehau5
Great! Do you mind expanding on why it doesn't work for you with emphasis on
the relevant, objectifiable aspects? (as objectifiable as you can be for an
anecdote, of course!)

~~~
milesvp
I'll take a shot.

What's actually being described in the article is entropy. Entropy counts
always go up, and it takes energy to fight entropy. Or to put it another way,
it takes more effort to produce simple writings, than it does to make
complicated writings (especially for things that appear inherently complex).

This means that debt isn't the right term here, because debt is something that
must be paid back by the individual, or organization that took out the loan.
It also requires a lender, which, if we're talking about technical debt, is
usually future you. In terms of publishing papers, the author isn't borrowing
effort from his own future, he's simply not doing work that he doens't need to
do. The problem the article is talking about is that others have to do work
because the author didn't do work. This is not debt, it's something else (I'm
sure there's a term for it in economics, the closest I can think of would be
some subtle variant of a negative externality).

While I agree that research would be easier if authors spent more time
distilling their ideas, that effort is a public good, and effort that will
largely not have a positive expected ROI for the author, even if there is a
huge ROI from a community collective point of view.

~~~
colah3
I agree that when you think about individual researchers it's a negative
externality -- in many cases, the researcher doesn't pay, but other members of
the research community do.

This is true in programming as well: often the technical debt falls on your
collaborators. You understand that crazy code you wrote and didn't document,
but no one else does.

For the debt perspective to really work, you need to think of the research
community as a team working together to advance science. Individual members of
that team take out debt to advance their work faster, but in the long run it
costs the team more.

