
How to Read a Paper (2016) [pdf] - DecayingOrganic
https://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-reading.pdf
======
TrackerFF
I know there are a lot of ML researchers and practitioners here - and I have
unfortunately only a very shallow experience with reading ML papers, more so
with the recent output.

But, I have to ask, how do you get a feel that the content actually looks
correct, and not just only quack? The improvements are usually in the 1%
range, from old models to new models, and the models are complex. More often
than not also lacking code, implementation / experiment procedures, etc.

Basically, I have no idea if the paper is reproducible, if the results are
cherry picked from hundreds / thousands of runs, if the paper is just cleverly
disguised BS with pumped up numbers to get grants, and so on.

As it is right now, I can only rely on expert assurance from those that peer
review these papers - but even then, in the back of my mind, I'm wondering if
they've had time to rigorously review a paper. The output of ML / AI papers
these days is staggering, and the systems are so complex that I'd be impressed
if some single post. doc or researcher would have time to reproduce results.

~~~
ssivark
> The improvements are usually in the 1% range, from old models to new models,
> and the models are complex. More often than not also lacking code,
> implementation / experiment procedures, etc. [...] no idea if the paper is
> reproducible, if the results are cherry picked from hundreds / thousands of
> runs, if the paper is just cleverly disguised BS with pumped up numbers to
> get grants, and so on.

Personally, I'm bearish about most deep learning papers for this reason.

I'm not driven by a particular task/problem, so when I'm reading ML papers, it
is primarily for new insights and ideas. Correspondingly, I prefer to read
papers which have new perspectives on the problem (irrespective of whether
they achieve SOTA performance). From what I've seen, most of the interesting
(to me) ideas come from slightly adjacent fields. I care far more about
interesting & elegant ideas, and benchmarks to just sanity-check that the nice
idea can also be made to work in practice.

As for the obsession with benchmark numbers, I can only quote Mark Twain:
_“Most people use statistics like a drunk man uses a lamppost; more for
support than illumination.”_

~~~
bonoboTP
(Not a Twain quote: [https://quoteinvestigator.com/2014/01/15/stats-
drunk/](https://quoteinvestigator.com/2014/01/15/stats-drunk/) )

------
tjchear
Can we apply the steps to reading source code?

Here's my take on adopting it to reading code:

1\. Read readme if available, read the list of source files to get a sense of
how the project is modularized. Identify the entry point. Identify type of
program from main entry point: is it a server, a CLI, or a graphical app?

2\. Run call graph analysis tool if you have it, so you can study callgraph
tree starting from main entry point. Read just the function names and start
making notes of how the execution works at various levels, e.g does it read
input then enter an infinite loop, does it wait on network packets, does it
use update/render loop, etc. Also make note of whether a function is
trivial/non-trivial based on quick glance at the code.

3\. Ignore the trivial ones, and read the non-trivial ones in detail. Make
note of the algorithm, data structures, and dependencies.

------
analog31
My spouse has a rule for science papers: First, look at the pictures. She
figures that people will put a lot of effort into their graphs and diagrams
telling a good story.

~~~
haecceity
Rats! I only use xkcd style plots.

~~~
jimbokun
I think that's likely to get you a lot of readers.

------
puhi
How do i find all the interesting papers?

Like i like to read about things like: ML, Scaling, Filesystem, Databases,
Algorithms etc.

I do get a lot of input through hn, friends, youtube, blogs but i'm not
getting my papers from direct sources. I don't have anything like nature or so
laying around either.

~~~
Mvhsz
Try to find a survey paper or review article [1] on the field you're
interested in. They summarize the current state of the art in a field and link
to the relevant papers. If the linked papers are behind a pay wall then you
can use arXiv as recommended in another comment by searching the
title/authors. The field usually wouldn't be as broad as "databases" but you
could probably find one on "distributed wide column stores". I think they're
usually published by grad students before they pick their thesis topic.

[1]
[https://en.m.wikipedia.org/wiki/Review_article](https://en.m.wikipedia.org/wiki/Review_article)

~~~
frayesto
You can also go here for almost all papers

[https://whereisscihub.now.sh/go](https://whereisscihub.now.sh/go)

------
wodenokoto
Is there a text that will explain the difference between a paper, an article,
a manuscript, a monograph and all the other words often used to describe
different kinds of written scientific material?

~~~
inimino
A dictionary? ;-)

But seriously, if you want to know the differences look up the etymologies and
then keep your eyes open for how each term is used in practice. That's all
there is to it.

~~~
wodenokoto
My dictionary defines manuscript as something written by hand rather than
typed.

Not exactly something that illuminates the usage in scientific literature

~~~
inimino
Doesn't it though? If you look at the etymology, manus (hand) + script (write)
directly gives you "hand-written", and then a hand-written document is more
likely to be the work of a single author, more likely to refer to an original
than a copy, and more likely to be old (before printing, typewriters, or
computers). Everything else is just associations that a word picks up over
time or in a specific community.

------
gchamonlive
This reminds me of How to Read a Book[1], which is also a great read.

[1]
[https://en.m.wikipedia.org/wiki/How_to_Read_a_Book](https://en.m.wikipedia.org/wiki/How_to_Read_a_Book)

------
m3kw9
I tried to read this paper using his approach but I didn’t know where to
start..

~~~
ajkelly451
Came here to say the same thing, you beat me to it!

~~~
m3kw9
This is like a level 1 joke, there is an easy established pattern to it, kind
of slap stick

------
Noxmiles
I want to know the content of the paper before reading the paper. The struggle
is real...

~~~
ooklala
Read the abstract?

------
hprotagonist
There's a 2016 updated version here :
[https://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/...](https://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-
reading.pdf) Should the link update?

