
Best Papers vs. Top Cited Papers in Computer Science - fbeeper
http://arnetminer.org/conferencebestpapers
======
sqrt17
As an NLP researcher, I see lots of NLP papers on top of AAAI. I think that
the high citation count is due to

(i) people in NLP actually bothering to tell others why their approach is
interesting

(ii) other people being interested in the same / a similar kind of thing
[avoiding the discipline-of-one problem that niche AI applications would have]
and

(iii) NLP having a reasonably developed "canon" about what counts as must-cite
papers. This canon is heavily biased towards US work, and towards people who
write decent explanations of what they do, but at least it makes sure that
people know about the big problems and failed (or not-quite-failed-as-badly-
as-the-others) solutions.

What you see in other conferences is that the "Best paper" awards get to (i)
more theoretical papers which still have issues to solve before people can use
the approach (nothing wrong with those!), in (ii) subfields that are currently
"hot". Whereas the most-cited papers are (i) more obviously about things that
a dedicated person could apply in practice, and (ii) in a subfield that is
obscure at the time but will become more popular in the following years.

------
jldugger
Reviewing SIGMOD, it appears that a lot of the citations earned are less about
innovative research, and more about the everyone using the software tools they
published.

And a survey paper in the field of big data analysis (survey papers are
citation bate, but won't be pulling in many grants or awards).

~~~
dagw
Once a paper gets big enough you basically have to cite it any time you touch
on vaguely related just to prove that you are aware of it, almost as a
shibboleth. My wife is an academic and on more than one occasion has she
gotten comments back from peer review along the lines of "good article, but
you why haven't you cited famous papers X,Y and Z".

~~~
edraferi
I really appreciate this practice. It makes it much easier to get up to speed
on something new.

------
ep103
Is it possible that papers that get the awards help give the scientists new
ways of looking at problems, while the papers that are frequently cited are
more likely to follow established viewpoints and back it with hard data I can
use to justify later experiments?

What I mean is, if a paper makes me think "wow, I've never though of this that
way before, I wonder if I could try something like that with this...." I
probably wouldn't cite it, right? Its not directly related. But I would
probably give it an award for best paper because it helped me come up with a
new approach to my own problem.

disclaimer: I am not a scientist.

~~~
physPop
Yes, this kind of momentum is certainly a factor.

Another issue is that "best paper" competitions tend to attract younger
scientists who don't have the same "star power" to generate citations.

Star power in science is nothing to scoff at either. If Einstein was still
about and writing papers you can be sure everyone and their grandmother would
cite them, even if the papers were useless. Now, while that is an extreme
example, many fields have their own "stars" that are credited with making
significant gains in a particular area.

~~~
raphman_
_> Another issue is that "best paper" competitions tend to attract younger
scientists who don't have the same "star power" to generate citations._

At all CS conferences I know of, 'best papers' are selected from all accepted
submissions without the authors having a say.

------
thomasahle
You could also see this as 'citation counting' not being a very precise
measure of paper quality/importance.

~~~
raphman_
This is certainly the case. On the other hand, "best papers" are not
necessarily really the most important papers. Often politics, e.g.,
proportional representation of all important sub-fields among the best paper
nominees, decide about whether a paper gets named "best" or not.

Anecdotally, I know of awesome papers that were not selected as "best paper"
and 'cute idea' papers without substance that got a "best paper" award.

~~~
seanmcdirmid
Or papers given the most 10 year influential award that aren't influential.
It's all relative!

------
Bill_Dimm
Nitpicking, but why are they claiming to provide MAP (mean average precision)
scores when their description and equation indicates that they are computing
average precision, not MAP. According to the definition of MAP [1] that they
link to, MAP is computed across multiple queries while average precision is
computed for one [2]. Furthermore, they truncate their calculation to only
consider the top 3 cited papers (i.e., they don't go all the way to 100%
recall), so it's not even really the average precision.

[1]
[http://en.wikipedia.org/wiki/Information_retrieval#Mean_aver...](http://en.wikipedia.org/wiki/Information_retrieval#Mean_average_precision)

[2]
[http://en.wikipedia.org/wiki/Information_retrieval#Average_p...](http://en.wikipedia.org/wiki/Information_retrieval#Average_precision)

------
psuter
Conference organizers are well aware that best paper awards are not perfect
predictors of importance or popularity. Many top conferences specifically
introduced separate awards ("most influential", "test of time", etc.) granted
e.g. 10 years after publication.

~~~
not_that_noob
Often times, the truly revolutionary ideas (and most likely to be cited in
future papers) are those that are not understood fully or perhaps even
reviled, and thus likely to not be awarded any significant honor on
introduction.

Einstien won his Nobel for work on the photoelectric effect, not relativity,
though obviously the latter mattered more.

~~~
privong
> Einstien won his Nobel for work on the photoelectric effect, not relativity,
> though obviously the latter mattered more.

The photoelectric effect led to quantum mechanics and the associated science
and technology. I don't think it's obvious that relativity mattered more. If
you judge by day-to-day use, one could plausibly argue that our understanding
of quantum mechanics has a bigger impact.

~~~
ep103
My high school science teacher told us that he was given the prize for the
photoelectric effect because that was a well understood mechanism by the time
of the awarding. It was generally understood that relativity was going to be a
bigger deal (concept wise), but since it was still so hotly debated / not yet
thoroughly accepted, the commission wanted to ensure that he was given a prize
before death (the award is not given post-humorously for some reason) and
given for something that wouldn't be later rejected.

~~~
Retric
"In 1905 Albert Einstein published a paper that explained experimental data
from the photoelectric effect as being the result of light energy being
carried in discrete quantized packets. This discovery led to the quantum
revolution."

So, he basically described photons as a particle for the first time. That's
also ridiculously huge.
[http://en.wikipedia.org/wiki/Photoelectric_effect](http://en.wikipedia.org/wiki/Photoelectric_effect)
Considering they had been generally thought of as a wave before that.
([http://en.wikipedia.org/wiki/Maxwell%27s_equations](http://en.wikipedia.org/wiki/Maxwell%27s_equations))
Setting the stage for a lot of wave / particle duality aka QM theory.

~~~
Someone
You are forgetting Sir Isaac :-)

I don't think he used the word photon, but he did describe light as particles,
and had (flawed, but that is science) arguments for that
([http://www.studyphysics.ca/newnotes/20/unit04_light/chp1719_...](http://www.studyphysics.ca/newnotes/20/unit04_light/chp1719_light/lesson57.htm))

------
raphman_
In 2009, Bartneck et al. did a scientometric analysis of papers presented at
CHI - the most prestigious academic HCI conference [1]:

 _" The papers acknowledged by the best paper award committee were not cited
more often than a random sample of papers from the same years."_

[1]
[http://www.bartneck.de/publications/2009/scientometricAnalys...](http://www.bartneck.de/publications/2009/scientometricAnalysisOfTheCHI/)

~~~
seanmcdirmid
CHI has a huge number of papers though, and consequently a large number of
best papers...so it is difficult for anything to stand out a priori.

------
j2kun
I think it's far more interesting to just see what the top cited papers are
every year (after the fact) than to compare with the best paper. Best paper
awards are given for a lot of reasons that aren't consistent across
conferences or even across years of the same conference.

------
eksith
A cursory browse shows an interesting pattern in the names of the researchers.

edit_ Perhaps I should clarify: It's entirely possible that exposure in the
West has a large part to do with the media, who often don't wade too deeply
into scientific matters.

------
zenciadam
Citation are just a game and are thoroughly gamed.

