
Debunking Princeton - friggeri
https://www.facebook.com/notes/mike-develin/debunking-princeton/10151947421191849
======
hooande
I hope this becomes a bigger story than the original. I can't tell you how
many times someone has thrown a junk science article in my face, thinking that
the issue in question was now settled. A survey with a small sample size,
outsized extrapolations and numbers that don't match the accompanying
conjecture.

There should be consequences for the people who publish these things. People
have a tendency to believe anything that someone in a lab coat says,
especially if it supports their point of view or anecdotal experience. In many
cases the people who do the research present it with few qualifications while
not standing behind assumed implications. If someone publishes sensational and
link baity findings they should say, unequivocally, "I'm willing to stake my
reputation on the idea this trend is real and will continue" or "These are
just data and I'm not willing to say that they have any bearing on reality".

Facebook may not have been right to dignify the initial post with a response,
but I hope it works for the best. They say that some attention is better than
no attention at all. It's important that this applies to self promotion and
persona creation and _not_ science. If somebody has something crazy to say,
they should start a personal blog. Those who want to intentionally attract
media attention should present themselves as such, instead of pretending to be
doing any kind of meaningful experiment and hypothesis testing.

~~~
throwaway_yy2Di
" _There should be consequences for the people who publish these things._ "

Many who are published deserve obscurity. And some who have obscurity deserve
more funding. Can you give it to them?

Do not be so eager to deal out censorship; for even institutional review
boards cannot see all ends.

~~~
deevus
Yes, Professor Gandalf.

------
devindotcom
Seems like Facebook is responding to the media interpretation of the Princeton
paper rather than the paper itself. I saw no problems with the paper: it
applied an epidemiologically-inspired statistical technique to Facebook using
a defunct precursor as exemplar. The limitations of this technique are
obvious, but it's an interesting idea.

People do "research" like this all the time - you throw shit at the wall and
see what sticks. Most of the stuff that ends up on the floor never receives
any attention at all, so you don't hear about it on first-tier news sites. But
when it's about Facebook, it goes viral, and suddenly is the subject of
intense scrutiny. They didn't bring this to the UN for a call to action. They
didn't start a company around it. They just applied an idea to some data and
wrote it up. And now the entire Internet is making fun of some exaggerated
version of their idea, summarized by Huffington Post hit-mongers.

Personally, I applaud these guys for putting in the work to test out a theory.
If it's not correct, it will go in the bin with the other ten million papers
with flawed theories, premises, methods, or other aspects that have been
published in the last day or two.

~~~
chaz
The problem is less about the technique, but about the flawed assumptions
about the data that they used, and how they assumed that it was a valid basis
for a paper. That undermines the whole thing, no matter the rigor of the
analysis.

The medium matters, too. I'm all for people discussing crazy ideas and seeing
what sticks, but when they wrote a paper about it and submitted it to
arXiv.org ("Submissions to arXiv should conform to Cornell University academic
standards."), they staked their reputation. Why wasn't this just a blog post?

~~~
bhouston
I think that discussing this in academia is worthy. One paper can start a
discussion, another paper can improve on it or disprove it and so forth. I
think that putting out a paper with a controversial conclusion is not wrong,
it is merely provocative. I hope there is more study on this and that it is
serious.

I also do not think they are completely wrong. I think there is general
consensus that in North America we have likely reached peak Facebook
engagement -- unless there is a game changer. The question is fairly open at
this point in time as to whether it stays steady, decreases slowly or fast
from this point.

~~~
chaz
It's worth exploring and discussing, and if academia wants to get involved,
their contributions are welcome. I just think that they shouldn't be surprised
when the lede is about "Princeton researchers" by mass media.

> I think there is general consensus that in North America we have likely
> reached peak Facebook engagement

I disagree. There are signs that US teens are decreasing engagement or maybe
not even registering: [http://istrategylabs.com/2014/01/3-million-teens-leave-
faceb...](http://istrategylabs.com/2014/01/3-million-teens-leave-facebook-
in-3-years-the-2014-facebook-demographic-report/). However, other age groups
are apparently doing well, and there's no sign that 80% of users will leave by
2017.

On the whole, they continue to grow. The question is whether the teens are an
early indicator that will apply to all demos/geos and that they will all
decline, or if there's maturation in the product and that it better fits an
older demo and the teens will grow into FB in later years. Or maybe something
else.

------
crm416
As a Princeton student, it's been pretty frustrating to watch this devolve
into (drawing from comments on this page) a "sham study by Princeton", with
this response being a criticism of "Princeton's methodology", and that
"Princeton was deserving of a response like this".

It's been pointed out by a few other HNers, but this type of logic does a
massive injustice and disservice to all the institution's undergraduate and
graduate students, as well as its professors, who work hard to produce some of
the highest-quality research in the world.

To say that this is a "Princeton study" is to present this as if it were
endorsed or produced by the administration or some department or even a
tenured professor. Instead, let's remind ourselves that this was a pre-peer
reviewed paper posted on _arXiv_ by two PhD students (who have likely been at
the university for a few years, tops). To paint this as the Princeton
community getting together as a collective and putting forth their best
attempt to "debunk" Facebook is just hilariously unfair.

Look, there's a thick anti-higher education slant on HN. People love
referencing the higher education bubble and the 'demise' of the current
university system or whatnot. But it'd be nice if we could keep things in
perspective here and at least do better than the media, who can't wait to
pounce on a Princeton vs. Facebook feud.

~~~
aristus
Let's try a thought experiment. What if two Princeton students published an
un-peer-reviewed paper on arXiv that said something useful, like revealing the
origins of life, or the perfect cookie recipe? Would you argue quite as hard
about how unfair it all is to call it a "Princeton study?" Or would you smile
and take the reflected credit?

Note that all the press now is about how "Facebook" responded to the study. It
was actually a single person, Mike, who posted a note to his personal wall
with some help from friends.

Being a member of an institution means you represent the institution in all
your actions. That's what being a member of an institution _means_.

~~~
judk
Mike is employed by Facebook as an audience researcher, which makes him more
of an official representative of his institution (in regard to the topic at
hand) than a mechanical engineering student is of theirs.

------
btown
What drove Cannarella and Spechler, two mechanical engineering Ph.D. students,
to prerelease a paper about network theory and epidemiology, without
coauthoring/consulting with epidemiologists or people experienced in viral
communications theories, is beyond me. But the title "Debunking Princeton"
seems to suggest this should be considered representative of all of the
quality of research Princeton outputs, which is certainly not the case. Many
posts have been made questioning the strength of the paper in question, and
the fact that one with such a link-bait title is rising on HN is unfortunate
in my view.

Once again, a reminder to everyone that there are no peer review requirements
for papers posted to arXiv. There is no evidence that the original paper was
ever accepted by any journal or conference, and not surprisingly given the
speculative nature of the study, the advisor of the two Ph.D.-candidate
coauthors declined to place his name on the paper. So as an institution,
Princeton is no more responsible for this paper than Obama is responsible for
that drink machine being broken down the hallway (thanks Obama), even though
it happens to be a drink machine affiliated with his country. In that regard,
with all due respect towards Mr. Develin, I'm going to have to "debunk"
"Debunking Princeton."

[Full disclosure: I am a Princeton alumnus.]

~~~
lbrandy
Is this a clever meta-troll? Or are you actually taking the title seriously?

It should be obvious that the title itself is part of the self-parody. The
whole thing is a giant exercise in reductio ad absurdum. In fact, the 'joke'
only works based on Princeton's reputation as an excellent university, if that
makes you feel any better.

[Full disclosure: I am a facebook employee]

~~~
001sky
TL;DR two doses of stupid aren't the way forward.

------
tokenadult
A suitably joking response for a "study" that didn't need to be taken too
seriously (and wasn't taken very seriously by most media outlets, including
the one that led a story with it). As we discussed yesterday here on Hacker
News[1], the study methodology was not sufficiently validated to convince most
people that Facebook will massively lose users, even if all the data were
correct. More likely, the study's model was just flat wrong.

[1]
[https://news.ycombinator.com/item?id=7104904](https://news.ycombinator.com/item?id=7104904)

~~~
hatred
Some people on the Wall Street did take it up a bit seriously perhaps going by
the price movement today.

~~~
poopsintub
My investment account seems to disagree too. The stock was down more than $1
before the market even opened based on that sham study by Princeton.

~~~
jakewalker
Why are you so sure that was the cause of the price decline?

~~~
poopsintub
The other big news was that Facebook was testing ads outside of their own
platform. The possibility of taking on Google's Adwords and adding more than a
few billion dollars in revenue should have increased the price?

~~~
patmcc
You're layering a lot of your believe about Facebook onto that assumption -
maybe people saw that news and concluded Facebook would fail in external ads
and lose a bunch of money, which decreased the stock price. It's hardly
unambiguously good news (like record profits or something).

------
md224
So, if this all snark, does that mean there's no "strong correlation between
the undergraduate enrollment of an institution and its Google Trends index"?
Because that would be a pretty interesting correlation.

I'm also curious why Princeton's search volume seems to have declined.
Obviously it doesn't mean Princeton is going to disappear, but what _does_ it
mean? Could be statistically insignificant, perhaps.

EDIT: Here are some comparisons:

[http://www.google.com/trends/explore#q=%2Fm%2F05zl0%2C%20%2F...](http://www.google.com/trends/explore#q=%2Fm%2F05zl0%2C%20%2Fm%2F08815%2C%20%2Fm%2F01bm_%2C%20%2Fm%2F01w5m%2C%20%2Fm%2F01w3v&cmpt=q)

Interesting how the initial spread seems to narrow.

~~~
hammock
>I'm also curious why Princeton's search volume seems to have declined.
Obviously it doesn't mean Princeton is going to disappear, but what does it
mean?

[http://www.google.com/trends/explore#q=harvard%2C%20princeto...](http://www.google.com/trends/explore#q=harvard%2C%20princeton%2C%20yale%2C%20uva)

Google trends represents share of search volume, not absolute volume [1]. As
the higher education bubble continues to grow, ivy league schools have not
grown as fast as other schools, and there are new schools being created as
well. All of this activity contributes to Princeton's (for example) share of
search decrease within their sector, and overall.

[1]en.wikipedia.org/wiki/Google_Trends

------
joelgrus
This is pretty funny, although they ruined it with their "HEY GUYS THIS WAS A
JOKE" at the end.

~~~
GBond
Judging from some of the comments here, that line at the end was apparently
necessary.

~~~
afhsfsfdsss88
FATAL: sense_of_humor.dll not found.

------
nilkn
For anyone curious about the author of this post...
[http://en.wikipedia.org/wiki/Mike_Develin](http://en.wikipedia.org/wiki/Mike_Develin)

------
DaniFong
Big data: 0. Smart analysis: 1. Snark: 100. :-)

------
abus
I see this as Facebook's willingness to use its unfair influence to discredit
anyone who dares challenge it. In other words, "if you have a page here, we
won't hesitate to use it against you".

~~~
scholia
Facebook takes a lot of flack: it's today's standard whipping boy. In this
case, it did respond to a fundamentally flawed research paper, but I doubt it
would have bothered if that paper hadn't been so widely and so shoddily
reported.

I though it was quite generous of Facebook to resort to gentle mockery rather
than shred its numerous stupidities.

------
smoyer
Hmmmm ... I think I believe both articles! Higher education and social media
are both in decline. Who needs critical thinking _and_ the ability to gossip
24/7? I'm going to go rewatch Idiocracy
([http://www.imdb.com/title/tt0387808/](http://www.imdb.com/title/tt0387808/)).

~~~
ballard
For anyone that hasn't seen it, Idiocracy goes after the present by
exaggerating the future.

It's a sad but true trend that people are too busy playing World of Warcraft
to understand much less resist what is going on around them: automating of
warfare[0], climate change[1], police militarization[2], sources of terrorism
(including JSOC ("American Taliban") killing American citizens without a trial
in countries without a declared war)[3] and peak oil [4]. That's a lot of
heavy shit that that isn't being addressed, which defers consequences to be
much more painful sooner or later. With commodities, often the consequences
are spread out gradually (epic rise in cost of gasoline) whereas others are
not so gradual.

That seems far more important than whether some pre-press study is important
or not.

References:

[0]
[http://www.ted.com/talks/daniel_suarez_the_kill_decision_sho...](http://www.ted.com/talks/daniel_suarez_the_kill_decision_shouldn_t_belong_to_a_robot.html)

[1] California is having the worst drought in five decades. North Texas is
having the worst drought in over a century.
[http://droughtmonitor.unl.edu/](http://droughtmonitor.unl.edu/)

[2] [https://www.aclu.org/militarization](https://www.aclu.org/militarization)

[3] [http://dirtywars.org/](http://dirtywars.org/)

[4] [http://www.theguardian.com/environment/peak-
oil](http://www.theguardian.com/environment/peak-oil)

------
S4M
Call yourself "Data Scientist" and publish graphics without units. Congrats!

~~~
owenversteeg
It's clearly a satirical post. I'm sure that if he was publishing a
nonsatirical article he would also not have implied that correlation is
causation.

------
debacle
> not all research is created equal

You mean "not all linkbait is created equal?"

------
lucb1e
> every Like for this post counts as a peer review. Start reviewing!

Wait did Facebook just start likewhoring themselves too?

------
curiousAl
Google Trends Extrapolation: The Nostradamus of our times.

~~~
stretchydeath
[http://www.google.com/trends/explore#q=google%20trends](http://www.google.com/trends/explore#q=google%20trends)

~~~
tethis
Google Trends predicts own demise?

------
adharmad
Nice comeback by facebook. Instead of defending facebook against the claims in
the paper (facebook is not dying! - a very hard-to-defend position) they
turned it around and used the same techniques used in the paper to show that
Princeton is dying. Offense is the best defense!

------
augustocallejas
One of the top comments on that page is that they loved the tone of article.
Personally I hate sarcastic responses (no matter how correct you are), which
sound very unprofessional. I hope to see less of these types of responses in
the future.

~~~
heyheyhey
Why do you expect them to be professional? Obviously, the response was in jest
and wasn't some formal response by FB's PR group. This was just done by a
couple FB workers who laughed at Princeton's methodology. To expect a certain
level of professionalism for someone's FB post seems a bit absurd and
pedantic.

------
yuvadam
Hey, Facebook, wanna prove Princeton wrong? How about publishing actual
metrics proving that user engagement is at an all-time high?

(Oh, you don't have metrics to prove that? Is that because users are jumping
ship?)

~~~
ajju
You mean the actual metrics linked to in the 2nd sentence of the post?
[https://www.facebook.com/photo.php?fbid=10101235005815241&se...](https://www.facebook.com/photo.php?fbid=10101235005815241&set=a.766892728661.2325161.4599&type=1&theater)

(Oh, you missed that? Is that because you didn't actually read the post?)

~~~
yuvadam
Those look like numbers from Facebook's SEC filings, which says all there is
to say about which metrics they're choosing to publish.

(Read the post, didn't miss it.)

~~~
ajju
Not sure what you expect them to publish beyond MAUs and DAUs by region. Those
are pretty standard ways to measure engagement and if I were Facebook I'd be
loathe to publish more granular data just to "quiet the critics".

------
dmazin
That was... unprofessional.

------
lockone
My guess is that the real research is the response to the paper and not the
conclusion presented in the paper. Kind of a shoot in the air to see which way
people run. The information gained by observing Facebook's response could
prove useful to other big companies that take a hit in today's media driven
society(internet). What works and doesn't work in their PR response. They
could also be tracking other data, such as stock prices of other social media
companies. It's what makes research so much fun.

------
snowwrestler
The article jokes about "the scientific principle 'correlation equals
causation,'" but I feel like I see people make this mistake all the time,
particularly with statistical concepts like expected value, and
epidemiological risk factors.

Statistics is descriptive--it's not predictive. It tells you about the data
you have. It doesn't tell you why the systems produced that data, and whether
they might produce very different data under different conditions.

------
jjcm
Sassy, but I feel like Princeton was deserving of a response like this. It was
a pretty poor study that had a lot of lazy data and speculation behind it.

------
protez
What is causation exactly? There's no way to prove any instance of causations
at all, if we delve deep into the problem of causation. Even though all data,
or correlations, support F=ma, a single exceptional case can disapprove the
"correlation," therefore we cannot assert force causes accelerations, and so
on, since no amount of correlations is ever enough to prove causation.

------
elwell
I wonder if this article would exist if the first one had been by Harvard
instead of Princeton (given that Harvard was the origin of Facebook).

~~~
username223
The apparent author is also a Harvard alum, fwiw.

------
xtc
Rather than refute the negative prediction for Facebook we're presented with a
sarcastic appeal to those who don't think Facebook could ever dissipate.

This only speaks more towards an argument that Facebook isn't being serious
enough with its own statistics. Of all organizations Facebook should be the
first to spot a trend especially with esteemed data scientists like Mr.
Develin.

------
tedsumme
Not to support the princeton conclusion but to initiate your debunk by
pointing at "like' trends first is silly.

------
nonconfermist
Maybe all sites on the Internet are getting less attention as the number of
things to do on the Internet increases.

------
yetanotherphd
Very nice. I think the bigger issue is that there is a huge amount of academic
research in which a fancy model (usually involving the latest academic fads)
is considered a substitute for serious statistics.

I think Facebook's models with linear time trends are actually much more
believable than the original paper.

------
bhartzer
I think there is a big difference here between Princeton, Harvard, and Yale.
Harvard and Yale clearly bought "likes" to their pages, while Princeton kept
it clean and thus doesn't have any fake followers or fake page "likes".

------
raverbashing
Oh and wasn't the Facebook research based on Google trends?

Meaning: people that can't type "facebook.com" on the address bar and type it
into google then click?

So... based on that, we can assume people are getting more familiar with the
internet maybe?

~~~
Sssnake
The original article isn't very long. If you want to discuss it, you could, I
dunno, try reading it? Yes, it is from google trends. Just like the data on
yahoo and google itself that they compared it to. So no, there is no rational
basis to assume that the data indicates people are getting more familiar with
the internet.

------
waylandsmithers
Yikes. I found this to be shockingly childish and the fact that facebook got
this defensive makes me think they believe the article has at least a kernel
of truth to it.

------
Ind007
Leaving all these articles, facebook has to die - this is my intuition.

------
joshvm
Good fun reading that, but.. an R^2 of 0.54? Not a convincing fit.

------
xordon
There are three kinds of lies: lies, damned lies, and statistics.

------
teaneedz
Isn't Facebook a carrier of bugs?

------
dynamic99
This is just awesome

------
jokoon
no, I still hate facebook.

------
dapvincent
"What's beef?"

------
charleswalter
The fact that Facebook even responded reeks of fear.

If the original article had been completely ridiculous, Facebook could've
laughed it off and wouldn't have had to respond at all.

It's like if someone tells me I'm fat. Because I'm in good shape, I wouldn't
react and just think the person is weird for telling me that. But if I was
anywhere close to overweight, you'd see a strong reaction of some kind from
me.

What the reaction is doesn't matter so much as the fact that there is a
reaction.

------
logicallee
If they weren't being satirical, this response would be a logical error, this
one:
[http://en.wikipedia.org/wiki/Tu_quoque](http://en.wikipedia.org/wiki/Tu_quoque)
(The logical error is that it could very well be true that Princeton is
declining, and proving this would not affect the results of the Princeton
study that Facebook is declining.)

However, the fact that they did so satirically sounds like when a politician
says, "that's preposterous" instead of "that's false."

In addition, the data they use is pretty weak. The first chart shows that
Princeton "crashed" or died between 2010 and 2011, since that's when that
graph tanked. Since that data isn't good to make their Tu Quoque argument
(that Princeton "will die") they are being satirical and ignore it. The second
graph shows that more and more Non-Princeton articles are published. But this
is due to more and more non-Princeton publications. Princeton has a fairly
static amount of output, as the world's universities started outputing
scholarly articles in English, you would expect Princeton's share to drop.

More interesting would be if its share of what it is trying to go for, Nobel
Prize Laureates, publication in Science and Nature, whatever - were on the
decline. This isn't addressed, just a global proportion of all scholarly
articles: not Princeton's aim.

However, Facebook's goal is to get a majority of daily active users.

The second to last graph actually shows a pretty good case that larger
institutions (by enrolment) correlate with search relevancy. But that is not a
case study of an institution whose enrolment fell, which is what it would take
to make a parallel case with the Princeton paper. They would have to pick an
institution whose enrollment fell with its relevancy, and then show that
Princeton is on the same track.

As it stands, it is not "longitudinal" but just a static cross-section of
enrollment and mentions. Perhaps enrolment is static at all major
institutions, regardless of search relevancy, and their enrollment remains
full even if they become irrelevant?

This small switcheroo is a major one, and shows why the article has to be
satirical.

Of course they did respond quickly :) It seems to indicate that Facebook did
some research, but then found their results too weak to publish straight.

------
lafar6502
Sciencey stuff didn't work? Time to call lawyers

------
trollingineer
Couldn't take this seriously given the writer is just mad he is a Data
Scientist at Facebook and not doing research at an academic powerhouse.

