
Facebook Overestimated Key Video Metric for Two Years - tshtf
http://www.wsj.com/articles/facebook-overestimated-key-video-metric-for-two-years-1474586951
======
coldtea
> _the tech giant vastly overestimated average viewing time for video ads on
> its platform for two years, according to people familiar with the situation_

They didn't "overestimate". They plainly gave false numbers to drive ad sales.

Coldtea's law: Never attribute to incompetence what can be explained by
profit.

~~~
billmalarky
"It is difficult to get a man to understand something, when his salary depends
upon his not understanding it!"

-Upton Sinclair

~~~
jgalt212
That is basically how Wall Street works. If a trader or salesperson is
printing money, management doesn't want to know how it's done. Plausible
deniability will at worst get you a failure to supervise penalty, whereas if
you understand how these folks are making money you could become complicit in
any illicit acts and thus exposure yourself to a much larger penalty and/or
jail time.

Of course, the government has proven themselves time and time again unwilling
to punish management so management might as well figure out how the illicit
gains are being made so they can replicate such processes when the big
producing trader or salesperson is invariably poached by a competing shop.

------
sean_patel
"How Facebook is Stealing Billions of Views" \- Kurzgesagt – In a Nutshell
exposed this with detailed analysis and evidence. Watch it here =>
[https://www.youtube.com/watch?v=t7tA3NNKF0Q](https://www.youtube.com/watch?v=t7tA3NNKF0Q)

This is neither a bug nor a "mistake". It's straight up fraud. Fraud because
Facebook uses these false #s to prove to their advertizing clients that they
are getting a HUGE ROI, when they really aren't. Also the investors and
analysts bump up their ratings and the stock goes higher and higher on false
ad view #s.

By counting a view as a legitmate view after the video plays only for 3
seconds, their "algorithm" counted billions of views even when the user has
not even seen it, because Auto-play is enabled by default and you have to opt-
out / disable it and no-one does it. It takes a person roughly 4 seconds to
scroll off their feed as they quickly "scan" their friends posts.

~~~
yladiz
Do you have a citation for the "4 seconds" figure? If it is 4 seconds, and it
logs a view after 3 seconds, that's pretty damning.

~~~
sean_patel
It's not publicly discussed for obvious reasons, but I worked closely with
Mudd for a couple of months last year. Here's a related interview where he
touches upon it.

[http://digiday.com/platforms/advertisingweek2015-facebook-
fi...](http://digiday.com/platforms/advertisingweek2015-facebook-finds-that-
fast-scrolling-millennials-consume-ads-2-5-times-faster/)

Facebook spend an awful lot of resources tracking all sorts of user behaviour
inside it's "walled garden".

I got really sick of it after a while and decided to stop working for them
(even as a contractor)

------
codehusker
On my programmer hand, I can see myself making a similar technical choice. We
have auto-playing videos, we shouldn't count the views less than 3 seconds
because that wasn't really an intentional view. But that means we should also
exclude them from the overall view count. I don't know if FB did that as well.

On my shareholder hand, this seems slightly like fraud.

On my advertiser foot, this seems slightly like fraud.

~~~
daveguy
That is an important question that wasn't clear in the article. If they
counted those sub 3 sec views in the count but not in the duration then it
would definitely be fraudulent. If they didn't include sub 3 sec in the count
or in the average then it is a valid technical decision. I am guessing they
were included in the count but not in the duration.

They should have a histogram of durations.

~~~
dismantlethesun
It looks like they didn't count it in either metric. Peoples' complaint is
that they excluded it from _all_ averages and all reports, which inflates
their average duration of video watched because videos less than 3 seconds
aren't included at all.

------
emcq
This seems slightly overblown. I suspect this was a metric useful for
engineering that found its way to external usage. That's because in many
recommendation systems in practice you want to filter out the spurious views,
and a simple way to do that is with dwell time.

It's not uncommon to require at least 50% of pixels in view for 1 second
before you have an impression for static images [0, 1]. AOL defines an
impression requiring 2 seconds for images [1]. Facebook likely did some
analyses to find that 3 seconds was a good cutoff for their site.

There are more sophisticated ways to estimate dwell time but they seem
uncommon in practice; perhaps due to their difficulty communicating to
advertisers what the impression metric actually means.

For sites with many bots or inbound marketing you often find users bounce
quickly which drives some of this timing. I'm a bit surprised it needs to be
that high for Facebook without many bots or users bouncing quickly. Perhaps
this is for mobile users scrolling quickly.

[0] [http://advertising.aol.com/specs/terms/aol-viewability-
terms](http://advertising.aol.com/specs/terms/aol-viewability-terms) [1]
[http://mediaratingcouncil.org/063014%20Viewable%20Ad%20Impre...](http://mediaratingcouncil.org/063014%20Viewable%20Ad%20Impression%20Guideline_Final.pdf)

~~~
partiallypro
It's not overblown if you're lying to advertisers about the quality of
impressions their ads received, and lying to investors about it
too...especially when your stock has rocketed to making Facebook easily in the
top 10 largest corporations in the world.

Not only that but Facebook actively brags about video impressions in their
conference calls. This is fraud. I think this is something that should be
investigated by the SEC to be perfectly honest. Doesn't mean they should be
charged, just investigated; because if they are misreporting this, what other
metrics are they misreporting to shareholders?

~~~
grandalf
> quality of impressions their ads received

I don't disagree, but FWIW the ads still had better ROI than Facebook's
competitors, so the oversight (while embarrassing) may not have actually
changed the willingness of ad buyers to pay for ads.

~~~
harry8
Better ROI than competitors. Did facebook tell you that?

~~~
cm2012
Any modern advertiser knows their ROI from different channels. As a big FB
spender, this doesn't effect us at all.

~~~
languagewars
FB has more information about other channels than you do, I.e. for all you
know they have algorithms favoring users to see FB ads that are on path that
usually leads to your site based on FB pixel tracking. Then all the multi-
attribution ROI of FB is nonsenses but you have no idea.

I'm not saying they are doing that, I am saying be especially careful with
anyone in advertising who is misleading and who doesn't try to protect their
reputation by being exact in what they say.

~~~
madeofpalk
Regardless, dollar for dollar, if people are seeing bigger conversions or
returns it doesn't matter whether they came from display advertising or
'organic content'.

If they buy something, who cares?

~~~
languagewars
If someone is intentionally adding correlations using predictive information
then they are stealing money you did not have to spend on traffic that was
inevitable. It is like standing inside your waiting room to hand out your
buisiness cards then claiming the sales commission.

------
bluetwo
"likely overestimated average time spent watching videos by between 60% and
80%"

Soooooooo... fraud? They say it didn't impact billings/revenue, but I have to
imagine they will now be in a position to give discounts, if not refunds.

~~~
sean_patel
Yes. straight up fraud. "How Facebook is Stealing Billions of Views" \-
Kurzgesagt – In a Nutshell exposed this with detailed analysis and evidence.
Watch it here =>
[https://www.youtube.com/watch?v=t7tA3NNKF0Q](https://www.youtube.com/watch?v=t7tA3NNKF0Q)

~~~
bbctol
That video is discussing a slightly different problem than the recently
discovered issue the article's talking about.

------
jtchang
I don't think this is overblown at all.

I expect to see a class action lawsuit because of this. If you are a big
advertiser and you can somehow show damages because of this error there may be
a case.

There are some adtech companies recently who focus purely on reporting the
metrics of your ad purchases. Kind of like acting as a third party. I forgot
exactly what they are called.

~~~
shostack
This assumes most advertisers can prove the value of a video view of a certain
length. I assure you as someone who deals with attribution on a daily basis
that most cannot and the ones that claim to would have a hard time proving it
in court. Attribution is hard.

------
pducks32
I don't believe for a second that this was unintentional. The amount of press
and mind share that Facebook has received for their "rocketing video efforts
that rival YouTube" was totally worth pissing off some ad exec. Very few
people will see this news compared to "Facebook dominates video" headlines
that have been everywhere. This was a smart play on their part.

~~~
donohoe
"Never attribute to malice that which can be adequately explained by
stupidity."

I can see exactly how things like this could happen in large corps. You'd be
surprised.

~~~
reitanqild
This might not be totally faor towards Facebook, but:

When it comes to _Facebook_ I guess there are slightly different rules:

A good rule of thumb for Facebook might be: expect malice.

We have seen it with the way the made everything public.

We have seen how they decided to sell out its own user a year after
proclaiming their love and how this wouldn't happen.

------
nostrademons
My data-weenie hat tells me that this is a silly metric anyway. As an
advertiser, you really care about the _distribution_ of viewing times, and in
particular what fraction of viewers watch the video all or most of the way
through. Arithmetic mean is virtually useless when the data has a power-law or
other non-linear distribution, and it's highly likely video viewing times
exhibits this.

~~~
teej
As someone who helps make decisions around ad spend... not really. I would
take a distribution if I was given it but it wouldn't substantially drive my
advertising strategy. Views (over 3 sec views is fine), clicks, viewthrough
conversion, clickthrough rate, and conversion rate are orders of magnitude
higher signal measures than the specifics of how far they got through the
video.

I'd imagine that view time distributions are way more important to new media
companies that rely heavily on video media like Buzzfeed or anyone putting out
content on Youtube.

~~~
cwilkes
What's the clickthru rate on a movie trailer? What is that going to tell you?
Do I click on ads on TV or in the newspaper?

Most video ads I see are more informational than a call to action. Who really
wants to click on an ad while in facebook on a phone?

~~~
teej
The advertising arm of movie companies are definitely looking at the
interaction rates of their trailer ads to help make decisions. The reason they
do so is because it is a valuable signal.

> "Who really wants to click on an ad while in facebook on a phone?"

Advertising for apps (read: games) is a massive, massive market.

~~~
ctrl-j
> > "Who really wants to click on an ad while in facebook on a phone?"

99% of the time I click on an ad, it's on accident.

------
laurihy
Then again, if a "video view" is defined as "watched over 3 seconds (50% of
the video visible in the screen, IIRC)", then it sort of makes sense that
"Average Duration of Video Viewed" doesn't include non-views.

For sure Facebook should attempt to name their metrics as descriptively as
possible, but also advertisers should make sure they understand how different
conversions are measured, what's included and what's not. Another example
would be "Clicks" metric, which included all engagement (i.e. likes, shares
etc), instead of just "Link clicks".

~~~
TazeTSchnitzel
It doesn't seem all that unreasonable, yeah. If a user scrolls right past a
video, should that really count towards the average?

~~~
RodericDay
A user below summarized the issue the way I understand it:

> _what happens is that views under 3 seconds aren 't counted towards the
> average. So if you have 999 999 views of <1 second each and 1 view of 3
> seconds, the reported average is 3 seconds / 1 view = average view time of 3
> seconds, while the true average is something like 500 000 seconds / 1 000
> 000 views._

This is massive fraud, especially if you're shouting from the rooftops "Look
at how much more attention people are paying to your videos in our platform!".

------
mikejb
I didn't fully understand what the bug was, but I found on a German news
page[1] with a slightly more detailed description:

 _Aufgezeichnet wurde zwar die Gesamtsehdauer aller User, geteilt wurde dieser
Wert allerdings nur durch diejenigen User, die das Video länger als drei
Sekunden ansahen_

So the average was not calculated correctly: They accumulated the duration of
all video views (including of those shorter than 3 seconds), but divided it by
the number of 'legitmate' views - i.e. only those longer than 3 seconds. So
you get a pretty big offset if you have many views under 3 seconds (which they
probably do, thanks to autoplay).

[1] [https://www.heise.de/newsticker/meldung/Facebook-Unklare-
Vid...](https://www.heise.de/newsticker/meldung/Facebook-Unklare-Video-Metrik-
fuehrt-Werbekunden-aufs-Glatteis-3330188.html)

------
cft
We are using Facebook Audience Network in random order with other ad network
for mobile ads. FB ads earnings are about 3-5% of other networks, with exactly
the same traffic. Other networks disclose the percentage the revenue split
between them and the publishers, but FB does not. What do they overestimate in
this case, I wonder?

------
bonniemuffin
I don't think average watch time is a particularly useful metric, because no
matter how they count views, there's some kind of arbitrary cutoff. If they
count a view as soon as the player finishes loading, they'll include a whole
bunch of people who didn't intend to watch the video, and in fact only
"watched" a few milliseconds of it before bouncing. So that will drag the
metric down artificially. I bet if they count views as soon as the video
starts playing, they could drag down average watch time by improving player
load time, because they'd count more unintentional "views" of very short
duration.

I hope/assume this is part of a suite of metrics that they provide to
advertisers, so you can understand it in the context of other things like
video completion rate or counts of views that reach X% through the video.

------
throwanem
It's like watching a wasp land on a nettle. You know somebody's going to get
stung, but you just don't care who.

------
aaron695
I don't get it, it makes sense?

Less than 3 seconds is not a view any more than seeing the picture of the
video is a view. It doesn't count I'd say?

Or do advertisers have to still pay for a 2 second view?

------
shostack
While this is concerning, smart advertisers were looking at the distribution
of view length percentages anyway as you can easily pull those columns in and
their derived metrics.

Likewise, smart advertisers look at lift in conversion metrics when possible,
in which case this stat is irrelevant.

That said, FB has not exactly helped things by making metric definitions a
little obfuscated in general.

Personally, I see a bigger concern is them giving 100% view through conversion
credit with a 1 day window by default as part of any website conversion action
tracking. There are very few cases (like some retargeting situations) where
you'd ever want to give full weighting for a VT, and while there is likely
value in VT's, I probably wouldn't give 100% credit to them by default given
the rate at which people scroll on mobile. But advertisers like to see big
numbers, agencies like to show big numbers, and so you have platforms like FB
aggressively try to push metrics like this and some of their rather loose
definitions of "engagement" without great explanation of the nuances or pros
and cons. These are largely left up to the advertiser to determine since, to
be fair, they are very subjective.

Savvy buyers know this and configure their reporting and tracking settings
accordingly because FB and PMDs give you those options. They are sometimes
just buried.

Ultimately, IMHO FB and Google's greatest defense towards any of these sorts
of claims is better and (more importantly) transparent attribution data and
tools. If they can prove their value on the bottom line, other things often
don't matter to many advertisers. Attribution is a tough nut to crack, but for
advertisers spending large sums, it is critical to be successful in these
channels.

------
restlessdesign
It is unreasonable, even for a developer, to count 3 seconds as a play. Anyone
working on a video product would know they were inflating the numbers.

~~~
prostoalex
The top-voted comment on this thread, by emcq, has a link to AOL's viewability
guidelines which states

"For purposes of Video inventory, an impression is considered “In-View” to a
user when at least fifty percent (50%) of the pixels in the ad are in the
viewable browser window for a minimum of two (2) consecutive seconds."

Google here
[https://support.google.com/dfp_premium/answer/4574077?hl=en](https://support.google.com/dfp_premium/answer/4574077?hl=en)
says "For in-stream video ads, 50% of the ad’s pixels must be visible in the
browser window for a continuous 2 seconds."

~~~
swish_bob
That's the IAB's standard viewable impression metric: [https://www.iab.com/wp-
content/uploads/2015/06/MRC-Viewable-...](https://www.iab.com/wp-
content/uploads/2015/06/MRC-Viewable-Ad-Impression-Measurement-Guideline.pdf)

------
Bedon292
Ok, what am I not understanding here?

If they are only counting a view as over 3 seconds. And then only averaging
the those views what is the problem? They tell me 1000 people viewed it (over
3 seconds), and then say the average view time was 10 seconds for those 1000
users. How is this an issue? Why do I care that 9,000 people scrolled past it
and weren't counted in the viewer numbers? Now if they were reporting this as
10,000 viewers and saying 10 second average, I see the problem. Is that what
was happening? Or what am I missing?

~~~
oneloop
My suspicion (based on nothing) is that they averaged viewing over viewings
with length >3 seconds, but _counted_ all views.

I.e. suppose your ad was displayed 5 times for 0.1 seconds and 5 times for 5
seconds. They would report this as "10 views, with average viewing time of 5
seconds".

~~~
tedmiston
I wish your comment was at the highest level. Everyone seems to be attempting
to make the same point in the other subthreads. The example here makes it
crystal clear.

~~~
oneloop
My guess was almost correct. The issue was indeed that they use "views with >3
secs" in some metrics and use "all views" in other metrics. Do one of the
other, don't mix them up.

[http://www.adweek.com/news/technology/facebook-gave-ad-
agenc...](http://www.adweek.com/news/technology/facebook-gave-ad-agencies-
inflated-video-viewing-metric-2-years-173676)

> UPDATE: Facebook authored a blog post today to futher explain the metric-
> based miscue. David Fischer, vp of advertising and global operations, wrote:
> "About a month ago, we found an error in the way we calculate one of the
> video metrics on our dashboard – average duration of video viewed. The
> metric should have reflected the total time spent watching a video divided
> by the total number of people who played the video. But it didn't – it
> reflected the total time spent watching a video divided by only the number
> of 'views' of a video (that is, when the video was watched for three or more
> seconds). And so the miscalculation overstated this metric. While this is
> only one of the many metrics marketers look at, we take any mistake
> seriously."

------
smithunsero
In 2012 there were reports that 80% of traffic from Facebook Ads were
automatic bots. So this is not the first time Facebook use tricks to get more
money from clients.

~~~
plopilop
From what I read [1] it's only one startup that claimed this, and I failed to
see any confirmation from any other company. For me, no replication from
companies that spend thousands every month on advertising = the startup messed
up with its stat tools or wanted to create a buzz around its name.

[1]: [https://techcrunch.com/2012/07/30/startup-claims-80-of-
its-f...](https://techcrunch.com/2012/07/30/startup-claims-80-of-its-facebook-
ad-clicks-are-coming-from-bots/)

------
sidcool
This looks like a flat out lie. Startups have to sometimes do that to woo
potential investors. Although I like Facebook overall, I do not agree with
many of their business practices.

------
breatheoften
What was the actual bug? Does this mean that no video had an average view time
less than 3 seconds?

~~~
sharkmerry
They dont count views for less than 3 seconds in their view count.

They didnt include views of less than 3 seconds in their "average view time"
metric, which is very misleading. Their metric was "average view time of
video's in our view count".

Which is not the same thing and the latter vastly inflates the effectiveness
of facebook video

------
bruinbread
This isn't anything new. Several YouTubers have "exposed" how Facebook tracks
views. It's not accurate, but it's the system that has been in place for a
while and they've defended it as such.

------
Arkaad
I am disappointed by FB. Next we'll learn that they do tax evasion.

------
supermatt
If face book only count a view as over 3 seconds, and they measure the average
view as only those videos over 3 seconds, what is the problem? Surely now its
less representative?

------
randomgyatwork
When will Facebook 'content creators'(users) get compensated for the value
they create?

------
yueq
Will there be a class action on this matter?

------
tn13
It might be inaccurate but does not matter much in advertiser perspective.
Given the conversions etc. eCPMs conversion to the real expected value.

------
malloreon
how much of their video income will they be refunding to advertisers?

