
Facebook Says It Found More Miscalculated Metrics - uptown
http://www.wsj.com/articles/facebook-says-it-found-more-miscalculated-metrics-1479303984
======
sfifs
I see a lot of commentary about conversion metrics and how does reach matter
etc. So let me share some perspective from point of view of mainstream
consumer products advertisers whose individual FB budgets easily push into
hundred million dollars plus zone.

The primary business objective of advertising for companies such as mine is to
build brand awareness, product benefit awareness and brand identity (eg. are
you partial to Coke or Pepsi? Nike or Addidas? etc). Doing this will influence
your purchase when you next shop for our categories (eg. your next shopping
trip to Walmart or Amazon). Note that we are looking to influence a future
shopping trip, any "digital conversion" we may get (eg. click thorough to our
product page or an online retailer) really is pure gravy from our point of
view.

The primary metrics of interest to us therefore are how many people did our
impressions reach (people - not cookies or devices), of the ones we did, how
many were in our desired target audience and what was our impression frequency
to them (at some point, high frequency becomes excessive). We don't even
really look at clicks or conversions unless there is a specific interaction
based marketing program.

Hope this helps explain how big advertisers look at media.

~~~
dzink
You are using Facebook ads for exposure, the same way consumer companies have
been treating TV. You are looking for mindshare. How is that working out so
far? Are you able to measure exposure to impact on future purchases?

~~~
lxchase
Actually for digital ads, there is a way to measure offline sales lift (albeit
a few shortcomings). It requires a lot of money (i.e. ~50K for a study, and
media spend 250K+). We are able to tie a digital impression to traceable
tender (credit card, loyalty card, etc). There will be an exposed group and
unexposed group who match each other audience-wise. Lift is then calculated
and vetted for significance.

For more info, I recently did an interview on an ad-tech site about this:
[http://www.thalamus.co/blog/lenny-chase-
interview/](http://www.thalamus.co/blog/lenny-chase-interview/) Happy to talk
more separately if interested.

~~~
shostack
Any insight into options for advertisers without that sort of budget to put
towards just a single brand lift test?

~~~
lxchase
Brand lift or Sales Lift? Brand lift is a bit different because there are a
lot more companies offering this due to more simplicity. The shortcoming of
Brand Lift is every vendor has their own methodology when it comes to deciding
what exactly is a marginal lift in brand awareness. I think the easiest option
for a lower budget is using a FB/IG brand awareness campaign and looking at ad
recall numbers
([https://www.facebook.com/business/help/1029827880390718](https://www.facebook.com/business/help/1029827880390718)).
However, we don't really know how they calculate this number and it would only
be useful to benchmark between campaigns (if you trust the number). The best
option right now is to find a vendor that polls the user who saw the ad, like
Vizu (Nielsen). Unfortunately, to get a stat. significant read, you probably
need a lot of results, and thus would need higher media spends.

~~~
shostack
Both actually.

I don't trust the black box of the ad recall stats they provide, so that's not
really an option. Is Nielsen's data any less of a black box for this sort of
data? And is it actually a decent sample? The older way Nielsen did things was
biased because people had to sign up to be a part of it.

And yes, the challenge is that to get a statistically significant read, we
need to throw a lot into a specific test, and that is hard to justify without
benchmarks. Kind of a chicken and egg problem if your brand is large enough
where it takes considerable display dollars to make a blip against your other
traffic.

I've been curious if things like Adobe's Attribution econometric modeling
tools help much here, but guessing that lack of data will be an issue there as
well.

~~~
lxchase
I agree with you 100%. No options are ideal and statistically correct which
sucks... a lot. As you mentioned there will always be selection bias and a
black box, just hopefully less, with Nielsen for example. Not many people in
our space even understand these problems exist, so I think you already lead
the pack in this regard. For the sales lift studies, the methodology seems
less "iffy" since its one to one and measurable results. One can argue there's
selection bias due to no cash data. I haven't seen any cheaper options to do
this while minimizing error also.

My client is in the same chicken-egg problem and the way we approached it is:
Going on blind is worse than the risk of a bad study. It's completely up to us
to do due diligence and ask the right questions to find holes in the vendor's
methodology and minimize the risk. We felt that the data gained will be
accurate directionally, albeit skewed.

I've never had first hand experience with Adobe's attribution models but from
what I gather, lack of data and similar to media mix modeling.

IMO, advancements in this space will be battling privacy concerns.

~~~
shostack
Yeah, privacy concerns are what will decide a lot of this. I'm very torn
because as an end-user I definitely have gotten more nervous with the current
state of tracking, but as an advertiser, these are arguably the toughest
problems in the industry and what keep me up at night.

Have you found any good off-the-shelf tools to help with the incremental lift
analysis that might be better suited for advertisers who are not as massive?
Our current plan is to get DCM in place, pipe everything we can through there,
and then let our Data Science team go nuts at trying to model it. Seems like
that's our best bet since tools like Adometry or other dedicated attributions
platforms don't make sense for us yet.

------
tangue
I work in a mid-size company where "the computer guy" manages the Facebook
page. One day my boss asked me _" WTH does reach mean, you mean with 50 bucks
we reached 100 000 people ?"_ Before replying, I read fb docs, a lot of blogs
and for me it was gobbledygook. I said it was marketing bullshit and that we
should focus on conversion rates. I felt guilty. Impostor syndrome kind of
guilty. Until now.

~~~
jseliger
I'm part of a small grant writing consulting firm, and we tried advertising
through Constant Contact's Ad Launcher program. We only want to reach
nonprofit executive directors and some heads of public agencies (for example,
housing authorities or cities managers). Facebook advertising was complete
garbage for us, or, alternately, none of us could target the ads
properly—including me. We got a fair number of "leads" but virtually none were
qualified clients.

Adwords used to work really well, but its effectiveness has declined in recent
years.

We've tried finding a LinkedIn consultant without any luck.

~~~
Sapph
Have you considered highly targeted value based cold emailing?

The HN community may be weary of sending / receiving unsolicited email but if
you provide massive value upfront and only contact prospects who would get 2x+
ROI from your product/services, you will not be seen as spam and will convert
a good % of the prospects.

I've run a lot of these type of campaigns and the key is to:

A) Only target prospects who have a real need for your product/service and
have the budget for it

In your case, I would go on different grant websites and add their grant
recipients to your prospect list. This way, you only target organizations who
actively use grants.

B) Do a bit of work for them to provide upfront value in your cold email

In your case, research a list of grants they're eligible for (you can share
the same list to nonprofits in the same space) and provide unique insights
about these grant givers are looking for based on your experience. Your value
upfront + value ideas will open the conversation and naturally lead to
prospects asking about about how you can help them.

You can get an idea of what the initial email could look like here:
[http://www.artofemails.com/sales-
freelancer#writer](http://www.artofemails.com/sales-freelancer#writer)

~~~
ams6110
I'm an executive at a nonprofit and I delete all unsolicited email unopened. I
just don't have time for it. I don't know how typical I am (small organization
and it's part time for everyone involved)

------
woliveirajr
> None of the metrics in question impact Facebook’s billing, said Mark Rabkin,
> vice president of Facebook’s core ads team.

This is the point. Might not impact the billing, but might lead announcers to
miscalculate which campaign to use, for how long, for which audience, in the
ending resulting in bad contracts with Facebook. That will be billed
_correctly_.

~~~
FussyZeus
Kind of like how service providers constantly have issues with their
infrastructure and support, but the billing website and auto-billing system?
That's always working.

Shows you where their priorities are.

~~~
YCode
To be fair, those systems have drastically fewer users connected and it's
comparatively easy to scope usage and subsequently resources for them.

Whereas with production systems usage and troubleshooting is far more
unstable.

~~~
gertef
Eh, some how the "purchase service" page is far more reliable than the
"downgrade or cancel my service" page.

~~~
ams6110
If there even is a page for that. Often that takes a phone call, a fax (!) or
even an appointment.

------
gthtjtkt
That reminds me, did they ever address the Click Farm problem that was
essentially rendering likes and views worthless for many advertisers?
[https://www.youtube.com/watch?v=oVfHeWTKjag](https://www.youtube.com/watch?v=oVfHeWTKjag)

Or the rampant Freebooting?
[https://www.youtube.com/watch?v=t7tA3NNKF0Q](https://www.youtube.com/watch?v=t7tA3NNKF0Q)

~~~
forgetsusername
It's hard to imagine them doing so, because up until a certain point
(advertisers leaving, which we ain't there yet) it's not in their interest.

It brings me back to something I complained about on these boards years ago. I
was working with a fairly small social gaming company about 10 years ago,
during which time Facebook was boasting about "300m, 400m, 500m users!".
Meanwhile, we were learning that almost half the users in our game were "fake"
accounts. It was a major issue, but wasn't in the interests of Facebook to
disclose during their hyper-growth. In fact, they made a point to state that
fake accounts was not an problem.

Clearly Facebook will use the metrics when they are convenient, despite their
invalidity.

~~~
hinkley
I had to learn the hard way that any number the development team gave the
marketing group would be used in a way we would come to regret. It's a big
part of my cynicism about the industry.

People remember numbers but lose the context. Especially people whose bonuses
depend on misunderstanding them.

------
snovv_crash
So you're telling me that they don't have a test which provisions a Facebook
with N users, calls the AJAX handlers for K clicks, from various users and
then checks that the metrics show that K clicks were counted, from the correct
users? That they don't do this for video watches, friend display-to-click
conversions, and every other metric they use to make decisions about what
happens on the site? Really?

~~~
gdulli
I don't believe in a conscious conspiracy within Facebook to defraud
advertisers. I absolutely believe in a pervasive, semi-conscious but unspoken
feeling that asking deep questions about the effectiveness of Facebook
advertising or the validity of ad metrics would raise conclusions that would
be definitively bad for Facebook, so it doesn't happen.

~~~
ethanbond
I interned on Atlas (FB's enterprise advertising platform purchased from MS a
while back) and this was the impression I got. _Everything_ has a dollop of
feel-good verbiage piled on top. We're not selling ads, we're "building more
meaningful connections between businesses and their customers."

The initial value proposition of online ads was click = confirmed intent. When
not enough people click, they construct increasingly convoluted notions of
purchasing seconds of mindshare at a time. Of course, at a certain point these
purchases become as untestable as a billboard on a highway, a big difference
being that highway billboards don't prop up industries, ecosystems, and even
geographical regions like online advertising does.

If someone calls shenanigans on online ads, it will be far more catastrophic
than if they did the same on billboards/radio/TV ads.

~~~
MorePowerToYou
+1 "construct increasingly convoluted notions..." to rationalize or support
anything is a huge red flag in this case and, imo, life in general.
Understanding the tax system requires trudging through increasingly obscure
rules? It's probably a broken, inefficient system. Health care billing system
rules are convoluted, vague, and complex? Yeah, it's probably pretty broken.
Need a convoluted, complex explanation to rationalize your support for Peter
Thiel despite his endorsement of certain political figures? Again, a huge red
flag.

~~~
VLM
Best analogized with computational astrology... Oh you used numbers? Then
astrology must be great source of advice.

------
DougN7
This isn't surprising if you've advertised on FB. I tried, and the results I
got were dismal (to be fair, it was for B2B which I think generally does poor
there).

~~~
the_watcher
I run B2B Facebook ads, and they work just fine.

------
malloreon
How much of the money they made from these bugs (the new ones and the older
ones) will they be returning to their advertisers?

~~~
pryelluw
Do they have a contractual obligation to do so? Im curious.

~~~
bomdo
Yes, if proven. If someone promises 1000 clicks for your dollar and delivers
only 999, they are at fault, have not fulfilled the contract and either you
should be reimbursed or they have to deliver the missing click.

That Facebook talk publicly about this issue (naming it a software bug) either
means some lawyer business is already happening in the background, or the
difference between what they promised and delivered is so obvious that it is
just a question of time. This article is bad for Facebook's image as an
advertisement platform and there is no reason to publish something like this,
other than liability.

That said, Facebook has good lawyers and will obviously make it exceptionally
hard for customers to prove Facebook is at fault. Additionally, their terms of
service for the self-serve ads[1] is filled with blurry statements such as "we
are not responsible for [...] technological issues [...] that may affect the
cost of running ads", which will make legal battles lengthy and boring.

[1]
[https://www.facebook.com/legal/self_service_ads_terms](https://www.facebook.com/legal/self_service_ads_terms)

------
keketi
I guess they moved fast and broke things.

------
0xmohit

      98% of all statistics are made up.
    

Its ok, Facebook.

\--

Earlier discussion:
[https://news.ycombinator.com/item?id=12561302](https://news.ycombinator.com/item?id=12561302)

------
anonbanker
I bet "miscalculated metrics" is a really good replacement word for "ad click
fraud".

------
zackkrida
Can I ask what has prompted this "investigation" of sorts by Facebook? Just
the election? I wonder who is putting pressure on them to straighten this out.

~~~
lxchase
As an anecdote, I know that the ad industry lately has been talking about
transparency and I'd like to think we're getting more advanced in tech that
people start asking the right questions. Or maybe I'm naive to believe that.

~~~
rhizome
The ad industry will never be "transparent" (i.e. clear and colorless),
because it has falsifiability issues. Transparency would mean that they admit
that most of the industry is based on guessing, that there's no science behind
huge commercial transactions.

------
mtgx
I think Facebook has proven by now that even if they aren't doing this on
purpose to trick advertisers into thinking their ads are much more effective
than they really are, at least the product doesn't seem ready for primetime
with so many "bugs" in measuring popping up.

If you're a small business, Facebook may screw you over big time, so it's not
worth the risk. I assume big companies just don't care.

------
dimino
Is it weird or normal for a company like Facebook to share this kind of
information with the public like this?

Feels oddly transparent. What is prompting this level of transparency?

------
chinathrow
"Lastly, Facebook said it plans to form a Measurement Council made up of
marketers and ad agency executives, and will also roll out a blog to more
regularly communicate updates about measurement."

I read that as "we have no idea on how to measure, so please tell us what is
important for you guys".

~~~
ethanbond
Wasn't Facebook's whole value proposition is that they knew more than the
traditional advertiser/publisher partnerships?

Such a sham.

~~~
jsprogrammer
I think the original proposition was: dumb fucks.

------
akerro
Facebook which supported Clinton finds those bugs just after elections that
Clinton lost.

~~~
tyleraldrich
What do advertisement metrics have to do with the election? As far as I know,
the story about "Facebook + US Election" was related to people
posting/sharing/therefore make trending fake new stories, which is unrelated
to the Facebook ads/ad metrics side of the business.

