
Truth goggles sniff out suspicious sentences in news - mjbellantoni
http://www.niemanlab.org/2011/11/bull-beware-truth-goggles-sniff-out-suspicious-sentences-in-news/
======
rubidium
Too bad it employs the PolitiFact database, as that itself is suspect to
having its own bias.
([http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection...](http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection_bias_politifact_rate.php))

Using a biased fact-checker to check facts doesn't really work well.
Constructing a non-biased fact-checker is the real challenge here. I'm not
sure it can be done, due to the great amount of spin that can be put on most
anything political.

EDIT: BTW, I'm not meaning to take sides on this, just pointing out the very
real difficulties in establishing a "fact-checker". It's worth knowing that
others have contested PolitiFact's methodologies. If the trustworthiness of
the statement is based on a person's judgement, then the fact-checker has
bias.

~~~
andrewcooke
have you read that article in detail? i have spent some time on it, and am
having trouble following the argument. at the start they say that
approximately equal numbers of statements are taken from either side. then
later they seem to be saying that a bias exists because the republicans lie
more than the democrats. what i'm having a hard time understanding is why that
is a bias - perhaps some republicans simply lie more?

again: it's not they are focusing more on one party than on another (they
don't: "the number of public officials subjected to PolitiFact's Truth-O-Meter
lens from each party is fairly even during the period under analysis"), but
that they catch more lies from one party (the republicans) than the other. in
particular, it appears that palin and bachmann are responsible for raising the
average number of lies for the republicans.

so why is this bias? why isn't it simply that palin and bachmann are liars?
what am i missing (i am not american, so don't know much of the context - i
just read the article and couldn't follow the logic of the argument)?

one possible alternative is that they are ignoring lies by democrats and
instead checking only true statements. but in that case you'd think there
would be a smoking gun - notorious statements from democrats that weren't
checked. the article doesn't give any evidence for that.

ps also, i take issue with your final disclaimer. you can make judgement calls
and still be better than coin flipping. just because it's not an automated,
exact science doesn't mean that it cannot edge closer to the truth. almost
everything in life is not exact, but we still know some people lie more than
others. one way to do this is by looking at what is given (for example, this
article) and checking whether it makes sense. maybe i have made a mistake - if
so, hopefully someone will correct me. eventually this thread will lend weight
to one conclusion or another...

~~~
temphn
> so why is this bias? why isn't it simply that palin and bachmann are liars?

I hesitate to get into this, but Peter Orszag has only been tagged by
Politifact once[1]. Yet at the time he was CBO director, there were many
people stating that CBO was distorting long term projections to show that
Obama's health care bill saved money. And recently the administration appeared
to acknowledge that these projections were (at a minimum) indeed
inaccurate[2,3].

    
    
      By scrapping the CLASS Act, HHS is losing about 40 percent 
      of the savings healthcare reform was supposed to generate. 
      The Congressional Budget Office’s most recent estimates 
      projected $210 billion in total deficit reduction from the 
      healthcare law, $86 billion of which would have come from 
      CLASS.
    
      HHS officials emphasized Friday that the healthcare reform 
      law is still projected to cut the deficit, since about 
      $125 billion of the most recent savings estimates are 
      still intact.
    

Now, you can determine for yourself whether you believe HHS and CBO's other
projections given that this program was found to be insolvent before it was
even launched. Please also note that there was no shortage of people
vociferously opposing[4] the healthcare bill before passage who called out
this particular provision as unsustainable due to adverse selection effects.

    
    
      The CLASS Act looks like a gift: It brings in $58 billion 
      in net tax revenues by 2019, lowering the deficit by an 
      equivalent amount because only minor costs will be booked 
      during that period. Under the CBO rules, the CLASS Act 
      technically covers one-quarter of the $250 billion 
      shortfall in funds needed to pay for health-care reform.
    
      The gimmick lies in looking only at the CBO's ten-year 
      budget window. The extra revenues are an illusion because 
      of the disaster lurking just beyond that horizon.
    
      In fact, none of the $58 billion is available to pay for 
      the House bill. The CLASS Act is so poorly designed that 
      the $58 billion reserve and all future premiums won't come 
      close to covering the generous benefits it's promising.
    
      Here's why the mechanics of the CLASS Act assure its 
      eventual collapse.
    

You may disagree, but this is one relatively cut and dried example in which
the critics of the healthcare bill appear to have been proven right. Indeed,
evidence-laden, numerical critiques of this particular "CLASS Act" provision
were all over the web along with charges of falsehood by Orszag.

Yet at the time when it might have moved the needle, Politifact was silent on
this issue. This may be what OP is referring to in terms of a selection issue.
They simply did not engage the CBO's projections, even though they were far
more amenable to classification as "true" or "false" than standard political
statements. They were also inarguably important.

[1] <http://www.politifact.com/personalities/peter-orszag/>

[2]
[http://www.nytimes.com/2011/10/15/health/policy/15health.htm...](http://www.nytimes.com/2011/10/15/health/policy/15health.html?ref=healthcarereform)

[3] [http://thehill.com/blogs/healthwatch/health-reform-
implement...](http://thehill.com/blogs/healthwatch/health-reform-
implementation/187697-hhs-to-suspend-class-program)

[4]
[http://money.cnn.com/2009/09/03/news/economy/health_care_cla...](http://money.cnn.com/2009/09/03/news/economy/health_care_class_act.fortune/index.htm)

~~~
Anechoic
Is there a particular statement from Orszag that you feel Politifact should
have addressed?

~~~
temphn
Well, this one is interesting.

<http://www.whitehouse.gov/omb/blog/10/03/04/No-Gimmick/>

    
    
      Recently, a lot of attention has been paid to a claim that 
      this deficit reduction is achieved only through a 
      business-as-usual Washington budget gimmick: paying for 
      just a few years of costs with many more years of savings.
    
      This charge is simply false—and let’s get the facts 
      straight. 
    
      ...
    
      Third, this is not a budget gimmick.  The purpose the 
      tried-and-true gimmick described above is to make a 
      proposal that adds to long-term deficits appear fiscally 
      responsible. But if that were the course we were taking, 
      we would expect to see a large fiscal hole at the end of 
      the first decade and larger and larger deficits in the 
      second decade. Instead, over the long-term, the savings 
      under the President’s plan are expected to grow faster 
      than the costs.  So, when the Congressional Budget Office 
      is done with its scoring, we expect it will find that the 
      President’s plan reduces deficits by roughly $100 billion 
      in the first 10 years and roughly $1 trillion in the 
      decade after that.  In other words, health reform should 
      reduce the deficit by growing amounts over the long-term.
    
      Put simply: Health reform will reduce the deficit in this 
      decade, and it will reduce the deficit by even more 
      thereafter.  There’s no gimmick in that.
    

A program which was supposed to save $86 billion is declared insolvent and
discontinued before it was put into production, by the people who were
intimately involved in designing it. They pushed hard to pass the bill and
made statements like Orszag did above.

In the interest of being nonpartisan, I think that history will look on these
kinds of statements in the same way we now look on Colin Powell's presentation
to the UN in the context of the runup to Iraq. At best the WMDs/savings just
weren't there, and it was an honest mistake. At worst, they lied. Somewhere in
the middle one might call them guilty of wishful thinking and exaggeration.
But whatever you call it, it's hard to continue to maintain that those who
claimed the presence of WMD/savings were empirically correct.

~~~
Anechoic
So you want PF to retroactively check people's predictions to see if they were
correct? That would be interesting to see, but it's clearly not what they do,
they address current statements based on current information.

Now was there any reason to believe that something in that statement wasn't
correct when he made it? (for example was it not going to be scored by the
CBO?) Otherwise I don't see the bias.

~~~
temphn
> Now was there any reason to believe that something in that statement wasn't
> correct when he made it?

Yes, at the time Orszag made the statement many commentators (including the
CNN link a few posts up) contested the CBO's projections and said they were
manipulated, with a particular focus on the CLASS Act. Orszag doubled down
just like Powell did before the UN, just like the yellowcake incident.

I happen to think this specific case is fairly cut and dried, but short of
Orszag admitting to a "lie" it is impossible to know what was in his mind.
(Though it should also be said that Politifact does not precisely distinguish
between whether a statement is empirically false and whether the speaker is a
'liar with their pants on fire').

Anyway, this is one example that stuck in my head as a numerical one which was
so important beforehand, and so checkable both at the time and today, that
even partisans on opposite sides might meet up to compare notes afterwards.

------
kls
There are few ideas that I see that have the potential to change the world,
but I would say this one may. A lot of people are too busy or to
intellectually lazy to research facts. As such they default to non-verbal ques
in candidates. The strongest being confidence.

A candidate can be confident while completely fabricating information and more
people will believe them than a candidate with no confidence and truth. Having
a tool like this to refute the information immediately will be valuable.

The trick will be the credibility of the "truth" provided. We all have biases
and keeping them out of our creations can be more tricky than it would appear.
No doubt if the product gains wide traction, many will accuse it of bias the
author needs to make sure that they have double and triple checked the facts
so that it can withstand the onslaught of accusations that will surly come
from both sides.

------
mjbellantoni
I've been thinking a lot lately about how technology can provide transparency
to the political process. Among these ideas have been "reading aides" like
this: tools that will provide verification and context for what you're reading
or watching.

Another such tool I've been thinking about would show you the relationship of
the writer and quoted sources (people or institutions) to the material being
covered.

For example, consider the case where spokesperson for a non-profit
organization is quoted defending a particular industry. This tool might tell
you that this person was once an executive at a for-profit corporation in that
industry or have some other non-obvious relationship.

If anyone knows of anyone working on something like this, I'd love to learn
about it.

~~~
espeed
I have been working on Whybase (<http://whybase.com/>), which allows you to
connect ideas and analyze statements made by people. You can use it to
discover insights into the reasons behind why things are true and how things
are interconnected.

~~~
noodly
Can you give some example ?

------
philwelch
"Goggles" "sniff"? Do people even pay attention to their own metaphors
anymore?

------
michaelkeenan
I've dreamed about the possibilities for automated fact-checking to improve
online debate, and considered attempting to write it myself (using data
sources like Wolfram Alpha and Google Public Data, and parsing natural
language with a big dirty pile of regular expressions, which can't be the best
approach but it's an approach I could attempt in a weekend).

For example, a program could examine a text for sentences matching "[X
country] is [comparison] than [Y country]" and check that against online data
sources. So if you're typing a comment into reddit or wherever, it would first
check your comment for claims it can parse, and alert you if you are mistaken
about something so you can change it. And if you are correct about something,
but didn't supply a citation, it could add it automatically.

This could have several awesome effects:

* It would make online debates more accurate, spreading truth and knowledge.

* It might set a standard of factual accuracy; people might come to expect assertions to be backed up with citations, and regard claims as suspicious if no supporting reference is provided.

* Just knowing their facts will be checked (possibly by a future version of the software being re-run on old forum comments and highlighting inaccuracies for all to see) might make people more careful in their assertions.

If voice recognition technology improves, then this might eventually fact-
check TV in real-time.

------
dylangs1030
This is really awesome, but I think it would be even better if a program could
be written such to flag sentences _without_ a third party judge. Using the
stylistic tendencies of lying, for example, in rhetoric. This might not be
possible or plausible, but if it could be written, I think it would be at
least equal to this technology.

~~~
cynwoody
There are at least two reasons that's impossible. First, liars are smart.
Second, not everyone spouting an untruth is a liar. If a person doesn't know
he's uttering a falsehood, then the fact is unlikely to influence his style —
or his speech spectrum or cadence or direction of gaze, etc.

~~~
dylangs1030
Well, I see a solution around the first reason. Just flag anything that is
worded in a suspicious way. The example, "Obama has the worst ratings in
modern times" _sounds_ suspicious. You shouldn't need a fact checker to know
you need to check that fact. If you could implement the same sorting method
common sense has to a fact checker, you might be able to make it work.

However, I admit, you might be faced with the problem that an overwhelming
number of statements would be flagged. I'm not sure how you would get rid of
false positives.

------
rplnt
Don't know why, but the page is completely broken when rendered in Opera.
Perhaps the custom font?

~~~
crabwalk
Looks fine to me in Opera Mac too, fwiw. (I run niemanlab.org, and I confess I
don't do any testing in Opera, so I honestly appreciate any bug reports.)

FYI, our browser breakdown: 31% Chrome, 28% Firefox, 15% desktop Safari, 11%
IE, 10% iOS Safari, 3% Android browser, 0.74% Opera, 0.26% Opera Mini, 0.23%
RockMelt (!).

~~~
rplnt
Might be some userscript I have installed, thanks for the feedback.

And as a bonus, browser stats from country I'm from:
[http://gs.statcounter.com/#browser-SK-
monthly-201010-201110-...](http://gs.statcounter.com/#browser-SK-
monthly-201010-201110-bar) (You can check more countries from this region, you
will see similiar results for Opera (Ukraine stands out though)). The reason
for that is that Opera was the best browser without any doubt (think pre-
firefox2 age). Only flaw was that it was not free - it was ad-sponsored. But
you could get it for free of course, and users in Easterish Europe didn't
really care.

------
jobu
_"... software that flags suspicious claims in news articles and helps readers
determine their truthiness."_

+1 for use of the word "truthiness". I can't believe the impact people like
Stephen Colbert have on our culture and lexicon.

------
noodly
If you're looking for truth in media, you're looking in wrong place.

~~~
michael_dorfman
What? The article is about direct quotes from politicians. There are a lot of
things you can blame the media for, but this isn't one of them.

~~~
noodly
The article is about "suspicious sentences in news", and media are news
producers.

This is one thing I can blame them for, due to: media needn't be objective -
that's not their mission - their mission is to make money, unless they're non-
profit (how many of them is ?).

They use deception, to make you believe that news they produced is objective,
and you shouldn't look for sources (which often are well hidden or absent),
and assuming that news they produce is compatible with your worldview, you
won't look for sources - you're their customer.

I'm sure you're able to divide media by political views they support - if
they're objective why it's possible ?

The use of deception, means they're not good source of truth.

The main reason I found HN and stayed here, is its news quality (almost no
political bs, and really close to the source).

~~~
slifty
So how do you propose we function as a democracy if any media source that
covers political issues is so full of shit that there is no point in even
considering it?

Your perspective is extreme, but it is a very important concern. If only
someone could try to create additional information layers to help you navigate
a medium that is potentially riddled with dubious claims. Maybe that person
could call the project "Truth Goggles" or something.

~~~
noodly
_So how do you propose we function as a democracy if any media source that
covers political issues is so full of shit that there is no point in even
considering it?_

From my perspective, for democracy, it's very important to have independent,
non-profit media that provide _raw_ stream of information - e.g. A/V stream
from congress, major events etc. - that while being boring would provide
unbiased stream of facts. For news - transcripts, photos, videos, documents,
all without commentary. That would allow to compare "revelations" served by
other media, with facts.

 _If only someone could try to create additional information layers to help
you navigate a medium that is potentially riddled with dubious claims. Maybe
that person could call the project "Truth Goggles" or something._

It would be great, if I could trust it - its sources (evidence) should be
explicit.

~~~
slifty
Sounds like we do, in fact, agree. The full title of the thesis is: "Truth
Goggles: Automatic Incorporation of Context and Primary Sources to Enable
Critical Media Consumption"

This tool won't tell you what to think; it will merely remind you to think.

------
buckwild
I don't understand how this can be any more effective than just cited sources
at the end of a paper...

~~~
DrewG
A lack of cited sources in an article doesn't necessarily imply that its
false.

~~~
jquery
And cited sources does not automatically make something true.

------
acgourley
This is excellent. There has been a certain class of "debate" sites that
gather opposing expert opinions on some controversy but they always had a
distribution problem. The people that need this kind of information most and
the least likely to go check these sites.

This is a great distribution mechanism of the output of those sites. If there
is a trending, controversial statement in the news it would be great to be one
click away from seeing smart people discussing both sides.

Now I want other kinds of goggles. Like goggles that spot popular
Anachronisms. Or trending memes.

