
Using Metadata to Find Paul Revere - decklin
http://kieranhealy.org/blog/archives/2013/06/09/using-metadata-to-find-paul-revere
======
startupfounder
Paul Revere, "in the Fall of 1774 and winter of 1775, I was one of upwards of
thirty, chiefly mechanics, who formed ourselves into a Committee for the
purpose of watching British soldiers and gaining every intelligence on the
movements of the Tories."[1]

Paul Revere was essentially one of the founding members of this country's
counter intelligence program against his oppressive government. He was the
first in a long line of Mark Felts, Bradley Mannings and Edward Snowdens of
this country.

If we have a system where these whistleblowers are stopped before they can
leak information on the system that catches them before they can whistleblow
there is no turning back. This would not be the UNITED STATES OF AMERICA
without Paul Revere and Edward Snowden.

[1][http://en.wikipedia.org/wiki/Intelligence_in_the_American_Re...](http://en.wikipedia.org/wiki/Intelligence_in_the_American_Revolutionary_War#Paul_Revere_and_the_Mechanics)

~~~
tzs
Did Paul Revere gather intelligence concerning both sides, and release
everything he had to both sides, so that he would help his enemies as much as
he helped his friends, and hurt his friends as much as he hurt his enemies?

If no, then the long line Revere is the first in does not include Manning.

~~~
BCM43
Do you have any evidence that Manning hurt his enemies?

~~~
mpyne
Manning's own statements to Lamo that the Army put him up for an award for his
counterintelligence work which was used to identify and 'break up' insurgent
cells.

~~~
BCM43
So... The army said so?

~~~
mpyne
I'll try to type slower this time.

Manning. Said. So.

------
LiamMcCalloway
Academic version: [http://db.tt/6UA1bXZq](http://db.tt/6UA1bXZq)

DIVIDE AND CONQUER: DISTORTED COMMUNICATION IN NETWORKS, POWER, AND WEALTH
DISTRIBUTION Wilson Perez-Oviedo

Cornell University and Banco Central del Ecuador

Abstract In a society composed of a dictator and its citizens, what are the
determinants of the political equilibrium between these two? What are the
conditions for a successful citizens’ revolt? What kind of strategies do
governments follow to prevent such revolts? The situation of these types of
societies can be understood as a game played between a leader, who has to
decide the distribution of the aggregate income, and a group of citizens who
have the opportunity to revolt if they are unhappy with the distribution.
Coordinated action by citizens is possible because they form nodes in a
communication network. However, communication through the network is
distorted, which could preclude the emergence of collective action among
citizens. The network structure and the distortion level are determinants of
the political equilibrium and wealth distribution. The model explains how the
dictator could use propaganda, cooptation, and repression to increase his
expected utility. Finally, the model is illustrated by applying it to cases in
Nigeria and Zaire/Congo.

~~~
dfc
Link to version that does not require signing up for and signing into drop
box:

[https://papers.ssrn.com/sol3/papers.cfm?abstract_id=522362](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=522362)

Big brother's little sister is just as bad. Why not give the ssrn link?

~~~
LiamMcCalloway
I didn't check whether the paper was publicly available and not on gated
publishers' websites, my apologies.

------
csears
So could we spoof bogus metadata to hide Paul Revere?

In other words, if a large enough number of people started start calling and
texting random Verizon customers, tweeting with random people from the middle
east, inviting random people with Muslim-sounding names to Google+ Hangouts,
and commenting on every Facebook like with "This ____ is the bomb!"... could
that tip the signal to noise ratio enough to defeat this type of analysis?

~~~
blktiger
If you haven't read it yet, check out Cory Doctorow's book Little Brother
([http://craphound.com/littlebrother/about/](http://craphound.com/littlebrother/about/),
[http://craphound.com/littlebrother/download/](http://craphound.com/littlebrother/download/)).
He explores ways that average citizens could fight back against technology
that watches our every move.

------
antoko
This is very clever and all, but from the British perspective - this is
exactly what they would have been looking for and Revere would be considered a
revolutionary and a real threat, so this is the system working as it is
supposed to.

This is the equivalent of saying that the NSA can use such metadata to find
muslim extremists that want to kill your children and implement sharia law.
(hyperbole courtesy of American media)

Changing the timeline such that "americans" are the underdogs rather than the
establishment isn't really helpful... is it? Or is this attempt to stoke
revolutionary sentiment?

~~~
ant512
It spins the story around in such a way that Americans can relate to the
dangers of a surveillance state.

You're suggesting that the NSA is just looking for bad guys and that they
_should_ be able to capture evil muslim extremists.

But now the US government decides to bring back alcohol prohibition. Drinking,
they say, is immoral, and so are the people who drink. Do you still have
nothing to fear?

Abortion gets outlawed, and its supporters are branded as immoral. Still got
nothing to fear?

Creationism gets accepted as a viable, scientific alternative to evolution and
gets taught in all schools, and people who believe otherwise are dangerous
subversives.

Are you now one of the bad guys? Are the NSA now interested in you?

I find it interesting that Americans can believe that the right to bear arms
will protect them from the world's largest military, should it ever be run by
tyrants, but at the same time believe that the government will always share
the same moral code that they themselves hold.

~~~
euroclydon
It's often been said during during debates about government waste and
inefficiency, that citizens do not actually want an efficient government. The
typical example is: would you want a boot to appear on your wheel, in your
driveway, the minute your license plate expires? Because that would be
efficient.

The average person feels that they are getting more benefit out of Google
searches, Gmail, Facebook, cell phones, and credit cards for that matter, then
they are being harmed by volunteering so much information to the corporations
who run these businesses.

If you think the U.S. is headed in the direction of outlawing alcohol or
teaching creationism is all schools, I respectfully suggest you find something
different to worry about -- maybe pot... wait, we are becoming more lenient on
it as well.

I find it a bit silly, that when the government adopts new technology to fight
crime, everyone brings up their pet boogey-man to try to strike fear in their
like-minded citizens. I honestly can't understand if you are in support of,
for instance, using web surveillance technology to capture and prosecute
abortion clinic bombers or not.

~~~
nhaehnle
> The typical example is: would you want a boot to appear on your wheel, in
> your driveway, the minute your license plate expires?

As long as they simultaneously provide the renewal for the license, why not?
It would even be a service, since I would have less of hassle dealing with it
and remembering to go to their office.

A government that is so efficient that it is basically impossible to even get
into trouble with these kinds of administrative things can be a very good
thing, as long as it is sufficiently good-natured.

The trouble comes in areas where there is more uncertainty, with laws that are
so flexible that it's hard to tell whether you're guilty or not even when you
know all the facts. If any prosecutor with an ax to grind can charge anybody
with a crime, you have a problem. If, in addition, there are efficient
databases that allow the prosecutor to find _evidence_ against anybody even
long after the fact, you have a recipe for totalitarianism.

~~~
nooneelse
> A government that is so efficient that it is basically impossible to even
> get into trouble with these kinds of administrative things can be a very
> good thing, as long as it is sufficiently good-natured.

Indeed. One route (the one that I'm trying to keep echoing in my head these
days) to thinking about such a nicely-efficient government is to ask how
something would go in Iain Banks' Culture.

------
danso
I don't know if David Simon's (creator of "The Wire" and "Treme") defense of
the Verizon phone records request ever made it onto HN's page:

Here it is, if you hadn't seen it: [http://davidsimon.com/we-are-shocked-
shocked/](http://davidsimon.com/we-are-shocked-shocked/)

Apparently it was so controversial that his site crashed from the traffic, and
he had to tell everyone to chill out: [http://davidsimon.com/nsa-and-fisa-
commentary-calling-it/](http://davidsimon.com/nsa-and-fisa-commentary-calling-
it/)

Anyway, why I thought of that in relation to the OP was, that I think some
defenders of the NSA and general government surveillance policies are just
unaware of how technology can fundamentally change things...As Google leaders
have been known to say, "Speed _is_ a feature"...and so it's not the finding
of information that makes the establishment of Google time in human
civilization, but how _fast_ Google allows us to do it.

So that said, Simon is one of the journalists I have the absolute highest
regard for...I'll be one of the many who think "The Wire" is the best TV drama
ever, both for its artistic take and for its illustration of how institutions
-- the police, the schools, the drug trade -- corrupt even the best of
individuals. "The Wire" is heavily based off of the year that Simon embedded
himself in the Baltimore homicide department...the book (which spawned a
network TV show) is the best book about the practice of journalism I've ever
read. After a year following the detectives, you'd think Simon would be pretty
much in cahoots with the police...but he followed up "Homicide" with "The
Corner", in which he spent a year embedded with drug dealers and their
customers...apparently most of the friendsships he made in the Baltimore
Police department evaporated after he published a book bringing sympathy to
Baltimore's downtrodden.

Anyway, I don't think Simon has a love for government or authority. But I do
think he's a little naive when it comes to advances in technology and their
consequences. When "The Wire" started, the police were focused mostly on
tapping pay phones. By the time "The Wire" ended, the police were surprised at
the advent of camera phones. So when Simon says he thinks the NSA and other
law agencies won't abuse their wiretap authority, _I believe him_...because in
much of his experience, the practical obstacles (such as, having to have an
officer watch a payphone all day) made it basically impossible for blanket
surveillance.

But technology is different...I think Simon's -- and others who I respect --
mistake is to think that the game is being played the same as it always is. It
may be the case that the NSA is staffed with as people as good and
conscientious as anywhere else...but it's naive to think (as was the primary
lesson in "The Wire") that the power they have will lead them astray...and to
those of us affected by it, it makes no difference if the violations were
intentional or accidental.

Anyway, back to Paul Revere and the OP...I think it's a great example. But of
course, what makes that educational scenario feasible is technology and the
ability to record information (metadata or whatever you want to call it) in an
organized way.

Frankly, I kind of thought anyone who read 1984 would understand how
technology changes everything. But yeah, I do think there are some well-
meaning people don't grasp the technology, and if they did, they'd have a
different opinion about the dangers of unchecked surveillance.

* edit: misspelled conscientious as 'contentious'

* edit: as an example of how much Simon continues to challenge the police as a citizen, here's an essay he wrote after the success of the Wire, in which he tried, as a citizen, to get the basic details of a cop-involved shooting, something that has always been public record. He eventually succeeded, and the revelations about the officer involved ended up jeopardizing the prosecution: [http://davidsimon.com/in-baltimore-no-one-left-to-press-the-...](http://davidsimon.com/in-baltimore-no-one-left-to-press-the-police/)

But as you can see if you read the piece, Simon is not to thrilled with how
the Internet has displaced newspaper journalism

~~~
yesbabyyes
In the words attributed to Joseph Stalin: "Quantity has a quality all its
own."

~~~
coldtea
Well, this particular quote might come from Stalin, but it's based on an old
Hegelian idea, that of a surplus of quantity becoming quality (add enough mass
and you get a black hole, temperature over 100oC and you get boiling water,
etc).

(Makes sense, since Marx, Lenin, Stalin et co were heavy readers of Hegel).

From Wikipedia: "One important dialectical principle for Hegel is the
transition from quantity to quality, which he terms the Measure."

[http://en.wikipedia.org/wiki/Dialectic#Hegelian_dialectic](http://en.wikipedia.org/wiki/Dialectic#Hegelian_dialectic)

Also check the ancient greek notion of the "sorites paradox", which deals with
a similar idea:

[http://en.wikipedia.org/wiki/Sorites_paradox](http://en.wikipedia.org/wiki/Sorites_paradox)

------
msandford
This is a really great counterpoint to the "if you have nothing to hide..."
crowd.

~~~
mseebach
Well, Paul Revere _did_ have something to hide.

~~~
msandford
Yes he did. The point is that having something to hide doesn't necessarily
mean you're a criminal or a terrorist. Or that in some situations what our
government today would classify as "criminal" or "terrorist" is in fact an act
of patriotism.

We all know that history is written by the victors so I won't suggest that
what Paul Revere and the other revolutionaries did was an unmitigated good.

But I will argue that needing to expend the resources it takes (salary,
benefits, management, etc) to have one human being monitor another does put a
pretty good check on the ability of those in power to stifle dissent. It also
provides some kind of a critical threshold over which the revolutionaries
can't reasonably be stopped. In many ways that's a good thing provided it's a
high enough level. Once the public opinion swings the best thing to happen is
for those in power to give up on suppression and start packing their bags.

------
quchen
While being an interesting read, the "old world" scenario was kind confusing
me every couple of lines. (Could be because I'm not a native speaker.)

~~~
aidenn0
If you're not a native speaker, you're probably also not as well versed in the
folklore of the US revolution. Paul Revere is a popular hero of the US
revolution, so using him as the target terrorist invokes "one mans terrorist
is another mans patriot" argument at the same time as it demonstrates the
usage of metadata.

~~~
gwern
And it cuts doubly, because a lot of the tactics of the rebels were what we
would call terrorism these days - tarring-and-feathering is simple torture and
terror tactics.

------
kevinpet
This is a lot of fancy math and graphics that don't differ all that much from
what a very simple analysis can give you. Only Revere belonged to five of the
groups of interest. Of the three who belonged to four groups, two of them are
in "top scorers" on centrality, and the third, with one of the two in
centrality, is in the final table.

~~~
rollo_tommasi
It's an exceedingly simple example used to illustrate the technique. The
methods included in the python library are robust enough to work just as well
on much larger and more complicated data-sets but that would be too difficult
to follow for someone who isn't familiar with the subject.

~~~
kevinpet
I think it's hard to understand what the point is without a result that
differs from the naive solution. That is, my naive "who's an important
terrorist" is "who belongs to the most terrorist groups".

~~~
sesqu
While belonging to the most terrorist groups will usually give the highest
score, it's not quite the same as what betweenness actually measures -
belonging to the most terrorist groups that are each large but do not share
many members.

------
rickyconnolly
You ʃhould have uʃed the long ʃ in this article

~~~
surlyadopter
The purfuit of happineff?

~~~
deadairspace
The character is ſ, and it doesn't go at the ends of words, so it would be:
The purſuit of happineſs.

------
dfc
kjh the author of this link has a greatintroduction to emacs:

[http://kieranhealy.org/resources/emacs-starter-
kit.html](http://kieranhealy.org/resources/emacs-starter-kit.html)

He also has a lot of good reference material on latex/org-mode/pandoc.
Definitely a good resource to have handy if you ever have a less-techie friend
who wants to get away from MS word and its ilk.

------
jroseattle
Makes me think of the ramifications of false positives.

------
lettergram
This would mean the larger your network, the larger your likely hood of being
a terrorist

~~~
pseut
From a predictive standpoint, that's not necessarily an error. Think of it as
"the larger your network, the higher priority to investigate for terrorism."

