
Ask HN: Can we help the police sort through the photos of the Boston Marathon? - jbaudanza
The police are going to need to sort through and catalog thousands of photos and videos. They may not be prepared or have the infrastructure to do this efficiently. Maybe we can build something to help.<p>Some ideas:<p>- A central place for people to upload photos and videos from the event<p>- A system to sort by time and location of the media<p>- Duplicate detection of photos and videos<p>- A forum to discuss the photos<p>- A way to crowdsource tedious tasks. For example: See if person X or item Y is in any of the photos of videos.<p>I'd love feedback on this idea. Would this be helpful? Does it already exist?<p>Edit: Getting lots of great feedback. If you're interested in helping, e-mail jon@jonb.org.
======
AYBABTME
One way I see one could crowdsource the problem without turning this into a
witch-hunt would be to collect as many pictures as possible and try to
recreate a 3D model of the scenes at various timestamps. Something like,

    
    
      * Go on project website.  There's a 3D mesh of the area.
      * Upload your picture:
       - Select a time frame you think fit your picture.
       - Navigate to the location you took your picture from.
       - Try to apply the picture on the mesh as well as possible.
      * Or, look at what people posted:
       - Try to correct other's time-positioning, or
       - Try to correct other's spatial-positioning
       - Fix the mesh for missing things.
      * Investigators can use that to look for clues on the scene.
    

Like OpenMap or Google Maps where people can send corrections. Of course, I
don't assume that this is trivial to implement. Just, an idea of how such
future problems could be approached.

~~~
GrantS
If anyone is attempting to build something like this, I'll just note that
automating this process (spatio-temporally registering photographs and
reconstructing 3D models of cities as they change over time) was essentially
the topic of my PhD thesis, but on historical time scales (1860s-2000s) rather
than minute-to-minute.

Slides from my CVPR 2010 presentation here:
[http://www.cc.gatech.edu/~phlosoft/files/schindler10cvpr_sli...](http://www.cc.gatech.edu/~phlosoft/files/schindler10cvpr_slides.pdf)

Videos: <http://www.cc.gatech.edu/~phlosoft/>

The best way to get a system running quickly would be to use Noah Snavely's
Bundler project for the 3D reconstruction:
<http://phototour.cs.washington.edu/bundler/> and, at first, trust most EXIF
tags for time.

~~~
aosmith
Super interesting, thanks for sharing!

------
trotsky
I understand you want to help, but i would have some serious reservations
about building a proof of concept crowdsourced surveillance analysis platform
and demonstrating whether it's viable. There are an incredible array of
surveillance tools involved in the fight against terrorism. I'm not sure
encouraging the general public to actively participate in a domestic
intelligence platform would have a net positive impact on society & our basic
rights.

~~~
siddboots
But given that there are no technical limitations for this being developed,
it's safe to assume that it will be developed at some point, if it has not
been already.

Wouldn't you prefer that something like this exists in the public domain?

~~~
gtank
Just because a given surveillance/warfighting/CNO technology COULD be
developed doesn't mean that one should immediately go do it.

------
gusgordon
The only practical way of making something like this work would be to collect
all the photos taken ~2 hours before the bombs went off near the finish line.
You could do this by searching Facebook, Twitter, and Instagram for geotags
and keywords/hashtags then sorting by date and time. Dump all the photos into
a database and do your best to sort by time (duplicates shouldn't be much of a
problem). I don't think there would be too many photos, probably less than
5,000. Then see if you can find what the bombs looked like before they went
off, then go back and find when they first placed.

If anyone is interested in doing something like this, I can help, so let me
know.

~~~
smoyer
Perhaps this could be combined this with the information below and we could
search pictures taken prior to the blast for black duffel bags that might
contain 6 quart pressure cookers.

~~~
pvaldes
I don't really like the idea of putting a lot of photos of traumatized people
on internet, photoshit happens, the last thing that those sons, women, family
could merit is to be lynched on internet.

...but

If you know or suspect that the bags were black, a reasonable solution that
maybe could have success without to blow up all the remains of privacy could
be:

1-to create a filter that makes a copy of the originals AND change all colors
of the bunch of photos in a named directory to i.e. soft blue... EXCEPT a
small range of black tones.

2- Thus, you could check or search then in the bunch with another script what
photos have those suspicious black tones. And you could choose to move those
photos to a different dir for a preliminary analysis, maybe saving a lot of
time.

2- or you could search for photos taken from same place in different times
(very frequent in a competition) that have black tones NOT changing of place
in a specific time. People moves, claps, node, etc, bags with bombs don't.

3-You could refine the script to permit to enter any desired tone also and
mask all the other colors. The goal is to find quickly a bag, but keep the
privacy of innocent people and victims that do not deserve this.

I thing that should not be much challenging technically.

~~~
smoyer
They shouldn't be too traumatized in the hours before the blasts ... and
that's when we'd be looking for the black duffel bags.

~~~
pvaldes
They are traumatized now, that's for sure :-(

To be exposed on internet again and again, or to put pictures of happy friends
and members of the family that now are in an hospital will add a lot of
unnecessary suffering and probably will enlarge the "satisfaction" of the
authors, the "success" of the operation, and I'm writing this with a very
angry face.

We have to avoid this.

If you have a photo of the marathon, any photo, not only of the finish line,
send this photo to the police.

To care and show respect and love for the victims, dead and alive, should be a
priority, in the same level as to put the authors in jail. (only my opinion).

------
stevenameyer
A similar technique was used to identify suspects in the 2011 Stanley Cup Riot
in Vancouver, Canada.[0] With people using the footage and pictures of the
riot and submitting identities to the police. I'd be interested in seeing
similar techniques used else where. It seems to have worked extremely well in
the case of this specific riot.

[0] <https://riot2011.vpd.ca/>

~~~
pbhjpbhj
This thread reminds me of the "hunt for things on Mars" online tools
<https://www.zooniverse.org/project/planet_four>.

You could basically tag all the specified features in images (in this case it
would be faces, bags) and add special comments if you thought you saw
something special (a person you recognised, a bag make, an apparent clue).

That would crowdsource the facial/bag recognition and identification aspects
without giving scope for trial-by-webapp.

------
shocks
I love the idea, but I think it impractical. It's too difficult stop it
degenerating into "this guy looks a bit shifty", "this guy is standing weird",
"this woman isn't laughing", etc.

Just look here[1] and you'll see it's already happening.

[1]:
[http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_m...](http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_marathon_attacks_please_upload_any/)

~~~
incision
Anyone thinking about enabling further crowdsourced investigative work would
do well to read that thread [1] with an eye to how quickly it got out of hand.

In my opinion...

Efforts to help with this should focus on helping people to submit any data
and information with relevant context they have in a consistent format to the
_relevant authorities_ , nothing more.

1:
[http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_m...](http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_marathon_attacks_please_upload_any/)

------
auctiontheory
[After reading Reddit thread.] My goodness. Flying while brown is bad enough.
Now do I have to worry about being tackled by vigilante do-gooders every time
I step out of the house with a backpack? Or, Cthulhu forbid, put my bag _down
on the ground_!

------
spikels
If you go to the police or FBI and offer help I am pretty sure they will say
no. Except in very controlled situations they are unlikely to share any of
their evidence. It is just not their culture.

However I wonder what you could achieve without their cooperation. Could you
"crowdsolve" a crime? Say a website where people post what they know and any
evidence they have.

~~~
cyphersanctus
The FBI will indeed crowdsource the evidence:
<https://news.ycombinator.com/item?id=5560609>

~~~
wbhart
They don't understand the term crowdsource. They think this means that the
crowd will be their source. This is explicitly spelled out in the article.

~~~
smoyer
In social science experiments, it's not unusual to have undergrads or grad
students classify the raw data for certain characteristics (a process
generally referred to as coding). If we produce (the crowd) a large amount of
data, do you really think they'll reject classification information? If
someone walked into their office and said "Look at this picture I took of a
guy putting something in a trashcan", I suspect that person would get pretty
immediate attention.

These agencies often appear to be slow because they've got political
bureaucracies as well as defined processes (chain of custody in evidence is
pretty important), but it's a mistake to conflate their slower speed with a
lack of intelligence.

------
jchung
Boston here. Couple of us had the same idea and are meeting tonight in the CIC
(101 Main in Kendall Square) to pull together an app. Designers and PR ppl
especially welcome. For now, please contact me at @jaredchung on twitter and
I'll rope you in.

------
startupfounder
Neartime is a very interesting piece of software that might be in the
direction this is going, could help in building a timeline of publicly
available (fb, Instagram, flickr, twitter, etc.) photos based on
lat/long+time. I believe it was only built for flickr.

[http://blog.logicalrealism.org/2007/09/23/neartime-find-
flic...](http://blog.logicalrealism.org/2007/09/23/neartime-find-flickr-
photos-taken-nearby-in-time-and-space/)

------
evan_
This is something I was wondering about as well. Would evidence gathered in
this way be admissible in court?

I worry that once the general public gets ahold of it, it would simply become
a "middle-eastern-looking person finder".

~~~
spikels
I'm no lawyer but I don't we why it wouldn't be admissible unless it somehow
violated the perpetrators rights. Even then since we are not the police the
situation might be different.

Anyone know for sure? There must be rules of evidence that cover this
situation.

~~~
berberous
I think there may be issues with authentication. Usually, you would have a
witness authenticate a photo ("yes, i took this with my camera at this day and
time, etc."). If it's a random photo on a website, it may be more difficult to
prove its accuracy.

~~~
spikels
Seems like there would be two stages: (1) solve the crime and (2) gather the
evidence. Seems like as long as the evidence is not tampered with it would
still be usable in court. For example uploading a picture shouldn't destroy
its value as evidence.

------
wavesounds
Google and Facebook both have facial recognition technology and the
infrastructure to handle this amount of data easily, one of them should step
up. It would probably be good PR.

~~~
tseabrooks
I find the idea that the Federal authorities don't already have the technology
and infrastructure necessary themselves somewhat laughable / unbelievable. Am
I way off base here?

~~~
danso
Yes you are. Have you been to a police station lately, like as a witness or a
victim, and have had your information processed?

Here's a very telling example that happened a few weeks ago in NYC, with
arguably the most advanced police force in the nation (NYPD). A woman was
violently beaten and robbed in a subway station. The mugger was wearing a
fraternity jacket with his nickname that, if you entered it into Facebook, you
would've found his public facebook page full of photos of him wearing that
jacket.

The police took _a month_ to release this brutal video. It took commenters on
Gawker, of all places, to doxx this guy on Facebook, who ended up getting
charged with the crime.

[http://abclocal.go.com/wabc/story?section=news/investigators...](http://abclocal.go.com/wabc/story?section=news/investigators&id=9053236)

Do not assume anything about the technological capability of law enforcement.

~~~
Evbn
You are conflating level of interest with technological capabilities.

~~~
danso
No, I'm not. Violent muggings are not a backlog type of crime for the NYPD
because they do not want a serial robber running around. You're right that
high-profile crimes will bring in new technology and capabilities, but if the
infrastructure isn't there (i.e. the tech skills of the officers in the field
and in the office), then it's not a given that the processing of mass digital
information is going to be smooth.

------
jbaudanza
The general feedback so far seems to be "Good idea, but could result in mob
justice."

How can we prevent this? Some ideas so far:

\- Moderation

\- Give people specific tasks: "Find this duffle bag" (thanks shocks)

What else?

~~~
trilobyte
What about just a way to forward on important images to authorities?

~~~
jbaudanza
I like that. If you flag a photo as suspicious, that flag doesn't necessarily
have to be publicly visible.

------
xauronx
I was thinking that same thing this morning. I've seen people on reddit find
where people live just from a photo out of their front window. I've seen
people track down stuffed animals that were made 20 years ago. The human
hivemind and it's capabilities has to be put to a valid use occasionally.

Anyhow... What specific goals would you be trying to reach? Identify every
individual who was there? Try to find photos from before the incident of
people with suspicious objects?

~~~
jeffreyrusso
I was down at the finish line (on the opposite side of the street) a couple of
hours before the bombing watching a family member cross the finish line. My
worry is that the crowd was so tightly packed that it might be tough to get
any footage of what was happening at ground level. If I had simply looked down
to see my feet, I wouldn't have been able to see them because we were so
crammed in. If an object was on the ground, or if someone carried a backpack
in at ground level, it seems like it could be missed by even the thousands of
people snapping photos at all angles.

That being said, I'm wondering if you couldn't use some sort of facial
recognition in conjunction with timestamps to track/detect unusual movements
(like someone who quickly moves into the area of the detonation, then turns
around and quickly moves out.)

------
nateaune
Some of us tech entrepreneurs in Boston built this site yesterday to aid in
the collection of photos and videos from the Boston Marathon.
<http://evidenceupload.org>

It can be used from a mobile phone to bulk upload photos/videos directly from
the phone and preserving all metadata (timestamp, lat/lon coordinates).

We used Filepicker.io for the upload widget, Amazon S3 for the file storage,
Heroku for hosting and Ruby on Rails.

------
WestCoastJustin
I suspect the software already exists and here is why: Following the 2011
Vancouver Stanley Cup riot [1] there was tens of thousands of photos and
videos posted to social media. Vancouver Police (VPD) where initially
overwhelmed with tips and digital media, so VPD setup a website [3] to
aggregate the data and identify suspects. ICBC (Vancouver BC's provincial
entity that issues drivers licenses) offered facial recognition software [4]
and their database (although I think this idea was scrapped due to privacy
issues). An organization called LEVA helped investigators analyze 1,600 hours
of video evidence [5]. The VPD riot website [3] puts the hours of video
evidence at 5,000 [6]. VPD used _a team of 40 LEVA forensic video analysts and
technicians from across the United States, Canada, and the U.K. was assemble
for an intense two-week period to review the evidence._ [5]

So, I think these people will step up and help if the government has not
already developed this capability. Wired states that "Data for the Boston
Marathon Investigation Will Be Crowdsourced" [7]. This process is likely
already underway.

p.s. since I mentioned the vancouver stanley cup riot, I should also mention
the vigilante justice that followed [8]. Obviously these two instances are
much different but I think it is worth mentioning. People were "naming names"
on-line, people were fired [10], people were expelled [9], people were
targeted with death threats [11]. IMHO this software can be very dangerous
when operated by the public.

[1] <http://en.wikipedia.org/wiki/2011_Vancouver_Stanley_Cup_riot>

[2] [http://m.theglobeandmail.com/news/british-columbia/riot-
inve...](http://m.theglobeandmail.com/news/british-columbia/riot-
investigators-overwhelmed-by-internet-leads/article2068499/?service=mobile)

[3] <https://riot2011.vpd.ca/>

[4] [http://www.straight.com/news/icbc-offers-facial-
recognition-...](http://www.straight.com/news/icbc-offers-facial-recognition-
technology-vancouver-polices-riot-investigation)

[5] [http://pipelinecomm.wordpress.com/2011/09/14/leva-
activates-...](http://pipelinecomm.wordpress.com/2011/09/14/leva-activates-
forensic-video-analysis-response-team-to-support-vancouver-police-department/)

[6] <https://riot2011.vpd.ca/faq>

[7] <http://www.wired.com/dangerroom/2013/04/boston-crowdsourced/>

[8]
[http://www.vancouversun.com/technology/Online+vigilantes+slo...](http://www.vancouversun.com/technology/Online+vigilantes+slow+police+investigations+report+Stanley/7478872/story.html)

[9] [http://www.cbc.ca/news/canada/british-
columbia/story/2012/09...](http://www.cbc.ca/news/canada/british-
columbia/story/2012/09/13/bc-alexander-peepre-riot-sentence.html)

[10] [http://www.cbc.ca/news/canada/british-
columbia/story/2011/06...](http://www.cbc.ca/news/canada/british-
columbia/story/2011/06/22/bc-rioter-fired.html)

[11]
[http://www.vancouversun.com/technology/rioting+teen+nathan+k...](http://www.vancouversun.com/technology/rioting+teen+nathan+kotylak+family+face+backlash+forced+leave+home/4972283/story.html)

------
zdgman
This is a great idea! I haven't seen anything like it and also I think you
kill of the forum to discuss photos and just attach comments to photos
directly (sort of like instagram). Wonder if there is anything legally against
chipping in like this?

------
uxrobinson
We have a technology that goes much further than the suggestions above. For
this use case, would need a little customization. If you want to get involved,
email me - barrie@cliplabs.com - jon@jonb, perhaps combine our efforts?

------
ashleyt
Despite negativity of this idea. I think you should totally go for it! How do
we know how useful it will be until it's done? There's no point speculating.

What harm can come out of this?

You could simply have all photo's uploaded and allow users to click and select
things they feel might be suspicious with comments added. If items are
highlighted by many users, it would show that it might be suspicious and it's
worth the police looking at! That way they could order the most important and
possible remove ones that they have personally looked over and think there are
no problems.

------
dewey
This reminds me of this project: <http://www.snapshotserengeti.org/>

------
Macsenour
I see a web app that lets the police point to a spot on Google Maps, and the
app provides a list of files stored that would show that spot in video or
film. And on the maps screen, show the angle of those pictures. Then allow the
police to highlight the file, and show the picture source point, and the V of
the picture window.

If that makes sense.

Thoughts?

------
koenigdavidmj
Amazon Mechanical Turk has been used for this type of thing (specifically
attempting to find Steve Fossett).

------
traughber
Great idea Jonathan. I think a really simple and quick-to-build tool similar
to what you suggest would be really helpful in situations like this. Right now
people are eager to help out but there isn't a great place to see all of the
photos and the best leads/analysis.

------
qwetryuop
there is a reddit thread already attempting to do this, but it's meeting some
resistance.

source:
[http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_m...](http://www.reddit.com/r/boston/comments/1cf5wp/2013_boston_marathon_attacks_please_upload_any/)

~~~
shocks
From reading the comment I think this evidence enough that it will be very
difficult to stop this from being a "mobsource" application.

------
CodeCube
Reminds me of the search for Jim Gray. Story here:
[http://www.lastwordonnothing.com/2011/03/17/searching-for-
ji...](http://www.lastwordonnothing.com/2011/03/17/searching-for-jim-gray/)

------
roc
Photosynth still around?

~~~
wavesounds
Great Idea! Not sure how you deal with time though... maybe one synth every 5
minutes?

<http://photosynth.net/>

~~~
roc
You don't. You can't, really.

You just let synth collect things into geo-locations to speed up the sorting
process. You still have to manually sort all images that overlay a 'point of
interest'. But the real boon is that you can quickly, intuitively and
effectively do that sort.

And if you see a 'person of interest' you can 'walk' down a possible path and
sort the images at intermediate points to see if you can establish a route or
potentially see them from other angles.

------
jbaudanza
If you want to volunteer to help build this, please leave a comment or send me
an e-mail. jon@jonb.org

------
FollowSteph3
You can't because where do u think the bad guys would go right away. Not only
that but they know where and what to look for. In other words you're giving
them all the info you have on them. Not the smartest thing to do ;)

------
adkatrit
it would be great if the news organizations that are capitalizing on
advertisements on boston bombing articles, would donate those proceeds to
victims.

~~~
aasarava
I don't know if it's fair to say news organizations are using the bombing to
gain financially. They are doing their jobs -- which is, providing a service
to gather and disseminate news. In order to do that service, reporters and
editors need to be paid.

You could look at another way: If the service didn't exist, many people would
likely not know the news, or the extent of it -- and so might not be compelled
to donate or otherwise help out.

~~~
adkatrit
I'm not implying that they are _using_ the bombing for financial gain, however
it is clear that horrific(and great, positive) events correlate with a certain
spike in revenue in the news industry. What I am proposing is charity. Suppose
event X creates Y dollars in ad revenue from it's initial spike in traffic.
This would be a donated percentage of that money. Let's not forget that
reputation is a key factor for creating loyal users.

If one had to choose between news service A vs news service B, knowing that
choosing A has the implication of assisting those in need, people are going to
choose A.

Call it pre-emptive humanitarian news campaigning.

~~~
geofffox
A week's on-scene coverage cost more to mount than the usual news from the
studio. Beyond that, live coverage 'bumped' many commercial avails. This was
an expensive week, not a profit center.

