
An Example of Forensic Science at Its Worst: US v. Brig. Gen. Jeffrey Sinclair - jzdziarski
http://www.zdziarski.com/blog/?p=3717
======
patio11
Ironically, the amount of pressure from the top involved in this (and related)
cases caused several prosecutions of sexual misconduct in the military to be
_dismissed_. You read that right. There's a doctrine called "unlawful command
influence" where, if the posture of the Commander on Chief on down suggests
that there is a "right verdict" independent of the facts, then the military
courts should dismiss cases rather than having military officers be,
essentially, ordered to convict.

A certain US president who is a Constitutional law scholar has run into this
at least three times and professes to be really surprised every time it
happens.

see generally: [http://www.nytimes.com/2013/07/14/us/obama-remark-is-
complic...](http://www.nytimes.com/2013/07/14/us/obama-remark-is-complicating-
military-trials.html?pagewanted=all&_r=0)

~~~
lotsofmangos
We have something similar going on in the UK with Cameron making commentary
and publicly taking sides on court proceedings.

[http://www.bbc.co.uk/news/uk-england-
london-25350419](http://www.bbc.co.uk/news/uk-england-london-25350419)

[http://www.bbc.co.uk/news/uk-politics-28014035](http://www.bbc.co.uk/news/uk-
politics-28014035)

~~~
hga
While that's not good, it's very different thing. Those two examples were of
normal, civilian courts, proceeding of an independent judiciary; a President
or Prime Minister has the "power of the bully pulpit", but no formal power.
Our minimalist Constitution goes so far as to say Federal judge compensation "
_shall not be diminished during their Continuance in Office_ ", i.e. the other
branches of government can't dock their pay.

In military courts, the judges and jury are in the chain of command, so as
patio11 put it, " _the military courts should dismiss cases rather than having
military officers be, essentially,_ ordered _to convict._ " Hence the formal
name "unlawful _command_ influence".

------
rayiner
What this really highlights is that lay people trust computer forensics people
to be experts, and often that trust is misplaced. Unfortunately, this isn't
the only example of misplaced trust. In several places, Zdziarski laments that
computer forensics doesn't live up to the standards of proper "science." Yet,
his trust in the rest of forensic "science" is also misplaced. Pretty much
everything besides DNA forensics, maybe fingerprints, is junk:
[http://lst.law.asu.edu/FS09/pdfs/Koehler4_3.pdf](http://lst.law.asu.edu/FS09/pdfs/Koehler4_3.pdf)
(page 4).

Indeed, the National Science Foundation has been looking into the field and
has been shocked by how unfounded it is:
[http://www.nsf.gov/pubs/2013/nsf13120/nsf13120.jsp](http://www.nsf.gov/pubs/2013/nsf13120/nsf13120.jsp)
("While the report acknowledges that 'the forensic science disciplines have
produced valuable evidence that has contributed to the successful prosecution
and conviction of criminals as well as to the exoneration of innocent people,'
it cites a need for systematic research to validate the various disciplines’
underlying assumptions and methodologies, adding that the 'forensic science
... communities will be improved by opportunities to collaborate with the
broader science and engineering communities.')

All that is just a polite way of saying: "holy shit your techniques lack real
scientific foundations; you guys need to hire real scientists."

~~~
Zigurd
On top of the junky aspects of forensic "science" are many bad practices.
1600+ cases in Massachusetts were called into question due to a crime lab
employee who told police and prosecutors what they wanted to hear. She was a
hero until she wasn't.

She was convicted of a crime and got a short prison term. The investigation
found she "acted alone."

Is she an outlier?

------
Radim
A sad read on many levels.

 _" I investigated a little. As it turns out, this particular manufacturer
first copies the iOS file system onto the Windows partition that their
software is running on, and then pulls the timestamp information off of the
copy of the data."_

 _" Irresponsible, to say the least. But this is the quality of the forensics
software assisting our government and military. Poorly written, over-priced
assumptionware."_

Also, the OP's professional frustration is endearing -- typical woes of an
expert who "knows and cares", observing snakeoil barons become rich on
bullshit :-)

~~~
wahsd
I think you should understand that America's economy has a long an illustrious
history of "snakeoil barons becoming rich on bullshit". It's kind of the very
basis of our whole system.

------
DanielBMarkham
_"...This, and many other types of artifacts are often either completely
overlooked by numerous commercially sold, expensive-as-hell tools, or in the
case of at least one tool – seemingly made up data. All of these came into
play in this case and would later play a role in its outcome..."_

So -- prosecutors and investigators don't understand the tools they use to
search phones, judges give them wide leeway in reporting such "evidence", AND
some commercial tools make up data to return?

Does this not deeply concern anybody in any of the branches of the government?
I hope I'm not being alarmist, but that statement, if true, seems to me to be
incredibly damming of the entire criminal justice system. It's like saying the
prosecutors determine who's guilty simply by selecting them for trial. The
rest of the work is simply acquiring and configuring the appropriate "tools"
(Note that it doesn't say they do this on purpose. I guess that's something.)

------
arjie
I wonder if anyone has ever been wrongfully convicted because of poor forensic
software. Those programmers are incredibly unethical people.

I'm familiar with the food-on-the-table argument, but wow, possibly getting
innocent people locked up just because you don't know what you're doing is
something. It can't feel good to receive the double whammy of knowing you're
incompetent and that you're incompetent enough to ruin lives.

Would I do it? I don't know. Through good fortune, I've never been, as a
programmer, in the situation where I worried about losing my job and being
unable to find one. So maybe those of us who feel we would never do something
like this (work on critical health software while ignorant, or on forensic
software while clueless) just don't know that it is easy when your livelihood
is threatened. Still really unethical, though.

~~~
FLUX-YOU
>work on critical health software while ignorant

This struck a nerve because that's how I feel in my position. I really had no
idea what went into the products that I'm not working on. I feel ignorant, but
someone's giving my code the okay and occasionally throwing some back, so I
must be doing something right. Someone said, "He looks like he can handle this
job" and here I am with no prior experience in this particular field (I did
have some programming experience). I wonder how many juniors these forensic
software companies employ?

On one hand, mission-critical companies should only hire people who are both
experts in programming, experts in safe and reliable programming, and experts
in the domain that they programming in. (note: actual experts, not "i've-done-
it-once-or-twice-so-i'm-the-department-expert-because-no-one-else-is-really-
working-on-it" expert)

On the other hand, it's incredibly hard to find all three of those things in
sufficient numbers to staff a team capable of handling large and complicated
software. With employees in general changing positions more often (sometimes
on whim alone, nevermind what the world needs), it's even harder to cultivate
5+ year employees who really know their way around the software and the
business.

There is just a huge volume of information and practice required to become
that individual, and you really run into daily practical concerns if we were
to only hire these experts. You have to be fed challenges that teach you so
you can eventually become a real expert, but you need competent oversight -- a
mentor, really. And sometimes you just don't have the luxury of tackling
problems that are 'small enough'. Sometimes your mentor really doesn't have
time to teach to the level of detail needed because let's face it, there's
still a business behind all of this. And then they should be competent in
security, performance, and a host of other skills because software is just so
connected and reliant on the other parts of the system.

So now you need someone with all of those previous skills and the ability to
teach people well. These newly minted experts will have to pass down the
knowledge as well.

How do you cultivate experts without also creating a little danger to your
customers and their customers and maybe even the general population? At some
point, that trainee is going to have to make change and implement things
without a safety net behind him. He's got to do this enough until he becomes
that actual expert, but that takes a really fucking long time, and I don't
think the market is really motivating people to take that route ("Why not just
make some websites for 80k a year and save myself all of that stress and an
early death?").

If someone else makes a big mistake, most of that time you spent to become
better becomes worthless when your firm hits the front page. Now you're
associated with someone else's problem and that's going to affect your chances
of a job. So not only do you have to watch out for your own mistakes, but also
your peers' mistakes as well. While you may not have to resort to flipping
burgers, you probably won't be able to get a job in a critical environment for
a while.

~~~
sanderjd
Not to diminish the _difficulty_ of accomplishing this, but there is a good
answer to your question: it requires both a strong _culture_ of quality
assurance (this is what you're talking about with mentors and watching each
others' backs), and a strong _specialization_ of quality assurance, made up of
the experts. Unfortunately, many software organizations treat the
specialization like a joke of a useless roadblock and only pay lip service to
the culture. I think this may be because lots of software truly doesn't need
to be very high quality, while other software truly does, and determining
which kind you're making and justifying the additional cost of quality (which
is enormous) is not straightforward.

In the case of this specific article, I find it odd that the blame is laid at
the non-forensic-expert software engineers, rather than the companies that
employ them, seemingly without the support of a tightly integrated set of
experts. There is nothing wrong with dividing labor between creating software
and defining and verifying what the software does. Both jobs are difficult in
specialized ways. It seems that, in the opinion of one expert at least, these
companies are just doing a poor job of QA.

------
kghose
So, I read through the article and could not find what was concretely wrong,
technically, with the data. Does he state that any where? Thanks.

~~~
justincormack
The details are of course confidential. He says:

"There are reasons I’m not going to dig into the details of the case. Certain
people that were involved could easily be pinpointed by revealing technical
details that could be pieced together with news reports, and help build a
story in your mind that would probably be inaccurate. The details aren’t so
important as the errors that were made. All you need to know from a technical
perspective is right here: some of the types of information that these
commercial tools were (and likey still are) misreporting is significant.
Evidence and timestamps of a device erasure event. Evidence of a backup
restore event. Application usage dates. Application deletion events and
timestamps. File access times. This, and many other types of artifacts are
often either completely overlooked by numerous commercially sold, expensive-
as-hell tools, or in the case of at least one tool – seemingly made up data.
All of these came into play in this case and would later play a role in its
outcome."

~~~
zo1
_" The details are of course confidential. He says:"_

I kept having to skip paragraphs to get to the juicy bits. Eventually, I gave
up and closed the tab. I don't much like this guy's writing style, as it
constantly goes off on tangents that don't really add anything to the story,
much less the baity title.

------
Symmetry
Sadly, forensic science can actually be a lot worse than the example here.

[http://www.huffingtonpost.com/2011/09/01/michael-west-
fabric...](http://www.huffingtonpost.com/2011/09/01/michael-west-fabricating-
bite-marks_n_944228.html)

------
cmb99
An Example of Pathetically Self-Aggrandizing Twaddle, Nearly at Its Worst.

~~~
kghose
You seem to have called a spade a spade and some folks don't seem to like
that. Yes, that page (the article and the byline) does seem a little anxious
to establish the author's credentials. But that should be taken separately
from the content.

I'm surprised that the article got so many up-votes considering it has so
little technical detail.

------
sp332
Jeffery Sinclair is the name of the first commander of Babylon 5. I had to
check that it was a real name, but it seems that it is also the name of a real
brigadier general.

------
RexRollman
This article makes me wonder: does forensic software undergo any kind of 3rd
party testing for accuracy? Or are we literally just taking someone's word
that it is producing accurate results?

~~~
jonstewart
There's some, but it's far too little to be comprehensive. Some law
enforcement labs have entire QA departments dedicated to validating internal
procedures, including how particular tools should be used.

------
munin
the courts and their interaction with science has always been tenuous, the
story of arson investigators, fire science and criminal justice is similarly
terrifying [http://www.newyorker.com/magazine/2009/09/07/trial-by-
fire](http://www.newyorker.com/magazine/2009/09/07/trial-by-fire)

------
contingencies
TLDR; "computer says no" all over again.

------
zibit
The OP should start hacking:

[http://www.sleuthkit.org/](http://www.sleuthkit.org/)

------
na85
The author states they were shocked at how Sinclair seemed to get off easy,
with just a slap on the wrist. I'm not sure why that's shocking.

It's been painfully obvious for more than a decade that there exists a certain
class of people in the US who are above the law, e.g. James Clapper.

~~~
pyre
Well, it looked like the powers-that-be wanted to make an example out of him.
So I could see it be surprising that he got a slap on the wrist after having
those "guns" pointed at him.

