
Convicted by Code: Defendants should be able to inspect code used in forensics - Figs
http://www.slate.com/blogs/future_tense/2015/10/06/defendants_should_be_able_to_inspect_software_code_used_in_forensics.html
======
donkeyd
I once nearly lost a contest because of a faulty SQL query on their side. If I
didn't get to see the query, I wouldn't have been able to defend my entry and
would've lost. Losing this contest would've been trivial, but if I applied
this to a trial, it would be horrible.

The error was that a 'group by' was used to find the number of unique entries,
even though they had leading spaces, that were part of their uniqueness. Group
by doesn't take leading spaces into account, leading them to get a different
result than me. I think that this could've happened to a lot of people, even
forensic IT engineers.

~~~
esnard
Actually it isn't specific to GROUP BY, the problem comes from the fact that
some DMBS (MySQL for example) which ignore leading and trailing whitespaces
(and other stuff like capitalization) while comparing strings, leading them to
count less unique strings that a stricter comparision would count.

------
6stringmerc
Okay so this article is bumping up against the hysteria that I'd categorize as
"semi-technology literate" yet makes some good points. Almost like talking
about how dangerous it is to walk through a minefield and then stepping on
one. There's a valid point in there somewhere.

Copyright reform is one of my favorite subjects, and for a multitude of
reasons. Should the prosecution be able to dump a case straight up without
recourse because the "Stingray" gathering tool is too lovely to submit to
review? Nope. Should breathalyzer code be held from review just because it's a
product made by somebody? Nope. Should FOIA be stonewalled or pay-walled and
inhibit the Constitutional freedom of the press? Nope!

Innocent until proven guilty is a very, very important premise for the US
legal system. It's backed up by both the Fourth and First amendments to the
Constitution. Any justification to put them aside for "War on ____" might seem
reasonable on the surface, until taking a closer look at multiple murder
evidence that comes from within the borders more often than on a laptop of a
Citizen who just so happens to be coming back from a foreign country and gets
worked for passwords under duress or has to forfeit hardware without recourse.

I dunno, maybe I sound like some kind of off-the-rocker dude by thinking about
such things, but I love my country, I'm willing to sit down and think about
this kind of stuff. It doesn't have to be extreme. Taking the small steps of
talking with one another about what we really value is important in my
opinion.

~~~
jfoutz
So... if i build and sell a "breathalyzer" to your local pd that simply
randomly selects a number between .08 and .16, you'd be ok with that? Pretty
much every other element of the system is biased toward conviction. Police
have quotas. Prosecutors need convictions. Private prisons get more money for
more convicts. It would be easy to dodge questions for a long, long time. Why
would police administer the test to a sober person? In this one weird example
a sober person registered as drunk. oh well, its still good for 99.9% of other
cases.

The right to confront your accuser is there for a reason. More and more,
software is the accuser. We all, at HN, struggle to make our code correct.
Step back, think a second, how do you QA a breathalyzer? How do you deal with
variations in sensor packages? Yes, they probably do more good than harm, but
are they "accurate"? how do you know that?

~~~
ubernostrum
Breathalyzers are known to be wildly inaccurate, to such an extent that in
some jurisdictions DUI is just defined in terms of a breathalyzer result
because they know it doesn't bear a reliable relationship to the actual BAC.

~~~
Cthulhu_
But is it accurate enough to say "you're drunk driving"? IIRC at least on this
side of the pond, if you're caught you either get fined, or you're taken to
the station for a more accurate breath test or a blood test.

~~~
ubernostrum
In the US, you have the right to request a blood test, but the blood test is
not required and the breathalyzer is considered sufficient evidence (again,
since the offense is defined in terms of the breathalyzer).

------
triggercut
There are similar issues with this in Structural and Mechanical engineering.
Engineers are expected to rely more and more on software to execute and
document complex calculations to verify designs, but how can you be sure those
underlying calculations/theorems/models are correctly implemented? Some
packages are constantly patching particular edge cases that get sent to them
from their users. Many issue announcements to warn of bugs that could cause an
incorrect result.

If a result from software led to a critical failure in a design, the onus is
most likely still on the Engineer.

I have seen cases where software is formally reviewed by independent
verification bodies, much in the same way your ISO 9001 compliance is. I can't
see why this wouldn't apply here. Have an independent party, who has signed an
appropriate NDA, asses and certify that your product does what it says on the
tin and audit it at regular periods.

~~~
irq-1
> Have an independent party, who has signed an appropriate NDA, asses and
> certify that your product does what it says on the tin and audit it at
> regular periods.

This may work for safety standards or financial liability, but it doesn't work
for the legal system, because "independent" parties are influenced by who pays
them.

~~~
triggercut
Yes, that's always a factor. Larger IVBs will tend to err on the side of their
reputation in the marketplace (which may also be a factor).

One could envisage a (horrible) situation where both parties engage an IVB (a
practice common in some areas of engineering) and then they both jointly
engage another body to effectively vet their processes and pass that joint
50/50 cost into their fees back to you.

Or the court appoints one and you agree to share the costs (probably wouldn't
fly in the US).

------
finance-geek
I think things will become even worse now that criminal "scouting" and even
vetting is being done via learning models. So you may not even find hard
filters or conditionals...instead the errors (or stereotypes?) would be
embedded deep inside some neural net. I'm not even sure how one would explain
that one to a jury.

~~~
TazeTSchnitzel
Via learning models? I wonder how long it takes before it just automatically
selects any black poor person.

~~~
DasIch
Machine learning doesn't make a difference between correlation and causation.
No doubt such systems already help reproduce the inequalities present in the
data they've been trained on.

~~~
TazeTSchnitzel
Yeah, that's what I'm thinking. GIGO basically: the US records it would be
using would suggest poor, black, mentally-ill etc. people are most likely to
be criminals. The software will pick up on that.

------
downandout
This defense attorney was creative for asking to examine the source code, but
that isn't the only way to cast doubt on the accuracy of the software that DNA
matched his client to the crime scene. He could simply obtain a copy of it and
have an expert run tests to determine a false positive rate and also what
types of scenarios cause the software to deliver false positives, then call
that expert as a witness.

~~~
struppi
...which could be much more expensive and sill not find the error you'd need
to defend the client. You'd essentially be black box testing without even
knowing what kind of errors you want to find. Or am I missing something?

~~~
downandout
It would be more expensive probably. However, under the current legal
framework, this would be the only way to question the accuracy of the
software.

------
joesmo
This is what happens when you have companies profiting off the misery of
others.

The biggest reason for companies wanting to protect their source code in this
case is that they already know their software is broken, like pretty much
every other software, and they don't want to fix it. The arguments against
losing money and such are total bullshit as courts have plenty of procedures
for disclosing materials only to the relevant parties present, not to the
public as a whole. These companies simply don't want to spend the money
auditing and making sure their code runs correctly because the only
consequence of that is wrongfully convicting someone they don't give a fuck
about.

I'd say, let them see the code and let the highest paid expert witness win.
That is, after all, the American way.

------
TazeTSchnitzel
The essential problem is that in such environments the process of doing a task
must be open to inspection, but software exists as a loophole that circumvents
making process public.

------
jhwhite
This use to be a problem in Florida with drunk driving arrests. The company
that makes the code for the breathalyzers wouldn't allow their code to be
reviewed by defendants. There was finally a precedent set that defendants
couldn't mount a viable defense without reviewing the code.

So for a while people accused of a DUI could wind up getting off, under the
right circumstances, by requesting the source code then getting refused by the
company.

The company finally allowed pieces of the code to be reviewed by the courts.

------
cm2187
I'm not sure I agree with that view. Independant testing by another lab should
remove any doubt on the validity of a forensic, rather than forcing companies
to open source their technology. And of course some form of
certification/random tests that ensures that the company providing the
forensic isn't a bunch of conmen.

~~~
mc808
But how can you verify that the two independent companies aren't both using
the same faulty code? You could ask someone under oath, but how would _they_
know whether an employee bought or stole code from the competitor, or they
both made similar errors in implementing a published algorithm, etc? As
mentioned regarding Volkswagen, black box testing doesn't necessarily cut it
(especially if the flaws are statistical in nature or triggered in unusual
circumstances, and more so if they are deliberate).

~~~
dogma1138
How can you currently verify that 2 labs aren't using the same faulty forensic
process when it comes to other types of evidence? Most forensic evidence isn't
100% conclusive things like DNA and fingerprint identification aren't a 1 to 1
unique match they are usually in the range of 1 out of 100-400,000 match.

This is sufficient for most cases because the likelihood of that evidence
being "wrong" when you combine it with other factors like motive, eye-witness
accounts and other non-physical evidence is very slim.

~~~
mc808
I'm not sure but would hope that traditional forensic labs and processes are
also open to scrutiny. E.g. if the defense suggests that some important mark
on the body could have been introduced while it was being handled, it should
be possible to show why that couldn't happen with the procedures that were
followed, and that the procedures were followed.

~~~
dogma1138
They need to meet the specific regulatory requirements in their jurisdictions
but it doesn't necessarily means that they are easy to inspect.

On the body part it's a bit more complicated it's more a CSI effect people
think that all cases have tons of physical evidence and that everything is
cool and flashy and high-techy in reality most cases have very little physical
evidence and labs might not be used at all most bodies are inspected at by the
county coroner's office which might be quite inadequate of collecting that
sort of evidence we come to expect by watching crime procedurals.

~~~
alsetmusic
> On the body part it's a bit more complicated it's more a CSI effect people
> think that all cases have tons of physical evidence and that everything is
> cool and flashy and high-techy

I sat on a jury a couple years back and the prosecutor's opening statement
addressed this issue in the first seconds of the case. He worked hard to make
a distinction between reality and television. This was a smart move, as some
of the people on that jury turned out to be dumb as rocks when we went into
deliberation. I'm sure this is true of most juries.

