
Can science solve an 80-year-old Indian mystery? - baazaar
http://www.swissinfo.ch/eng/jaeger-lecoultre-_can-science-solve-80-year-old-indian-mystery-/42474992
======
nerdponx
> The ISV (Inter-Session Variability) algorithm provided a decent comparison
> score of 0.79 for Kanchan Prabha Devi which was much higher than an image of
> Zubeida Begum Dhanrajgir. However, the results of the CNN (Convolutional
> Neural Network) algorithm were not as clear-cut. Devi only managed a score
> of 0.42 while Zubeida got 0.50.

> So, which algorithm to trust?

This is an excellent example of the danger in using algorithms that produce
point estimates, but not variability estimates. Whether you model it or not,
any inference or prediction exercise implies a probability model (else you
wouldn't need to conduct inference at all). Failing to make that model
explicit, whether by nested cross-validation or an analytical technique, leads
you down a dangerous road of making arbitrary comparisons And breaking near-
ties on the basis of "field knowledge", which can turn out to be a little more
than glorified gut instinct.

~~~
sandGorgon
Thank you for this!

we are planning to have a consultant come in a build image comparison of
selfies with ID cards (for a fintech mobile app). What kind of validation and
variation scores should we ask for to explicitly model the probability ?

~~~
nerdponx
I don't think there's anything available for CNNs other than resampling
techniques like nested cross-validation.

I'm also a fan of sensitivity analysis, where you jitter one of the inputs, or
one of the model parameters, and see how robust the output is to local noise.

ISV I'm not explicitly familiar with. But there's no reason you couldn't use
one of the general purpose techniques from above.

That said, resampling is very computationally intensive, so I can understand
why it's not that popular in conjunction with similarly intensive iterative
fitting algorithms like backpropagation and MCMC. I didn't mean my post to be
a criticism of the research as much as I wanted to just put a warning out
there about an issue that researchers without deep statistical backgrounds
tend to overlook.

------
grif-fin
Not getting into the story just commenting on the title 'Can Science
solve...'. It boils my curiosity when sentences start with 'Can science solve
X, Y or Z?'. It is referred to as if science is a tool.

I mean is there an alternative to science that we question in that way? -_-

~~~
gnaddel
I would not read this as questioning science but as the science-journalism
equivalent of asking whether the 'mystery' can be 'solved' at all.

~~~
grif-fin
When a sentence is in form of 'Can X do/solve/manage/become/etc ... Y?' it is
both simply asking and challenging X at the same time.

Challenging science is absurd as is challenging the reality.

~~~
livingparadox
"Can the president outrun a cheetah?" is not the same as challenging the
president's qualifications as president.

It _is_ challenging the idea that the president can outrun a cheetah, however.
So, "Can science solve X?" is not the same as "Is science effective?".

