
Best Practices for Applying Deep Learning to Novel Applications - mindcrime
https://arxiv.org/abs/1704.01568
======
AndrewKemendo
It's good general advice, but frankly I think it doesn't address some of the
major pitfalls - namely the top one: personnel.

In the very beginning it's stated:

"In this report I assume you are (or have access to) a subject matter expert
for your application."

In my experience this is where it goes off the rails for most of the crowd
that she is addressing. Not because they don't have someone, but because who
they have isn't really a "subject matter expert."

It's a muddy term anyway especially in the field of Machine Learning.
Excluding for a moment the huksters and bold faced liars, within ML there is
WIDE variance in competence, domain specificity and application specific
capability within the field.

The biggest capability gap that I have encountered when working with fantastic
ML folks is that the ones that understand the mechanisms/algorithms/approaches
best, are actually pretty terrible at delivering production code. That's not
for lack of capability, it's simply because the bulk of their time has been
spent in research - so they approach things very differently than application
focused engineers. This is extremely relevant in this case because this is an
application specific paper.

There are a plethora of mine-fields in applications of ML, some of which are
outlined here from a systems approach, but the majority of which are personnel
issues in my experience, and "culture" issues - not to be confused with
"culture fit" problems that exist elsewhere.

~~~
sndean
> It's good general advice, but frankly I think it doesn't address some of the
> major pitfalls - namely the top one: personnel.

Just for some context [0]: Leslie works at a facility where there's
(generally) no shortage of subject matter experts. Recently, within the last
~6 months, he's been pulled into at least a few groups where lots of
biologist, chemist, and others who essentially know nothing about DL are
trying to get projects off the ground. The shortage he's experiencing is
probably on the ML/DL side.

[0] I work with Leslie and have received his advice on projects.

~~~
AndrewKemendo
In fact the USG broadly, specifically the DoD is taking a huge step forward to
try and build ML into everything.

I was invited to talk with the Defense Innovation Board small group this last
Tuesday with SecDef Mattis, Eric Schmidt et al. and it's actually becoming a
HUGE deal, one that nobody inside the USG is prepared to address, so I'm not
surprised Leslie is bombarded with these requests.

I'll go back to the point though that the USG is critically short of competent
ML personnel.

~~~
sndean
> I'll go back to the point though that the USG is critically short of
> competent ML personnel.

Definitely agree with that. And from what I've heard they're having a terrible
time trying to hire. Especially where we're at, where a PhD required, the USG
won't be able to compete on pay (?).

~~~
AndrewKemendo
It's a mix between these three things:

1\. Being ethically opposed to working for the DoD

2\. Terrible (relative) wages

3\. Working environment dominated by bureaucracy

Some seriously big obstacles considering the options.

------
gwern
Looks like good advice to me. Like a more DL-focused version of "A Few Useful
Things to Know about Machine Learning"
[http://www.datascienceassn.org/sites/default/files/A%20Few%2...](http://www.datascienceassn.org/sites/default/files/A%20Few%20Useful%20Things%20to%20Know%20about%20Machine%20Learning.pdf)
, Domingos 2012

------
comicjk
> Let’s say you want to improve on a complex process where the physics is
> highly approximated (i.e., a “spherical cow” situation); you have a choice
> to input the data into a deep network that will (hopefully) output the
> desired result or you can train the network to find the correction in the
> approximate result. The latter method will almost certainly outperform the
> former.

This aligns with my experience (computational chem PhD). When applying a
strong, general-purpose mathematical patch to an existing model, use as much
of the existing model as possible. Otherwise the patch will have a hard time
fitting, and maybe be worse than what you started with. Philosophically, this
also comports with my thinking (it's the modeling equivalent of Chesterton's
Fence
[https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence](https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence)).

------
DrNuke
Physics is deterministic in states and always tends towards an equilibrium, so
novel results from DL may still fit some continuum math model without being
stable or even real. Domain expertise, on the other hand, helps prepare data
for ML algos in such a way that results will come (or not) within the
boundaries of reality and hopefully stability. I am trying both approaches for
some materials science goals of mine and am curious to see what happens, now
that powerful hardware is cheap enough on the cloud to put some ideas to work.
Side point is all this was just impossible five years ago, so I am grateful
and excited to have this opportunity.

------
bluetwo
Was kind of hoping for some examples of novel applications no on has thought
of. :-)

------
lngnmn
Surprisingly good and sane paper, without all that hipster's bullshit.

The emphasis on the quality of the training data and, most importantly, on the
evaluation and careful choice of heuristics on which the model to be build
upon, is what makes the paper sane.

There is no shortage of disconnected from reality models based on dogmas,
while, it seems, there is acute shortage of the models properly reflecting
some particular aspects of reality.

Data and proper, reality-supported heuristics (domain knowledge) are the main
factors of success. Technical details and particular frameworks are of the
least importance.

This, BTW, is why it is almost impossible to compete with megacorps - they
have the data (all your social networks) and they have the resources,
including domain experts, without whom designing and evaluating a model is a
hopeless task.

