
Machine Learning: The High Interest Credit Card of Technical Debt (2014) - jwmerrill
http://research.google.com/pubs/pub43146.html
======
PaulHoule
This article keeps going around.

There are lots of things wrong with "Machine Learning" practice that are
problems of the practice instead of the subject itself.

One of the most dangerous things that comes out of academia is the "project
orientation", that is, the idea that something is "done" at a certain point in
time.

The "technical debt" aspect comes in when you are managing a product or a
program, which can be thought of as a series of projects that influence each
other.

The "paradigm" is that somebody wants to learn machine learning so they
download the same set of digits that Hinton used in his papers, and then they
replicate the result, and then they announce they are done. When you see
everybody is chasing the same problems (Hinton's digits, the Kaggle of the
day) you start to realize that the setup of a problem which can be solved by
machine learning is actually worth more than the learning.

~~~
yid
> you start to realize that the setup of a problem which can be solved by
> machine learning is actually worth more than the learning.

This is a fantastic point, but I'd like to also caution readers that they
should spend some time understanding the elements of statistics and machine
learning before jumping into theano or Weka tutorials. Lately, I've seen a lot
of voodoo ML in the valley, starting with billboards on Highway 101 that
pretty much _just_ say "Artificial Intelligence & Machine Learning" along with
the name of some company, like its some sort of panacea.

------
mallamanis
An upcoming NIPS paper seems to be a followup

"Hidden Technical Debt in Machine Learning Systems" D Sculley*, Google
Research; Gary Holt, ; Daniel Golovin, Google, Inc.; Eugene Davydov, Google,
Inc.; Todd Phillips, Google, Inc.; Dietmar Ebner, ; Vinay Chaudhary, Google,
Inc.; Michael Young, Google, Inc.; Jean-Francois Crespo, Google, Inc.; Dan
Dennison, Google, Inc.

(see
[https://nips.cc/Conferences/2015/AcceptedPapers](https://nips.cc/Conferences/2015/AcceptedPapers)
)

Unfortunately, I haven't been able to find the paper yet.

~~~
mizzao
That's because NIPS 2015 will be held in December. Final papers are due Oct
30th :
[https://nips.cc/Conferences/2015/Dates](https://nips.cc/Conferences/2015/Dates)

------
malux85
The worst are the organisations that don't have any reproducibility in their
work.

Models in production that are total black boxes. There's no way to
deterministically recreate them.

But I treated it as an opportunity --- was able to teach some engineering to
the data science guys and they taught me some data science :)

~~~
binarymax
It is surprising how often we take reproducibility for granted in the software
world. With unit tests and continuous integration scenarios it becomes such a
fundamental concept that it would be unheard of for developing products
without them.

Then all of a sudden you have terabytes of data updating a model faster than
you can keep up with it, and after a couple months, starting from scratch to
get the same model doesn't seem plausible anymore. Two years of production
time and you have a honking mess.

Also it is easy to fall into the trap of prototype-to-production, when you've
developed a nice looking demo that suddenly has all of the real customers and
none of the engineering rigor.

It is an interesting time to be in software :)

~~~
malux85
I agree - it's so challenging to keep up with Terabytes and Terabytes of
moving data ... but reproducibility is so fundamental to science in general.

I'm not disagreeing, I'm just wondering if there's somehow a middle ground.
What do you think? Can you share any experiences or tips?

I am running my own company at the moment, and my main dataset is about 2.5
TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix).
It takes too long to iterate the whole dataset, so when I'm training my
machine learning classifier I subsample the large dataset then compare
previous model results to the new model, and inspect the differences.

~~~
binarymax
If I have one piece of advice to give (that you probably already know), it is
to have a sound way to measure results and ensure that you are improving.
Usually that involves custom tooling that falls within the integration test
layer. So if 'inspecting the differences' is a reproducible and statistically
reliable way to know that you are good - then you probably are!

------
tim333
Previously

[https://news.ycombinator.com/item?id=10006293](https://news.ycombinator.com/item?id=10006293)

[https://news.ycombinator.com/item?id=8775772](https://news.ycombinator.com/item?id=8775772)

------
Havoc
These finance analogies are getting out of hand...

~~~
tosseraccount
too subtle. better stick to "climate science".

