
Neglected machine learning ideas - rbanffy
https://scottlocklin.wordpress.com/2014/07/22/neglected-machine-learning-ideas/
======
j2kun
I think this article exemplifies the difference between the kinds of things
you see codified in books and on the internet, versus what's active research
and well known folk lore in academia. And maybe it highlights the substandard
search mechanisms for published research, or the difficulty of learning from
published research papers. But it's definitely not about neglect, at least not
for most of the topics listed in the article.

For example, the recent AAAI 2014 conference had a bunch of papers on online
algorithms for various problems. [1] Likewise, COLT had five or so papers on
online learning. [2] Same with KDD [3], SODA [4], and the many other
conferences this year that accept papers about ML.

And learning in the presence of noise? Unsupervised learning? Feature
engineering? I am literally doing multiple research projects in all of these
areas right now! The only way I can imagine that you think they're neglected
is that you just don't know where to look for them, because these topics are
all over the place in my world. For example, one common term for "feature
engineering" is "representation learning," and this was a big topic at this
year's SDM conference, specifically w.r.t. data mining in networks.

Why can't you find a book you like for topic X? Maybe it's because researchers
have little incentive to write books. You folks in industry could fix that.
What with all your ridiculous market valuations of various mobile apps, surely
you could scrape together enough funding to convince the experts in their
field to write a book.

[1]:
[http://www.aaai.org/Conferences/AAAI/2014/aaai14accepts.php](http://www.aaai.org/Conferences/AAAI/2014/aaai14accepts.php)
[2]: [http://orfe.princeton.edu/conferences/colt2014/the-
conferenc...](http://orfe.princeton.edu/conferences/colt2014/the-
conference/accepted-papers) [3]:
[http://www.kdd.org/kdd2014/program.html](http://www.kdd.org/kdd2014/program.html)
[4]:
[http://www.siam.org/meetings/da14/da14_accepted.pdf](http://www.siam.org/meetings/da14/da14_accepted.pdf)

------
murbard2
One often neglected idea is transduction. The idea is that you can learn to
label data better by augmenting your labeled dataset with unlabeled data.

The unlabeled data helps discover the structure of the data which in turns
helps the supervised learning.

There is a lot of 2-phase approach to this problem, by first discovering
feature with unsupervised learning and then using those features in supervised
learning, but in many cases it's possible to do both at the same time.

If you have a generative model for your data, you can treat the labels as
potentially missing values and learn the joint distribution.

~~~
benanne
Afaik what you're talking about is more commonly referred to as "semi-
supervised learning". Transduction is a more specific case of semi-supervised
learning where you know the test set, i.e. you know which data points the
model will have to make predictions for. That means you can exploit this data
for training the model, for example by unsupervised pre-training or with a
pseudo-labeling approach.

~~~
murbard2
Bayesians don't need a test set 8)

~~~
turnersr
Why is that?

~~~
murbard2
Because the prior on your parameters smooths out the prediction.

Most cookbook techniques such as ridge regressions, cross-validation, etc have
a Bayesian interpretation as a prior on the parameter.

Bayesian techniques allow you to use all the data available.

That said, sometimes they are computationally expensive, and it's better to
approximate them by using a test set.

~~~
moultano
Unless you have an infinite regress on priors for your priors, and
uncomputable Komolgorov penalties on the structure of your model, I think you
need a test set. (This means you need a test set.)

~~~
ced
There's an interesting chapter in MacKay's book on Occam's razor. I'm not sure
how I feel about it, but it's very thought-provoking.

------
cf
These are great topics that are often neglected by machine learning textbooks.
Some of the reason has to do with machine learning textbook writers not really
doing research in Reinforcement Learning or Time Series. For things like
Online Learning the author cites a great book but nothing for a more
mainstream audience has been written yet.

Much of this stuff is being actively worked on though. If I could give one
practical tip. Read KDD conference papers. Those are very applied and usually
very accessible demonstrations of what techniques are out there, what problems
they are typically applied to and importantly how well they worked.

Excellent post.

------
saosebastiao
Really good summary. I find it interesting that Reinforcement Learning, Online
Learning, and Time Series modeling are all in the neglected category. They are
all methods that seem to fit very well in both autonomous robotics and
finance. I would venture to guess that they aren't really neglected, just
hidden behind the firewall of Intellectual Property.

~~~
cnaut
I agree. I wouldn't consider online learning neglected. The article mentions
vowpal rabbit which is being used at Microsoft

------
dj-wonk
> "online learning" ... and the subject is unfortunately un-googleable

That's not a surprise; "online learning" is a pretty misleading name. Here are
some better alternatives:

    
    
      * incremental learning / training
      * stream-based learning / training
    

So, let's use these terms instead -- then it will be easier to find in your
favorite search engine.

Here is one example using the term "stream-based learning":

[http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.1.79...](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.1.7903)

~~~
idunning
I wouldn't say its "misleading", it is a correct usage of the word online
(especially contrasted with offline learning) and is the name of the sub-area.
You won't find many (most?) papers in the area with those search terms
unfortunately.

~~~
dj-wonk
These so-called "online" techniques do not require being online (as in
connected to the internet); they simply require being exposed to a stream of
ongoing training data. They learn incrementally. This is why I say the term
"online" is misleading.

See:
[http://en.wikipedia.org/wiki/Online_machine_learning](http://en.wikipedia.org/wiki/Online_machine_learning)

> Online machine learning is a model of induction that learns one instance at
> a time.

~~~
scottlocklin
Online learning existed before the internet, dammit.

------
chmullig
This is actually a great overview/review of some of the things that I've
encounter regularly at work, but never seen discussed seriously in a book or
class.

------
c54
Hierarchical Temporal Memory based algorithms such as the Cortical Learning
Algorithm (CLA) implemented by NuPIC [1] should also be on this list
(unsupervised, online, realtime)

[1] [http://numenta.org/](http://numenta.org/)

~~~
deong
I think most people would classify those as AI rather than machine learning.

------
aganders3
I would love to see more books/articles/blogs on unsupervised learning and
ensemble techniques. E.G. - can I use k-means clustering as input to train a
naïve Bayes classifier?

~~~
saosebastiao
Interesting...I should write a blog post about some techniques I've used that
are similar to your example (modulo specific algorithms). Is there something
specific you are trying to accomplish?

~~~
aganders3
I would read it! I work with a specific kind of high(er) dimensional medical
imaging data, and I think unsupervised learning could be used for
classification and foreground separation. K-means is giving me some promising
preliminary results, but I'd like to assign samples continuous probabilities
rather than binary classifications. I'm relatively new to ML but trying to
incorporate it into my research, so I apologize if any of that doesn't make
sense!

~~~
electrograv
K-means clustering has been successfully used to extract features in a "deep
learning" style architecture (with good results at image recognition). You'll
probably find this useful:
[http://web.stanford.edu/~acoates/papers/coatesng_nntot2012.p...](http://web.stanford.edu/~acoates/papers/coatesng_nntot2012.pdf)

------
Teodolfo
A lot of these ideas, with the possible exception of conformal prediction,
have quite a lot of academic literature. They are hardly neglected.

~~~
joe_the_user
I've scanned a lot of literature on various AI fields over the last few years
(And Jeesh, we should say AI if we're talking getting superb, actually-
intelligent algorithms as opposed to the work-a-day, reliable algorithms that
"machine learning" arguably already has).

I would contend that there can be a "significant" seeming amount of literature
on field X but field X may still wind-up not pursued in the larger scheme of
things.

Often what happens is a single individual or small circle, gets interested in
a given field and researches it among things for as long as the funding
persists and then once the funding dries up they move on. Or one person has
tenure, keeps researching but everyone else moves on because it doesn't look
like a way to keep getting funding.

Even more, as the author mentions, a big question is what approaches are
taught as _the way to do it_ (and I guess it again comes down whether you're
aiming for just machine-learning/a-better-heuristic-statistics-for-big-data or
if you are aiming for moving towards intelligent algorithms, even if
intelligence means just flexible adaptivity).

Yes, you can find lots of results if you search for "online learning", say.
Otoh, for whatever given algorithm that has mindshare currently, is there a
quest to find an online version? My sampling of the literature says no and I
happen to agree with the author that online processing could be an important
piece of artificial intelligence advances.

------
aaronsnoswell
Was anyone else taken with the images interspersed throughout that blog post?
Does anyone know where they are from?

Edit: found them thanks to Google image search:
[http://www.darkroastedblend.com/2014/01/machines-alive-
whims...](http://www.darkroastedblend.com/2014/01/machines-alive-whimsical-
art-of-boris.html)

~~~
6cxs2hd6
I saw a nice attribution at the end of the post. (But possibly it was added
belatedly, e.g. after seeing your comment.)

~~~
pdenya
> Images by one of my heroes, the Ukrainian-American artist Boris
> Artzybasheff. You can find more of it here.

Agreed, very nice attribution. It wasn't added in response, I noticed it 8
hours ago (last night for me).

------
pizza
The invisible hands re-ranking HN posts continue to astound me with their
reach..

~~~
dang
We're experimenting. Expect to see more of this.

[https://news.ycombinator.com/item?id=8122403](https://news.ycombinator.com/item?id=8122403)

~~~
pizza
Interesting, I'm glad to see more thought applied to rankings. What kinds of
posts are you hoping to see more often?

~~~
dang
Substantive posts that would otherwise have fallen through the cracks.

