
Machine Learning Library for C++ - optiminimalist
http://image.diku.dk/shark/sphinx_pages/build/html/index.html
======
pilooch
Very interesting, but as a daily practitioner I am skeptical.

First, this is a _lot_ of code! As a C++ machine learning programmer, I am
impressed as I know the pain (someone explains why, see comment
<https://news.ycombinator.com/item?id=5613797> ).

Second, it contains a version of Blas and ublas as well as LBFGS and more,
much more, coded from scratch as it seems. This seems too much for an ML
library, and a lot to maintain.

This makes me skeptical of performances and maintenance of the code, but it
would be fairer to try it first.

Still very impressed.

~~~
Schneekatze
Hi, shark developer here.

First of all, we are glad that our library is discussed on this board! We are
happy for every feedback we can get!

Regarding Performance: we try to get the key algorithms as fast as possible.
And for the hardest parts we rely not on ublas, but use optional bindings to
ATLAS. Speed was one of the key design criteria. We hope that we achieved
that. Clearly this is no guarantee that every algorithm is fast, but in this
case: just add a ticket!

Please bear in mind, that Shark is still in beta stage, and we are heavily
developing it (I am right now working on the family of multi class SVMs). So
for example parallelism using OpenMP is not fully integrated.

~~~
jhartmann
You should consider using Eigen for linear algebra, I have personally found it
much better performance wise than using bindings to ATLAS or other more
standard linear algebra solutions. ML algorithms tend to be multistage (think
about the update of weights with momentum in a Neural network for example),
and the primitives available in ATLAS or a blas library are really too low
level. Eigen since it generates code for a whole complicated expression can
blow a standard linear algebra library out of the water for a certain class of
problems. For others obviously the highly tuned vendor BLAS code would win,
but I've seen huge speedups by using Eigen it fits well for complex ML
operations.

~~~
Schneekatze
This is true, even though we would rather switch to Armadillo due to it's
easier handling and better high-level behaviour.

Right now the linear algebra library we use -ublas- has the same behaviour as
Eigen for BLAS1 type expressions. So it tries to generate optimal (non-SSE)
code. Only for BLAS2 and 3 we fall back to the ATLAS-routines which has the
same performance as Eigen on the interesting problem sizes.

//small edit In the end it is not so interesting whether the BLAS1-type
expressions are fast as they make up < 1% of run time performance. The big
chunks are the data processing inside the matrix-matrix multiplications of the
Neural Networks and similar entities.

~~~
jhartmann
You forget that if you can do the whole weight update in a single shot
operation that the data doesn't have to go through the cache multiple times,
and at least on the problem sizes I am working on FP bandwidth isn't what
kills it, but the memory bandwidth. Back to the NN example: If you can do the
matrix multiply and the application of the delta weights in a single loop
iteration you get much better cache behavior.

Another thing about code generation, I am also using a hacked version of Eigen
as well in a project I'm working on that can do the tanh and derivative of the
tanh so the NN activations go quite abit faster since you can generate
vectorized code for the whole calculation that will visit the memory location
exactly once. While true the calculation of the weight updates is the most
time spent, I saw 3-4x speedup in the activation code doing it in a single
operation due to better memory access patterns and less loop iterations.
Better memory access patterns can also have synergistic effects on other code
because there is less cache pollution happening. By being fast and loose and
introducing a few other copies of the matrix data in my case, my performance
falls off a cliff when it no longer fits in the cpu cache nicely. 10x
difference in the particular case I am remembering.

As always performance is part art, part science and perhaps it won't matter as
much for the general case, but for my specific implementation and my matrix
sizes Eigen has made a measurable difference for me compared to other
solutions.

------
JacobiX
The library supports some very useful algorithms. Both supervised and
unsupervised ones. But I can't use it for our commercial non GPL projects.

~~~
jhartmann
Flame Suit On / Rant Mode On

I actually really don't understand why anyone uses GPL for a library. I've
been doing open source for a long long time, and love the GPL. I have code in
the Linux kernel, and believe free software AND open source software are great
solutions to very real problems in software engineering. Having open code just
gives people more options, and I firmly believe it will win over time as far
as quality is concerned.

I just think only providing libraries to other GPL code is stupid. It just
limits the usefulness of the software. LGPL is great here, you get the core
changes contributed back to your library from a greater group of people and
everyone wins. Limiting a library to GPL means a large population can not use
your code, those writing applications that can't be licensed under the GPL.
Limiting choice is BAD. The whole reason you should be creating and using free
software and OSS is to not weld the hood shut. GPL should be for applications,
LGPL just limits choices for libraries. Down with the GPL for libraries!!!

Flame Suit Off / Rant Mode Off

~~~
to3m
LPGL isn't vastly better than the GPL as it makes it difficult to link
statically and/or release for closed platforms. Its advocates would probably
see these as plus points, but I'm not sure that they're going to increase
uptake.

~~~
thwest
Difficult, but not really a significant part of the challenges for delivering
closed source binaries across platforms.

On linux, you have to build distribution specific binaries that match the
shared library versions in the package manager.

On Windows, you generally put all of your shared libraries in your
application's folder, since there are plenty of bad actors who install DLLs
without versions in the filename to system32. Leads to duplication on the
system but its a generally accepted bad practice.

(Can't speak to shipping LGPL libs on OS X).

~~~
to3m
I was thinking more of Xbox360, Playstation3, iOS, etc. - none support user-
replacable files. I don't even think any even support dynamic linking... )

------
hackinthebochs
This looks awesome. I've been itching to try out some ideas I have after
having gone through Bishop's book, but I've been hesitant to write the
algorithms from scratch. Now I'll have to decide between learning matlab or a
library such as this.

~~~
tlarkworthy
You should definitely use a scriptable ML library. The process is very
iterative and not suited to a compiled language like C++. I use skilearn alot,
but also the matlab toolboxes or R are great. At its heart ML is alot of
stats, so use something built for maths, not C++. It doesn't really make sense
to break out C++ until you know _exactly_ what algorithm and settings you need
_and_ your application is real time.

~~~
bch
If the OP was thinking of writing his own algorithms, and this is a linkable
library with that heavy math already implemented, couldn't he write bindings
for Python/Lua/Tcl/Ruby and have everything he needs for script-ability, or am
I missing something?

~~~
andrewguenther
You aren't missing anything, you are absolutely correct, but the question
isn't "can they?" the question is "will they?"

~~~
bch
I'm in the Stanford/Coursera machine learning course right now, and something
like this is nearly excactly what I've been looking for.

As some others have said, GPLv3 is off-putting, but there is the LGPL mlpack
lib (<http://www.mlpack.org/>) (also C++). Personally, project-wise, the only
way this could be improved is if the project were pure C, and a BSD, MIT, or
similar license. Quite looking forward to checking these out, though.

~~~
tlarkworthy
honestly you guys are crazy if you think shark is gonna help you _learn_
machine learning. Its ideal for deployment of ML on things like embedded
computer, robotics, games etc. where real time learning is required. Machine
learning requires _alot_ of experimentation and C++ is a terrible medium for
that. There are _loads_ of good machine learning libraries implemented for
python and matlab. Pretty much every good paper in machine learning is
accompanied by an algorithm implemented in matlab or python or R. _learn_
using those reference designs. Once you figured out what you want, then deploy
on a system in C++ by all means using shark. I do robotics for a living, and I
do go from scripting to C++. Unless its absolutely necessary I avoid C++. Only
things like vision which is so CPU hungry that it has its own computer do I
require C++, every other algorithm stays in python.

~~~
jhartmann
Actually you forget that performance when you need to train for days at a time
is critical, if I use Octave/Matlab/R my current project might take months to
train instead of weeks. All my ML code is high performance threaded C++. I
recommend you use a good template linear algebra library like Eigen, you can
do plenty of experimentation in C++. I find with a set of a few modern
libraries and the required experience a C++ programmer is just as if not more
efficient than a Python/R/Matlab programmer. It comes down to the skill of the
programmer and the proper choice of libraries.

~~~
tlarkworthy
True that matlab octave and R are all rubbish for performance. I use python +
numpy which all delegates to BLAS for the hardcore linear algebra stuff. I
don't normally find C++ gains me all that much. You can also do GPU
acceleration pretty easy using theano (e.g.
[http://deeplearning.net/software/theano/tutorial/using_gpu.h...](http://deeplearning.net/software/theano/tutorial/using_gpu.html))

So I reckon my GPU accelerated python still beats a C++ pthreads approach, and
is alot faster to develop on.

Your mileage may vary, from what you said you probably know what you are
doing, maybe GPU is not applicable. I was really replying to the initial
comments that said they want to start learning machine learning on a C++
system. Training for days suggests you are doing something hardcore like
MCMC/DBN/Guassian Processes, learners should not start there though....

~~~
jhartmann
I'm doing deep belief networks with dropout, and don't have access to GPU's
with good double precision performance. I used to write graphics device
drivers, so GPU computing has a special place in my heart and definitely agree
with you there performance wise. It is funny though that my little laptop is
hitting training times similar to some papers where people are using low end
GPU's though, its amazing what you can do when you pay attention to
performance.

I suspect my tuned C++ code will work quite well on a Intel MIC, and that is
probably where I'm going to go when I have more resources to throw at the
problem. I do know that Theano does use Alex's C++ CUDA code under the covers
and I have done lots of reading of some of theano's code looking at
implementation details to help developing my code. I just am not a big python
(or most scripting languages actually) fan, perhaps I'm just too old school
and written C, C++, C# and Java too long. If it doesn't smell or feel like C,
I feel like Scotty in Star Trek 4 when he was making the transparent aluminum
on the mac.

------
swalsh
Does anyone know if there's a good ML library written in C# or F#

~~~
profquail
I don't know of any written in F#...maybe I'll have to write one :)

You might also ask on the fsharp-opensource mailing list, maybe someone has an
F# ML library I don't know about:

[https://groups.google.com/forum/?fromgroups#!forum/fsharp-
op...](https://groups.google.com/forum/?fromgroups#!forum/fsharp-opensource)

