

GPU-Accelerated Deep Learning Library in Python - rbanffy
https://github.com/hannes-brt/hebel

======
kyzyl
Hm. And here I was thinking I'd seen most of the accelerated deep learning
implementations. Although this one seems a touch incomplete as far as features
go. There are some other for those not aware:

[1] [Python] pylearn2 by the Lisa Lab @ uMontreal. Probably one of the
beefiest of the bunch. Has a pretty extensive feature set, but IMO isn't the
simplest to use.

[2] [Python] MORB by Sander Dielman. Just restricted boltzman machines, but
very nice and intuitive to use. Also built on Theano.

[3] [C++] CUV by the AIS lab at Bonn University in Germany. Not strictly deep
learning, but it's a fast CUDA-backed library written in templated C++ with
bindings to other languages, including python. It's been used to implement
deep learning models.

[4] [Python] DeepNet by Nitish Srivastava and co. at U of Toronto. I don't
have as much experience with this one but it's built on cudamat and implements
most of the main model types. Interestingly, they've taken the approach of
using Google protocol buffers as a sustainable means of defining model
configurations.

[5] [MATLAB] DeepLearnToolbox by Rasmus Palm. If matlab is your thing, here
you are. Implements most models you are likely to want.

Not all of these developed equally as actively developed, but there is some
good stuff above and I haven't found many instances where what I wanted wasn't
(somewhat) readily available in at least one. I'm sure I'm forgetting one or
two.

[1] [https://github.com/lisa-lab/pylearn2](https://github.com/lisa-
lab/pylearn2)

[2] [https://github.com/benanne/morb](https://github.com/benanne/morb)

[3]
[https://github.com/deeplearningais/CUV](https://github.com/deeplearningais/CUV)

[4]
[https://github.com/nitishsrivastava/deepnet/tree/master/deep...](https://github.com/nitishsrivastava/deepnet/tree/master/deepnet)

[5]
[https://github.com/rasmusbergpalm/DeepLearnToolbox](https://github.com/rasmusbergpalm/DeepLearnToolbox)

~~~
hannes-brt
Yes, Hebel doesn't have a ton of features and a kitchen-sink of different
models yet, but I hope that's going to change. There are lots of things that
are quite easy to implement in the current framework, such as: \- Neural net
regression \- Autoencoders \- Restricted Boltzman machines

There's a lot of interest for convolutional networks and the best way to
implement it will be to wrap Alex Krizhevsky's cuda-convnet, like DeepNet and
PyLearn2 have, but this will require a bit more effort.

With respect to other deep learning packages, Hebel doesn't necessarily do
everything differently, but depending on your needs it may be the best choice
for a particular job.

PyLearn2 is big and monumental and although I haven't used it much personally,
it seems to excellent. But as you mentioned, it's not necessarily easy to use
and if you want to extend it, you have to learn the Theano development model,
which takes some time to grok.

DeepNet is quite similar to Hebel in its approach (even though it offers more
models right now). However, DeepNet is based on cudamat and gnumpy, which I
have found to often be quite unstable and slow. Hebel is based on PyCUDA which
is very stable and according to some preliminary tests I did runs about twice
as fast as cudamat.

So, the idea of Hebel is that it should make it easy to train the most
important deep learning models without much setup or having to write much
code. It is also supposed to make it easy to implement new models through a
modular design that lets you subclass existing layers or models to implement
variations of them.

~~~
kkjkok
Question: do you/will you plan to support converting GPU nets to CPU, perhaps
by keeping weights and architecture definition separate from PyCUDA dependent
structures during serialization?

I have found that using a trained net for preprocessing can be accomplished
using very limited resources (read: Core 2 Duo laptop). This is one of the
very nice features of DeCAF, which _could_ allow for some interesting
applications on embedded devices.

Great work by the way - I look forward to testing it out soon!

~~~
hannes-brt
That would be possible, but since Hebel is mainly meant to be used in research
I don't think it's a big priority now. The most important reason to do this
would be to allow development on laptops and workstations without NVIDIA cards
and to run the finished model on CUDA hardware later.

As far as embedded devices go (I assume you're talking about ARM cpus etc),
they are probably too underpowered to run Neural nets anyway, or models would
have to be written in highly specialized C.

------
hannes-brt
Hello Hacker News, this is the Hebel developer. Thanks for the attention and
the Github stars!

I'll do my best to answer any questions in the thread. Looking forward to you
trying out Hebel and your pull requests ;)

------
diab0lic
This is very interesting especially for those users more interested in running
deep neural nets than programming deep neural nets. I was initially
disinterested because it is missing several important features (at the moment)
such as auto encoders and convolutional neural nets. A quick peek inside the
example folder revealed the fact that you can specify what you want computed
using YAML instead of specifying how to compute it in code means that as long
as you're using something Hebel does implement you can quickly experiment with
structure and parameters without worrying about programming errors. Very
useful for a researcher more interested in playing with the structure of a
deep neural net.

It also serializes the model and results to disk by default. This is great for
loading up the model later on another, possibly less performant, machine and
performing your classification / etc...

Of course PyLearn2 covers this feature set and more, but isn't as easy to get
started with.

I'll make an effort to use this when I can, unfortunately my two current
projects involve a convolutional neural net and an auto encoder. :(

TL; DR: Specifying structure using YAML instead of coding neural net. Working
at a higher level than other libs such as Theano.

------
thearn4
Serious question to those with experience in the area: is the term "Deep
learning" more than a buzzword?

~~~
ot
It is not a buzzword, as it describes a very specific concept: learning based
on a neural network with multiple (say >= 3) hidden layers.

It is not a new idea, but it has been viable only since the last few years
thanks to both computational power advances (huge clusters, GPUs, big data),
and algorithmic breakthroughs (sampling algorithms, stochastic optimization,
contrastive divergence, ...).

~~~
ljlolel
Of course, it's also a buzzword.

~~~
ot
Yes, often in research when there is a hot topic like this everybody tries to
jump on the bandwagon, and now the term is so widely spread that attaching
"deep learning" to anything makes it sound cool.

What I meant is that at least this has some specificity, in contrast with
terms like "web scale", "big data", "machine learning", "2.0", that are so
broad that can be attached to anything.

~~~
ma2rten
What's wrong with "machine learning"? I think it's no less specific than
"artificial intelligence" or "computer science".

------
sitkack
All of my GPU resources are OpenCL based. The only place I could run it is on
AWS. Someone really needs to make a CUDA compiler with an OpenCL backend.

~~~
diab0lic
I have this same issue and am forced to run all my experiments on AWS. Once
you get a workflow down it is pretty easy and the AWS resources provide an
acceptable level of performance. Though I suppose if you're just toying around
/ exploring it can be discouraging to go through all of that and incur AWS
fees.

~~~
sitkack
But you must have an Nvidia card around some where to test your codes out
first? It looks like the best bang for the buck is a GTX 760.

~~~
diab0lic
If I had a 760 I'd likely just run all my experiments right on my device. As
it stands my only machine is a Early 2009 macbook pro with a Geforce 9600M.
The nice part about pylearn2, and theano is that the symbolic expressions
compiled by theano can run on your processor too, albeit much slower. You can
always test to make sure 1 to 2 epochs work locally before sending it off to
AWS for GPU computation.

I'd be very willing to buy a 760, or even a 770 at their current prices. The
only thing holding me back is that i'd have to buy an entire computer in which
to place the card. Haha :D

If you're interested in just how fast video cards can be for deep learning as
compared to CPUs take a peek at the results on [0]. That is for older model
GPUs, and they're an order of magnitude faster. Though as I understand it the
Geforece 5xx cards are superior for scientific computing as compared to the
6xx and 7xx series which are more gaming oriented. (May still outperform 5xx
due to raw speed at the cost of some additional CPU time). Have a look at the
appendix on [1] for more info on Fermi vs Kepler GPUs.

[0] [http://deeplearning.net/tutorial/lenet.html#running-the-
code](http://deeplearning.net/tutorial/lenet.html#running-the-code)

[1] [http://fastml.com/running-things-on-a-gpu/](http://fastml.com/running-
things-on-a-gpu/)

~~~
kkjkok
The convolutional neural network code that pylearn2 and the Toronto group use
is specifically tuned for GTX580 cards - users have reported factors of 2x-10x
slowdown using Kepler series cards. In general, most users (of pylearn2 at
least) highly recommend a GTX5xx device.

I personally use a GTX570, and it is pretty decent, though not spectacular.
Costwise, it is reasonably priced, and "good enough" for most of the networks
I have tried (minus ImageNet...)

A key problem is limited GPU memory in the Fermi series, as it is difficult to
fit a truly monstrous network on a single card. Krizhevsky's ImageNet work had
some very tricky things to spread it across 2 GTX580s, and the training still
took a very, very long time.

~~~
diab0lic
I didn't realize the performance dip was so dramatic. As of late I've been
considering acquiring some hardware and I guess I'm going to have to keep this
in mind. If ones were to buy one today I don't even think I can find stores
that sell the GTX 580. Would probably need to search on ebay.

Thanks for the heads up friend!

~~~
kkjkok
Check out this discussion - it may help you decide what card to get. There was
also an email somewhere about how TITAN is currently not any faster than a
580, though no hard numbers.

[https://groups.google.com/forum/#!topic/pylearn-
dev/cODL9RXP...](https://groups.google.com/forum/#!topic/pylearn-
dev/cODL9RXPVd0)

Once again, my 570 is slower than a 580 (about 2x), but "good enough" for now.

------
yconst
Looks good! What performance gain could be expected when using GPU vs CPU for
Deep Learning? It'd be great to get some figures.

~~~
willwill100
Depends, of course, on the architecture but you could be looking at a ~x50
speedup.

~~~
hannes-brt
Yes, you can't come up with a hard number, but a 10x speedup is really the
baseline. 50x and more are not out of the question.

------
daureg
Looks a lot like Theano
[[http://deeplearning.net/software/theano/index.html](http://deeplearning.net/software/theano/index.html)]

~~~
paulgb
Theano is for compiling mathematical expressions to CUDA, it doesn't implement
any deep learning itself. If anything this may be comparable to pylearn2.

~~~
hannes-brt
Yes, this is true. Theano is just a GPU-computing framework that you can use
to implement any kind of numerical model. Hebel implements some specific
models, but it uses PyCUDA as a backend.

------
aldanor
Went to the docs page, looked for benchmarks, haven't found any.

Went to the github page, looked for unit tests, haven't found any.

Other than that, looks promising.

~~~
hannes-brt
What kind of benchmarks were you looking for? Benchmarks with respect to speed
or MNIST results? For MNIST, using the DNN example, Hebel gets results that
are in line with reported results on the MNIST homepage [1] for similar models
(pure Neural nets with cross-entropy error and no distortions etc), which is
about 1.3% error.

There are some unit tests in Hebel though [2], even though coverage is not
complete.

[1] [http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/) [2]
[https://github.com/hannes-
brt/hebel/blob/master/hebel_test.p...](https://github.com/hannes-
brt/hebel/blob/master/hebel_test.py)

------
tomrod
Awesome! I'll be testing this one out when I get some free time.

------
X4
In which WebApp project did you use deep learning?

What problem did you solve with it?

------
ye
Needs cool demos of practical applications.

~~~
diab0lic
If you execute the code shown in the README it should run an experiment
against the MNIST dataset[0]. If it performs as well as other deep learning
examples[1][2] you should see a test error rate of less than 1% on
classification of handwritten digits. This is pretty impressive, especially
considering the fact that the collection contains a few digits that I cannot
even recognize myself.

[0] [http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/)

[1]
[http://deeplearning.net/tutorial/lenet.html#lenet](http://deeplearning.net/tutorial/lenet.html#lenet)

[2]
[http://neuralnetworksanddeeplearning.com/chap1.html](http://neuralnetworksanddeeplearning.com/chap1.html)

