Ask HN: Whats the best way to learn C++ for Deep learning? - wpmoradi
======
krat0sprakhar
I'm not sure how C++ for DL is different from regular C++ so take my advice
with a grain of salt.

I took a class under Bjarne Stroustrup and he highly recommended Tour of C++
[0] as the best way to learn modern C++ for someone who already has some
programming experience. That and of course, Effective C++[1] by Scott Meyers.

[0] - [https://www.amazon.com/Tour-C-
Depth/dp/0321958314](https://www.amazon.com/Tour-C-Depth/dp/0321958314)

[1] - [https://www.amazon.com/Effective-Specific-Improve-
Programs-D...](https://www.amazon.com/Effective-Specific-Improve-Programs-
Designs/dp/0321334876)

~~~
randcraw
I'd agree with the shout out for Tour of C++ as a good first book. Unlike most
C++ intros, it's concise (192 pages) and up-to-date enough to exemplify C++'s
standard forms (esp. 98 and 03). If you plan to develop in C++ you'll want
other books with more examples and reference, but these will shift the focus
more onto trees than forest (at ~1000 ppg).

Accelerated C++ by Koenig and Moo is another nice short overview of C++, but
assumes more familiarity with C and OOP principles.

------
siekmanj
I think python is generally used for AI and data science stuff, so you'd
probably be better off learning that.

That said, I think one of the best ways to learn how to do something is dive
right into it. I'm currently in the process of writing a deep learning library
in C:
[https://github.com/siekmanj/sieknet](https://github.com/siekmanj/sieknet)

So if you're dead set on C++, I'd say the best way is to try to build
something cool right off the bat.

~~~
eindiran
I just checked out your github repo and you've got a really cool project going
there. What made you decide to start writing it?

~~~
siekmanj
Thanks! Mostly it was out of a desire to learn more about the actual
algorithms behind AI, and trying to figure out if it's a field I might enjoy
working in. I really enjoy building things from the ground up, and
understanding how each part works.

------
spott
What is your reason for learning "C++ for deep learning"?

This will kind of define _how_ to go about doing it.

I can think of a few different reasons you might want to do this:

* You DL code in another framework is slow, and you have some custom C++ you want to write to speed it up. For this, you probably want to learn both high performance C++ and/or CUDA kernel development. You can probably avoid diving completely into C++, and just call a couple of optimized routines from python (and whatever other deep learning framework you are using in python). In this case, it might be worth looking into Tensor Comprehensions.

* You are taking someone elses DL algorithm and moving it to C++. In this case, you still don't need to be diving too much into C++, Ideally, you probably want to use something like ONNX and Caffe, and then call into them with some python wrapper. If you want the whole thing to be in C++, then you are still going to use something like ONNX and Caffe, but you will be writing mostly application code to support it.

* You want to help with the development of Pytorch, Tensorflow, Caffe or anther deep learning framework. In this case, you need to know CUDA kernel development, performant C++ idioms, and good C++ application development idioms.

These are pretty different flavors of C++, which makes it hard to give you a
good answer.

~~~
mi_lk
I'm very much into the last point, do you have more detailed
resources/suggestions?

------
valedra
You can look at examples from DyNet
([https://dynet.readthedocs.io/en/latest/cpp_basic_tutorial.ht...](https://dynet.readthedocs.io/en/latest/cpp_basic_tutorial.html))
or the C++ Pytorch bindings
([https://pytorch.org/cppdocs/](https://pytorch.org/cppdocs/)). I would
strongly recommend python though, as most deep learning work is done with it.

~~~
jdonaldson
Seconding this. Keep in mind you're not in a "normal" C environment when
dealing with GPUs. It's a profoundly negative experience dealing with chip
manufacturers (Nvidia) that push proprietary hardware and actively thwart your
efforts at producing stable and correct code. It made me realize why Linus had
such animosity towards them.

------
make3
With 1.0, PyTorch now has an official legit C++ frontend:
[https://pytorch.org/cppdocs/frontend.html](https://pytorch.org/cppdocs/frontend.html)

PyTorch was apparently used in ~43% of the ICLR papers (a big prestigious deep
learning conference)
([https://www.reddit.com/r/MachineLearning/comments/9kys38/r_f...](https://www.reddit.com/r/MachineLearning/comments/9kys38/r_frameworks_mentioned_iclr_20182019_tensorflow/))
and it's market share is growing at an incredible pace. I would expect it to
beat Tensorflow in popularity at least for research very soon.

That's definitely where I would start if I absolutely had to do deep learning
in C++. Otherwise I would do Python, which is likely used by 95% + of the
research field.

------
madenine
Just like if you were going to learn Python for Deep Learning, you're going to
need 3 things:

1\. Understanding of Deep Learning

2\. General familiarity with the language

3\. Understanding of packages/tools that will help you accomplish your goals.
If this were python we'd be talking Numpy/TensorFlow/PyTorch/Keras/etc.

We're talking C++, I'm guessing you'll want to work with less abstraction than
TF and the like, so BLAS, LAPACK, and maybe the whole CUDA family (CUDA,
cuBLAS, cuDNN, etc).

Otherwise, TensorFlow, Caffe, CNTK, and other popular deep learning frameworks
are available for C++, even though they might be better known and more heavily
used in other languages.

------
gmiller123456
If you look into deep learning, I think you'll quickly learn that the
programming language doesn't matter that much. I think a great place to start
would be the "Two Minute Papers"[1] Youtube channel. A lot of what he presents
there is deep learning material, with links to the actual papers and often the
source code as well as a plain English explanation of what's going on and why
it's important. After looking at the source code of a handful, I think you'll
realize that focusing on learning a single language won't be of any use.

I think it's also important to point out that it's not a great way to ask for
help without giving some details about your background and your motives. If
you've been programming in any language for quite a while, you'll need a
completely different approach to "learning C++ for deep learning" than someone
who's never programmed before. Likewise, if your goal is to work in the
industry producing cutting edge work, you'll take a completely different
approach than someone who just wants a rough understanding of the details just
to satisfy your curiosity. So, without providing that context, you're going to
get answers that are all over the place, almost all of which will be useless
to you.

[https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg](https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg)

------
mooneater
Might be 3 different skills here:

1) learn c++ 2) learn deep learning 3) learn c++ with deep learning: much
easier once you have done 1) and 2)

~~~
wpmoradi
I am familiar with Python and learned deep learning that way. Now would like
to learn C++...

~~~
HankB99
C++ is IMO a bit at the deep end of the pool. I would begin with tutorial
information (Meyers and Stroustrup) and work with that. At some point you will
get tired of tutorial exercises and start to see how you can use C++ for the
topic of DL. I can't comment on Peters as I am not familiar with that.

Hmmm... It seems not possible to search for "The C++ programming language" on
amazon. As fast as I type '++' Amazon deletes the characters. >:( Luckily a
search for "The C Programming language" also lists the 4th edition of
Stroustrups book. ... And after searching via Google, the Amazon page now
works. Weird!

------
ergodic
If you are looking for a computational graph toolkit that is based on clean
C++ go for Dynet.

It is very complete. Similar to pytorch (which borrowed some ideas from Dynet)
and backed up by top academia (not big tech, hence its lower profile). It has
a python wrapper as well, as everything nowadays.

If you are into deep learning for sequence to sequence (machine translation)
Marian is also C++ centered. Fastest MT around. Complete (transformer and the
like), extendable. Not that reading friendly from what I have heard.

Disclaimer: Python type of guy here.

------
gaius
Plenty of examples with CNTK [https://docs.microsoft.com/en-us/cognitive-
toolkit/cntk-libr...](https://docs.microsoft.com/en-us/cognitive-toolkit/cntk-
library-api)

Also Nvidia have a course
[https://courses.nvidia.com/courses/course-v1:DLI+C-AC-01+V1/...](https://courses.nvidia.com/courses/course-v1:DLI+C-AC-01+V1/about)

But I am also curious why... this is deep-end stuff if you’ll forgive the pun.
Why not start with Keras and only dip into the C++ if you need to?

------
jahewson
DL libraries are pretty complex and sparsely documented which can make for a
painful learning experience, even if you already know C++. Maybe go and learn
some graphics programming with C++. You’ll encounter plenty of matrix
manipulation, shaders, OpenGL and the like, and there are plenty of really
good learning resources and the immediate visual feedback makes for a
compelling experience. Then you can return to DL and understand where things
are coming from.

Make sure you learn how to use a debugger. You’ll need it.

------
skinner_
Besides the already-mentioned pytorch and tensorflow bindings,
[http://dlib.net/](http://dlib.net/) is an option if C++ is really a hard
requirement. It is opinionated and incomplete, basically being a one-man-show,
but it has some weird-but-cool ideas like each neural network topology having
its own template type.

------
jason_slack
I guess for me, having been using c++ as my preferred language for 25 years I
just now read api's. When new C++ features are added I get to know then by
practicing using them in small programs then into production code.

Stroustrup and Meyers are the two most well known names. I buy all their books
and I read them and practice all exercises and problems if the book has them.

------
mothsonasloth
Since half the people are ignoring the question and chipping in their own
language recommendations. I will join the bandwagon.

Use Java and enjoy C++ like syntax with 80% equivalent performance (based on
my experience)

Then try this library -
[https://deeplearning4j.org/](https://deeplearning4j.org/)

~~~
mindcrime
Yes, DL4J is a very nice DL framework. I definitely second the recommendation
to give it a shot.

------
wpmoradi
Or C++ for general software development purposes...

------
timwaagh
Deep lesson: learn C#. decrement.

