
Interactive workflows for C++ with Jupyter - SylvainCorlay
https://blog.jupyter.org/interactive-workflows-for-c-with-jupyter-fe9b54227d92
======
FlyingSnake
C++, despite being a really powerful language, has a really primitive build
system. Developers have to wade through a complex soup of
Makefiles/CMake/Gyp/ninja etc just to get a project set up. There's conan for
package management, but it pales when compared to Cargo or Gradle. Moreover
setting up Conan is a full time job in itself, and it's not without weird
quirks either. I've not tried Bazel yet, and would love to hear if it's as
good as Xooglers say.

C++ on Jupyter looks amazing, and I really hope it makes starting with C++
easier.

~~~
superbatfish
The conda package manager and it's subproject conda-build are fantastic tools
to solve this problem. I write scientific software (primarily a mix of C++ and
Python), and I use conda for literally every dependency I need, including pure
C or C++ dependencies. Most of them are already available via the conda-forge
project, and the ones that aren't are easily integrated by writing my own
recipes to package with conda-build.

~~~
dimatura
I write in a mix of python and C++ as well. As such, I've also gradually moved
towards using conda. What's your dev workflow for the code that's not a
dependency like? Specially if the code you're developing is C++. Do you make a
conda package of it as you're developing? I've found that too much of a hassle
to rebuild. I usually just use CMake with the conda environment dir as the
CMAKE_INSTALL_PREFIX.

~~~
superbatfish
Yeah, that's what I do also. conda-build makes it easier to use your code
downstream, but doesn't obviate the need for build scripts in general.

However, if you're using conda, you might be able to at least _simplify_ your
build scripts, even though you can't eliminate them.

If you are okay with requiring your users to have conda, then you can exploit
that fact to simplify your CMakeLists.txt in some ways. For instance,
find_library, etc.. can be replaced with hard-coded links to
${CONDA_PREFIX}/lib/...

(I'm not saying that's necessarily "best practice" for all projects, but it's
a nice option to consider, especially in the early phases of development.)

~~~
dimatura
Ah, that's an interesting shortcut.

------
ddavis
Very cool! I watched the SciPy talk on xtensor (same developers as xeus) and
the project looked pretty awesome. Many people in my field used CINT for a
long time and cling is so much better. It's nice to see it used outside of
particle physics.

One thing I disagree with is the idea that the lack of a good interactive
environment for C++ makes it difficult to teach. C++ is a compiled language,
so learning how to compile a C++ library or program is _part_ of C++. I feel
like C++ beginners should go interactive _after_ learning how to compile a
project. I write that with first hand experience; my first programming
experience was with writing interpreted C++ in CINT and I feel like it
hindered my ability to eventually understand what a real C++ program was.

~~~
jcelerier
> so learning how to compile a C++ library or program is _part_ of C++.

I entirely disagree. This is only for historical reasons, the language itself
does not care at all about the compilation model. This mindset is what is
keeping C++ back, both for the industry AND in students' minds.

> I feel like it hindered my ability to eventually understand what a real C++
> program was.

The moment you have a .o file it's not a C++ program anymore, but a platform-
specific object file. You aren't learning C++ but windows / mac / linux's
native binary production toolchain.

~~~
ddavis
> The moment you have a .o file it's not a C++ program anymore, but a
> platform-specific object file. You aren't learning C++ but windows / mac /
> linux's native binary production toolchain.

Then that is at least part of the learning curve of seriously using C++ so I
think as of today it's (potentially) dangerous to start with only interpreted
code and wait to be introduced to compiling, build tools, etc. They should be
taught together. I also think it's not the greatest environment - and I hope
it changes, perhaps projects like this will help!

------
gcc_programmer
I think jupyter with C++ is an amazing idea, and I hope it will help more
people to start with, in my subjective opinion, one of the most important
programming languages today.

That being said, I look down on all the commentators who say that C++ has a
primitive build system/package manager etc. Lets be clear: C++ has _no package
manager_ , and , in my view of the world, no compiled language has a native
build system, its just that most compiled languages dont separate compilation
from linking and also most languages support modules :)

C++ is _hard_. Throwing some fancy frontend and and a REPL will only make you
realise that sooner. The complexity is mostly unjustified, but the language is
so powerful that you could do anything with it: from trading strategies to
power plant control systems to game engines. I have been studying C++ for 9
years, have used it for 7, and have been a professional developer for only 2.5
years, and I think I have only scratched the surface. I have used python for a
lot less time but I feel like I am more comfortable with it because _Python is
trivial to learn_ (to a person with a comp sci background ofc). What you think
you are cutting away (dependencies/build jnfo) is something essential to
learning C++ and how everything comes together. Better learn it sooner than
later imho.

Rant over

~~~
weberc2
> no compiled language has a native build system, its just that most compiled
> languages dont separate compilation from linking and also most languages
> support modules :)

Most modern, mainstream compiled languages have a build system that wraps
around the compiler and linker. Invoking the compiler and linker are
responsibilities of the build system, and frequently modules are compilation
units.

Modern C++ is a neat language (I used to be a C++ developer) but it's tooling
alone makes it unsuitable for any project where rapid iteration is remotely
important.

~~~
blackflame7000
You're right about most compilers having a build system MSVC being probably
the biggest.

Modern C++ when combined with Qt makes rapid iteration very doable. In my
opinion Qt should be the standard c++ model.

~~~
Matthias247
QT follows a certain style, which is mostly object-oriented and heavily PIMPL
based (it thereby compiles fast).

The consistent style is actually something nice. However lots of C++
programmers will not necessary like the OO style because it will lead to
relatively slow programs. I guess with all the indirection which is going on
and the metadata-based reflection and signal/slot system it might not really
be ahead of even easier to use managed languages, which are even more
convenient to use.

If you look at the Boost (or also C++11/14/17) libraries you will often find a
very different programming style, which is heavily based on templates. This
one is a lot harder to read and understand for non-experts, but it allows to
achieve an even higher performance. The authors might think that this is the
preferred model for C++.

~~~
blackflame7000
Qt is vastly more complex than the boost libraries due to the meta-object
compiler and qmake build process. Furthermore the signals and slots are passed
using lock free mechanisms and you will find they hold their own in
benchmarks. The biggest advantage is the thread safety Qt affords. Not sure
why you equate templates with expert programmer because often times template
systems offer unacceptable build times on large programs. (The boost template
object serialization library for example) Qt lends itself to writing more
correct and simpler to digest code which any expert programmer will tell you
is far more important than fast code

~~~
Matthias247
I didn’t imply in my post that boost is the better choice for experts. Exactly
because I also don’t think that is always true. I only mentioned that lots of
Boost libs are harder to use for beginners than Qt base once since one often
needs to understand a good amount of templates. whereas Qt is mostly based on
a sniper object oriented model with inheritance.

------
aldanor
I can’t help mentioning my own project -
[https://github.com/aldanor/ipybind](https://github.com/aldanor/ipybind) \-
which provides a lightweight Jupyter interface to pybind11 [1], allowing to
quickly sketch working C++ extensions in the notebook; all features of
pybind11 like numpy support are immediately available. Been using it myself
quite a lot for prototyping C++/Python code.

[1] [https://github.com/pybind/pybind11](https://github.com/pybind/pybind11)

------
periram
Finally! Thanks a ton to both Cling developers and Jupyter. Personally, a lot
my programming involves prototyping and playing with a lot of different ideas
and implementations before settling on a solution. Compiled languages put a
steep barrier to this way of thinking and that is why Python has always been
my favorite. I have come back to C++ thanks to Cpp14, 17 and great new
features like this!!

------
rawland
To install this, make sure to start with plain miniconda2, instead of
anaconda2.

I was fighting against QuantStack's modified software stack for two hours
until I discovered this:

[https://github.com/QuantStack/xeus-
cling/issues/74#issuecomm...](https://github.com/QuantStack/xeus-
cling/issues/74#issuecomment-343681758)

Fortunately, they updated the README.md in

[https://github.com/QuantStack/xeus-cling](https://github.com/QuantStack/xeus-
cling)

~~~
superbatfish
Good tip! IMHO, I always recommend the Miniconda installer over the full
Anaconda installer, except for absolute beginners who would struggle to
understand basic concepts like environments or the PATH environment variable,
etc.

------
muxator
> Most of the progress made in software projects comes from incrementalism.
> Obstacles to fast iteration hinder progress.

I would like to print it in very big letters and put it at the entrance of
slow-moving IT departments. "cost of change" is likely one of the most
important metrics to measure the effectiveness of a team.

Being risk adverse because of fear of change leads to stagnation.

"more Jupyter notebooks and less spreadsheets" could be a useful
simplification for conveying this change of POV.

~~~
taeric
And yet,"more data in spreadsheets" is a truism. The spreadsheet model is
empirically one of the most successful ones we have.

Worse, so called "notebooks" are neat, but they are just as bad for
engineering best practices as spreadsheets are.

Not to say that you shouldn't be able to grow from freeform practices. You
should. Just don't mistake them for being somehow superior to spreadsheets.

~~~
jononor
You can very easily move some functions out of a notebook and into a proper
module, that can be included in a program / production system. Autoreload
extension in Python Jupyter notebooks makes it extra nice. It is also easy,
and convenient to write py.test style tests inside a notebook, just call them
manually inside the block. Then when you move it out to module, you have tests
that will automatically be picked up by your testrunner.

~~~
taeric
You can. Most don't. And moving something from a free form environment to a
tested continuous delivery one, is not automatic.

Luckily, the performance constraints of most environments are such that python
is not an automatic deal breaker nowadays. That said, correctness of code
proofs are usually different from correctness of machine learning algorithms.
Such that mixing them seems to just fool both sets of practitioners.

~~~
jononor
My point is mostly that it that Jupyter far from as bad as spreadsheets (at
least commonly used implementations).

Correctness of machine learning is a complicated problem generally.

~~~
taeric
Meh. With some of the practices I've already seen, at least spreadsheets
always show the data. You send me a spreadsheet, I can verify most of it. Send
me a notebook, I can typically only audit what you did.

------
harry8
This is how I'm doing C++ rapid interactive. Combined shell and C++ in a
single file that compiles itself and either runs itself or runs objdump on
itself. I find it really useful and fast to test something in junkcode. Not
quite the same problem as interactive C++ with Jupyter but there's some
overlap. Overlap with godbolt as well - find this quicker perhaps because I'm
more used to it?

[https://github.com/hal88/junkcode/blob/master/template_scrip...](https://github.com/hal88/junkcode/blob/master/template_script.cpp)

I post in the hope that it is a useful idea to someone and also that people
respond with their similar ideas that might be useful to me.

------
rocqua
This looks downright amazing.

I only wish it had come a few months earlier. I've had to switch from python
in a jupyter notebook to C++ for performance reasons, and I really felt the
loss of incrementalism.

~~~
cbcoutinho
You can create your project into python library by wrapping in ctypes or using
the cffi package. I do that with some of my numerical simulation code and it
doesn't take away from the interactivity of python. Once I get a feature more
or less complete, I re-write it in c/fortran and compile it away. This
maintains performance as well as focus, because the 'interesting' thing you'll
be working on is quick and dirty using python, and the stuff you've already
okay'd is in a compiled language out of sight.

~~~
rocqua
It is this transpiling from python to C++ that I want to avoid. Especially
because there is a lot of iterative development on the C++.

However, I'll essentially never be able to run my code from a notebook because
it runs on a cluster I need to SSH into. Thing is, I'd like to develop that
code locally and iteratively.

~~~
cbcoutinho
But I thought you said you already re-wrote it from python to c++ anyways? I
was trying to point out it doesn't have to be either/or. That being said, I
think the fact that you need to get access through ssh is another hurdle all
together.

~~~
rocqua
I switched once I needed the performance, but that was not near the end of
developtment. The switch was made far from a point where the code, or even the
algorithms were finished.

Thus it was required to do a lot of incremental development in C++. It
wouldn't make sense to add in a manual transpilation at every increment, and I
needed the actual C++ code to judge performance.

------
mempko
Looks like the world is finally catching up to the Future that Temple OS lives
in.

~~~
Mathnerd314
TempleOS doesn't do scatter plots.

------
TisenoT
>This also makes C++ more difficult to teach. The first hours of a C++ class
are rarely rewarding as the students must learn how to set up a small project
before writing any code. And then, a lot more time is required before their
work can result in any visual outcome.

Install a c++ compiler, create main.cpp, `g++ main.cpp`

What am I missing?

~~~
flukus
> What am I missing?

The students don't know anything about the CLI. They might not know what a
directory is or how to cd to it. They may have to install multiple packages.
On certain operating systems they'll have to set some PATH variables.

I'm not saying it shouldn't be that easy, but due to failures in earlier
education it's not.

~~~
ryanpepper
When I was taught C++ at University, we were just told to download Quincy
which bundles a (pretty old) version of GCC with a text editor on Windows.

Once we had the basics down they introduced Linux, the command line interface
to GCC, etc.

~~~
AstralStorm
Formerly, djgpp package. And the equivalent Windows IDE, can't remember its
name but it shipped with Mingw.

------
scardine
I could not make it work on a mac. It almost works but looks like there is no
mac package for xeus-cling at the conda repos.

Tried also to clone the github repo and compile locally but got tired of
fixing dependencies and stuff after a few compiler errors.

------
rb808
I'd like to use Jupyter notebooks to do support tasks - eg check on system
hosts and do reports, maybe restart processes. I think it could be perfect.
I'm just hesitant because no one else seems to use it this way. Any advice out
there?

------
westurner
QuantStack/xeus-cling [https://github.com/QuantStack/xeus-
cling](https://github.com/QuantStack/xeus-cling)

QuantStack/xwidgets
[https://github.com/QuantStack/xwidgets](https://github.com/QuantStack/xwidgets)

QuantStack/xplot (bqplot)
[https://github.com/QuantStack/xplot](https://github.com/QuantStack/xplot)

------
mcemilg
Okay. If its works amazing, so it can be work on other compiled languages?
This will be amazing and it can be game changer.

------
peheje
Cool. For anyone looking at using Python when speed is important should also
take a look at Numba.

~~~
superbatfish
Yes, numba is great; I've had a good experience with it for some functions.

Some problems aren't suited well for it though, especially those that involve
Python dictionaries or sets. For those problems, switching to C++ results in
MUCH faster code (std::unordered_map is way faster than Python's dict,
especially for small elements).

~~~
peheje
As I'm aware of Python dictionaries are not supported by Numba, however it
will still run in "object mode" which won't give you much performance boost. I
tend to always specify the Numba function as njit or objectMode=False. But yea
true, Numba does have shortcomings, maybe dictionaries and classes will be
supported in the future.

------
deepGem
Actually, I've found Xcode to be a very easily usable prototyping tool for
C++. I haven't used it in any large project, but it's super simple to just
prototype some concepts. Having that in a browser is even better.

------
BeetleB
It would be great if we get C++ on Jupyter. At the moment, I use Org Babel to
do these kinds of things. It's a halfway solution.

------
boltzmannbrain
I really hope this works as promised. I'm curious if they'll add support for
boost (www.boost.org) though.

------
kasperset
How is it different from R notebook implementation of C++ chunks execution?

