
Want to call C from Python? Use D - atilaneves
https://atilaoncode.blog/2020/02/19/want-to-call-c-from-python-use-d/
======
michelpp
Interesting article, but I wouldn't say introducing a 3rd language is the
easiest way to call C code from Python, that distinction goes to the CFFI
library by Armin Rigo et al. I've used it to wrap several C libraries and it's
amazingly powerful. Bonus it works with CPython and PyPy.

[https://cffi.readthedocs.io/en/latest/](https://cffi.readthedocs.io/en/latest/)

~~~
oefrha
Don’t forget the builtin options.

First there’s PSL ctypes,[1] which is supposedly inferior to cffi to some
extent, but hey it comes with your CPython installation, no third-party dep
required.

Or you can write a C/C++ extension module.[2] A bit more heavy-handed and
there’s a learning curve, but once you’ve done it once or twice it’s really
not hard, and it affords you great flexibility.

[1]
[https://docs.python.org/3/library/ctypes.html](https://docs.python.org/3/library/ctypes.html)

[2]
[https://docs.python.org/3/extending/extending.html](https://docs.python.org/3/extending/extending.html)

~~~
atilaneves
The solution in the blog post _is_ an extension module. Written for you, by
the compiler. Instead of a learning curve and "it's really not hard", it's "no
learning curve" and "I struggle to find how this could be easier".

~~~
oefrha
I would take a tested and true solution sanctioned by CPython core devs any
day over a novelty third party library that gets another language toolchain
involved.

Also, the solution seems to only cover what’s possible with ctypes already. If
don’t need the flexibility there’s little reason to hand roll an extension
module. (Not saying ctypes doesn’t have its problems; e.g. when you pass in a
wrong number of arguments instead of throwing a TypeError it segfaults.)

~~~
loeg
ctypes does not parse C headers and doesn't know about enums or function
parameter types — only that they have a symbol in the .dynsym section of a
.so.

Maybe you're thinking of cffi? Cffi does basically what this blog post covers,
except without D. It requires a C compiler at some stage in the process, but
you can pregenerate during 'build' phase and ship a header snapshot to avoid
requiring the C compiler at runtime.

~~~
oefrha
I didn’t say ctypes takes the same approach — I even mentioned ctypes’
shortcomings. I only said ctypes makes what this does possible already.

~~~
atilaneves
No, ctypes does not. It can't handle macros, and it requires users to define
the structures in Python themselves. It's miles away from what's in the blog
post.

------
WalterBright
> I wrote a little project called dpp because I’m lazy

I can confirm that Atila is a very lazy man. He's always figuring out ways to
have his computer do the work for him. He's so successful at avoiding work,
people keep giving him more work to do, making him a very hard working lazy
man.

~~~
canada_dry
> always figuring out ways to have his computer do the work for him

One of my earliest programming mentors mused that the reason I might well
become a decent programmer was because I was _both extremely stubborn and
horrifically lazy_. He'd observed that I would re-write code until it was the
most efficient possible to automate a needed task that could save me
time/effort in the future.

~~~
ugexe
It’s a common observation, and indeed known as one of the Three Virtues of a
great programmer according to Larry Wall

[http://threevirtues.com](http://threevirtues.com)

~~~
FeepingCreature
I think arrogance should replace impatience, which has redundancy with
laziness. (Or possibly hubris, which is the weakest.) Arrogance makes you
start projects that you are not skilled enough for, and grow in the doing.

~~~
taejo
Aren't hubris and arrogance more or less synonyms? With hubris being perhaps
more specifically _arrogance about one 's abilities_

~~~
FeepingCreature
Right, but that's not exactly how it's used in the post.

------
jakogut
I'm currently working on a project using Cython to wrap and expose a large and
complex C API (CEF) to Python. One of the most time-consuming, tedious, and
error prone tasks is translating functions and structures to Cython syntax,
including managing the GIL. Also, keeping the translated API up to date with
new releases.

I'm curious to see how this D approach compares, both in workload required to
expose and use an API, and the flexibility of the end result. There are a lot
of quirks and special syntaxes in Cython to allow for not just interop between
C and Python, but also writing C in Python-like syntax mixed with actual
Python. This is an area I assume this approach would be lacking.

Regardless, as someone who loves mixing C and Python, this is an encouraging
reason to look more at D.

~~~
jessermeyer
Is there a reason you chose not to build / refactor the project as a library
to call from Python?

~~~
jakogut
Can you clarify your question? I'm using Cython to write extension modules
that directly interface with CEF's C API. CEF is about 40k lines of C++ that
programmatically controls Chromium, which itself is written in C and C++.

~~~
jessermeyer
I see. Thanks, that does change my understanding of your situation. My revised
question with your clarifications in mind is, is there a reason you're not
calling CEF's C api directly from Python? Are your extension modules doing
some heavy lifting themselves, or are you using Cython just to handle the glue
between the language interface?

~~~
jakogut
> is there a reason you're not calling CEF's C api directly from Python?

Such as by using CFFI? Cython's strength, in my opinion, is in the fact that
it is its own language resembling both C and Python that compiles to down to
C, rather than just an FFI.

> Are your extension modules doing some heavy lifting themselves, or are you
> using Cython just to handle the glue between the language interface?

I would say the majority of the application logic is written in plain Python,
and Cython makes interfacing with it, creating higher level interfaces, and
marshaling data far easier than with CFFI or using the C API directly. This is
pretty subjective though.

------
jononor
I often use C++ and pybind11 to do this. pybind11 takes an approach very
similar to boost::python (template magic). It works rather well.

Though most of the C code I write can compile as C++ also. Why write in C
then? It's still the lingua franca in microcontroller land... Just want to
reuse the code on PC and Embedded Linux, and Python makes high-level testing
much nicer.

~~~
aldanor
Pretty much this. Add to that pybind11's exceptional numpy compatibility
(including easily mapping C types to structured numpy dtypes that can be e.g.
converted to a dataframe), and there's not many choices left if you're not
planning on reinventing a wheel or two.

(disclaimer: pybind11 contributor)

~~~
pletnes
Is pybind numpy aware? This is interesting. Having never used pybind, where
can I learn more?

------
dekhn
It's really crazy how many ways there are to implement FFI in Python. I
started in the "old days" (Makefile, a bunch of CPython C code to implement
argument parsing, etc), switched to swig (experienced both the good and bad
sides), hoped that GCC-XML would make it easier to convert complicated C++
classes to Python interfaces (big mistake), then hoped that LLVM 'would solve
the problem', saw Google release CLIF (still can't compile it from scratch on
a modern linux system), and the large number of "late-binding" approaches. Of
all of them, I was most skeptical of Cython but have slowly come to appreciate
its sublime cleverness.

------
jaked89
D & Nim (especially Nim) are very appealing from a language perspective, but
the lack an IDE undermines the advantages they bring.

I personally won't consider using a language for a meaningful project unless
it has a decent debugger, and reasonable refactoring; ie. at least find
references & rename.

Beef is building on an IDE, which is a refreshing take, but it's still too
early.

~~~
JNRowe
My own pet peeve about D is the lack of local documentation. If I install my
system's compiler packages¹ I receive basically no documentation at all. That
is a huge difference to the experience with nim/rustc/go/$others which all
provide a nice fat $lang-doc package in their Suggests field, and all of those
contain full standard library docs/tutorials/etc.

1\. gdc or ldc on Debian. The experience /may/ be different with the dmd
repositories, but being external they're of less interest to me.

~~~
WalterBright
D comes with an interesting utility called `dman`. Run it from the command
line and as an argument give it a D topic and it'll open a browser on the
topic.

    
    
        dman dman
    

will, of course, open the browser on a page about dman.

~~~
JNRowe
Which, as I noted, is still a very different experience from go/nim/rust in
Debian. It wasn't a pet peeve about no documentation, it is about how I can
interact with it compared to to other languages.

------
atilaneves
I describe how easy it is to call C code from Python by using D as the glue
layer. AMA.

~~~
cannam
So behind the scenes is a utility that scans the #included headers, runs the C
preprocessor on them, then uses Clang code to pick out the declarations, wraps
them for calling from D, and redefines any C macro values as D constants? Do I
understand that correctly?

(The other half of the puzzle being a tool called autowrap that wraps the
resulting D for consumption by Python.)

I'm not sure whether I personally would ever use this, but it's a very
interesting exploration of the subject.

~~~
atilaneves
> Do I understand that correctly?

Yep!

> (The other half of the puzzle being a tool called autowrap that wraps the
> resulting D for consumption by Python.)

Correct.

> I'm not sure whether I personally would ever use this, but it's a very
> interesting exploration of the subject.

Python was an example. Want to call it from C# instead? Same code. Excel? Same
code.

~~~
jcelerier
could you explain the weird fascination of the D community with Excel ? there
are so many "D" things for Excel and it seems really weird as an outsider - I
don't remember the last time I've seen an actual Excel in the wild and not
libreoffice or openoffice

~~~
VHRanger
Since D has a fairly small community, I imagine its a handful of resourceful
users in the financial sector fixing problems they're forced to deal with

~~~
bachmeier
There's also the fact that the Excel userbase is massive, so it's a sensible
target.

------
andrewprock
Is swig no longer a thing in the 2020s?

I've always done this using the bundled Python/C integration path, or when
there was too much stuff, use swig.

[http://swig.org/](http://swig.org/)

~~~
atilaneves
I used swig once. Never again.

~~~
andrewprock
I'm curious, what was your negative experience?

~~~
atilaneves
Literally all of it.

------
marmaduke
[https://github.com/davidjamesca/ctypesgen/wiki](https://github.com/davidjamesca/ctypesgen/wiki)

does this without needing a compiler, by parsing the header files and
generating the ctypes wrappers

~~~
atilaneves
There's no way to parse headers without a compiler.

------
skocznymroczny
"Envious of C++’s and Objective C’s credible claim to be the only languages
that can seamlessly interoperate with C (due to header inclusion and
compatible syntax)"

Wouldn't Nim and Zig also fit the bill?

~~~
atilaneves
The last time I looked at the Nim documentation, one had to manually declare C
and C++ functions. I don't see how that's different from pretty much any other
language that can call C (i.e. pretty much all of them). Then there's
preprocessor macros.

I don't know how Zig does it, I'd have to look.

I expand on why dpp is the way it is in my DConf 2019 talk:
[https://www.youtube.com/watch?v=79COPHF3TnE&t=8s](https://www.youtube.com/watch?v=79COPHF3TnE&t=8s)

~~~
nimmer
Nim has an automated generator for C/C++ wrappers or translating from C.

Also there are very easy ways to create Python modules in Nim:

[https://robert-mcdermott.gitlab.io/posts/speeding-up-
python-...](https://robert-mcdermott.gitlab.io/posts/speeding-up-python-with-
nim/)

[https://github.com/yglukhov/nimpy](https://github.com/yglukhov/nimpy)

~~~
atilaneves
Thanks for the links.

That's nice, but not the same - users have to intend to write a Python
extension in Nim. With autowrap, you can call _unmodified_ D code (and as
shown in the blog, C code as well with dpp). That means code that was never
meant for consumption from another language can be made available.

~~~
nimmer
You don't need to modify the code, just add a wrapper.

Nothing prevents generating a wrapper automatically but 99% of the time that
is not very useful. The large majority of Python extension contain Python-
specific code to access or return Python objects or act in a pythonic way.

~~~
atilaneves
> You don't need to modify the code, just add a wrapper.

Still not the same.

> Nothing prevents generating a wrapper automatically but 99% of the time that
> is not very useful. The large majority of Python extension contain Python-
> specific code to access or return Python objects or act in a pythonic way.

Except, and I cannot stress this enough, at work we're using this to
succesfully call into D production code without any Python-specific
_anything_. It just works. The article wasn't even about that, it's about
making it work just as seamlessly for _C_.

------
abrax3141
Okay, but what if I want to call python libraries from C (or any other
language)? What we need is some sort of .net-like global interop in the Unix
world.

~~~
dralley
You can't really do that without including the entire Python runtime in your
application, and cpython is (so I've heard) absolutely horrid at being
embedded in things.

~~~
abrax3141
Why can’t we create a --server mode standard where every language can be
started up in that mode and serves a standard API with standard linker types
(where possible)?

------
justlexi93
Scripting languages by nature will always be slower than C. There's a tradeoff
between program speed and development speed

------
fithisux
Hopefully the dream to compile a D library to another language (something like
swigd) it will be interesting

~~~
atilaneves
That already works with Python, Excel and C# now. There's a link in the blog
post to another blog post to that effect.

------
hatsunearu
Or you know, learn any of the native options that let you call C things
relatively painlessly.

------
epicgiga
How many languages do we need again? The amount of time we spend playing
around with special case languages and frameworks. Would be nice if machine
languages followed the lead of natural languages, with a single one slowly
pushing the rest to the sidelines.

~~~
macintux
What a terrible world that would be. For a while, 20 years ago, it looked like
Java was going to be the one language to rule them all; dodged that bullet.

Kitchen sink languages are my least favorite. Give me something with a tight
focus on a problem space.

