
Python Internals: PyObject - kercker
http://www.gahcep.com/python-internals-pyobject/
======
harrisi
Although it's been posted here (only twice, though!) before, Philip Guo put up
a 10-hour series of lectures on CPython internals[0]. I am nearly done with
them, and have found them very useful and well-paced. Posting here because the
author of this article seems to have stopped the series, and anyone interested
in this stuff should definitely check out the Guo lectures.

[0]: [http://pgbovine.net/cpython-internals.htm](http://pgbovine.net/cpython-
internals.htm)

~~~
giis
+1 Guo series is excellent. I completed them twice & very confident about my
python skills more than ever before :)

~~~
pgbovine
Thanks everyone! Recording lectures has been great since it doesn't take much
extra work than not recording; just takes some time to render the final videos
and upload. Ideally I'd have time to split the videos up into smaller chunks,
but alas free time is diminishing nowadays.

~~~
harrisi
Would you have any interest in providing someone (I would volunteer) the
videos so that they can package them in smaller, topic-specific pieces for
easier consumption? I don't know anything about editing videos, but if your
writings and lectures have taught me anything, it's that I'm capable of
learning things and using that knowledge to do interesting and useful work.

As an aside, you are one of my main sources of inspiration. I was a high
school dropout with several terms of failing out of community college courses,
but after reading your PhD Grind series/book, I'm now a (nearly) straight A
student and am very determined to go as far with my education as I can. I
would be honored to help you in any small way I can to express my gratitude!

Thanks again for unknowingly improving my life.

------
dfox
Contrary to what the article says, PyVarObject is still PyObject at the same
time and PyVarObjects have nothing to do with mutability (certainly, it's
trivialy obvious that not all PyVarObjects are mutable as immutable tuples are
PyVarObjects, while mutable lists are not)

The thing that is handled by PyVarObject are types whose instances can differ
in their size, with tuple and bytes objects being probably most common cases
when it is used.

------
partycoder
If you try to embed or extend Python in C you will be exposed to PyObject very
quickly. The official Python documentation can walk you through that process
(visit the embedding or extending parts). The PyObject structure mentioned in
the article also comes with many functions to manage them programatically,
cast them to other types, etc.

~~~
p4wnc6
I'm curious if you know much about jobs doing this kind of thing. I've worked
a ton with Python, and know a lot about using Cython to write extension
modules, wrap C++ code, etc.

I really want a job where I find performance bottlenecks in research or
business analytics or financial analytics code, migrate parts to robust Cython
implementations, and then build and maintain the broader Python libraries and
APIs surrounding that.

But what I'm unfortunately finding is that basically nobody uses this stuff
anywhere in practice. I've interviewed at a ton of places doing everything in
C++ entirely, and I just see their code and I think, my god, why would anyone
choose this? You can write the critical stuff in C++ just as you are, and then
so easily wrap it for Python, then write the other 99% of the code in super
easy, dynamic Python, even with various kinds of validating systems, or using
typing and type annotations to avoid certain kinds of bugs, and just
everyone's life is easier.

But they are just so entrenched in the old way of doing it that nobody is
willing to try.

I've been consistently amazed at how few jobs in the monthly Who Is Hiring
thread are tagged even just with NumPy. I think I've seen maybe one job in 6
months that mentions Cython, and then I found two others outside of HN and the
interviews with those places didn't work out.

How to find a job doing this stuff is like my main vexing work problem at the
moment. It's so useful, but it's sort of locked outside of some kind of energy
barrier that quant teams seem unwilling to cross.

~~~
osullivj
Suggest you look for an opening on JP Morgan's Athena project, or Bank of
America Merrill Lynch Quartz. Both those platforms are C++ on the inside with
Python APIs. The originators have gone independent here:
[http://www.wsq.io/about-us/](http://www.wsq.io/about-us/) The site says
they're hiring.

~~~
brianwawok
A few trading companies do this kind of work also.

Since you really want to write C and Python... Do you care if you have to
write trading code to help billionaires make more money?

~~~
p4wnc6
I don't think I would be a good match for bank culture or HFT, but if the work
you describe is in a small-to-medium sized asset manager or hedge fund, then I
would not mind, and would find it enjoyable. I worked at such a place a number
of years ago and it was one of the better jobs I've had.

The trouble I find is that a lot of small-to-medium sized asset managers and
hedge funds don't have very good technology practices in general, and are
pretty far from sophisticated use of low-level Python. There are often a lot
of political reasons why they cling to Excel/VBA, R, or MATLAB, and there's
almost always some political battle happening between the people that want to
rationalize the system to a proper software design, and people who just want
to keep cranking on the hodge podge of existing tools.

I'd probably be fine with some trade-off regarding all of that, if the pay was
acceptable and there was a strong commitment to a healthy work/life balance.
But among finance firms this is extremely hard to find.

So even though I am not at all opposed to doing this work in finance, I still
find the volume of acceptable job openings in that field to be too small to
make it realistic.

------
denfromufa
Here is blog post about python list implementation, one of 2 main data
structures used internally: [http://www.laurentluce.com/posts/python-list-
implementation/](http://www.laurentluce.com/posts/python-list-implementation/)

------
denfromufa
Cython can be used to auto-generate optimized Python C-API code that gets
source-mapped line by line to original Cython code in the form of HTML
annotations. This stuff is very hard to write by hand.

