

Python 3 Q&A by a prolific core developer - hynek
http://ncoghlan_devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html

======
sutro
Unconvinced.

------
zanny
I think all the outrage is that Python 3 behaves like a new language in how
everything needs to be ported (albeit at a magnitude less complexity than
going up the language chain or across the playing field, like C to Ruby or C#
to Java) but it still incurs the new language cost, while being the new
version of the old language.

If people just thought of Python 3 as the replacement to broken text model
Python, like how XHTML tried to supplant HTML and failed, and then HTML5
supplanted XHTML with some quirks, the same thing happens here.

Also, I adore the comments about the GIL. I understand the benefits of using a
single language in every circumstance, but (even) a JIT scripting language is
not sufficient for all problems. I guess it depends on if a polyglot code base
is more complex than handling multithreading and the corresponding memory
complexity and gimmicks, and that is developer centric.

Still, I would much rather just stick processor bound code in openMP C
functions and call those from Python when need be. It seems like the "right"
answer to the performance problem, without losing much productivity.

------
gvalkov
It's just one of the little things, but I'm still very conflicted about
_print_ as a function. The reasoning is sound, but I just can't seem to get
used to it. It feels especially cluttered and clumsy with new-style string
templates:

    
    
        print('ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper())
    

versus:

    
    
        print 'ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper()
    

Despite the more consistent naming (ConfigParser -> configparser), simplified
api (iteritems() -> items()) and all other syntactic improvements, I somehow
still find 2.x code more enjoyable. Writing small scripts in Python has kind
of lost its charm for me.

This, of course, is all very subjective and I'll probably grow over it in a
few thousand lines of code. I hope you're all less sensitive to the little
things that annoy you.

------
toyg
Unreadable on iPhone, what's up with its stylesheet? Safari won't even
zoom...?

~~~
hynek
Yeah, it seems to be the default RTFD style…try:
[http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs-...](http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs-
python-
notes.readthedocs.org%2Fen%2Flatest%2Fpython3%2Fquestions_and_answers.html)

------
wololo
I wish python 3 would make default parameters immutable. It seems to go
against the "simple", "explicit" python philosophy, not to mention that (it
seems) most people tend to learn about the feature through debugging.

description: <http://effbot.org/zone/default-values.htm>

~~~
hynek
Yeah sometimes I wish there were a warning whenever someone defines a function
with [] as def arg.

~~~
wololo
I know pylint and pychecker do this, but not pep8. Opinions on those?

------
smeg
Python3 and Perl6 can both fuck off.

I look forward to the day PyPy is considered the real Python. Look at PyPy's
homepage (<http://pypy.org/>), doesn't even mention Unicode as a significant
feature. Instead it talks about speed, security, concurrency, and
compatibility with the current _real_ Python (2.7.2) - all the things real
Python programmers care about and expect the Python developers to focus on.

PyPy may be some way off but I want to find the developers and hug them for
setting the right vision and _trying_.

I just donated $50 and if you hate Python3 you should too.

~~~
sho_hn
Or if you really like Python 3 (as I do!) you can donate those $50 to Python 3
support in PyPy: <http://pypy.org/py3donate.html>

------
dmbaggett
From the article:

 _Back in reality, though, complaining about the GIL as though its a serious
barrier to adoption amongst developers that know what they’re doing often says
more about the person doing the complaining than it does about CPython._

It was a solid, well-argued piece up to this point. You do yourself and the
Python community a disservice by writing off your critics as ignorant. It
sounds petulant and childish, and is wrong.

There are valid arguments on both sides of the GIL argument, but neither
side's advocates are ignorant or bad programmers.

~~~
bwood
In my experience with Python, the issue of the GIL comes up far too
frequently. I use Python first as a prototyping environment, and once I have a
proof of concept I like to optimize for performance (without re-writing the
entire program in a more appropriate language). Most of the stuff I do is CPU-
bound, but also involves large amounts of data. So, without writing C code as
an extension, I can either suck it up and deal with single-core performance,
or I can use the multiprocessing module and hope that inter-process
communication isn't too expensive or difficult to code (and no, it's not
always as simple as just using multiprocessing.Queue). I am under no delusion
that the GIL will ever be "fixed", but I am disappointed that I spent years
working with Python before I got into more CPU-intensive tasks and eventually
hit the brick wall that is the GIL. And of course, comments like you
mentioned, which attack my competence as a coder, are not constructive.

~~~
pwang
I'm interested to hear about your CPU-bound Python programs that also involve
large amounts of data. Normally such problems are I/O bound and are better
solved in a data parallel way. (Graph problems can sometimes be an exception,
but those are still generally I/O bound.)

For writing extensions, have you considered Cython?

~~~
bwood
Sure thing, here are two examples:

1) My current project at work is a GPU-accelerated keyword-matching engine.
The project was started before I joined the company, so I had no say in the
choice of Python. Keywords change infrequently, while we analyze a continuous
stream of incoming text. There are several million keywords, ranging from
small to enormous in size. Aho-Corasick
([http://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_mat...](http://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_matching_algorithm))
is a pretty ideal algorithm for this scenario, which we use for the GPU
matching kernel.

AC requires some preprocessing of keywords into a deterministic finite
automaton (basically a suffix trie). This is very expensive for a large number
of keywords with a large number of characters. The DFA grows to something like
10GB while being built.

Meanwhile, the main engine loop has to be running continuously, while updating
keywords in the background. The engine is a service available to other systems
on our network, so it uses multiple threads for concurrent I/O. The problem is
that the GPU performance is so ridiculously high that the CPU can't keep it
fed with data. I've profiled it and this is not a memory-bound problem...the
CPU simply cannot keep up with the document streams that we send to it.

The concurrent I/O threads cannot reasonably be split across processes because
they need a shared memory space for the data structures driving the engine. So
clearly, the background keyword updating is a problem if it runs in the same
process as the rest of the engine. I spent a lot of time trying to figure out
how to get the keyword updating working in its own multiprocessing process.
It's a complete hack to work around the failings of Python (I can go into more
depth about the implementation issues if you'd like). And this is why I loathe
the GIL.

We use Cython for some aspects of the code, but the keyword updating has
yielded very little gain. It's difficult to rewrite parts of the keyword
updater as more optimized Cython because it uses some language features that
do not seem to be supported in Cython.

2) For a personal project, I need to do a lot of timeseries processing. I'm
using Python to prototype, with the intention of either optimizing it
eventually or possibly rewriting it in a more suitable language. I've found
parsing timestamps to be particularly CPU-intensive, while working on
gigabytes of data. Most data I send to a multiprocessing process will have to
be returned in some form eventually, so communication costs are huge. So huge,
in fact, that I only see a 10% speedup from splitting the workloads evenly
across six cores. Profiling reveals that the majority of the "processing" time
is actually just waiting on data getting sent back to the main process. This
would not be a problem with a shared memory space.

~~~
baq
re 2) - <http://docs.python.org/library/mmap.html> ?

~~~
bwood
Interesting, thanks for the tip! I'll look into it and think about whether it
makes sense for my timeseries analysis.

------
ash
Why can't I use Python 2 modules from Python 3 program? Is there any effort to
make it possible?

~~~
hynek
There is no such thing as "Python 2" or "Python 3" modules. There are just
modules that will go belly up because they are incompatible with one or
another.

I presume you mean that you can’t access your old Python 2 modules from a new
Python 3 installation. That’s just because Python installations usually don’t
share their modules (i.e. site-packages). You can try to install them using
Python 3 (usually just python3 setup.py install) and see if they work or not.

~~~
lrem
There's a deeper question here: why doesn't the interpreter contain a
compatibility layer for the old code? There's no really good reason for the
old modules to go belly up in the new interpreter. At least no other than the
man months needed to implement that layer. But then, that probably would still
be orders of magnitude less than migrating all the libraries that keep people
from migrating...

~~~
hynek
That’s essentially what __future__ imports are for – things that go further
would be simply too much work and Python 3 was about shedding weight, not
shipping essentially two interpreters. (while they are very similar from
outside, the innards couldn’t be more different)

~~~
lrem
Cool. So this way porting things to Py3k got easier. That perfectly explains
why all the critical libs have been ported fast and no one remembers about
Python 2 anymore, doesn't it?

Shedding weight is sometimes a step in a good direction. But we need to draw a
line at some point. As the history of Python 3 adoption shows, that point
maybe was not optimal.

Edit: forgot an angle:

Also, __future__ is fixing the incompatibility of what people are using with
something that they can't use. I'm not sure if it's the right problem to
solve.

~~~
hynek
As it has been pointed out for several times: the adoption of Python 3 is
doing just fine. It has never been expected that everybody will be using
Python 3 today.

But look at all those current efforts at Canonical, Django, Twisted…it’s not
like nothing is happening and we expect a knot to burst. Far from it! Porting
started slowly and has gained a momentum by now that has surprised myself.
It’s not like we changed the language completely like perl6 did.

In the result, we’ll have a better Python for it.

------
thu
The following situation happens with other data than just unicode. This seems
an argument for static typing in general.

    
    
      The reason this approach is problematic is that it means the traceback for an unexpected UnicodeDecodeError or UnicodeEncodeError in a large Python 2.x code base almost never points you to the code that is broken. Instead, you have to trace the origins of the data in the failing operation, and try to figure out where the unexpected 8-bit or Unicode code string was introduced. By contrast, Python 3 is designed to fail fast in most situations[...]

~~~
gardentheory
Well the lighter solution and more Pythonic solution is strong typing. If a
Unicode string and a byte string could not be combined then the problem would
not exist AFAIK. For example "a" + 1 is an error, why is u"a" + "s" not an
error?

~~~
masklinn
> For example "a" + 1 is an error, why is u"a" + "s" not an error?

"a" + b"s" is an error in Python 3.

~~~
gardentheory
Sorry, that was my point. Strengthening the types in python3 solves the
problem without static typing being needed at all.

~~~
masklinn
ok.

Static types solve it earlier though, especially for little-exercised code
paths which could be forgotten or missed in testing.

------
dmbaggett
In the last few hours, this has been silently added to the post, and I cannot
post a reply on that site:

 _Jumping on the internet to say that “they” (specifically, the people you’re
not paying a cent to and who aren’t bothered by the GIL because it only
penalises a programming style many of us consider ill advised in the first
place) should “just fix it” (despite the serious risk of breaking user code
that currently only works due to the coarse grained locking around each
bytecode) is also always an option. Generally speaking, such pieces come
across as “I have no idea how much engineering effort, now and in the future,
is required to make this happen”._

Fine, then don't bother. But don't insult us or call foul for our pointing out
the downside impact _on us_ of this decision. Your assumption that we are
naive rubes who don't know how to code is really, really wrong.

I have a very good idea how much engineering effort would be involved in
fixing the GIL, and I am well aware that Python has involved many person-
millennia of gratis work, and am appreciative of both. However, I still
disagree with the Python devs' obviously entrenched position that fixing the
GIL isn't worth the effort, and I will continue -- even when shouted down by
the likes of you -- to advocate for the GIL's removal or some equivalently
good solution. (As I said earlier, I am not opposed to STM solutions, but the
current one performs unacceptably without special-purpose hardware.)

Why? Because I have a single, selfish interest in this. I depend heavily on
Python now, and want the language to be better. I have written many lines of
Python 2 code that rely on the threading primitives in the standard library.
Perhaps it was foolish of me to expect that the threading model offered by the
standard library, modeled on Java's threading primitives, would some day work
in the same way as Java threads do in practice. Nonetheless, I am left with a
real world problem: my CPU-bound threaded Python code does not scale well to
multiple cores. I need the GIL fixed, or to rewrite my Python code, or to
migrate to another language that supports the standard model of threading
programming that real-world programmers have been using for several decades,
and which has built-in support from all major operating systems. Or, sure,
wait for STM to be ready for prime time and migrate my thread-based semantics
to the new STM-based semantics.

The best path right for me right now is migration to Jython or IronPython. But
then we are still unsupported orphans, living in the third world of Python
2.X.

I guess it comes down to: do you want people to actually use this language to
write programs they want to write, or do you want Python to be an advocacy
platform for "correct" programming? Python's pragmatism has always appealed to
me, so the ivory tower reaction to the practical concerns around the GIL
really seem dissonant. (And this is coming from an MIT AI Lab Lisp guy who
would rather write everything in Lisp. But Lisp lacks Python's wonderful,
high-quality third party libraries and developer-centric pragmatism regarding
unit testing, documentation, etc.)

I know you are tired of hearing people bitch about the GIL, but, really:
people write multithreaded programs. They should work as advertised, using
native OS threading primitives and taking advantage of the native OS thread
scheduler. Why does Python offer threading primitives if the language is not
meant to support, from a practical standpoint, multithreaded programs?

~~~
old-gregg
_...the standard model of threading programming that real-world programmers
have been using for several decades..._

What standard? And whose "real world"? The need for threads has always been
controversial even among OS kernel devs. UNIX/Linux/BSDs have twisted and non-
trivial threading histories peppered with religious wars similar to this one.
And which "several decades" are you talking about?

There is no such thing as "standard threading model". To some, a thread is
just a flavor of fork() with a wrong parameter and plenty of "real world"
programmers continue to believe that kernel-level threads is a hack. And
please, do not make it sound like Python threads are useless. Far from it.

Python threads are not what you are used to. That's pretty much TL;DR of your
comment.

 _...Python's pragmatism has always appealed to me, so the ivory tower
reaction to the practical concerns around the GIL really seem dissonant..._

I feel like they are being dragged into it though. The original motivation
behind GIL support has always been a pragmatic one: removing GIL will make the
entire codebase more complex, harder to hack on and will complicate and slow
down the development/maintenance of the libraries. That's pretty pragmatic.

But a fairly vocal groups of users started to claim, similarly to you, that
programming with threads is _supposed_ to work like they expect it to work
according to make-believe "threading standard", to which GIL supporters
(correctly, IMO) replied that shared memory + locks is not the only/best
approach to concurrency. It is easy to be offended by this answer but it
doesn't invalidate their point.

~~~
dmbaggett
_Python threads are not what you are used to. That's pretty much TL;DR of your
comment._

Correct. I am used to programming language threads that work the way computer
scientists and programmers have typically described them -- for example, as in
this (I hope uncontroversial) Wikipedia article:

<http://en.wikipedia.org/wiki/Thread_(computer_science)>

When I say "standard model of threading", I am not talking about nuances of
call conventions to the underlying OS thread primitives. I am talking simply
about running multiple streams of instructions, bytecodes, or other units of
computation in parallel, within a single OS process.

~~~
haberman
That Wikipedia article defines threads in terms of operating systems. Only one
small part of that article concerns how threads are exposed to programming
languages.

You can't talk about running multiple streams of instructions or bytecodes in
parallel without talking about the nuances of how they share memory. Semantics
of a multithreaded memory model are a highly "opinionated" thing -- there are
lots of possible ways to define it, and the definition can have widespread
effects on efficiency, ease of programming, and the guarantees that the
runtime can provide. For example, an important aspect of a Python memory model
would be that no Python program can SEGV the interpreter due to a race
condition.

I recommend the following reading to get an appreciation for how much really
goes into a memory model and how far from "simple" or "standard" it is:

    
    
      http://en.wikipedia.org/wiki/Memory_model_(computing)
      http://en.wikipedia.org/wiki/Java_Memory_Model
      http://www.kernel.org/doc/Documentation/memory-barriers.txt
    

Python is a lot harder to define a good memory model for than say Java,
because in Python lists and dictionaries are primitive objects. If you say:

    
    
      x['A'] = 1
    

...that is a single operation that must not corrupt the dictionary, even if
multiple concurrent threads are mutating it. In practice, this means that you
need to either make every such mutation wrapped in a lock (which adds a lot of
locking overhead) or you need to use lock-free data structures (which are
still relatively experimental and architecture-specific).

~~~
dmbaggett
I agree that a so-called dynamic language like Python is at something of a
disadvantage because it must make atomicity guarantees that lower-level
languages like C need not.

I still don't think it's reasonable to conclude that typical programmers are
fine with their threads not really running in parallel, or that the GIL isn't
worth bothering to fix, even though fixing it would be hard. In my original
post yesterday, I pointed out that as the language footprint has grown,
Python's disadvantage in this respect has increased: it is much harder to
remove the GIL now than it was in, say, the 1.5 era when there actually was a
(problematic) GIL removal patch.

We've gotten way off track, but the original point I was trying to make was
that 1) the GIL really is a problem for not-purely-theoretical programs
written by competent developers, and 2) that the 2->3 transition, by
complicating the language and increasing the workload for the alternative
implementations, has made it less likely than ever that the GIL problem would
be resolved.

And, indeed, Nick explicitly confirmed this by saying the GIL is basically a
dead issue for the CPython devs. His post made many good points about the
merits of the 2->3 transition, and in particular pointed out some ways that 3
has reduced work for the alternative implementations, but I remain unconvinced
overall. And not out of ignorance or incompetence, as he implied.

~~~
haberman
I still think your position is unreasonable, because your inherent assumption
is that the GIL is a "problem" that needs a "fix." This terminology is
appropriate for a situation where the status quo could be improved without
giving up any of the benefits of the current implementation. But this is not
the case; removing the GIL in the way you advocate would add CPU and memory
overhead that everyone would pay, even in the single-threaded case. And this
is to say nothing of the practical problems of maintaining compatibility with
existing C extensions.

The GIL is not a bug, it's a threading model. You wish the threading model was
something else. You insist on your particular vision of an alternative
threading model without acknowledging its downsides. You make no indication
that you have actually considered or tried the alternative concurrency models
that CPython _does_ support, like multiprocessing, greenlets, or independent
processes. You make no objective arguments for why your desired threading
model is better than the ones that are currently available, except that you
could avoid changing your code. You accuse Python of failing to live up to
some accepted standard for what a "thread" should be, when in fact no such
standard exists, especially for high-level, dynamically-typed languages like
Python. If anything, newer languages are moving _away_ from shared-state
concurrency; see Erlang, Go, and Rust.

I don't think you have malicious intentions, but I urge you to reflect on what
you are demanding and whether it is reasonable. What may look to you like
"obvious" brokenness that demands an "obvious" fix is really a lot less clear-
cut than you seem to think it is. I feel for the Python developers who have to
deal with this complaining all the time.

~~~
comex
To Python-level code, Python's threading model is pretty much exactly the same
as that supported in all "fast" languages such as C and Java (even Go, ever
pragmatic, has locks). Given that Jython already allows true multithreading
and PyPy is trying to emulate it with STM, it's reasonable to see the GIL more
as an implementation bug that won't be fixed for practical reasons than as a
threading model... even if Python _also_ supports alternate threading models
that are perhaps better for most applications anyway (if strictly less
powerful).

~~~
haberman
> To Python-level code, Python's threading model is pretty much exactly the
> same as that supported in all "fast" languages such as C and Java

Yes, but Python also exposes higher-level operations like table manipulation
as language primitives.

> Given that Jython already allows true multithreading

That may be, but as I mentioned this has an inherent cost, both in CPU and in
memory. Therefore it is not a strict improvement over CPython, just a
different direction.

------
seunosewa
From the article: "The only downsides of this approach are that it means that
CPU bound Python code can’t scale to multiple cores within a single machine
using threads, and that IO operations can incur unexpected additional latency
in the presence of a CPU bound thread."

Response: The only downside? As a Python user suffering from JVM envy, I have
to say that that's a SERIOUS downside!

Here's why: (1) Python is slow. Almost any real life program in pure Python
will have CPU-bound components. (example: BBCode parsing on my forum).

(2) Most programs that need to scale won't need to scale beyond a single
machine (the most active web forum on my continent runs on a single quad core
server)

Therefore the need to be able to scale CPU-bound python programs to multiple
cores on is a very real need. Even though we accept that removing the GIL is
hard, let's not insult real-life Python users by suggesting that their needs
are not real.

~~~
hynek
The article doesn’t say multi-CPU scaling isn’t necessary. It says that
threads are usually the wrong answer anyway.

There are great process based ways to scale out – look no further than Erlang
to see it’s true.

~~~
seunosewa
I use Python for my day job. I've experimented extensively with Java and
Scala. I must say this: Threads are AWESOME. Threading is the most flexible
model of concurrency because you can can build programs that EFFICIENTLY
implement "alternative concurrency models" like message parsing and STM with
threads. Threading is supported natively by every OS. And fast.

Erlang is hyped as the ideal model for concurrency, but in practice is a niche
product that's primarily useful for programs that are almost pure IO - chat
servers, routing components like proxy servers and packet switches.

The Erlang model does NOT apply to python, anyway, since Python processes are
nothing like Erlang processes. Unlike Erlang processes, Python processes are
very heavyweight and message passing between them is costly.

~~~
hynek
If you’re using python, the performance gap between processes with message
passing and threads with locking is the last of your problems, believe me.

The big difference is that processes are much more robust and testable. The
cases where threads are really needed are fringe cases and – while it’s a pity
– Python doesn’t seem the right language if you don’t want to go the
Jython/IronPython way.

The bigger problem is that people are used to go for threads by default
although only few are able to write bug-free threaded code. It’s obsolete but
prevalent performance wisdom and the fact that threads were really popular in
the Java world.

~~~
ak217
This is a really short-sighted perspective.

Proper support of threading inherently allows more performance and flexibility
than multiprocessing. On top of them, you can build powerful, Pythonic
abstractions like concurrent.futures.ThreadPoolExecutor.map and STM, and on
top of those, even more powerful abstractions that help the developer avoid
concurrency bugs.

I'm really excited for PyPy. That is a project full of people who are not
afraid to quickly iterate on powerful ideas that can make Python a high
performance language that it deserves to be, instead of resorting to calling
MT programming a "fringe case" and ad hominem attacks.

~~~
hynek
I wish you’d read the article before making your accusations.

~~~
ak217
I have. There's an ad hominem attack in the middle of it. Otherwise it's a
great article.

I understand that everyone here is acting in good faith and wants Python to be
better, and the article otherwise contains lots of great information presented
in a reasonable manner. You bring up lots of good points too. But other
statements like the ones I mentioned are overly broad or brash.

~~~
hynek
I think it’s just attrition by explaining the GIL problem over and over again.
Nick is one of the major core developers and probably just fed up by the
topic. So this bit of snark is all the pay he’ll ever get for his work on
CPython.

~~~
scott_s
I really liked the article. But look at how much of this thread is spent
talking about that bit of snark. I think that mixing the snark in with all of
the rational reasons for Python 3's existence and not working on the GIL made
some people less receptive to rational arguments. In other words, I don't
think it was worth it.

