
Python performance is a fool's errand - ntoshev
http://whoosh.ca/blog/fools_errand
======
Goladus
Code optimized for performance is usually optimized based on environmental
constraints. The more you optimize, the more the code will look like a
description of instructions to manipulate that environment, rather than a
succinct description of your solution. There is an inherent conflict. C++
doesn't solve this problem, it's just one of the possible compromises.

With Python and similar high-level languages, there are a few layers of
"environment" that you have to tailor your instructions for. You have to
understand how Python manipulates the machine resources and then you have to
understand how to manipulate Python to manipulate those resources. The result
is code optimized for python on a particular machine, which might not look
anything like what you'd expect being familiar with one or the other but not
both.

~~~
stcredzero
The need for optimization is just another example of "leaky abstractions."
Ideally, you'd just let the JIT take care of it for you, but it's not able to
do everything. There are Smalltalk parsers that use lots of if-then
equivalents, because these are faster because of bytecode level optimizations,
even though using Dictionary lookup would be clearer and shorter. That's an
example of a leaky abstraction in the compiler->bytecode->JIT implementation
of the language itself. What's worse, is that the structure of the code -- why
it's written in that way -- is highly cryptic. You can't write a bunch of if-
then in a way that reveals that they are faster. The Mercurial project's
implementation is optimized to ensure high locality of reference for data on
disk. You can't make stuff like that intention revealing. You must resort to a
comment.

I think this reveals a weakness in languages in general. The issue of
performance is mostly _orthogonal_ to most of the business problems you're
solving. The fact that optimization issues get embroiled in code -- this is
something we should try to eliminate.

"Correct then fast" is just a compromise. You minimize the amount of
optimization done to preserve as much readability as possible. We should have
languages that allow us to optimize without mucking up code. (Is it possible
with Lisp Macros?)

------
noss
I think this rule about clarity-of-code > micro-performance can be generalized
to programming as a whole.

~~~
ntoshev
In the case of a full text search engine, you really _need_ that performance.

I'm not sure clarity and performance of code are inherently conflicting
requirements; in any case the author wouldn't have this gap in performance
between the two versions if he used C++.

~~~
tdoggette
Which is why you don't use Python where performance is one of the biggest
priorities. I'm a big fan of Python, but it's not for everything.

------
seunosewa
There is a perfectly adequate solution to the specific problem that inspired
the above post. It's in the second comment. The author simply didn't know
enough about Python at that point.

