
Python idioms I wish I'd learned earlier - signa11
http://prooffreaderplus.blogspot.com/2014/11/top-10-python-idioms-i-wished-id.html
======
ims
I think the example in #4 misses the point of using a Counter. He could have
done the very same for-loop business if mycounter was a defaultdict(int).

The nice thing about a Counter is that it will take a collection of things
and... count them:

    
    
        >>> from random import randrange
        >>> from collections import Counter
        >>> mycounter = Counter(randrange(10) for _ in range(100))
        >>> mycounter
        Counter({1: 15, 5: 14, 3: 11, 4: 11, 6: 11, 7: 11, 9: 8, 8: 7, 0: 6, 2: 6})
    

Docs:
[https://docs.python.org/2/library/collections.html#counter-o...](https://docs.python.org/2/library/collections.html#counter-
objects)

~~~
frindo
Python noob here. What does the _ mean in "...for _ in range(100))"?

~~~
danudey
To add to specifics:

_ in python is commonly used as a variable name for values you want to throw
away. For example, let's say you have a log file split on newlines, with
records like this:

    
    
        logline = "127.0.0.1 localhost.example.com GET /some/url 404 12413"
    

You want to get all the URLs that are 404s, but you don't care about who
requested them, etc. You could do this:

    
    
        _, _, _, url, returncode, _ = logline.split(' ')
    

There's no special behaviour for _ in this case; in fact, normally in the
interactive interpreter it's used to store the result of the last evaluated
line, like so:

    
    
        >>> SomeModel.objects.all()
        [SomeModel(…), SomeModel(…), SomeModel(…), …]
        >>> d = _
        >>> print d
        [SomeModel(…), SomeModel(…), SomeModel(…), …]
    

Which I think is basically the same behaviour; you run some code, you don't
assign it, so the interpreter dumps it into _ and goes on about its day.

------
dllthomas
_" Because I was so used to statically typed languages (where this idiom would
be ambiguous), it never occurred to me to put two operators in the same
expression. In many languages, 4 > 3 > 2 would return as False, because (4 >
3) would be evaluated as a boolean, and then True > 2 would be evaluated as
False."_

The second half of this is correct, but it has nothing to do with whether the
language is statically or dynamically typed. It's a tweak to the parser,
mostly.

~~~
azernik
It's not just a tweak to the parser, and it does have to do with the type
system, but you're right that it's not about static typing.

The issue is that there are languages (like C) where typing is static but
weak, so e.g. booleans are also integers and can have integer operations like
'>' applied to them. In other words, the problem is that in C True == 1 and 1
> 2 is a valid expression. In Python, which has strong(er) types, this
expression can have an unambiguous meaning if you want to add the feature to
your parser.

~~~
dllthomas
You can implement it entirely in the parser if you can avoid name capture - it
may or may not be implemented entirely as a tweak to the parser in practice,
but it's fundamentally a syntactic thing.

Your discussion of types here is all wrong - it's true that C treats booleans
as if they were integers, but _Python does, too_ :

    
    
        >>> (3 > 4) < 2
        True
        >>> 3 > 4 < 2
        False
        >>> 3 > (4 < 2)
        True
    

It has nothing to do with types.

~~~
myg204
I think the comparison is more readable as (here with the >>> of the python
shell):

    
    
        >>> 0 < x < 1
    

as opposed to

    
    
        >>> 1 > x > 0
    

Following the number line and placing x there is nicer IMHO. Other than that,
very nice trick.

~~~
dllthomas
In terms of coding style, I agree with you that following the number line is
usually going to be clearest. I think there might be some situations where
descending is better than ascending, but certainly both are radically better
than what I did above (low > high < lower).

As an example for what I was specifically trying to show here, though, that
doesn't let me distinguish things quite as clearly.

I believe that (a < x < b) gives the same value as at least one of (a < x) and
(x < b) for any value of x.

    
    
        x < a:
            a < x gives false
            a < x < b gives false
    
        a < x < b:
            everything gives true
    
        b < x:
            x < b gives false
            a < x < b gives false
    

So there's no way to get both of the parenthesized versions to disagree with
the unparenthesized version in a single example.

------
shackenberg
If you were underwhelmed by this blog post have a look at:

Transforming code into Beautiful, Idiomatic Python by Raymond Hettinger at
PyCon 2013

[https://speakerdeck.com/pyconslides/transforming-code-
into-b...](https://speakerdeck.com/pyconslides/transforming-code-into-
beautiful-idiomatic-python-by-raymond-hettinger-1) and
[https://www.youtube.com/watch?v=OSGv2VnC0go&noredirect=1](https://www.youtube.com/watch?v=OSGv2VnC0go&noredirect=1)

Topics include: 'looping' with iterators to avoid creating new lists,
dictionaries, named tuples and more

~~~
glitchdout
Sorry for the offtopic but why did you add `&noredirect=1` to the end of that
youtube url?

~~~
shackenberg
sorry, was only an accident.

------
__luca
Sincerely, Transforming Code into Beautiful, Idiomatic Python – by Raymond
Hettinger... [http://youtu.be/OSGv2VnC0go](http://youtu.be/OSGv2VnC0go)

~~~
scep12
I was lucky to watch this video while first learning the language. Every
beginner (coming from another language) should watch this to understand the
idioms of Python.

~~~
__luca
Ya, me too. It's also a very funny talk. :)

------
euphemize
One of my favorites:

    
    
        >>> print "* "* 50
    

to quickly print a separator on my terminal :)

Previous discussion on python idioms from 300 days ago:
[https://news.ycombinator.com/item?id=7151433](https://news.ycombinator.com/item?id=7151433)

~~~
Animats
That's cute, but the result of a bad design decision.

Python overloads "+" as concatenate for strings. This also applies to lists.
So

    
    
        [1,2,3] + [4,5,6]   yields  [1,2,3,4,5,6]
    

This is cute, but not what you want for numerical work.

Then, viewing multiplication as repeated addition, Python gives us

    
    
        [1,2,3]*4  yields [1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
    

This is rarely what was wanted.

Then there's numpy, which has its own array type, needed because Python's
array math is slow. Numpy arrays have different semantics - add and multiply
perform the usual numeric operations. You can mix numpy arrays and built-in
lists with "+" and "*". Mixed expressions are evaluated using numpy's
semantics. Since Python doesn't have type checking, it's possible to pass the
wrong kind of numeric array to a function and have it treated with the wrong
semantics.

Moral: when designing your language, using "+" for concatenation is tempting,
but generalizing that concept comes back to bite you.

~~~
wodenokoto
Then what would you have

    
    
        [1,2,'q',[1,('a',2)]] + 4 
    

yield? The reason why numpy lets you do math operation on each element in an
array is because you can safely assume that each element is a number. You can
assume absolutely nothing about the types of the elements in a list.

~~~
Animats
"TypeError: cannot add 'str' and 'int' objects."

Just because you can define semantics for nonsense doesn't mean you should.

~~~
dalke
I'll modify the example slightly to something which doesn't have a type error:

    
    
        [1,2,'q',[1,('a',2)]] * 4
    

With element-by-element operations, that would be

    
    
        [1 * 4,2 * 4,'q' * 4,[1,('a',2)] * 4]
    

giving

    
    
        [4, 8, 'qqqq', [1 * 4, ('a', 2) * 4]]
    

applying that again, assuming that tuple * scalar is also applied element-wise
gives:

    
    
        [4, 8, 'qqqq', [4, ('a' * 4, 2 * 4)]]
    

and ends up with

    
    
        [4, 8, 'qqqq', [4, ('aaaa', 8)]]
    

I can't think of any case where that's meaningful.

Also, what should this do:

    
    
        x = [1]
        x.append(x)
        print(x + x)
        print(x * 4)
    

? Currently these print:

    
    
        [1, [1, [...]], 1, [1, [...]]]
        [1, [1, [...]], 1, [1, [...]], 1, [1, [...]], 1, [1, [...]]]
    

because the print function knows how to handle recursive definitions. Do all
of the element-wise operations need to handle cyclical cases like this? I
think numpy can get away with not worrying about this precisely because, as
wodenokoto pointed out, it can assume a flat structure.

------
ghshephard
Wow - that's really, really great list.

In particular, #7 is something that I didn't even know existed, and I've been
hacking around for 2+ years.

Instead of:

    
    
       mdict={'gordon':10,'tim':20}
       >>> print mdict.get('gordon',0)
       10
       >>> print mdict.get('tim',0)
       20
       >>> print mdict.get('george',0)
       0
    

I've always done the much more verbose:

    
    
       class defaultdict(dict):
    
           def __init__(self, default=None):
               dict.__init__(self)
               self.default = default
    
           def __getitem__(self, key):
               try:
                   return dict.__getitem__(self, key)
               except KeyError:
                   return self.default
    
       mdict=defaultdict(0)
       mdict['gordon']=10
       mdict['tim']=20
       print mdict['gordon']
       10
       print mdict['tim']
       20
       print mdict['george']
       0
    

I'll be sure to make great use of the dictionary get method - I'm embarrassed
to admit how many thousands of times I could have used that, and didn't know
it existed.

~~~
bhy
Do you know there's also collections.defaultdict ?

~~~
ghshephard
I do now! I had known (and used) collections.OrderedDict, but had never used
defaultdict. Millions of keyerrors later...

I clearly am going to have to spend a few hours today grokking everything that
collections.* has to offer. Thanks very much.

------
rnhmjoj
This is something I do instead of writing a long if-else:

    
    
        opt = {0: do_a,
               1: do_b,
               3: do_b,
               4: do_c}
        opt[option]()

~~~
baddox
Do you consider that to be idiomatic? I've been out of touch with the a Python
community for a few years, but back then I wouldn't have considered that
remotely idiomatic, and if I was on a team writing software, I would have
argued that we shouldn't be writing code like that.

~~~
davidrusu
I've done this before, I think of it as a more powerful form of a switch
statement.

I'd love to hear why you think it's not ideal.

~~~
zo1
" _I 'd love to hear why you think it's not ideal._"

I'm not the poster you posed that question to. But for me, the one big
drawback of using that idiom is that the function signatures have to be
identical. So you either have to resort to args/kwargs, or you have an
additional intermediary method between the actual "guts" of what you're
calling, and the "switch" statement.

Or you live with the fact that you're passing unused/unnecessary parameters to
your functions.

~~~
myg204
Good point, and in that case, I would say this idiom is not so great really.
For functions with the same signature, it's a fine solution I think.

------
ckuehl
_> There is a solution: parentheses without commas. I don't know why this
works, but I'm glad it does._

It's worth mentioning that this is a somewhat controversial practice. Guido
has even discussed removing C-style string literal concatenation:

[http://lwn.net/Articles/551438/](http://lwn.net/Articles/551438/)

You may wish to consult your project's style guide and linter settings before
using it.

~~~
smoe
I personally don't like this style of using multiple strings. Makes radical
changes of the text cumbersome.

I think in most cases it's better to use triple quotes. And if the content of
these variables isn't exclusively shown in the shell, you should use
translation files anyway.

~~~
philsnow

        $ cat triple.py
        def foo():
            print """this is a triple quoted string
                     this is a continuation of a triple quoted string"""
    
        if __name__ == '__main__':
            foo()
        $ python triple.py
        this is a triple quoted string
                         this is a continuation of a triple quoted string
    

This is really warty. In bash you can mostly get around this with <<\- for
here documents (which removes leading indentation, so even if you're defining
a here doc at a place that itself is indented, you don't have to align the
text to the left column yourself). The man page in my version suggests it only
works for leading tabs, not spaces, though.

e.g.

    
    
        $ function usage() {
                cat <<-END
                        this is a here document that
                        spans multiple lines and is indented
                END
        }
        $ usage
        this is a here document that
        spans multiple lines and is indented
        $

~~~
Spiritus
Use `textwrap.dedent()`?

~~~
noinsight
I normally just do this for multiline strings:

    
    
        s = "\n".join(["one","two","three"])

~~~
philsnow
This happens at run time whereas IIRC the string literal is crammed pre-joined
into the ,pyc / .pyo file.

The performance hit of doing this kind of thing really adds up in a larger
app.

------
desdiv
I'm not much of a Python guy, but that chained comparison operator is sweet!

Sure, it's just syntax sugar, but it saves a lot of keystrokes, especially if
the variable name is long.

Is Python the only language with this feature?

~~~
kazinator
Common Lisp (and other dialects):

    
    
       (< a b c d)   ;; T if a < b < c < d
       (<= a b c d)  ;; T if a < b < c < d
    

Also:

    
    
       (lcm a b c d ...) ;; lowest common multiple
    
       (+) -> 0
       (+ a) -> a
       (+ a b)  -> a + b
       (+ a b c) -> (a + b) + c
    
       (*) -> 1
       (* a) -> a
       (* a b) -> a * b
       (* a b c) -> (a * b) * c
    

Is it just syntactic sugar? (< a b c) evaluates a, b and c only once, which
matters if they are expensive external function calls or have side effects.

    
    
       (and (< (foo) (bar))
            (< (bar) (baz)))
    

isn't the same as

    
    
       (< (foo) (bar) (baz))
    

By the way, this could be turned into a short-circuiting operator: more
semantic variation. Suppose < is allowed to control evaluation. Then an
expression like (< a b c d) could avoid evaluating the c and d terms, if the a
< b comparison fails.

~~~
hasenj
But can any lisp dialect do:

    
    
        a < b >= c
    
    ?

~~~
davidrusu
(< a (>= b c)) same number of operators but a few extra parens

~~~
kazinator
Here (>= b c) will return a boolean value, which you're then comparing to a.

~~~
kazinator
Which means it could be done, but the relational operators (any one of which
could be the leftmost constituent) have to all be defined as macros which
analyze the rest of the expression with alternative evaluation rules.

------
RyanMcGreal
I came across this when I was first learning Python and it has always
impressed me:

    
    
        from random import shuffle
        deck = ['%s of %s' % (number, suit) for number in '2 3 4 5 6 7 8 9 10 Jack Queen King Ace'.split(' ') for suit in 'Hearts Clubs Diamonds Spades'.split(' ')]
        shuffle(deck)

~~~
ajuc
I never liked how people in Python use stringWithSpaces.split instead of a
list. Just feels wrong somehow.

But I've seen it many times so it's probably pythonic

~~~
RyanMcGreal
As DaFranker points out, it's just easier to type than

    
    
        ('Hearts', 'Diamonds', 'Spades', 'Clubs')
    

and has less opportunity for typos and syntax errors. If I was concerned about
performance I would replace it with a tuple, but it was Good Enough for a
quick example.

~~~
ajuc
Sorry, I realised I sounded negative. Nothing against you, it's just my
private opinion.

It's not about performance, but about one more incidental step when reading
the code. Small cost, but it's there.

~~~
emmelaich
One reason I like it is that you're probably going to be reading in the
possible values from a stream or arg or config file anyway.

But you have a good point. YAGNI is YAGNI.

------
wodenokoto
Can someone direct me to a comparision of subprocess and os? I keep hearing
subprocess is better, but have not really read any explanation as to why or
when it is better.

(I'm glad I'm not the only one who was thrilled to discover enumerate()!)

~~~
willvarfar
The OS module interacts directly with the OS rather than abstracts it, so a
lot of the functions in it have the "may not be available on all platforms"
apology.

Subprocess uses OS under the hood but offers an abstraction that mostly works
on all platforms, e.g. the way that "communicate" is implemented on Windows
differs considerably from how its implemented on Unix.

------
leephillips
I was grateful for the example of multilined strings, mysterious as it is. The
lack of any way to do this has been an annoyance of mine for quite some time.

------
tomp
Some comments:

1\. Am I the only one that really _loves_ that `print` is a statement and not
a function? Call me lazy, but I don't mind _not_ having to type additional
parentheses.

5\. Dict comprehensions can be dangerous, as keys that appear twice will be
silently overridden:

    
    
      elements = [('a', 1), ('b', 2), ('a', 3)]
      {key: value for key, value in elements} == {'a': 3, 'b': 2}
      # same happens with the dict() constructor
      dict(elements) == {'a': 3, 'b': 2}
    

7\. I see

    
    
      D.get(key, None)
    

way too often.

8\. Unpacking works in many situations, basically whenever a new variable is
introduced.

    
    
      for i, el in enumerate(['a', 'b']):
        print i, el
    
      {key: value for (key, value) in [('a', 1), ('b', 2), ('a', 3)]}
    
      map(lambda (x, y): x + y, [(1, 2), (5, -1)])
    

Note: the last example (`lambda`) requires parentheses in `(x, y)`, as `lambda
x, y:` would declare a two-argument function, whereas `lambda (x, y):` is a
one-argument function, that expects the argument to be a 2-tuple.

~~~
philh
On the subject of "call me lazy", I really like the % syntax for string
interpolation. I'd like perl-style interpolation even more. "".format() is
going in completely the wrong direction for me. (I don't think % is being
removed, but I think it's discouraged.)

> lambda (x, y): x + y

This syntax is removed in python 3:
[http://legacy.python.org/dev/peps/pep-3113/](http://legacy.python.org/dev/peps/pep-3113/)

~~~
0942v8653
I'm still not sure why that was deprecated. It's much cleaner. I still use it…
:/

~~~
GFK_of_xmaspast
Ideology.

------
rectangletangle
I'm a fan of Python's conditional expressions.

    
    
        foo = bar if qux is None else baz
    

They're particularly interesting when combined with comprehensions.

    
    
        ['a' if i % 2 == 0 else 'b' for i in range(10)]
    

Though this particular example can be expressed much more concisely.

    
    
        ['a', 'b'] * 5

~~~
plikan13
Before the conditional expression was introduced to Python I sometimes wrote
things like

    
    
        foo = [baz, bar][qux is None]

------
polemic
I work with python full time, and the last (#10 string chaining) is one of the
few times the syntax had caused me grief, due to missed commas in what were
supposed to be tuples of strings. The chaining rules are one of the few
sources of apparent ambiguity in the syntax, especially when you include the
multiline versions.

------
jemfinch
Most of these idioms actually make me sad.

When I first started using Python around 1999, it didn't even have list
comprehensions. Code was extremely consistent across projects and programmers
because there really was only _one_ way to do things. It was refreshing,
especially compared to Perl. It was _radical_ simplicity.

Over the decade and a half since then, the Python maintainers have lost sight
of the language's original elegance, and instead have pursued syntactical
performance optimizations and sugar. It turns out that Python has been
following the very same trail blazed by C++ and Perl, just a few years behind.

(At this point Python (especially with the 2 vs. 3 debacle) has become so
complex, so rife with multiple ways to do even simple things that for a small
increase in complexity, I can just use C++ and solve bigger problems faster.)

~~~
stdbrouw
Are we reading the same article, though?

It's certainly up for debate whether named tuples and enums, various kinds of
metaprogramming and decorators might be making the language more complex for
fairly little gain... but this article talks about the `enumerate` function,
about string formatting and dictionary comprehensions. Simple, straightforward
stuff with no downsides.

------
_navaneethan
There is an sweet in India called [0] _phirni_ when you start eating from
outer layer to inner layer, you will feel like _walking in heaven_.Now you are
in outer layer.I hope you enter to the inner level and feel the python still.
:)

[0] [http://www.wikihow.com/Make-Phirni-%28a-Rice-and-Milk-
Dish%2...](http://www.wikihow.com/Make-Phirni-%28a-Rice-and-Milk-Dish%29)

------
tinkerdol
_" Missing from this list are some idioms such as list comprehensions and
lambda functions, which are very Pythonesque and very efficient and very cool,
but also very difficult to miss because they're mentioned on StackOverflow
every other answer!"_

Can anyone link to good explanations of list comprehensions and lambda
functions?

~~~
RyanMcGreal
This seems like an easy intro to list comprehensions:

[http://www.pythonforbeginners.com/basics/list-
comprehensions...](http://www.pythonforbeginners.com/basics/list-
comprehensions-in-python)

------
retroencabulato
Nice list, but I was confused by the arguments to the dict .get() example
until I looked up the definition.

------
TheLoneWolfling
I wish there was an interval set in Python's builtins.

I also wish that ranges were an actual proper set implementation - so you
could, for example, take intersection and union of ranges.

And I wish that Python had an explicit concatenation operator.

~~~
Spiritus
You mean like the built in `set` object?
[https://docs.python.org/2/library/stdtypes.html#set](https://docs.python.org/2/library/stdtypes.html#set)

~~~
omaranto
I think he or she meant a specialized set data structure that stores sets of
numbers which can be written as finite unions of interval. Typically besides
set operations you'd also want (1) that only endpoints of the intervals are
stored, so that the structure is compactly represented in memory, (2) to be
able to recover the canonical representation of the set as a sorted union of
disjoint intervals.

~~~
TheLoneWolfling
This is exactly what I meant, thank you.

------
panzi
Oh, wow. I didn't know the dict comprehensions. Since when do they exist? I
always used:

    
    
        d = dict((key(x), value(x)) for x in xs)

~~~
medecau
[http://legacy.python.org/dev/peps/pep-0274/](http://legacy.python.org/dev/peps/pep-0274/)

I'm not entirely sure how to interpret the PEP header. It dates back to 2001
and was updated in 2012. It's probably in python since 2.3 but maybe
2.7(2010)/3.0(2008).

------
sherjilozair
Is there any such collection of advanced Python patterns, aimed at Python
programmers with more than 2-3 years of experience?

~~~
tomkinstinch
I'd love other answers to the same question. Here are a few I've come across
in the past:

[http://www.rafekettler.com/magicmethods.html](http://www.rafekettler.com/magicmethods.html)

[http://sahandsaba.com/thirty-python-language-features-and-
tr...](http://sahandsaba.com/thirty-python-language-features-and-tricks-you-
may-not-know.html)

[http://www.siafoo.net/article/52](http://www.siafoo.net/article/52)

------
flares
haha.. in #1, the easter egg "not a chance" :) :)

