

Wield Python's super() like a Jedi - raymondh
http://rhettinger.wordpress.com/2011/05/26/super-considered-super/

======
guygurari
It seems to me that code that uses super() in these creative ways will be very
difficult to maintain. You need to understand the subtleties of the MRO just
to find out which method is being called. Pitty the poor Python programmer who
stumbles into such code without being aware that super() may not call the
class's direct base.

I suppose similar criticism can be leveled against many other dynamic
techniques, except that this one sacrifices more readability than I'm
comfortable with.

~~~
jessedhillon
super() supports arguments which allow you to specify which class's
implementation to lookup, which is an explicit way to resolve ambiguities.
Although well-designed multiple-inheritance should try to avoid those
ambiguities in the first place -- a class shouldn't inherit from multiple
bases with conflicting implementations of a method.

I think in a case like the one you describe, where the direct ancestor's
implementation is circumvented to access a different ancestor's
implementation, it should be done by calling

    
    
      super(Ancestor).someMethod()
    

That should improve understanding.

~~~
Peristarkawan
I disagree, that's extremely fragile. It might work for the particular class
you are implementing. However, it might cause _subclasses_ of that class to
break, since those subclasses will have a different MRO than the current
class, and you could end up skipping more of the MRO than you intended.

------
mrshoe
Personally I'm not in favor of enforcing such strict conventions, especially
in a dynamic language like Python. Also isn't this pattern he repeats
throughout:

    
    
        class Shape:
            def __init__(self, **kwds):
                self.shapename = kwds.pop('shapename')
                super().__init__(**kwds)
    

Better written like this?

    
    
        class Shape:
            def __init__(self, shapename=None, **kwds):
                self.shapename = shapename
                super().__init__(**kwds)
    

It seems like there are some things about Python's which the author doesn't
like, and he solves them by adding a lot of boiler plate to each of his
classes. It's probably a better idea to just embrace the Python way or use a
different language.

~~~
narm60
These two aren't equivalent, the first example requires the shapename keyword
arg, raising a KeyError when it wasn't specified

~~~
Peristarkawan
That's true. The second one might be better written as:

    
    
      def __init__(self, shapename, **kwds):
    

Or, in Python 3:

    
    
      def __init__(self, *, shapename, **kwds):
    

(making shapename a required keyword-only argument)

~~~
reinhardt

      def __init__(self, shapename, **kwds):

is _not_ better written; it allows shapename to be passed positionally which
makes cooperative inheritance fragile.

~~~
Peristarkawan
The version that I was rewriting:

    
    
      def __init__(self, shapename=None, **kwds):
    

also allows that.

------
Peristarkawan
It seems to me that the example of combining built-in dictionary classes is
naively optimistic. For starters, OrderedDict, as it happens, does not use
super! It calls the dict super-class methods directly. Since dict happens to
be the next class in the MRO, this doesn't really matter for the purpose of
this example, but I can envision a scenario where some plucky programmer
inherits from both OrderedCounter and some other dict subclass, and the result
doesn't work because OrderedDict does the wrong thing.

And OrderedDict isn't the only one. Maybe for some reason I would like to have
an OrderedCounter where all the counts default to 42. So I do this:

    
    
      class DefaultOrderedCounter(defaultdict, OrderedCounter):
          pass
      doc = DefaultOrderedCounter(lambda: 42)
      doc.update('abracadabra')
    

Which results in:

    
    
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "c:\python32\lib\collections.py", line 507, in update
          _count_elements(self, iterable)
        File "c:\python32\lib\collections.py", line 63, in __setitem__
          self.__map[key] = link = Link()
      AttributeError: 'DefaultOrderedCounter' object has no attribute '_OrderedDict__map'
    

Whoops! Apparently defaultdict doesn't use super either. Of course a better
way to do this would be to subclass DefaultOrderedCounter and just override
the __missing__ method by hand, but that's not the point.

The article goes into "How to Incorporate a Non-cooperative Class", which
basically says "wrap it up in a proxy class". But that's not really going to
work here, since the result would be two separate dicts, with the
defaultdictwrapper methods operating on one dict, and the other methods
operating on the other.

~~~
Peristarkawan
Okay, I give. What's the markup for posting code snippets on this site?

~~~
lmkg
Put two spaces at the beginning of a line for that line to be displayed
monospaced.

    
    
      Example

------
draebek
I didn't see it linked in the article nor mentioned here already but several
of the problems with `super` have been discussed in "Python's Super is nifty,
but you can't use it"[1] for quite some time.

[1]: <http://fuhm.net/super-harmful/>

~~~
mikeklaas
That article is about Python 2's super, which is a different beast. Lessons
were learnt in the design of Python 3's version.

~~~
Peristarkawan
Not really. The only thing that changed about super in Python 3 is that you
can call it with no arguments, which is just a short-hand for the way it was
already used in Python 2. Everything in that article applies equally to both
Python versions.

------
lachenmayer
Nice article, but one thing struck me when reading the code examples posted in
the article: What's the use of using __kwds in __init__?

    
    
        class Shape(Root):
            def __init__(self, **kwds):
                self.shapename = kwds.pop('shapename')
                super().__init__(**kwds)
    
        class ColoredShape(Shape):
            def __init__(self, **kwds):
                self.color = kwds.pop('color')
                super().__init__(**kwds)
    

I would personally just have written this somewhat similar to this:

    
    
        class Shape(Root):
            def __init__(self, shapename):
                self.shapename = shapename
                super().__init__()
    
        class ColoredShape(Shape):
            def __init__(self, shapename, color):
                self.shapename = shapename
                self.color = color
                super().__init__(shapename)
    

But then again I'm by no means what anyone could call a Python 'Jedi'. Can
anyone explain to me why using __kwds in this way would be better? Would it
not lead to having to look up the declaration of the class for the exact names
of arguments to pop() every time?

~~~
Peristarkawan
The reason for using keywords is that there is no guarantee that the next
class in the MRO after ColoredShape will be Shape. When the instance is a
ColoredShape, it will be, but if ColoredShape is subclassed with a more
complex inheritance structure, then something else could end up in between
them in the MRO.

Thus, ColoredShape.__init__ can't simply pass on shapename to the super
method, because that might not be the argument expected by the next __init__
method in the MRO. Instead, each method needs to gracefully accept arbitrary
arguments and then pass on whatever it received (minus what it consumed).

------
jamesli
Am I missing some points here?

It seems that the examples are tightly coupled and favors more on inheritance
than on composition.

------
oinksoft
A true Jedi favors composition.

~~~
uriel
One of the reasons I'm starting to love Go more than Python.

In Go something like super() is superfluous.

~~~
jessedhillon
The only reason why you don't need super in Go is because you cannot embed
multiple interfaces _except_ if their sets of methods are disjoint. This
eliminates super's job of allowing programmers to differentiate between base
classes with conflicting method implementations.

Without this, Go requires you to design your class hierarchy in a way that is
cleaner, but the super concept is still not superfluous -- because you do
sometimes need to reference an ancestor's implementation in Go.

You embed an interface and then explicitly reference the embedded interface by
name. The Go interface

    
    
      type ReaderWriter interface {
        Reader
        Writer
      }
    

is exactly like this Python

    
    
      class ReaderWriter(Reader, Writer):
        pass
    

To get at a particular superclass implementation, you pass the class to super,
like so:

    
    
      def read_and_write(self):
        super(Reader).read()
        super(Writer).write()
    

which in Go is:

    
    
      func (rw *ReaderWriter) ReadAndWrite() {
        rw.Reader.read()
        rw.Writer.write()
      }
    
    

So it's not superfluous -- there's an equivalent syntax to super which
involves referencing embedded interfaces.

~~~
Peristarkawan
Sigh. No, that is not at all how super works in Python. The equivalent code to
that Go function would be:

    
    
      def read_and_write(self):
          Reader.read(self)
          Writer.write(self)
    

That is, if you want to get at a particular superclass implementation, then
you just call it directly. super() is used when you're _not_ calling a
particular superclass, instead relying on Python to compute the correct "next
method" for you.

"super(Reader).read()", if you actually try it, will just result in an
AttributeError. This is because super(Reader) returns an _unbound_ super
object, not a bound super object that you can actually dispatch from. Why this
advanced usage might be useful is beyond the scope of this thread, but if
you're curious then just google for _python unbound super_ and hit the Feeling
Lucky button.

~~~
jessedhillon
Yeah you're right, I was mistaken. super uses the MRO of the first argument,
or inspects the stack if none is given.

BTW your sighing is not necessary.

~~~
Peristarkawan
I apologize. It was late, and my supply of patience for the day was exhausted.

------
fferen
The " __kwds" technique is actually something I came up with independently to
solve a different problem when I was using PyGame: ridiculously long argument
lists when constructing sprites. In this case it significantly increased the
flexibility and brevity of the code, although perhaps with a more complicated
inheritance tree it wouldn't be worth it.

Nice to see this pattern being acknowledged somewhere else; I thought I was
the only one that did that.

~~~
raymondh
It's a great technique. I'm sure your PyGame code benefitted in a big way :-)

------
silentbicycle
This is a particularly unnecessary title change.

------
veyron
how does the runtime performance of super() compare to the hardcoding method?
(from your example, using dict.__setitem__ as opposed to super().__setitem__ )

~~~
raymondh
In CPython, calling super() has about the same cost as calling any other
builtin like len() or int(). The MRO itself is precomputed and stored in
inst.__class__.__mro__.

Contrasting dict.__setitem__ with super().__setitem__, the latter adds one
function call. In addition, both forms require a builtin lookup and an
attribute lookup.

In PyPy, much of the overhead of builtin lookups, function calls, and
attribute lookups is automatically optimized away.

~~~
kingkilr
Unfortunately on PyPy super is less efficient than refering directly to the
parent classes method, however it's fixable (and will be fixed).

~~~
veyron
Do you have some sort of timing measure?

~~~
kingkilr
No, I haven't timed it myself, I just know how it's implemented and what code
we're generating internally.

Edit: To be clear I would _not_ advocate not using super() on account of this,
as I said it'll eventually be fixed.

------
lza
It's a super() good overview of super(). Nice to see a post that uses a real
example.

------
kalelias
tl;dr

Just looks like super/parent like in other languages. Did i miss something
from not reading this?

~~~
masklinn
> Did i miss something from not reading this?

Everything? Mostly the part where, Python being an MI language, a given class
A has multiple parents. And these parents are a sequence, not a set, which
define a Method Resolution Order, and that method resolution order usually
isn't breadth-first either, so Python's super is a graph walker. And because
it's explicit, any node in the graph can stop the walking.

This means permuting two parent classes can have significant impact on the
semantics of the graph walking, therefore the results of calling super.

~~~
kalelias
Thanks for the short explanation. I totally forget about python being MI.

------
exDM69
Ah, the "call super" antipattern revived.

~~~
__rkaup__
How's it an antipattern?

------
mclin
Have to deal with enough languages as it is. Don't want to further add to my
brain mess by learning Python 3 syntax just yet.

~~~
raymondh
super() was introduced in Python2.2 many years ago. Everything in the article
works with very old versions of Python.

Here's a link to the Python 2 version of the examples:
[http://code.activestate.com/recipes/577721-how-to-use-
super-...](http://code.activestate.com/recipes/577721-how-to-use-super-
effectively-python-27-version/)

~~~
bendotc
super() was introduced in CPython 2.2, but it changed in Python3:
<http://www.python.org/dev/peps/pep-3135/>. The link you posted uses super as
"super(cls, instance)", but the main article uses Python 3's super, which can
just be called as "super()", and it figures out the class and instance.

So no, super really has changed, and the syntax in the article does not work
with very old version of Python.

~~~
bretthoerner
> So no, super really has changed, and the syntax in the article does not work
> with very old version of Python.

Oh come on, he's saying if you do the _incredibly minor_ syntax adjustment
that all of the actual meat of the article still works in Python 2.

~~~
nicpottier
I upvoted the parent, because that syntax distinction is actually fairly
giant, always having to name the class and also changing that on renames etc,
is a PITA.

It is actually one of the things that annoys me most about Python, I guess
this is the first thing that actually tempts me with Python 3. Damn.

