
Static types in Python - tabbott
http://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/
======
CJefferson
I just tried taking the article's advice, and running mypy on some of my code
(completely unannotated) to see what came up. Some interesting things have,
already made 3 patches.

One comment to the article author: Tell people how to install mypy. I started
by trying 'pip install mypy', then wondered why I didn't end up with a mypy
executable (you need pip3 install mypy-lang, mypy is an unrelated package).

~~~
tabbott
Thanks for pointing that out, fixed!

------
clusmore
Honestly I haven't looked into MyPy a huge amount, and haven't used it yet.
But from reading the docs on it I feel like it's taking the wrong approach. In
my opinion, Python is all about duck typing, so having a static type checker
that seems to focus on concrete types feels wrong to me.

Consider the first example,

    
    
      def sum_and_stringify(nums: List[int]) -> str:
          return str(sum(nums))
    

The type here is way more specific than it needs to be. Not only will this
work on any sequence type (anything that supports __iter__), the element type
only needs to support __add__ and __radd__ (to work against the implicit seed
value of sum).

I really feel like in order to get the best of duck typing, you need a good
syntax for describing types by _usage_ , not by explicit name. Please let me
know if I'm missing something, and this is actually more expressive than the
documentation suggests.

~~~
dbcurtis
Well, I like to put that another way -- I think Python is all about protocols
(in the Python sense of the word protocol). I think that is what most people
mean when they say Python is all about duck typing, and it communicates the
concept more precisely.

The concept of 'type' is overloaded with too many meanings these days. It
started out as a what-storage-to-allocate and which-instruction-to-generate
(int v float) concept (FORTRAN I, variables starting with I..N are ints,
others real). Deep class hierarchies muddied the waters by overlaying the idea
of whether or not a class implements a particular property or method, and how
to locate the property or method that you want.

Python shows us that types and protocols/interfaces are two very different
thing. In Python, asking 'isinstance()' is usually a bad idea unless you are
doing something very low level, low level as in next stop is a C extension.
The missing predicate in Python is 'hasprotocol()'. Abstract base classes can
be used to accomplish that: isinstance(foo, SomeABC), but I think the intent
could be more transparent with different syntax.

~~~
dllthomas
> Python shows us that types and protocols/interfaces are two very different
> thing. In Python, asking 'isinstance()' is usually a bad idea

Interesting perspective. This would seem to be pretty well described by
parametric polymorphism with constraints (Haskell-style polymorphism with
typeclasses). In that context, too, we are often well served staying as
generic as possible, and asking at runtime about the type of something is a
bit of a code smell (and only possible where you have a Typeable constraint
available).

~~~
clusmore
Yes. Haskell's type deduction is exactly what I was thinking of as I read the
article. You write a function using what you need and it deducts the
requirements based on usage, giving you the most general type. I think this is
actually very close to compile-time duck-typing. Actually I think if you took
the class declarations out of Haskell and removed the instance lines (but left
the subsequent implementations), and had the compiler deduce typeclasses for
you, you'd get something that felt very much like compile-time duck-typing.

~~~
int_19h
If you want an even closer example, it would be OCaml with its structurally
typed object system with inferred types. For example:

    
    
        # let f obj x y = (obj#m x)#n y;;
        val f : < m : 'a -> < n : 'b -> 'c; .. >; .. > -> 'a -> 'b -> 'c = <fun>
    

Here I defined a function taking 3 arguments, called method m on obj, passing
x as argument, then called method n on the result of that, passing y as
argument.

Note the _inferred_ type signature for this: the first argument is of type <m
: 'a -> <n : 'b -> 'c; ..>; ..> \- i.e. any object that has a method named m
(and possibly some other unspecified members - that's what ".." means), with
said method m having a signature that allows us to pass some 'a, and returning
another object of type <n : 'b -> 'c; ..> \- i.e. something with a method
named n that accepts 'b and returns 'c. 'a and 'b' are then used as types of x
and y, respectively, and 'c is the the result of f.

So effectively, this is full-fledged duck typing in the vein of Python
objects, but captured on type system level.

~~~
clusmore
>Note the inferred type signature for this Wow, that's really powerful. Thanks
for showing me this!

------
Roboprog
I am glad to see that this is optional. We need languages that mix
compile/parse time (static) and run time (dynamic) types, to mix what fits
best to each situation.

Of course type checking can catch _some_ problems. The more extremist elements
of the static types proponents seldom seem to ask the _cost_ of adding types,
and making some otherwise generic/overloaded operations harder to implement
(e.g. - excess clutter in some "wordy" type systems, particularly in the C
lineage where declared identifier names follow the other verbiage). We need
more tools that allow us to easily nail down fixed interfaces to avoid errors,
but still allow flexible / dynamic / metaprogramming type code when needed,
without having to jump through hoops in either case.

Don't be a static typing extremist! (or throw the baby out with the bath-
water, either, I guess)

~~~
wyager
A sufficiently good static type system won't make generic operations harder to
implement. It can even make them easier to implement in terms of guiding you
towards good generalization techniques.

You can also do full Python-style "duck typing" statically. It's called
structural typing, with native support from languages like typescript,
purescript, etc. It's used a lot in web dev because it works well with
Javascript's object model.

~~~
duaneb
> A sufficiently good static type system won't make generic operations harder
> to implement.

This is true for the JVM definition of 'generics', but it doesn't mean that
arbitrary operations on a collection of types can be easily expressed with a
given (static) type system. Run time type information + a dynamic type can
ameliorate some of this, but that's shoehorning optional types into a static
type system, and I don't think that's quite what you meant.

Think about a generic operation to diff two data structures: you need access
to the members of the struct in a generic way. I don't think this is _easily_
solved with static types.

~~~
wyager
Can you give a specific example where you think you need dynamic types? The
diff operation you described sounds like it could be made generic pretty
easily, but I may be misunderstanding the goal.

~~~
duaneb
> Can you give a specific example where you think you need dynamic types? The
> diff operation you described sounds like it could be made generic pretty
> easily.

How do you figure? This would require macros or runtime type introspection.

A specific example might be "serialize this json type without having to
annotate the fields with how to serialize them".

~~~
wyager
> This would require macros or runtime type introspection.

Not if you structure your data using an appropriate data structure, like a
map. After all, this is all structs in dynamic languages are.

> A specific example might be "serialize this json type without having to
> annotate the fields with how to serialize them".

Aeson does this safely and statically using Haskell's powerful static
Generics.

    
    
        data Foo = Bar Int String deriving (Generic)
        instance FromJSON Foo
        instance ToJSON Foo
    

You can get named fields like

    
    
        data Foo = Bar {baz :: Int, qux :: String}

~~~
duaneb
Most static typing systems have no parallel to 'deriving', which is to my
understanding compile-time computation (aka a macro).

~~~
dllthomas
"deriving" is compile time; "Generic" is runtime inspection of the shape of a
value (in terms of constructor applications). Something (at least vaguely)
similar to Generic exists in many statically typed languages - including
(IIRC, IIUC) Java and Go.

~~~
duaneb
> Something (at least vaguely) similar to Generic exists in many statically
> typed languages - including (IIRC, IIUC) Java and Go.

Sure—but it's in spite of static typing, not because of it. You certainly give
up any guarantees of avoiding runtime errors by detecting typing issues at
compile time.

~~~
dllthomas
Sure, it's orthogonal to static typing (I wouldn't quite say "in spite of"). I
wasn't trying to weigh in on the general topic, just to clarify what was
happening where.

------
krylon
I still think the way Common Lisp handled typing was very convenient: Typing
was basically dynamic, but one could use static typing in specific places, and
one could even tell the compiler if this was for speed or for safety.

I like the idea of a hybrid type system where statically typed and dynamically
typed parts coexist, OTOH, I can imagine it being difficult to implement.

~~~
junke
Agree.

However:

    
    
        s/was/is/g
        s/could/can/g
    

Premature burial is a little bit scary, I take it you are doing this because
Halloween is near.

~~~
Roboprog
Languages never die. But companies stop making new projects with them - for
some number of 9s percent of companies. E.g. - FOOGOLTRAN is a five-nine dead
language, if 99.999% of new projects don't use it.

~~~
Roboprog
I guess as long as a language is used in at least 11% of projects, it can
avoid being "one-nine" dead :-)

------
Animats
Optional static typing is fine. But this approach to it is not good. The
_correctness_ of the type information is optional. PEP 484: "While these
annotations are available at runtime through the usual __annotations__
attribute, _no type checking happens at runtime._ " (Italics in original)
That's just not right.

If you take that seriously, compilers can't use type information to optimize.
It feels like Von Rossum is making life hard for the PyPy compiler project
again. If they had hard type information, they could do serious optimizations.
Guido's CPython still uses a CObject for everything internally, so it doesn't
benefit much from this.

The direction that everybody else seems to be taking is that function
parameters, structure fields, and globals are typed, but local types are
inferred as much as possible. C++ (with "auto"), Go, and Rust all take that
route. Type information on function parameters and structure fields is an
important item of documentation; if the language doesn't support that, you
need to put it in separate documentation or comments. But writing elaborate
type declarations for local variables is no fun, and the compiler can usually
infer that info anyway.

~~~
ceor4
Frankly, each of your three paragraphs seems unrelated to each other, so I'm
still not sure what your argument is.

1) Not type checking at runtime is exactly what people mean by static typing,
because the definition of static typing means the checks are done before
execution.

2) This does not make it harder to optimize, on the contrary if they generated
type assertions it would considerable slow things down on the boundary of
typed and untyped code.

3) C++, Go, Rust, Haskell they are fully statically typed, the only difference
is they support varying levels type-inference which means you don't need to
"type out the type" but it's really orthogonal to the whole
optional/static/dynamic typing issue.

~~~
catnaroek
C++ and Go don't have type inference. All they do is set the type of new
variables to whatever they're first initialized to. Type inference means
figuring out the types of variables from _how they 're used_. The litmus test
for type inference is inferring function argument types.

~~~
ceor4
I'm sorry, but that's not the definition of type inference.

"Type inference refers to the automatic deduction of the data type of an
expression in a programming language."

[https://en.wikipedia.org/wiki/Type_inference](https://en.wikipedia.org/wiki/Type_inference)

Which you'll notice C++ and Go, are listed as languages with type inference.

But you're right that they don't have as nearly as strong type inference as
rust, which doesn't have as strong as haskell etc.

~~~
steveklabnik
Rust and Haskell have very similar type inference. The only major difference
is that Rust doesn't infer type signatures, but that's a design decision, not
a power issue.

------
scrollaway
> _For the many programmers who work in large Python 2 codebases, the even
> more exciting news is that mypy has full support for type-checking Python 2
> programs, scales to large Python codebases, and can substantially simplify
> the upgrade to Python 3._

Woah! Excellent news. Really cool that they found a way to use this
opportunity to bolster Python 3 migration.

------
tabbott
Original author here; I'm happy to answer any questions about the mypy
experience!

~~~
thristian
Does Sphinx (or any other automated-documentation tool) read the MyPy
annotations, or would I have to type them in twice, in different syntaxes, if
I wanted types in my docs + linting?

Or does type documentation not matter that much, once it's being enforced by a
tool?

~~~
RubyPinch
there is at least this, others might also exist
[https://pypi.python.org/pypi/sphinx-autodoc-
annotation](https://pypi.python.org/pypi/sphinx-autodoc-annotation)

------
StavrosK
I am extremely excited about this. I already verify parts of my applications,
but better support (especially in libraries) will make this that much more
useful. It's an essential part of the checking pipeline (with pyflakes, pep8,
McCabe, etc).

------
aikah
Started to use python recently, coming from Go, I found:

\- the official docs are useless, return types are often not documented (wtf
?) and docs aren't structured and clean. Python documentation is a giant mess.

\- mypy is a god sent I used it from day one to document my Python code, but
it needs better integration with python doc tools, 3rd party linters and co.

I'm ok with optional type checking. A team can make it mandatory during
continuous integration while not breaking legacy code, but I hope mypy will be
directly integrated to Python in the future and not a 3rd party tool.

~~~
ubernostrum
Could you give some examples of functions in the standard library (or built-
ins) which don't document their return types?

~~~
hawski
I sometimes have to write some python script. The documentation documents
return (and expected) types, but in a manner that is a bit alien to C or C++
programmers. It is more talky. It is probably in dynamic nature of python.

Instead of a list of possible returns with some kind of prototypes you may be
left with description: "The returned object is always a file-like object whose
file attribute is the underlying true file object.".

See for example documentation for tempfile.NamedTemporaryFile [1]. This time
possible returns are not in the separate paragraph as is in many other cases.
You can't click anything expect TemporaryFile(), so you have to search for
proper reference. The information is there, but instead of quick glance you
have to parse and search more text.

[1]
[https://docs.python.org/3.6/library/tempfile.html](https://docs.python.org/3.6/library/tempfile.html)

------
singularity2001
@OP(tabbott), you can add to "Benefits of using type annotations":

Improves IDE integration, especially IntelliJ PyCharm's jump-to-source

------
wyager
> It has a far lower false positive rate than the commercial static analyzers
> I’ve used!

Hoo boy. That's not a good thing. A consistent type system doesn't have "false
positives". Reading between the lines, this means "the type checker is
incomplete and allows incorrect types."

What is the story here with constraints, polymorphism, etc?

------
jbritton
I wish PEP 484 put more thought into backward compatibility with Python2.

Currently you need to import things like Callable, Tuple and these don't exist
in Python2. It is especially odd since I am using MyPy comment syntax, so MyPy
needs import statements even when all types are inside comments.

I like the way Haskell puts type information above the definition.

~~~
price
In Python 2 and also in Python 3.4 and earlier, you get Callable, Tuple, and
the rest from the `typing` module on PyPI:
[https://pypi.python.org/pypi/typing](https://pypi.python.org/pypi/typing)

This is a direct backport of the same `typing` module that's in the stdlib
starting in Python 3.5.

(I'm one of the mypy core developers and work at Dropbox. We're using PEP 484
and mypy on a lot of Python 2 code!)

------
weberc2
Why doesn't the import cycle trick work with the Python 3 syntax? This doc
seems to suggest that it should:
[http://mypy.readthedocs.io/en/latest/common_issues.html#impo...](http://mypy.readthedocs.io/en/latest/common_issues.html#import-
cycles)

~~~
ddfisher
It should work with the Python 3 syntax if you quote your type annotation.
I.e.

    
    
      def f() -> "Bar": ...
    

instead of

    
    
      def f() -> Bar: ...
    

(This is normally used for forward references.) This makes it slightly less
nice, but not a terrible workaround overall.

------
baq
i've been skeptical about mypy's usefulness. i'm absolutely thrilled to be
proven wrong here. thanks for blazing the trail.

------
meshko
Here's what I don't understand -- every discussion about static typing is full
of people saying static typing is so good and useful. Everyone i talked to,
everyone i work with, we all prefer static typing. And yet... Javascript and
all that.

~~~
imtringued
Javascript runs in the browser natively. It's primary advantage is that you
don't need to install anything other than the browser. The reasoning behind
node.js is that a lot of front end engineers no longer need to learn and use a
seperate language for the server and maybe even can reuse libraries between
the server and browser.

Javascript is the bash of the internet. It's not that great but it does have a
"killerapp".

------
mixmastamyk
Nice, how come some of the type names are capitalized? When the builtins are
lower-case? e.g.: List compared to list()

~~~
ddfisher
Good question! There are separate capitalized types for List, Dict, etc
provided as part of the `typing` module in order to let you specify element
types. Types are all normal Python expressions, so if you wrote `list[int]`
you'd get a runtime error, because `type` objects (i.e. the `list` class)
aren't subscriptable.

------
fuzzythinker
Don't see c/c++ target in the roadmap, which seems like an ideal use case for
typed python besides correctness. Thoughts?

~~~
ddorian43
What ? You mean like export to c/c++ code? Isn't python too dynamic for that ?

~~~
fuzzythinker
I'll say python isn't any more dynamic than ruby and groovy, and they have
fairly stable and usable static compile target release/ports [1]. Python has a
few [2] attempts, but none of them is close to stable or really useful.

[1] [https://crystal-lang.org/](https://crystal-lang.org/)

[http://groovy-lang.org/releasenotes/groovy-2.0.html](http://groovy-
lang.org/releasenotes/groovy-2.0.html)

[2]
[https://github.com/shedskin/shedskin](https://github.com/shedskin/shedskin)

[https://github.com/codeblazer-io/blazescript](https://github.com/codeblazer-
io/blazescript)

[https://github.com/serge-sans-paille/pythran](https://github.com/serge-sans-
paille/pythran)

~~~
ddfisher
Don't forget Cython [1]! My understanding is that it's quite useable (though
it unfortunately uses a different static type system/syntax than mypy).

[1] [http://cython.org/](http://cython.org/)

~~~
fuzzythinker
Didn't forget it, just didn't include it since to me, it's more like a totally
different (and not pretty) language that tries to look like python. I should
however include Nuitka [1] since it is fairly usable, it's just the generated
version isn't much faster than its interpreted python version.

[1] [http://nuitka.net/](http://nuitka.net/)

~~~
Macha
Why does the same not apply to Crystal then?

~~~
fuzzythinker
Guess you can say Crystal is a totally different language as well. It's just
to me, Crystal still has the same feel as ruby and Cython doesn't. Again, it's
just my opinion, I won't argue if you feel otherwise.

------
wyldfire
Type annotations and mypy are a great idea, sure. But IMO the low hanging
fruit is from static analysis tools like pyflakes, prospector, bandit. Most
projects don't even capitalize on these and your time is better spent
remediating the findings from these and/or carving out the exceptions where
necessary.

------
sumitgt
This sort of reminds me of how TypeScript got started. Bracing to hear about
TypePy in coming days :P

~~~
zo1
And what exactly do you think they would add to Python that is not already
there?

------
agentgt
Does PEP 484 plan to allow runtime reification of types (ie type information
during runtime via reflection aka Java/C#)? I made a cursory check of it and
it is unclear but I would assume no.

I know some may disagree but I have found runtime reification of types fairly
useful in the Java world (partly because Java does not have good code
generators like OCaml/Haskell and partly because Java is not that expressive).

------
lyschoening
Mypy is great, but it still has some ways to go. For instance, the type
checker is thrown off by descriptors (__get__ and __set__ are ignored).
Further, the typing module needs some work. The types in the module are
sufficient for annotating simple programs, but the more "meta" the programming
becomes the more walls will pop up.

------
leshow
IMO adding static types is good step but it sort of misses the point of modern
type systems.

The good thing about types is not that you can just annotate (or not with good
inference not) and validate that some type is what you expect it to be. It's
that with a proper type system you can develop better abstractions.

------
vram22
Interesting. I had blogged about mypy here, a few years ago:

[https://jugad2.blogspot.in/2012/12/mypy-python-variant-
with-...](https://jugad2.blogspot.in/2012/12/mypy-python-variant-with-goal-
of.html)

but had not tracked it much after that. Must check it out again.

------
Dowwie
Can someone point me to a live, public instance of Zulip that I can use to
experience a real demo?

~~~
lima
[https://zulip.tabbott.net/](https://zulip.tabbott.net/)

It's amazing. Zulip is Slack done right. Their threading/light-weight topic
approach is second to none.

------
foxhop
The python 2.x "comment" version should have been parsed from the docstring...

------
whalesalad
It bothers me when the constructs of the language are not used for things like
this. Example, the syntax for a list of ints is `List[int]` as opposed to
`list(int)` where list is an actual fn in python and so it makes more sense.
Obviously list(int) has far different meaning, but List is a totally foreign
thing in Python. I think it makes sense for a DSL to remain as close to the
parent as possible.

Another example of this is most of the prismatic/schema-esque frameworks for
python to do validation on datastructures. Instead of saying foo must be {
"name": "String" } (where there is a framework-specific meaning behind the
literal word "String") we should take advantage of the built-in types and say
{"name": basestring}.

End rant.

------
RodericDay

        sum_and_stringify(nums: List[int]) -> str:
    

This looks so bad. I think Python is making a big mistake with this.

The commented version is much more acceptable.

~~~
zanny
Its basically the same syntax as Rust, which I honestly like. I mean, I didn't
like the syntax to begin with (I have always been a stickler for how name:
type introduces a superfluous colon over type name syntax) and using two
glyphs for returns is horrible, but since several languages are now doing
this, if you are _going_ to do it, you might as well keep it consistent.

~~~
RodericDay
ddfisher up there is already talking about stuff like

\----

It should work with the Python 3 syntax if you quote your type annotation.
I.e.

    
    
      def f() -> "Bar": ...
    

instead of

    
    
      def f() -> Bar: ...
    

\---

Yeah, you need to write `from typing import List` to get the generic List
class that you can parameterize with an item type. As you pointed out, you can
use the builtin lowercase-l list without importing anything, but that doesn't
allow you to specify the item type.

\---

imo, this is brutal. I'll just wait and see whether I'm proven right or wrong,
but the extremely poor aesthetics of the whole thing bode badly.

------
singularity2001
Sorry for posting this issue here:

python3 type_annotations.py

    
    
        def sum_and_stringify(nums: List[int]) -> str:  

NameError: name 'List' is not defined

EDIT: Solved: Either just list or List[int] with import:

from typing import List

Thanks for all the help guys! By the way I took the example from the mypy
page. A classical case of 'critical line missing from example code' (as
perfectioned by Bjarne Stroustrup back when)

~~~
ddorian43
I think you have to import that.

~~~
singularity2001
import List??

At least this works: def sum_and_stringify(nums: list) -> str:

~~~
dcre
The built-in constructor is `list`, not `List`.

~~~
singularity2001
Thanks for all the help guys! by the way I took the example from the mypy page

    
    
        def sum_and_stringify(nums: list[int]) -> str:  

TypeError: 'type' object is not subscriptable

so either just list or List[int] with import, got it.

A classical case of 'critical line missing from example code' (as perfectioned
by Bjarne Stroustrup back when)

------
vegabook
...and so we risk shuffling even further away from what made Python great.
Simple pragmatism. Now in addition to unicode hair tearout, xrange forcefeed,
dict access strait jacket, print function dogmatism, and extraneous unused
async library which was already implemented 14x elsewhere, we are moving
towards strictly typed Python?? If you want all this stuff, surely there is C,
Ocaml, Go, or Typescript, etc, all of which are much faster?

~~~
protomikron
It is a common misconception that typing makes stuff more complicated and
inconvenient. Some type systems are quiet _bad_ (not powerful enough, so they
do more harm than good), but that does not mean that the idea of typing itself
is bad, but their implementation is. In my opinion Python kicks ass, because
of world-class libraries for nearly anything, and not because it's dynamic (a
broad term, I would say it's strongly typed) und you do not have to declare
types (which can often be inferred).

You listed 4 specific languages, but the following things are each
inconvenient in them:

    
    
      - C: it's inconvenient to implement a quick prototype
      - Ocaml: it's inconvenient that the stdlib is so small
      - Go: it's inconvenient for scientific/numerical code
      - TypeScript: it's inconvenient to run server-side
    

Obviously you can do everything in any language (Turing, blabla), but the
question is, how much effort (time) does it cost.

Python rocks in all of them (edit: the _big_ exception being frontend webdev),
in particular for quick prototyping. I think mypy is an exceptional and fine
contribution to the eco-system and I really appreciate their effort.

~~~
llekn
Being in the middle of a struggle with Javascript tooling (Javascript, ES5 v/s
ES6, typescript, webpack v/s System.js, shims, pollyfils, npm downloading the
internet, etc, etc) I can't but imagine that there should be a better way.

How much I would like Webassembly to be ready to open the gates to other
languages on the frontend land. Anyone here has more information about
webassembly / python on the browser? I would love to know what is the current
state of the art of running python on the browser and contribute to make the
dream come true.

I have on my radar Brython [0], py.js [1](seems to be abandoned) and pypy.js
[2]

[0] [http://www.brython.info/](http://www.brython.info/) [1]
[http://pyjs.org/](http://pyjs.org/) [2]
[http://pypyjs.org/](http://pypyjs.org/)

~~~
aumerle
[https://github.com/kovidgoyal/rapydscript-
ng](https://github.com/kovidgoyal/rapydscript-ng)
[https://github.com/QQuick/Transcrypt](https://github.com/QQuick/Transcrypt)

