While I really like type annotations I hate that they’re little more than IDE hints.
There’s nothing to stop calling functions with incorrect datatypes at runtime and I’d consider this one of the single biggest weaknesses of python right now
There is no runtime typechecking, but with an offline typechecker like mypy they're still far more than just IDE hints. In my experience you can catch most typing problems that way.
PHP went with runtime typechecking, and it isn't pretty. The type system is much less powerful than Python's because it's really hard to add complex runtime checks to an already existing dynamic language. You can make a function check whether its argument is an array, but you can't make it check whether it's, say, an array of integers, because that would require walking through the entire array, or making every array do expensive bookkeeping in case it's needed later.
If you want more complex types you have to resort to phpdoc annotations, which are only useful for offline checkers, like Python's.
When I wrote PHP we used Psalm for offline typechecking. Between that and PhpStorm the runtime checks didn't add a lot of value, and they even made mocking a lot harder at times.
On the other hand I imagine runtime checks are more useful for legacy codebases that are too messy for static analysis. I didn't work on those.
The fact you can rely on every API to allow duck typing is a core feature of the language. It has pros and cons, of course, like all features. But it is not an accident, it's by design.
And type hints have been designed with this in mind. It will stay that way.
If this is feature makes you unhappy, Python will make you unhappy. It's important to know what are the values that are important to you and chose your toolbox accordingly.
Isn't Python just a language? AFAIK there are many Python interpreters (or implementations), like CPython, Jython, IronPython,....
Couldn't a Python interpreter enforce type checking and raise errors whenever a type doesn't match? I'd be happy to use such an interpreter in many scenarios.
Tons of libs wouldn't work this way I guess, probably including stuff in the standard library. So you could really just use a subset of the Python ecosystem. I'm not sure it's worth the effort.
> I think it should be up to the API to specify which types are allowed, not the developer just hoping it works.
This assumes:
- the dev is thinking about the API
- the dev has the knowledge and experience to do it
- the dev has the resource/constraints that allow to do so
- dev has an interest in doing so
- this is a good investment for that particular project
Thinking in black and white does not make for good language design.
I'm glad Guido designed Python and said no to so many requests to make it no Python. Because he sure been pressured to.
> When you’ve worked on a sufficiently large and complex code base maybe you’ll understand.
I do.
I also understand Python projects that have a favorable cost/benefit ratio for type hints are not that many.
When you're used to hammer you see everything as a nail, and people loving a specific language feature tend to want it everywhere.
A huge number of dev in the Python community are not professional nor even experienced programmers. And a equally big number of projects have a size or a nature that makes type hint a bad value.
When you’ve worked on a sufficiently diverse project base maybe you’ll understand.
I really enjoy being able to write about 90-95% of python with type hints, the static checking and IDE autocomplete is fantastic, and the remaining just be like "ok computer just trust me".
Or when I'm doing data wrangling in jupyter and I never use type hints cause it's all just REPL rapid prototyping and I get autocomplete from the objects.
I think type hints make sense in some specific cases. Especially if it's an external API that your code is providing, or for when the usage is not clear.
It's good for checking the internal logic, but being optional is important in the python context.
I wrestle with this too, but I think it can be useful if you understand the limitations and usefulness of it for yourself. I know that the type hints are way for me to think more carefully about what I want the functions to do and how I'd like them to respond. I'm not expecting the type checker to enforce correctness or optimize better, I'm simply hoping that the tools will help me write the best code I can for my uses. I think it's analogous to a writer working with an editor: the writer is allowed to write whatever they want, but the editor's job is to analyze it and provide suggestions to make it better. The writer has to decide what to do with those suggestions. Sometimes hinting gets in the way and I hate it, but sometimes it helps me. I try to use the help and ignore the interruptions.
Check out Pydantic: https://pydantic-docs.helpmanual.io/. It lets you define model classes and enforces the type hints of the models' fields at runtime. Plus you get free serialization/deserialization to dicts, JSON, or arbitrary Python classes. It's _really_ nice.
> There’s nothing to stop calling functions with incorrect datatypes at runtime
You can enforce typechecking of your code on the CI with a tool like `mypy` if you want additional safety. But you'll never get the same guarantees you'd get in OCaml or Rust. It's a tradeoff between flexibility and safety.
There's actually value beyond them just IDE hints. They can help expose mixed type operations (e.g. foo = int + float) and other places where you may be unintentionally casting between types. Eliminating those can result in notable performance improvements, particularly in tight loops.
What are your thoughts on purely static tools on that for Python?
Basically my latest project that I was thinking about is to write a symbolic executor for python that would simply preserve all of the required-type information for all of the execution paths of the program.
Also the other project is to write a tool that allows you to walk all of the possible executions of a given piece of code to find which functions are being called from it - incredibly useful for refactoring of the old code.
Does something like this exists already that you've used and found useful?
> simply preserve all of the required-type information for all of the execution paths of the program
A type-flow analysis? Due to Python language features such a meta-programming and runtime code execution, you may find that at the end of this all your reported types will come out as 'Any' and the tool will be useless!
What depth of context are you expecting to have to use to avoid megamorphisation?
The literature tells us more than a couple of levels is likely not tractable. That’s not going to enough is it?
I'm not an expert in Python, but I am in dynamic analysis of other similar languages. Python lets you redefine methods, doesn't it? How do you accomodate the fact that while your routine is running someone could be redefining all your methods on another thread?
Nobody's yet managed what you're suggesting being able to do. Maybe you've got some new idea that nobody's even thought of before?
> But your language semantic model does include redefining methods on a concurrent thread. So you need to handle that.
I don't need to handle that at all if I say that "you can only redefine methods during initial startup which is defined as importing all of the necessary modules". At which point I can run my static analysis tool and it will preserve semantics.
If you're doing meta-programming by redefining methods used in other threads in one thread, then you're clearly redefining the whole idea of your programming language.
> There's no need to be abusive.
People like you (Ph.D in Ruby) have spent too much time doing their doctorate. You went down the rabbit hole of specialisation and now you think you can generalise your knowledge to everything.
I am going to disappoint you:
a. Python doesn't (practically) do multithreading
b. I think you're doing something really wrong if you're trying to do monkey-patching of methods from one thread to another (I don't care which language you're programming in)
c. Plenty of people in this thread have given links to
actual tools resolving the problem you're saying is unsolvable
d. I have an impeccably clear picture of how I am going to solve the given problem - the only reason I have asked the question is because I simply wanted to see prior work on the subject.
Yet you're forcing your opinion. Opinion which is not based on a single quote/whitepaper or a URL.
So I am not being abusive.
>
But to be even more clear: I am not trying to resolve the problem in the general case (it's pretty obvious you can't). I am trying to resolve the problem which fits 99.9% of the scenarios I have to deal with on a daily basis. And sadly there are _no_ tools for that at all.
> I simply wanted to see prior work on the subject. ... not based on a single quote/whitepaper or a URL.
For example start with Olin Shivers' doctoral work on k-CFA, and why he needed a k! And then follow the trail of citations from there forward.
Then look at David Van Horn's doctoral work on analysis:
> There is, in the worst case—and plausibly, in practice—no way to tame the cost of the analysis.
> The empirically observed intractability of this analysis can be understood as being inherent in the approximation problem being solved, rather than reflecting unfortunate gaps in our programming abilities.
> As the analyzer works, it must use some form of approximation; knowledge must be given up for the sake of computing within bounded resources.
That approximation is the 'Any' I was talking about.
And he's specifically talking about your idea of symbolic execution and why it doesn't work
> if we place no limit on the resources consumed by the compiler, it can perfectly predict the future—the compiler can simply simulate the running of the program, watching as it goes.
Get rid of that poetry please and give me a single reason why PyCharm’s algorithm that infers types within the functions in Python should not work; but actually works reasonably well in practice.
As a matter of fact all examples given in the papers you provide deal with generalised second-order cases of functions accepting functions which can be resolved by simply allowing to annotate said functions with the types by hand instead of inferring aforementioned types.
I wrote something like this. It used the bytecode, simulating types on the stack.
Functions with annotations had to be called with compatible types, functions without annotations were symbolically executed for each call site. Constructors were special cased so that the object attributes were set up correctly. Types for a stack slot or local could diverge and be narrowed depending on the path through the program.
The amount of metaprogramming used in practice in python made using it for real programs feel insurmountable. Also, I'm pretty sure performance would have been a problem beyond the toy programs I got it working on. But it was a fun exercise.
Very similar is https://google.github.io/pytype/ which also infers and adds annotations, as well as providing a type checker which in some ways works better than mypy (recursive types!)
Not Python exactly but I have these situation. I am writing business server in C++ that exposes JSON based RPC API and Javascript client library to access said API.
I have no idea where is that claim about high productivity of these free form languages came from. In C++ in addition to the nice intellisense compiler checks everything before sending program to debug. So I do not need to do manual testing during runtime to discover typo in my code.
In javascript I can write literally any crap and the system lets it go. Intellisense sucks big time as well. As a result of this and usual typos I have to literally go and do a lot of clicking to see if this part blows up at runtime.
So all in all I'd say productivity in C++ is way better. And C++ is not a stellar example of easy language. Actually it is easy for me. Not because I am super-duper programmer but your basic coding with the help of standard library is a piece of cake. I do not really need to be an architecture astronaut and dive into any esoteric stuff. Simply not needed
To answer your question, Python is several times more expressive than C++ given that is is a higher level language, and since fewer lines are needed for a given program it is likely to be quicker to write.
To quote Martelli from Google talking about how the Youtube startup outran Google Video's might so much that they had to buy them:
"Eventually we bought that little start up and we found how 20 developers ran circles around our hundreds of great developers. The solution was simple! Those 20 guys were using Python. We were using C++. So, that was YouTube and still is." [1]
Google took months to copy new YouTube features, while YouTube could catch up in a week after Google innovated.
[1] Google Books, search term 'python interviews google vs youtube'
2) I do use Python but mostly for administrative/deployment/similar tasks as a scripting language. I see no proof that Python is "several times more expressive". I would really appreciate if you could kindly point me to a real study that shows exactly how it is more expressive and on what in particular the time is saved (you can of course omit the examples where Python is used as a glue language to stitch together few math libraries written in C/++).
I do have my own experience. Of course it is statistically meaningless. My friend and I have implemented simple GUI desktop app (custom video player with some features) as a bet. My buddy did it in Python and I did it in Delphi. I won: 1 hour vs 4.
"An empirical comparison of c, c++, java, perl, python, rexx and tcl"
Lutz Prechelt
IEEE Computer 33 (10), 23-29, 2000
80 implementations of the same set of requirements, created by 74 different programmers in various languages, are compared for several properties, such as run time, memory consumption, source text length, comment density, program structure, reliability, and the amount of effort required for writing them. The results indicate that, for the given programming problem,“scripting languages”(Perl, Python, Rexx, Tcl) are more productive than conventional languages. In terms of run time and memory consumption, they often turn out better than Java and not much worse than C or C++. In general, the differences between languages tend to be smaller than the typical differences due to different programmers within the same language.
Did some reading. The first thing that puts me on alert about the result is this:
" Most programmers in the script group used the associative arrays provided by their language and stored the dictionary words to be retrieved by their number encodings"
For C++/non-scripting they either used plain arrays or custom build tree in one case. So they had to write lots of code.
Sorry but there are associative arrays in C++ standard library and search and this and that. Building the program they've specified would be a no brainer in modern C++. This article is irrelevant for current situation.
Awesome. I will try this next Monday on a code base at work, I hope it will improve readability. I admit I'm a little skeptical about the adoption of optional typing if it's not going to affect performance but maybe this will change my mind.
Haven’t been involved in the Python community for awhile but isn’t already a thing? Really cool project but thought I saw something like this already existed. If I’m right, is there anything that stands this apart from its counterparts?
What's the benefit of this over simply adding the type annotations directly? I guess this is mostly for those unwilling to understand types?
Especially given the admitted limitations of inferring types, such as the add exmaple discussed; this seems to be fixing an anti-pattern problem. As this those who would build a project in Python that would largely benefit from these annotations, would be most suited to just spend the couple of hours needed to truly apply it themselves.
One of the core requirements of the type hints design is that they don't affect the Python language.
You can still write python programs without them.
You can add them only partially.
You can also add them later.
Python stays Python.
MonkeyType lets you take a code base that is untyped, or partially typed, for historical or because it was not worth it at the time, and turn it into a fully typed project quickly.
It makes the transition between the exploration phase into the industrial phase much easier.
For me, it's kinda fantastic to be able to fiddle with a design, changing my mind again and again, without having to fight the type system, then once things are settled, add an additional safety net on top.
It's also very reassuring knowing I can just hack things, knowing that later on, if I want to take it to the next level, I have the option to change my mind about type hints.
Totally get that, and agree that this is an aspect if Python that is awesome for iteration on new ideas without many barriers.
However, this seems to make the assumptions that when you, as you say, transition to the industrial phase:
- MonkeyType will get all of the assumped types correct, which at any scale I doubt. Not because of the ability of the projects contributors, but because they warn about this in their own documentation
- If you're doing this at an industrial scale, and you haven't already documented/thought about/understood what these functions are. I'd appreciate that it may help intially, but still runs in to the problem I was discussing above.
Have you actually spent time adding type annotations to existing python code? In my experience it's a huge pain compared to other languages. Documentation is sloppy about types because people aren't used to worrying about them, so even looking at the code it's really hard to even determine the actual types of things without just running it and poking around.
Not sure why the initial tone of your response is quite so dismissive. But I do work with Python on a daily basis, even though this shouldn't be the deciding factor on the validity of any point. Discuss the point, not the individual, if it's so easy to disprove.
It seems that you're talking about poorly documented code, not how type checks effect this. Python code should have at least docstrings for types, or a scrawl with pen and paper describing this. If not, this is a failing of the project, and it's codebase. Not something you should rely on an outside tool to fix.
Python is jokingly said to be "the second best language for most things". To achieve this, it has to be able to be decent for many different purposes such as setuping machine learning models, scripting SIG system, doing test automation, powering a web site, provisioning linux servers, scrapping web page, batch renaming files and teaching code, analyzing numbers from a hdf5 file, being the client to a database/ftp/ssh/imap system.
It means it can't just be purely functional/OOP/imperative, be must borrow from each paradigm. Be statically typed or not, but gives features from both. Etc.
Because you don't want type hints when you are in jupyter or scripting your linux box. But you may want it when your Saas reach 3 millions lines.
> It's been explicitly stated that Python will never make the type hints mandatory
Type hints are already mandatory if you want to use data classes. This is unfortunate - the two features really should be orthogonal.
At least when Guido was still BDFL, there was little resistance to type hints encroaching on unrelated areas of the language, and hence becoming less-and-less optional.
There’s nothing to stop calling functions with incorrect datatypes at runtime and I’d consider this one of the single biggest weaknesses of python right now