Honestly, I've recently written a few 300-500 line programs in Python using type hints and I'm never going back. And I'm not even using mypy often, if ever.
My editor now has a fairly deep understanding of my code, and can tell me of all sorts of surprising errors I'm making before I run the code. There have been a few times where I found an error, went into the editor and saw I had missed a error message about that line. Shout out: Using LunarVim with LSP and TreeSitter.
The other thing I'm enjoying is that libraries can use them to make things happen more automatically. I believe it was Typer (the CLI argument parsing library) where if you declare an argument to a function as "files: List[Path]", it understands that the argument will take one or more files, or "-" to mean stdin. If you just say "file: Path", it understands it is a singular file.
I was curious about type hints when they first came out, wasn't really expecting to use them but they seemed cool, but now I'm totally sold.
>My editor now has a fairly deep understanding of my code, and can tell me of all sorts of surprising errors I'm making before I run the code.
I agree this is useful, but to me the most useful feature of types are the self-documentation. It's crazy the difference between a non-typed library with shitty documentation vs a typed library with shitty documentation.
The types on stitty-documented code often do depend on the types actually being checked though. Otherwise you need to author to be diligent in their types, but if they're bad in their other docs, that seems unlikely.
Not typing is kinda a crazy way to write programs when you think about it. It can be beneficial in certain niche use cases like data science where code is always terrible. For everything else, types are a huge boon for the developer and everyone consuming their work
I think data science is a perfect example in favor of types -- the code is often terrible because of the lack of typing. Pandas has notoriously poor developer ergonomics, and I recall painfully poring over type errors across the board -- lists, dataframes, numpy arrays, etc. are all iterables, so they can be interchanged in some contexts, but not in others.
Had I had MyPy back when I was working in data science, I would've saved countless hours and headaches.
What Pandas does is notoriously hard to fit into a compile-time type system. Certainly too hard to go into the brains of scientists who didn't grow up coding.
No, the code in data science isn't bad because of the lack of typing. The code is "bad" mostly because those writing it are relatively fresh from starting to program. Also there is more pressure to make things possible, often just to run it once, and neglect repeatability or scaling to larger code bases. Different emphasis. That doesn't mean an experienced full stack developer would do Data Science better, because he might lack a lot of skills that matter more in that domain.
> That doesn't mean an experienced full stack developer would do Data Science better, because he might lack a lot of skills that matter more in that domain.
This resonates with my experience. I had the opportunity to work on a DS codebase written entirely in Scala with all the typing, parallelism, actor model, whatnot. Basically I joined the company because of this technical factor. It was fun until I figured out that DS was "typed IF-THEN-ELSE written by Java devs in Scala returning stuff the users complain about with high reliability within milliseconds". Now I am happy to be back to the single threaded untyped Python world. Still no bugs in production, because we validate all requests to death, have unit tests and integration tests running on real data not on mokups. Basically we follow the principle: if the integration test passes, our typing is just right, or at least good enough. Funnily all the typing errors we catch are caused by wrongly typed data, coming from the productive system written in a typed programming language... what a strange world.
> integration tests running on real data not on mokups.
I can see you are enjoying the life outside of a highly regulated industry. Having certain kinds of production data in tests (or feeding that to test environment) would be a major audit finding in any finance or healthcare company.
...thereby destroying the production-like features for which you want it for testing, which you then need to recreate and reintroduce, so you might as well just synthesize test data in the first place, since that's what you end up doing anyway, in effect.
I know you probably just want to be annoying, but really, there is a world of difference between completely synthetic and anonymized data. Without talking about specifics, anonymizing is just the operation of making the process of deanonymization a lot of harder. "Hard enough" is usually specified in some form by the regulator. You can identify an individual by their ECG data, for example, it's just really hard...
No, in actual practice you don't scrub the stuff you actually need to test.
> I know you probably just want to be annoying, but really, there is a world of difference between completely synthetic and anonymized data.
No, I’ve spent ~20 years in healthcare, with this issue as a frequently recurring issue.
> No, in actual practice you don't scrub the stuff you actually need to test.
In actual practice, the stuff you really need to test often overlaps with the stuff minimally required to scrub to legally deanonymize the data. The most common scenario I’ve seen trying to do this is both creating most of the work of generating synthetic data and failing to legally deidentify the source data.
What I wrote was meant in the context of data science. You can not do ML without having access to real data, not even in highly regulated industries. Obviously you won't touch PIIs. But whether the real data is sitting in your train/test/validation data set or you use it for integration tests, doesn't make any difference from the perspective of an audit.
This is the largest difference. When there's no expectation of code lasting beyond a very short lifespan, why go through the effort to future proof things, improve maintainability, have better ergonomics, etc?
Sure. And once you land on something worth keeping, you clean it up then. But between time point 0 and then, a whole lot of code gets written to be run very few times.
A pandas dataframe types to something like Iterable[SomeKindOfProductType[int, str, str, ... (78 other columns)]]. The formal type of a dataframe in the middle of a transformation is... not very useful to know.
You wouldn't typically type tabular data in any language. Give it (at most) a DataFrame<StoreTransaction> type if you must, where StoreTransaction describes the structure of a row - maybe declaring only columns that model generation code was doing typed operations on (e.g. numerics vs strings) to avoid the need for reflection.
Either you type the tabular data at compile time or you don't get type checking of tabular data at compile time.
The number and types of the columns aren't necessarily known at compile time. Which leads to runtime errors. Even in a statically typed language, such dataframes are a kind of "dynamic typing escape hatch". As complexity of a software increases, such mechanismus of dynamic typing and throwing runtime errors creep up all over the place.
Sure. If you're dealing with untyped data at runtime you can't type it at compile time. Not a new issue, and handled all the time in otherwise typed languages.
> What Pandas does is notoriously hard to fit into a compile-time type system. Certainly too hard to go into the brains of scientists who didn't grow up coding.
I'm not sure if that's true. Doesn't Pandas handle ETL and some anaysis? There is nothing inherent to ETL that makes it a hard problem with compiled languages.
In your opinion, what does Pandas do that's hard to do with compile-time languages?
I didn't say "with compile-time languages" but "with compile-time type systems". And many similar tools in a statically typed language will necessarily create a way to have one static type that doesn't care what the data inside actually looks like.
This even starts with basic Numpy and handling tensor objects. It's not easy for a type checker to understand what operations you can do with what shape of tensor. Worse, most often you don't know (or want to know) some of the dimensions or even dimensionality of some of the objects. Then it is impossible to check all of this at compile time.
> This even starts with basic Numpy and handling tensor objects. It's not easy for a type checker to understand what operations you can do with what shape of tensor.
That doesn't sound like a Python problem.
Instead, it sounds like the natural consequence of numpy being designed in a way where their data types aren't organized into subtypes, and leave that as runtime properties. This is a natural reflection of numpy's take on vectors, matrices, and tensors, which in terms of types are just big arrays with runtime properties.
To put things in perspective, in C++, Eigen supports static dense vectors and matrices whose size is specified and known at compile-time. I'm sure Python doesn't impose addition static type constraints than C++.
Of course it's not a Python problem, all similar tools have the same "problem" that they can't easily fit that stuff into their type systems, so they invent some way to not care about it.
It isn't a matter of "compile time": explicit type declarations and definitions can often be formally sound but practically worthless.
Significant types in ETL-style applications typically come from outside (e.g. a certain CSV column in the input file contains a date in YYYYMMDD format, or maybe YYYYDDMM, figure it out, and don't forget time zones or your accounting will go wrong).
Then types are mostly complex but obvious and easily deducted (e.g. multiplying matrices of compatible shapes necessarily gives a matrix of a certain shape, why should the program say anything more detailed or lower-level than "do a matrix multiplication" or "do a tensor product"?); they are an often dynamic and unpredictable property of the data, not a useful abstraction.
The source code shouldn't need to say anything about the type of the resulting matrix explicitly, perhaps. But why shouldn't the type system keep track of shapes and deduce the accurate type for the result of said multiplication?
Totally agree, I would love for data scientists to use types and a lot of other good practices. I work primarily in mlops and have been trying to do this for some time, it just mostly feels like a losing game.
The output of most analytics is a report. For longer lived processes it seems easier to just hand off
Have you tried it? Python has gained most of its popularity before type hints, and I would (wildly) guess 99% of Python programmers don't even use type checking at all.
Static typing is completely viable and I hear those arguments mostly from those who didn't use Python for a long time. The added productivity makes up for a little more debugging while the program is running.
I'd also posit that many Python codebases are somewhat less "untyped" than one might assume. Django for example does a lot in its ORM and Form classes to make sure the right stuff goes in the right slot.
If you mean “Python programmers” to be random sysadmins that write hack code residing on a random EC2 instance then maybe you’re right. But most I get the sense you haven’t looked at many Python libs lately.
The added productivity of non-typed Python is such a ridiculous myth to anyone who has to maintain significant Python code bases. Sure, it makes you more productive for one-off exercises but the moment you’re having someone else inspect your code the productivity gains vanish. This largely true of dynamic languages more generally except in niche caches (e.g., ORMs) where types are strongly implicit.
I say all this as a mathematician that is used to reading papers full of symbols packed with implicit information. The code I’ve read from gung-ho dynamic typing advocates is some of the most painful drivel I’ve ever had to wade through.
Python libraries aren't representative of Python code. The majority of Python programmers (not "just sysadmins" wherever that slur comes from) never publish or contribute to any libraries.
I can compare Python's productivity to other languages, and it beats all of them so far. Another lesson I learned from working with statically typed languages: The quality of the code is more dependent on who writes it than on the language or even the ecosystem. But people are productive much sooner in Python than they are in C# or F#, and then we can work on the foot-gunning.
Python's productivity isn't a myth. This is proven by its popularity and by successful projects across academia and industry alike.
You might be more productive in Python versus anything else for the scale and complexity of the software you are involved in, but you cannot make sweeping generalisations like this. I also think you fail to understand that in many situations, static type systems actually improve productivity, especially at scale or for non-trivial domains. This is of course why Python is now trying to retrofit them.
There's also the argument that one is most productive in what they know. As someone who has spent far too long with Haskell and OCaml, Python's semantics and libraries often seem surprising and counterintuitive to me.
I "fail to understand" something you actually never did or experienced, right? I know that a lot of companies are maintaining large Python codebases, and the "lack of typechecking" might be one of the biggest pain points, but the total pain is a lot less. That was always my experience and most people who say differently haven't actually used Python that ...
I know that Typescript helped me over Javascript. I don't get that benefit from Mypy yet, but the "pain" just isn't as great as it was with JS. And I've seen what inexperienced programmers do when a static typecheckers try to force or encourage them to do the right thing. It's not necessarily the right thing, just a lot more complicated than it needs to be.
> I "fail to understand" something you actually never did or experienced, right?
No, I have worked at a variety of large institutions that have millions of lines of Python. I work with it often. The use of Python is more about it being accessible for non-developers, which is why they also teach it in schools; and the need for a standard. It is the new "Visual Basic". Again, it is not more productive in all situations.
If you write more code that is untyped than typed, you are harming whoever comes after you. Untyped code is an absolute dragon even if it is properly documented (which it practically never is). Glorifying "productivity" and putting it above the greater good is essentially everything that's wrong with the world, at every level.
That’s quite a strongly worded opinion. It sounds like it should be utterly trivial in this case to show that typed code causes less harm. I’m happy with whatever definition of harm you’re happy with. So where can i review the data supporting such a strongly worded position?
On a quick search, for example https://link.springer.com/article/10.1007/s10664-013-9289-1. "This paper describes an experiment that tests whether static type systems improve the maintainability of software systems, in terms of understanding undocumented code, fixing type errors, and fixing semantic errors. The results show rigorous empirical evidence that static types are indeed beneficial to these activities, except when fixing semantic errors." is pretty much as strongly worded as scientific journals can accept.
- Static type systems are an effective form of documentation
– Static type systems reduce the effort to fix type errors
– Static type systems may not be helpful in preventing semantic errors
It would have been awesome to have evidence for #3 but even just #1 & #2 could be useful, albeit to a much lesser extent. Could we quantify by how much?
It’d be important to quantify because if static types mean a developer can be done sooner, then they have spare time to spend improving the software. So do these differences with static types translate into benefits for the developer?
For the second paper, looks like the test task is quite small and the language used requires more work from developer than a modern static type system with type inference and generics. Still the difference between static and dynamic typed variants is rather small. I wonder if the situation would change if the project took weeks or months instead of hours, but that would be an expensive experiment. Also, it would be interesting to see if type inference and generics would increase productivity but in the real world we are stuck with Java and Go anyway.
Other take home message is that the famous 10x programmer was found in both groups. The one with dynamic language was faster, just as the argument for dynamic typing goes.
There is no conclusive and objective evidence that static typing helps at all. Studies are at best inconclusive, certainly a far cry from confirming the superiority of static typing. Everything else is subjective. To have a recent and complete opinion about the one thing you necessarily need to have less of a recent and complete experience with another.
The benefits of masks against spreading respiratory viruses are easily and objectively proven. Completely different thing.
Well, while you're waiting for that study to ride down from heaven on a white horse clad in gold, the rest of us are going to just quietly get on with being (subjectively) more productive.
The number of times I've stared at some random Python function deep inside a library wondering what does this function return? Even the docs often don't say.
I took a random stab in pandas but they’ve adopted type annotations so that defeats the point. So i thought old-school code bases - zope sprang to mind, but that’s not a good example either because they use docstrings and it’s kind of obvious. So i tried gunicorn on a whim and they have even more attention spent on their docstrings.
I just want to sample this feeling of lost confusion in python so that i can figure out how I’d deal with it.
I was actually looking at urllib3 yesterday trying to work out the return value from HTTPConnection.connect which calls this _new_conn function internally:
def _new_conn(self):
"""Establish a socket connection and set nodelay settings on it.
:return: New socket connection.
"""
...
try:
conn = connection.create_connection(
(self._dns_host, self.port), self.timeout, **extra_kw
)
...
return conn
(I've chopped it down a bit to emphasise the important bits)
What's the type of "conn"? It certainly wasn't clear to me that it's a socket.
But just now I looked at upstream and I notice they've added type annotations, which do greatly improve things:
When i hover over connection.create_connection i get a docstring that says it returns a socket.
That docstring isn’t enough for the problem you had though, e.g. I wasn’t clear from that if they meant a socket.socket or maybe some special socket derivative type so i hit go to definition and create_connection is a short method - i can see it is indeed a socket.socket being created, configured and returned.
I’m guessing you’re not using any developer tooling, maybe a straight text editor i guess?
This kind of tooling has been around for a long time, i remember using rope and jedi in vim back around the late 2000’s i think. It’s gotten a LOT better since PyCharm appeared mid 2010’s i think - first i recall using it must have been around 2014-ish. These days, it’s even better. I would really strongly encourage adopting some kind of tooling that lets you navigate and has some basic refactoring - rename is usually 80%+ of the value there so it doesn’t need to be complex. A few ergonomics like having the ability to jump to test and call from repl with a keystroke - you’ll turbo charge your comfort and productivity.
That has to be our difference i think because it took one mouse move and one key chord to completely bottom out.
On the small scale, Python is very productive. Once you get to large codebases with a rotating roster of many developers, it becomes a net negative, which is why type specification languages are gaining major inroads. I love Python, but I wouldn't write anything for production at any scale in it anymore. But for small scale stuff, it remains my favorite! (Although, I like static binaries languages, like Go, for a lot of things these days because of the deployment simplicity)
What's your definition of "large codebases" here? My experience is that on Python codebases large (~100k lines) and small, type annotations are generally a net negative, particularly with a "rotating roster of many developers", because of their complexity and detrimental effect they have on readability. As the OP says, incorrect types are worse than no types at all.
How is typing productive in large codebases? If I see a bunch of imports related to typing only, I have to juggle them around in my mind together with stuff that really matters.
Not to mention the notorious circular import problem, that is much more likely in a big codebase.
If I have a custom class, that class has to be passed as a type hint for functions. What If I use that class in another module, but have to import it inside this one that has a type-hint only?
Well, let me introduce you to their genious solution:
from _future_ import annotations
from foomod import FooClass
def myfun(foo: FooClass):
return foo.bar()
I also find most of the "import typing" stuff to be clutter.
I worked at a place that was heavily into adding typing to existing code, some of which was almost a decade old. They spent more than a few person years on that effort, and it's still going on.
Not once did I find the additional typing useful in my actual day-to-day work.
Scaling a code base doesn't start or end with type checking. Some of the answers to what you are implying is: Use interfaces when using OOP. Consider dependency injection or other Inversion of Control patterns. Maybe don't use OOP as much...
Python has a tremendous amount of momentum because it has become so popular outside of the traditional software development world, but other languages are picking away at it, even given that momentum. And while I am in a bubble, like everyone else, in my bubble, and those I talk to in my bubble, all see large a shift away from Python.
Adding stronger typing, like type hints, could reverse that, of course.
The reason not to is performance and reliability. Typing helps with the latter, Python is just bad at the former.
Major companies build things entirely in Javascript, too. Popularity doesn't mean anything. Honestly, for a small web app, I'd pick PHP / Laravel over Python. Deployment is way simpler.
I have been developing almost exclusively in Python for the last 4 years (occasional C#, Java and PHP) and although my entire team disparages PHP it is really quicker and fewer characters for most stuff.
Luckily IDEs can auto add the imports, but sometimes a slightly mistyped piece of code auto adds an import I need to delete, I love avoiding imports and circular imports in PHP. The other nice thing is avoiding random python behaviour differences in MacOS, Windows and Ubuntu. Code that works fine in CircleCI but not locally. Works on a Mac but not on windows etc… I didn’t experience that with PHP.
Not sure there are a huge number of long term PHP devs who switch to Python or the other way around to have a reasonable opinion.
This is a rather uninteresting statement, as we all live in bubbles - knowledge of them helps, but really just puts you in a different bubble. Still a bubble.
I'd say all metrics point to the popularity of Python increasing, while there is no data indicating that people are fleeing Python en masse. Or at all, actually.
In my view it's both. I'm one of those "scientific" programmers. I've used type hints, just to learn it, but tend not to use them in practice. Bugs caused by typing errors seem to be extremely rare in my little world.
I've also programmed extensively in Pascal, assembly, and C. I get it about types. Don't forget C has 8 different types of integers that have to be kept straight. ;-)
On the other hand, I realize that any of my programs is but a little nub of code sitting on top of large and expertly maintained libraries. I'm grateful for whatever the authors of those libraries are doing to keep their sanity, and am not surprised when I see type hints in those codes.
If you have programmed extensively in Pascal, assembly and C, you aren't "one of those scientific programmers". Or at least not the newer generation that got started on R, Python and maybe SAS...
I'm not sure I'm following your comment correctly, but you seem to be implying that adding type hints in Python results in some lost productivity?
If so, I would counter that adding types isn't particularly onerous and adds very little overhead to development time, and over time they'll tend to increase your productivity since you'll make less errors, IDEs can give you better hints, etc.
As to the percent of Python developers who use types / type checking, it seems to me that's been changing somewhat rapidly over the past few years. It's certainly greater than 1%.
I disagree, dependencies can change causing types to change which requires you to alter hundreds of lines across multiple repos.
Each new line or character is a line to maintain, there is a cost associated with that. Tests have costs but are extremely valuable, they are a no brainer to me, types are way less obvious.
I have spent a lot of time maintaining python types. I have no idea how many bugs it has prevented. I am not sure it is a large number.
I know there have been bugs in types, bugs in mypy and dependency changes which created work which only existed because types were in place. There have definitely been days worth of types maintain.
There are the extra type imports, extra time to run tests and integration tests etc.
It is not as straightforward as you might initially assume.
I have absolutely tried it and worked on many code bases big and small before typing happened. I've also worked on other languages and paradigms and seen what works and doesn't.
In my experience its the other way around. Most people who are a fan of dynamic typing have worked in systems with types. The world is largely coming around to the view that types make more consumable and safe software
For me, python is mostly a calculator, and it seems faster to try a calculation and immediately see if it runs and/or looks correct, rather than try to implement type hints et al (though I still use type hints occasionally).
If you find that you are returning silly types like that, then it is an indicator you are writing your function badly and should refactor it to be simpler
It is a silly type for a function to return, because you have three levels of data structure nesting, with not one bit of information about the meaning of any of the values except their types. If I call your function and then do result[1][2], what does that mean?
This is exactly the kind of situation where you're supposed to use a named tuple or a data class to make it clear what's what.
Learning any new syntax, feature, etc takes time. IME learning type annotations can pay major dividends. For example, I wrote some type-annotated code the other day and mypy pointed out a couple places where I was passing the wrong type, which saved me from having to fix those bugs later, thus saving time for "actual work."
There is a recurring cost to complex type annotations and that is that every single developer working on that repo (could be thousands) has to unravel the complex types, their aliases and the instance objects to modify any code.
This as opposed to a legitimate type system where complex types are classes themselves which don't need unraveling. There is no need to alias a type and there is no need to maintain separate complexities for the object and the type.
I agree that complexity could be an issue with Python's typing, which has some flaws, although it's been improving.
Type aliases are okay if they're not overly complex, having used them in Python and other languages to good effect, e.g. to avoid duplication or for clarity.
I'd suggest using type annotations where it's not overly onerous--which is probably most places--to get the benefits without much effort but perhaps avoid them or use simplified types where it's too complex.
I also agree that there are places where typing is unnecessary and doesn't add much value, such as in certain scripts.
Like many things in programming, it's all about finding the right trade-offs.
Yep, that's gross. Having come from a position where people where forced to actually add typing like this to make it through code review, I feel the pain.
Totally agree. I'm on a project using python for the first time and really can't believe how horrible it is to not have type checking, especially given that it takes time to run the code as a glue job.
I use them for the same reason. I've started sprinkling them into projects where I can. Other devs on the team seem to like seeing them, but have higher inertia when it comes to writing them.
Have you (or anyone else on this thread) used retype or MonkeyType? We have a codebase right now that's mostly lacking type hints. But, now that we have added automatic flaking into our build (and pre-commit hooks), I'd like to get some more mileage out of that toolchain we just invested into, but it would be a hassle to go back and add type hinting to everything.
What I'd like to do is integrate something into our pre-commit hooks.
It's not really as necessary as you think -- making mypy/pyright pass your CI ends up being a lot of busy work to make the type checker happy when, to me, all the value is realized when it's used as a development aid. If your test suite passes that's honestly more than good enough.
You have the right attitude. I have worked with delusional individuals who one day "soon" expect to mypy check their entire several hundred thousand line legacy code base. This is supposed to be a boon to developer productivity. I hope it outweighs what was lost getting to this point.
Not yet, but that might be coming. I have one personal project set up to run mypy as a pre-commit, but I'm not (quite) ready to force that on my coworkers.
This article is so hopelessly mis-informed, that it's practically hard to read without screaming "that's not how any of this works!" Most of their main arguments - type hints can be wrong, they can be ignored, and they don't inform you in a useful way about program state because of this - all evaporate as soon as you start using the correct tooling.
My stack is pycharm, mypy, pydantic, sqlalchemy stub, several mypy plugins, black, isort, tabNine completion, and a bunch of custom scripts/Make to automate the common tasks. I'm not even sure I'm on the same planet as the author. I can practically spot incorrect type hints, mismatched types, weakly defined interfaces, from orbit, that's how well they are highlighted. Since every type is known, between pycharm and tabnine I can type 2-5 keys and spit out entire lines of code. And since it's all typed, automatic refactor is a breeze.
I remember the dark ages, before type hints. It was rough. I lived in the REPL. Write 3 lines, run, Exception. Write 3 more lines, run, NoneType error. Ad nauseum. Now I write for hours on end and don't wince when I hit execute. And I don't have to go bug coworkers with questions like "hey what keys are in this dict when we redirect from the oauth endpoint".
Sure, I have my gripes, but it's mostly with the lack of power of the type system, like lack of higher kinded types. But I'm confident it'll get there.
Come to the typed side. We have less mental fatigue and screaming at neverending runtime oopsies.
> My stack is pycharm, mypy, pydantic, sqlalchemy stub, several mypy plugins, black, isort, tabNine completion, and a bunch of custom scripts/Make to automate the common tasks.
Christ. What a mess. With a stack like that, it's a stretch to say you're "writing in Python" anymore.
> Most of their main arguments (...) evaporate as soon as you start using the correct tooling.
You silly! Just use these carefully curated list of 28 different tools (that I got to after years of trial and error) and you'll have a somewhat acceptable tooling and type support in Python.
Anything that can use an LSP server will give you most/all of those benefits. Your favourite editor + LSP, that's the tooling. It's very much worth it.
That list is an editor, a type checker, a package to enforce type checks at runtime, 3rd party type annotations for a popular SQL ORM, some plugins for the type checker (probably the pydantic one), a code formatter, an import sorter, and (optionally) a Github co-pilot alternative.
To be fair, I don’t think you need that entire stack. Just using VSCode + pyright (which is a one button install in VSCode) + type stubs for libraries your using without native hints, gets you 99% of the benefits.
If you really want, you can even throw in a little pre-commit hit hook that runs pyright and prevents you from commuting code with type errors.
I think the person you're responding to is just trying to be thorough in case it is helpful to someone else setting up python tooling; the only parts of what they wrote that are relevant to type checking are mypy and pydantic.
> pydantic is primarily a parsing library, not a validation library. Validation is a means to an end: building a model which conforms to the types and constraints provided.
Anyone whose used Typescript or even strong typed languages knew what to expect from Python + mypy. No surprises there. But pydantic's use of type hinting to create a remarkably dense serialisation library, and then fastapi's leveraging that to create REST interfaces along with automagic documentation generation - well that wasn't when I was thinking would happen when I saw the type hinting proposal.
You can write type stubs for other libraries. Over past year my team had a 100k line codebase that we’ve gotten mostly typed. But we still use a lot of other untyped libraries like tensorflow. So in past couple months we’ve started adding type stubs to the parts of the api we use. While tensorflow/numpy/etc are massive libraries most people only use a small subset of api. You can write type stubs for the subset you use. It definitely takes time but I find it very helpful to have useful type checking of tensorflow so I bit bullet and started writing stubs. I explored upstreaming them to type shed but got lazy on dealing with CI issue although I hope to revisit it later.
If you expect type hints to be perfect and completely disallow you to write type unsound code then this is right, if you expect type hints to be useful when developing then this is very wrong. The automatic type hints produced by pyright on completely untyped code are fantastic.
I was curious of seeing if pydantic was mentioned, thanks! We use most of the same tooling as you, I'm always interested in improving our tooling, would you be willing to share your mypy plug-ins names or maybe most useful tools for inspiration?
My thoughts exactly. Obviously, Python Type hints are an attempt to add type-check to a language "after the fact" and keep it optional. So it's unfair to compare that to a static typed language and conclude that it's bad.
I work with large code bases (100K-1M lines) and can't look back to the time we didn't use Type Hints. Would maybe dismiss it only for scripts or very small projects.
I use this very strict config with mypy and it's effectively a typed language. I rarely have any errors in runtime now, at least from type-related problems like assigning 'vals = 3' and later trying to call 'vals.append(4)'. I sometimes use 'Any' because it's basically required for things like loading JSON or retrieving the results from a GET request, but I have several strictness settings that ensure I never pass around wild Any values and I try to keep their use to within a single function or method and always return a well-typed value. Sure it's all in my IDE right now, but if I import 1 or 2 packages suddenly my well-typed Python code starts to act more like a typed language in runtime, too!
disallow_any_unimported = true
disallow_any_expr = true
disallow_any_decorated = false # true if you never plan to use Any
disallow_any_explicit = false # true if you never plan to use Any
disallow_any_generics = false # true if you never plan to use Any
disallow_subclassing_any = true
disallow_untyped_calls = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_decorators = true
no_implicit_optional = true
warn_redundant_casts = true
warn_unused_ignores = true
no_warn_no_return = true
warn_return_any = true
warn_unreachable = true
strict_equality = true
strict = true
What opensource project is closest to "doing python the right way", in your opinion?
I'm in "python is not suitable for production" camp, would be happy to change my mind.
> I remember the dark ages, before type hints. It was rough. I lived in the REPL. Write 3 lines, run, Exception. Write 3 more lines, run, NoneType error.
That's why I switched (back) to typed compiled languages and never went back.
Python types just work. Just use these 30 libraries, spice up your IDE with settings and voila /s.
This is garbage and moving backwards as a programmer. I loved python when it was easy to use with any editor and with few plugins. Now I'm just going to switch to a statically typed language where I don't have to install a zillion plugins to get first class typing.
I think the difference is largely a culture problem. Most python devs think of typing as a “nice to have” extra, whereas someone using a typed-at-compile-time language see it as an essential feature of good code. You can see the results of this in the fact that a lot of popular 3rd party python libraries still don’t have type hints.
Even with typing, how often do you see Any in python vs Object in Java?
> I think the difference is largely a culture problem. Most python devs think of typing as a “nice to have” extra, whereas someone using a typed-at-compile-time language see it as an essential feature of good code
I love statically typed languages. I also love python. But pythons "type annotation" system is garbage. It is neither here, nor there and developers spend an inordinate amount of time figuring out types.
While other languages typing is a productivity booster, pythons type annotation absolutely isn't, especially with complex types.
But a language relying so heavily on IDE means that the language is severely lacking.
People not using said "modern" IDEs are just stuck out of using the language. Is this progress?
Even with modern IDEs, there is a difference between what different IDEs do. So now the language is stuck with different IDEs doing different things. Is the progress?
It also means that anyone developing a future IDE can no longer support these languages without support. Is this progress?
God forbid someone tries python on a new architecture where new the entire complex toolchain is not going to be available. They just can't use this language. Is this progress?
I'm pretty sure the IDE uses either mypy or pyrite under the hood.
I don't know how this is a sign of python lacking. Works just fine in jupyter notebooks, too, and that is hardly an "IDE" (though it does have Jedi autocomplete).
Python type annotations in the language are nothing more then allowing you put any Python expression in the "type slot" and have it be syntatically valid. They don't do anything. All the functionality is 3rd party packages taking advantage of their existence.
My codebase is more complex than str, text, dict, int, bool.
So developers are creating complex custom types with union[custom_type_1, custom_type_2] where each custom type could have more unions of other custom types. These custom types then get imported everywhere. It is utter garbage.
I don't follow. So some 3rd party library will define a type like Routes = Union[RouteSet, List[Union[str, Route]]] and this is bad? The complexity is already there, you just don't want to see it?
That gets called with all the types in that nested union but now the type checker can’t help you if GoogleRoute and Metaroute have different attribute names or that .append isn’t a method on sequences but is on lists.
Like that unwieldy nested union already exists in your code, adding the type just documents it and the type checker makes you handle all the cases.
Is there something specific about imports that don’t work with type alises?
Definitely types, but it's still weaker than in other languages. Just the other day found a maddening bug, where Django ORM query accepted `date` type comparison against a datetime field, and of course that resulted in crazy things in production. Other languages would have caught this compile-time.
You’ve never used a compiled statically typed language. I’m sorry to say it but your comment would sound naive to you if you had. The article is spot on.
Adding type hints to Python has increased my productivity by at least a factor of 10. They allow you to reason about code in a local function without having to track back up dozens of call sites to ensure you're getting what you think you're getting. That alone is worth the price of admission. Both when editing code or reviewing someone else's. It's fantastic, particularly in a very large code base.
The editor experience only increases that factor as you can navigate a lot more freely. Even for small code bases type hints are a revelation. Knowing if something is nullable or not, alone, has caught 100's of bugs in waiting.
Most of the complaints here feel like attacking a strawman, particularly the ones arguing what's the point if you're not running mypy. Run mypy! It's like saying what's the point of writing unit tests if you don't run the test suite. For any sufficiently large code base you absolutely should be running mypy with any plugins for your framework of choice added.
For heterogeneous dictionaries use `dict[str, object]` and then you're forced to use the existing duck-typing mechanisms you're used to. Or, you refactor into better types and/or structures.
Is there friction? Sometimes. You don't have to annotate everything. If the cost is too prohibitive for a particular structure or function, ignore it. Sometimes there are mypy bugs that make writing correct code impossible. # type: ignore[reason] that sucker. Don't throw the baby out with the bath water.
We use a flake8 linter to enforce that all new or modified functions at least annotate their return type which makes interacting with functions quite a bit nicer, and encourages all devs to at least do the minimum. Usually you find that most args are appropriately typed at the same time.
Because ecosystems are what makes people productive and languages are just a means to access them. Python's ecosystem is absolutely massive and high-quality.
Most popular languages have massive ecosystems. Don't see Python's being a differentiator except for a few specialized areas. In fact, the project I'm on runs on spark wrapped by python libraries, so really its the scala libraries that provide all the heavy lifting.
I'm baffled by people loving type hints in python when strongly typed languages have been widely available for so long. This is why people liked java and c#.
There are way more differences between Python and Java than just "having explicit types". If that was the only difference, your comment would make more sense.
I would even go so far as to say that Java's type system is the very one that left such a bad taste in people's mouth that many people swore off explicitly typed languages for a decade or two. It really was that bad, especially before the last few years. It added a lot of extra boilerplate for minimal practical benefit. People used Java because it was fast, cross-platform, and less likely to go horribly wrong than unmanaged languages like C and C++. I doubt very many people used it because they just loved typing tons of boilerplate or because they thought the type system was good enough to catch tons of errors.
The type systems in Rust, TypeScript, and other more recent languages are far more expressive (and capable of catching real-world bugs) than what Java offered. If you think Java has a good type system... things have come a long way since then. Java gets updated continually, and some people will surely jump at my comment and claim things are great these days, but, no. Java is not up to par, even today. C# has done a better job of keeping up, in my opinion, but even it still lacks extremely useful features like proper Sum Types.
I don't have much experience with Python's type hinting, so I can't comment as directly there.
When a language like typescript supports too many obscure features, or multiple ways to do the same thing, the curve becomes too steep to be proficient.
Yeah, it doesn't prevent passing unexpected type as object to Print func and getting runtime exceptions. And returning "object" type isn't helpful if we want to prevent runtime errors. But then again, it can be written to spit out compile time errors: https://dotnetfiddle.net/XfKVaa
You misunderstand the ecosystem. Many folk program in Python because it’s largely forced upon them. It’s often the easiest language to work with for data science, ML Eng, or data engineering, despite many frameworks actually running in the JVM. It’s simply more accessible. The appeal of Python has not been its provenance or design for over 15 years, but rather the ecosystem.
Most engineers I work with are very happy to work with python. They would not say it is forced upon them. You are free to provide a much better alternative (that does not exist as far as I know).
Same thing happened to the JS world with typescript (though typescript seems more powerful than python type annotations).
Its fine though - I'm glad the culture is trending towards understanding that strong typing saves you time rather than causes you extra time. This is true even for very small and simple programs, and even if you're the only developer.
The typing debate has been ongoing for decades, and it's switched back and forth a few times. JS, Python and Ruby all came out in the 90s, after there were plenty of statically typed languages being used at the time. Alan Kay has argued that types are too restrictive and usually don't describe the kind of data the program is really about. Maybe that's not as true for an advanced typing language like Haskell. And maybe Kay had C and Pascal in mind, when he thought of objects as being the basis for a proper dynamic type system.
C,Pascal, even C# and C++ need type annotations mostly to actually make the program work. The type systems in Typescript and F# are more about helping the developer than helping the compiler.
Python type hints are slightly different because there are type hints as a syntax feature and then there is type checking, which can be static like mypy or during runtime like Beartype. Every type checker comes with its own type system, though they are hopefully somewhat similar...
TypeScript has a more powerful type system, but arguably it's less powerful because the hints are invisible to runtime code. Python doesn't do any runtime type checking by default, but the hints are accessible in code, which unlocks a fair amount of power. For example, Strawberry is a GraphQL server library that allows you to define your GraphQL schema using the same syntax as dataclasses.
I really like C# and have done a bunch of work with it in the past! But I also love Python for reasons other than just static vs dynamic typing.
To be honest, I think a lot of Python [or arbitrary dynamic language] developers can be quite sloppy, throwing dictionaries around everywhere because it's easy and leading to some really hard to read code. Type hints guide people away from that sloppiness while still allowing access to most of the features that make dynamic languages so useful and expressive.
It's a best-of-both-worlds situation and I'm here for it.
Many languages also demonstrated Python-like succinctness with strong static types and type inference. E.g. OCaml dates back to the 90's and ML dates back to the 70's.
If the type system is not powerful enough it will be extra works just to get around the inadequacy..
If c# have some support of sum type and structural subtyping like Python I will be using that... I mean even Haskell does not have proper records.. Python is not definitely the best, but quite nice among the current options.
My team started with an untyped codebase a year ago. I added a rule to CI that every pr that touches files with type errors must decrease the type errors in touched files by at least 5. When we started we had like 30k type errors. A year later and it’s about 5k left. This was a small wrapper script over mypy/pyright that just called them twice once on master and once on your branch to compare error counts.
I did occasionally get pushback on it but stricter checks will be inconvenience at first and I argued it’d help over time. Now that we’ve had it a year it’s pretty well accepted in same way many would accept run formatter.
The problem with these sorts of "rules" is they create needless changes outside of the PR's primary focus. The PR's purpose is to do X: add a feature, fix a bug, whatever. Now because I've done X, you want me to do Y, a set of totally unrelated changes that ultimately are a distraction.
Why wouldn’t you submit two different branches to review? One where you fix errors for types and another for the feature with the fixes:
(git main) > git checkout -b abc-1234/type-fixes
<do work on types and commit it>
(git abc-1234/type-fixes) > git checkout -b abc-1234/feature-branch
<do work on feature and commit it>
Then you can submit the feature branch as a PR on top of the type fixing branch and a PR for that branch to main. You only see the type fixing changes on that PR and only see the feature changes on its PR.
Mypy can be configured to ignore modules based on their names.
Though honestly I think it’s worth it to just bite the bullet and spend a full day or two to go through and fix every single error. After this you will have a much easier time navigating code base. I mean, your “new code” is probably modifications to your old code or calling your old code right? So you want to have at least the boundary between new and old typed. It’s quite satisfying anyway, as you will likely learn and discover tons of bugs on the way.
With any large legacy Python codebase, that's usually a month-long undertaking for one engineer, if not longer!
Python by design encourages duck typing, which means that you'll have plenty of places that simply take multiple different types, and just slapping a union is usually non-trival and not very beneficial either (eg. you'd be randomly patching things with fakes and mocks in tests).
If you think it's a single day job, I am sure my company (and plenty others) would pay you gladly your single daily rate to get our codebase migrated in a day :-D
For any large code base, it'll go on for months, since it gets done between higher priority work. You'll discover some problems, sure. But if they were really important, you'd have hit them earlier. My experience is these sorts of efforts create a lot of busy work and not much more.
You can pipe the diff into specific flake8 checkers. Disable the check by default so it's not run over the entire codebase, and have a separate line for running specific checks on just the diff. Eg:
this article is so full of... not-very-educated thoughts on gradual typing in a dynamic language that it's hard to know where to start.
A few concrete criticisms:
1. If you're using a dynamic language, then _by definition_ the language will not enforce your static hints at runtime. However, good news! Python has always been strongly typed, and _does_ enforce types at runtime!
2. A number of the examples given would be _impossible_ to statically type in most compiled languages (those without dependent types). It's hard to know where to go with a critique saying that you can't fully represent heterogeneous values in a dict, given that you can't do this _at all_ in many statically compiled languages.
It seems to fall back on saying "use of the type system requires too much discipline to be useful". This might be an interesting criticism on its own; however many current users of Python can say, from experience, that the required discipline is not a high enough hurdle that it prevents us from using types consistently and correctly.
There is plenty to critique about Python's static typing, but this article would have been better titled "I'm confused about Python's static type system and want somebody to show me how it's actually used in real-world scenarios".
This seems like an inaccurate and condescending put-down. What is the definition of "educated" in this context? Is there a book or generally well-known resource?
The author is clearly thoughtful and curious. Near the top of the post he says "what I’m hoping for is that someone will come along and tell me that I got it all wrong. “When you do it as follows, the system works: $explanation” Because, you know, I’d really like to have static typing in Python."
> 1. If you're using a dynamic language, then _by definition_ the language will not enforce your static hints at runtime. However, good news! Python has always been strongly typed, and _does_ enforce types at runtime!
And yet, the author points out that this is a working program:
foo: int = 'hello'
print(foo)
In what reasonable sense can Python be said to "enforce types at runtime" here?
I think the top comment was less polite than it should be, but I would echo that the post seems to have a bit of a weird understanding of gradual typing?
The Any type is to allow incremental typing. You start out with dynamic code, and progressively 'typify' it by adding more and more annotations, with Any working as a stopgap where for the moment no valid type exists, until you finally have a fully typed program. Inserting Anys because you're lazy is like casting things to Object in Java because you're lazy, abusing it like that is just incredibly sloppy and bad code.
If someone is really struggling with this, simply use a linting rule to ban Any in 'mature' code.
Like the author said, the weird thing is that it goes both ways. I'm pretty sure if you try to return `any` from a function with an explicit return type in TypeScript you'll get pinged for it.
noImplicitAny is different than what's being discussed (I think). Returning something typed as `any` in a function that has a different return type is totally fine: it's not implicit (you've cast it to any), and it passes (you're saying the type is literally anything after all).
Python is strongly typed. It respects types at runtime e.g., `"1"+1` causes TypeError.
Python is not statically typed.
For comparison, C is statically typed but it is weakly typed e.g., printf function has no idea about actual types of its arguments (in particular, errors in the format arg may produce non-sensical results silently).
Can a string be printed? Yes! Then Python is (correctly) allowing typesafe behavior here. Your (incorrect) annotation does not in any way contradict the typesafe-ness of printing a string (or an int). They're both equally printable.
This is just a very, very bad example, plain and simple. It 'looks' bad to a superficial reading, but in practice it demonstrates only what is already known about static typing in Python - it is enforced (or not) separately from the interpreter.
It seems that your point boils down to "Python enforces types at runtime if you define 'enforce types at runtime' not to include 'enforcing that data you declare to be of a certain type actually is of that type'".
Am I the only one baffled by this way of thinking?
Python does and always has enforced types at runtime. This is called duck-typing. If you're not familiar with the concept of strong+dynamic, it is easy to see how this could be confusing. This may help. https://stackoverflow.com/questions/2351190/static-dynamic-v...
Static type annotations by definition are not enforced at runtime. This has been true of every language that has ever used static typing, including C, C++, Java, Rust, Typescript... you name it, statically typed languages only enforce their static typing at compile/analysis time.
Yes, it is possible to annotate types in Python incorrectly. It's possible to do this in all other languages that allow type-unsafe behavior (whether natively or via reflection, etc). This may be less common in some languages than in others, but it is fundamentally possible in the vast majority of languages that perform static typing, because those types are fundamentally enforced at analysis time, not runtime.
The author of the article seems to be unfamiliar with these distinctions, and maybe you are too. It's fair to complain that these distinctions are confusing and make things harder for programmers. Nevertheless, very few languages have ever been designed with a type system that acts exactly the same at analysis time and runtime. It's a very difficult problem, partly because these are all abstractions from the perspective of the underlying computer, which only understands boolean logic and integer/floating point math, and has virtually no other notion of types in any formal sense.
Type systems are a complex topic, and as someone who has been paying attention to them for a while, it's frustrating to see uninformed discussion of them show up on Hacker News. That said, it's perfectly understandable that people are confused by these distinctions, and rest assured there's a lot of effort going into developing better languages that suffer _less_ from these issues.
For the everyday working programmer, however, there are currently lots of tradeoffs to be made, and Python's approach to static+dynamic typing is actually pretty usable, all things considered, which is why the community overall has embraced the new static typing despite its blemishes.
Duck typing is really enforcement of interfaces, not types themselves. Everyone who has passed a string into a Python method expecting a list of strings has run into this. It works, but did you really want to process a character at a time?
Also, some languages like Java do actually enforce types at runtime. If you've ever called a Java method dynamically with the wrong types, you'd know. You can run into ClassCastException, or worse (ClassNotFoundException if you have the wrong classloader.) This is a runtime error.
> Duck typing is really enforcement of interfaces, not types themselves
A type system is just a set of rules that must be followed. The rules can be as weird or as simple as you want. Just because <int> + <string> is allowed doesn't mean it's not enforcing types.
Not really a criticism, just trying to push to expand your idea of what a type system is.
> Yes, it is possible to annotate types in Python incorrectly. It's possible to do this in all other languages that allow type-unsafe behavior (whether natively or via reflection, etc). This may be less common in some languages than in others, but it is fundamentally possible in the vast majority of languages that perform static typing, because those types are fundamentally enforced at analysis time, not runtime.
It is pretty damn hard to make this mistake in languages that enforce static typing. I mean, you would have to go out of your way to do so. In Python, however, it is trivial to write the wrong type signature or to modify the body of a function so that it no longer matches the signature.
"Fundamentally possible but terribly unlikely, as opposed to the Python way of doing it" would be a more accurate description.
I think your (and the article's) argument could be summarized as "static type annotations without automated enforcement by actually running a type checker considered harmful".
Since this argument is not meaningfully different from "comments that are lies considered harmful", it seems fair to expect reasonable people to dismiss it as uninteresting.
It is meaningfully different because, as the article says, comments are known to not be machine verified and thus to potentially be wrong. With type hints, they’re usually verified, but not always, so you’re more likely to trust them and get a nasty surprise when they turn out to be wrong.
However, there are other aspects to the article’s argument, such as the fact that it’s easy to end up with type annotations going silently unenforced even if you do run the checker.
> it seems fair to expect reasonable people to dismiss it as uninteresting.
Do you think that's a fair treatment of someone who disagrees with you but has been respectful of your opinion so far?
I realize proglang debates are flamewar territory. But I have been very careful to state I find Python type hints puzzling and less useful than they should be. It is obviously an interesting opinion shared by many others, not just me or the article's author.
As a fan of statically typed languages, I do indeed find some Python programmers engage in a form of Stockholm's syndrome. It usually takes the form of the assertion "I never found a bug that was a type error". Have you or anyone you know ever said this?
In my mind, the opposite statement of the one you attribute to me would be "wishful thinking and a positive attitude is enough to catch mistakes, it's not necessary to have help from automated tooling". In this day and age, I cannot disagree more.
> I think your (and the article's) argument could be summarized as "static type annotations without automated enforcement by actually running a type checker considered harmful"
It illustrates your mindset, in my opinion. I think type annotations without checking them (or without a satisfying implementation of said type checking) are mostly pointless.
You also left out the part where I remarked on the dismissive tone of your reply.
No, you're not the only one. All of these gymnastics to justify the behavior of the string-declared-int baffle me.
Is it "legally" baffling, i.e., do I understand why this behaves in the way it does purely mechanically, and do I understand the arguments the gymnasts are making? Yes.
But from an idiomatic, colloquial, linguistic, syntactic, programming-historical, or programming-cultural standpoint? Baffled.
Even calling it gradual typing is baffling. It's just adding metadata-that-doesn't-look-like-metadata that sophisticated programs that aren't part of the Python bytecode compiler can use. The definition I know is the same one from Wikipedia:
> Gradual typing is a type system in which some variables and expressions may be given types and the correctness of the typing is checked at compile time [...]
Maybe not by default but there is tooling that makes it work that way — that’s exactly how the python build system works at my company. I don’t know the details but when I “build” python code (creating bytecode pyc files) it produces compiler errors and fails to build if I e.g. try to pass an int to an f(x: str). This has usefully caught bugs before I ran some expensive/time-consuming scripts.
I guess you could say this doesn’t count as it’s a separate program doing the type checking but IMO there’s not a clear distinction between that and e.g. having a first type checking pass in the “same” compiler program.
> While these [type] annotations are available at runtime through the usual __annotations__ attribute, no type checking happens at runtime. Instead, the proposal assumes the existence of a separate off-line type checker which users can run over their source code voluntarily.
Right, it’s definitely still a voluntary feature. What I meant is if you opt in and make it part of your project’s standard build procedure for python code, you effectively get the behavior and benefits of “compile-time type checking” (to the extent that you actually use type annotations in your code).
>> Gradual typing is a type system in which some variables and expressions may be given types and the correctness of the typing is checked at compile time [...]
> This is definitely not what Python is doing.
When I read "compile time", I mentally replace it with "before runtime". There isn't a real compilation step in Python, so it's reasonable to accept static code analysis as part of "compile time" in that definition.
The point is that type annotations will let a static analysis tool make sure that the program is correctly typed. This can be enabled easily in most of the modern IDEs and editors. You could just decide to block the code's deployment in CI/CD if the type checker raises issues, for example.
typically works fine (or worse, works fine in debug mode and causes unpredictable UB-related bugs in release mode).
std::mem::transmute here is a no-op at runtime; all it does is subvert the static type checker.
The difference between statically-typed languages and things like Python with type hints is _not_ that the former has any runtime type checking; it’s that subverting the (static!) type system is more difficult and requires more obvious ceremony, and that static type checking is required whereas with Python it’s optional.
If you hacked rustc to disable static typechecking you would get a similar experience to Python with type hints but without any mypy-like tools. (Of course, this would be hard to do in practice, because Rust, like most statically-typed languages, uses types not just for typechecking but also for code generation and method dispatch.)
Agreed. Static typing is usually enforced during the compile stage in other languages. It's specifically meant to catch problems before running the code (hence the "static" in its name).
This said, I find the linting/type-hint-checking stage of Python cumbersome and confusing :(
I agree with most of your comment, but I think there's a miscommunication problem here.
When Python "enforces types at runtime" this is the actual types, not type hints. Type hints are not a runtime artifact. The "actual type" here is "string", and enforcing it means Python would not allow invalid operations on it without error'ing. A programming language that will let you do basically anything to any value because it doesn't enforce type safety is of course C. Python is safer than C.
It also won't let you call thing.method without thing.method definitely existing.
Best of all, all of these are enforced at compile time.
Now, C absolutely will let you convert an int to a pointer, or a char to an int, or an array to a pointer, or... well, it will let you convert lots of things to lots of things. Some of those things are unsafe unless you are quite sure what you're doing.
> A programming language that will let you do basically anything to any value because it doesn't enforce type safety is of course C.
False on several grounds. You can't call something that isn't a function. You can't call a function on something that doesn't have it. You can't use an int as a pointer or an array without casting it. You can't use an int as a dict (not that C has any built-in idea of what a dict is...)
> Python is safer than C.
Depends on what you're doing, and what kinds of errors you are more prone to.
Sure you can. That's exactly what you do when you call a function returned by `dlopen` -- it might be a function, but it might also just be garbage. It might even be garbage that's coincidentally marked as executable and does some stuff before eventually crashing!
> You can't call a function on something that doesn't have it.
This is also very easy to do -- you can default-initialize a structure containing function pointers, and then call one of them. You'll be calling some random crap on the stack, which might or might not do anything of interest. But either way, C will happily let you do it.
C isn't completely untyped (there are, as you've observed, things it will forbid at compile time), but it's about as weak as a static typing system can be.
I think you're talking past the comment you're replying to.
In both examples, you have tell the compiler exactly what the types are. In the first one you have to cast the return value of `dlsym` to a function pointer, and at that point, as far as the compiler is concerned, it's a function. In the second example you have to explicitly declare the struct containing function pointers, so of course you can call its members.
So it's strange to suggest that C is somewhat untyped; it's not untyped at all. C just makes it very easy to force the compiler to assume different type for a value (that's necessary for low-level code like the examples in the comment you replied to; and of course that makes it very easy to do unsafe things, I don't think anyone disputes that).
And about the "weak typing system" comment, note that nobody agrees on what exactly that means. Just as an example, see how this page[1] defines "weaker" vs "stronger" types, and how it's completely different from the way you're using here. In general, I find people call type systems that don't do what they expect or want "weak", but that's not an useful discriminator: at that point you might just call it "bad typing" to dispel any illusion that it has any objective meaning.
I didn't say it was "somewhat untyped." I said it has a very weak static typing system.
Just because "weak typing" means something different in Haskell doesn't mean it can't be used meaningfully (and beyond "bad") in the context of C. C's typing is historically referred to as "weak" because C's notion of casting doesn't distinguish between type and value transmutation: pointer-to-pointer casts don't convert the referent (because they can't), which in turn gives the C compiler very little leeway in proving that the program's types as declared have any particular meaning at runtime.
Compare this to Python, which is "strongly" typed in the same sense: doing `y = str(x)` on `x` means that `type(y)` actually is `str`, and not merely a promise to the runtime. It's enforced, which makes it strong.
My point about "weak typing" is this: if someone says "hey, I'm designing a new language, its type system is going to be weak", they gave you no information whatsoever about the type system other than maybe "some people on the Internet will probably complain about it".
That's why it's useless to say things like "it's about as weak as a static typing system can be" and "has been historically considered weak".
If you had explained your objection to C's type system (like you have now), I wouldn't have said anything. And I've been on the Internet long enough to have seen C's typing is historically referred to as "weak" because X for *many* values of X, so I disagree that that's the only or even main objection.
You know what I meant and what error I was clarifying for the comment I was replying to. Yet you went out of your way to point all sorts of irrelevant mistakes in my post, when the gist of it was right.
Do you feel it was more important to correct me on these trivial things, or was my reply more or less correct when fixing the conceptual error in this sentence?:
> "In what reasonable sense can Python be said to "enforce types at runtime" here?"
Here's a nitpick of my own to your post:
> "Some of those things are unsafe unless you are quite sure what you're doing."
Wrong. Type (un)safety does not depend on you "being quite sure of what you're doing".
PS: assume I know C and that I've programmed complex things using it. No need to catch yourself with statements like "not that C has any idea of what a dict is". Assume we are both C programmers. It will make you sound more polite.
1. You might need to re-read the site guidelines about assuming good faith.
2. Your last two sentences were, I think, not needed for the point you were making. They were also somewhere between gross generalizations and flat-out wrong, and felt like a gratuitous slam on a language that wasn't even the topic. I thought that deserved a response, even if it wasn't your main point.
3. I may assume that you know C. I don't assume that everyone reading this exchange knows C.
Python can't be said to enforce types at runtime. AFAIK, it doesn't say that. In fact, the original article starts off with "I'm expecting to be able to run a static analyzer", then it goes on to say "This isn't caught at runtime".
Yes, but is it caught at static analyzer run? Yes, I typed it into vi and ran mypy on it and it said "error: Incompatible types in assignment (expression has type "str", variable has type "int")."
But I didn't have to run mypy on it, my editor told me: "Expression of type "Literal['hello']" cannot be assigned to declared type "int"".
Maybe the word "educated" wasn't the right choice in this case, can you offer a better one? I honestly couldn't make it through the entire article, despite several attempts. It is full of inconsistencies and what seem like wilful misunderstandings to justify the authors conclusion.
Particularly as the author starts off saying "run a static analysis tool" as if knowing that's how you go about it, and then saying "it doesn't catch it at runtime".
> In what reasonable sense can Python be said to "enforce types at runtime" here?
Python is both strongly typed and protocol-typed. In the case of `print(...)`, the protocol is that the received parameter responds to `str(...)` (i.e., has `__str__`).
(What Python isn't is statically typed. But "strong" and "static" are completely different typing dimensions.)
> In what reasonable sense can Python be said to "enforce statically declared types at runtime" here?
It can't, because Python doesn't have statically declared types. It's a strong, dynamically typed language.
Python also doesn't allow you to declare static types. It allows you to declare type hints, which are a sort of adjacent and potentially unrealistic (or just plain incorrect) typechecking proof that doesn't correspond at all to runtime behavior.
If your statement were absolutely true then a product like mypy would neither exist nor provide value.
Python has the capacity to have some form of gradual typing built-in at the language level, but it does not, which has led to the enormous confusion we have before us. The SBCL implementation of Common Lisp is a great example of a strong, dynamically typed language that is able to, when possible, detect errors statically at compile time. That tech is 30 years old now.
There is no reason why the Python bytecode compiler could not statically or dynamically detect this programmer error at either compile-time or run-time present in this statement:
x: int = "x"
Compare to Common Lisp, well known for being dynamically typed (so dynamic that it's possible to change the class of an object at run-time!):
(declaim (type integer x))
(defvar x "oops")
; ==>
Unhandled SIMPLE-TYPE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {10005D05B3}>:
Cannot set SYMBOL-VALUE of X to "oops", not of type INTEGER.
quitting
compilation unit aborted
caught 1 fatal ERROR condition
Instead, this job in Python is haphazardly relegated to the program readers and IDE implementers.
> If your statement were absolutely true then a product like mypy would neither exist nor provide value.
I don't see why that would be the case. The value of Mypy is that it's essentially a proof language over something that resembles Python[1]: it's a way to do some amount of static typing without having to maintain a separate copy and representation of the program.
And sure: Python could detect inconsistent type hints at runtime. But why would it? It's already strongly typed, and the presence of type hints usually means that someone is already running Mypy or another checker during development. It's not clear that there's a significant advantage to be gained, particularly one that justifies the additional overhead.
[1]: "Resembles Python" because mypy does not actually evaluate any Python. It doesn't know what types your program has at runtime; it only knows type hints and a few small rules (for things like string literals) and trusts the developer to reconcile those rules with Python's runtime behavior.
Do you think other languages should do the same, because very few do any kind of runtime type checking (e.g. bounds checking) due to the performance penalty. Why focus specifically on python's lack of runtime type checking?
Anecdotally, my code in python is functionally fully typechecked and runtime type checking would solely penalize me by slowing down my code. There's no need for it because the type checker correctly proves the soundness of the program's types, in exactly the same way that Java's or C++'s do.
The point is precisely that you don't need to suffer the costs of runtime type checking if you have a solid typechecker. Lisp's approach here is worse than the one adopted by python/js/ts/go/etc.
What other languages do that though? C doesn't, Java doesn't outside of particular cases. If you can trick or mislead the compiler, the runtime often has no type information at all, so you'll get, at best, a ClassCastException, and more often than not get a segfault.
Python enforces its runtime types at runtime. 'hello' is a perfectly valid argument to print. Python will raise a TypeError if you try to use it to index a list though.
For 1, there's some unfortunate but important semantics. Python does not enforce type _Hints_ at runtime, as Hints are in many ways fancy comments. But as pointed out, python is a strongly typed language (at least on the sliding scale of strong to weak), and types are known at runtime (call `print(type(var))` to see them). And calling `1 + 'a'` will result in a TypeError exception, unlike say JavaScript.
I believe this is the relavent passage in the article the GP is referencing, which is incorrect if you take "runtime" to mean something like when the line is executed: "Python is a dynamically typed language. By definition, you don’t know the real types of variables at runtime."
It _is_ however correct to say you don't know the real type _before_ runtime, at least python does not.
> In what reasonable sense can Python be said to "enforce types at runtime" here?
In [1]: foo: int = 'hello'
In [2]: foo + 1
TypeError
Input In [2], in <cell line: 1>()
----> 1 foo + 1
TypeError: can only concatenate str (not "int") to str
>>> foo: int = 'hello'
>>> type(foo)
<class 'str'>
Python's type annotations are annotations. Types are being enforced at runtime, it's just that some are thinking that annotations have some effect on the interpreter, when they do not.
This is a demonstration that it has not enforced what many would reasonably believe is a static type declaration.
But we have (to quote another commenter) "uneducated" people making a supposedly boneheaded mistake to assume
x: T = y
is a declaration that x will only contain values of type T, or otherwise something will be detected by cpython as invalid (at either bytecode-compile-time or run-time).
Instead, this line is more properly interpreted as "documentation reasonably believed to represent a type saying something about values that x can be assigned at runtime, that's not in a docstring, that some programs may query."
I can totally forgive anybody who believes the former over the latter, and expresses their belief in a statement like that which you've quoted.
Type hints are not static type declarations. Python does not refer to them as such, nor does Mypy or any other typechecker for Python type hints.
Absent of active machine checking (e.g., via mypy), type hints should be considered exactly what you said: another form of documentation, one that might just be wrong.
This is my point precisely: it looks like a static type declaration to anybody who has seen what a static type declaration looks like in any statically or gradually typed program in the last 50 years.
But in Python, it's not (and documented in a PEP as not), and it causes massive confusion.
That's the way it works in Ruby with RBS, and it will work the same way in JavaScript should Microsoft's proposal for type hints make it into ECMAScript[1].
Come on. It’s uneducated because it’s just a naive hot take. The education in this case comes from experience and basic understanding of the underlying issues. The author does not demonstrate possessing either of these attributes.
The "print" function will take any Python object and call its "__str__" method. Not sure if it will do it to a string, but the outcome would be the same.
As far as I remember, the "__str__" method is present on all objects in some form, so print will happily take any type.
Personally, I took exception to the following from TFA:
> Python is a dynamically typed language. By definition, you don’t know the real types of variables at runtime. This is a feature.
In "you don’t know the real types of variables at runtime", this is just flat wrong. One doesn't know the real type of a variable until runtime. It seems to me the author maybe has a bit of confusion between weak typing and dynamic typing here.
I get what they mean though. In a typed language you know the type of a variable at compile time. In dynamically typed languages there is nothing to enforce this and a variable can be passed with any type. So there wording may be off but the point stands.
> One doesn't know the real type of a variable until runtime.
In the program
def x(f):
return f(3)
what is the "real type" of `f` at runtime; say, at the instant just before the call to `f`? How can you tell?
If your answer is that `f` has type "function-like", then that's a much weaker kind of a type than even type hints offer, let alone a real type system.
> what is the "real type" of `f` at runtime; say, at the instant just before the call to `f`? How can you tell?
We can tell by looking at what python triggers TypeErrors for.
>If your answer is that `f` has type "function-like", then that's a much weaker kind of a type than even type hints offer, let alone a real type system.
f implements the Callable interface - meaning it's either a function/method or an instance of a class with a `__call__` method.
So you can pass e.g. `print` or an object with a `__call__` method. If you try to do `x(5)` that's an immediate runtime TypeError because `5` can't be called.
f must also accept one int argument. It can have additional optional arguments, and it can also accept other types (e.g. `f("foo")` can also work!), but it must work on one int. The one int can already be optional - the function can also work with zero arguments.
Calling e.g. `"foo".join(5)` fails with a TypeError because 5 isn't Iterable, and calling `"foo".join([1], [2])` fails with a TypeError because it gets more arguments than expected.
Right, "function-like". I thought OP was meaning the static type of a variable, which genuinely can't be known fully at runtime in general. But it turns out that they meant to write "... until runtime", not "... at runtime", which makes this all a bit moot. And invites more confusion about the two disjoint usages of the word "type" in the PL community, as seen liberally spattered about elsethread, but that can't be helped :-)
> 1. If you're using a dynamic language, then _by definition_ the language will not enforce your static hints at runtime. However, good news! Python has always been strongly typed, and _does_ enforce types at runtime!
So PHP is not a dynamically typed programming language according to you? It enforces type hints at runtime just fine.
There is nothing in the definition of dynamically typed languages that says they can't use type hints to perform runtime checks.
Was going to say the same for some implementations of Common Lisp, notably SBCL, which by default treats type declarations additionally as assertions. Subject to limitations it can use declared types and infer more types at compile time to produce more optimal assembly and give warnings if it detects type errors, or interestingly warnings if it has a chance to use e.g. a fast assembly add on a fixed sized int if you help narrow its inferred type with an explicit declaration one way or another. But unless it can prove a type always holds, it will also by default check declared types at runtime.
Most people I know writing Python don't use type hints because they are too much of a hurdle for very little payoff. The tooling that pays attention to type hints is slow as molasses or difficult to use or understand. The type hints themselves are of dubious use.
As a fan and advocate of static typing, I find myself advocating for type hints anyway, but I must agree I often can't reply anything to my coworkers' objections, because Python type hints are truly not that useful.
> The tooling that pays attention to type hints is slow as molasses or difficult to use or understand.
I use mypy daily on big codebases, it is fine, not fast, but fine.
> The type hints themselves are of dubious use.
They tell you when you make type errors, they help you understand what type things should be. The same as in literally every other language with static typing. There is nothing special here, nothing different. python with a static type checker running in strict mode is not fundementally different from Java's static type checking.
> As a fan and advocate of static typing, I find myself advocating for type hints anyway, but I must agree I often can't reply anything to my coworkers' objections, because Python type hints are truly not that useful.
> The same as in literally every other language with static typing
No, obviously not the same, otherwise I wouldn't be complaining. They are not even on par with Typescript, which I'm not a fan of either.
> [type hints] tell you when you make type errors
Not according to other comments I seem to be getting here. Other people are arguing type hints are not primarily for checking, but a form of notation for documentation. Seems wasteful, and I wish the Python community and tooling decided instead that they are for actually checking them.
> What do they lack that would make them useful?
Standardized, go-to tools that work in the build pipeline and that catch most errors without taking a long time to do so.
I haven't found mypy to fill these requirements. It's so bad I cannot convince my coworkers to make the effort to write more type hints.
> No, obviously not the same, otherwise I wouldn't be complaining.
What is the difference?
> They are not even on par with Typescript, which I'm not a fan of either.
Go is not on par with Typescript or Python, I still don't think it is okay for people to just say fuckit and `interface{}` it all and it is still shit to work with code that does use `interface{}`. At least Python with mypy has null safety, something that Java and Go does not have. There are some places it is worse than other statically typed languages, others where it is better.
> Standardized, go-to tools that work in the build pipeline and that catch most errors without taking a long time to do so.
It is mypy. What actual type errors does mypy not catch that you want it to catch? Why can't you use it in the build pipeline? I do it every day, it catches all the errors it should. The one complaint I can maybe see is the duct type compatibility complaint, and it is not really something that comes up that often for me, and definitely not something I would say invalidates the whole concept.
I'm not going to repeat myself, I already told you.
I find mypy slow, unsatisfying, inconsistent, and it fails to catch many type errors. No, I'm not going to go look in my work laptop to give you an example.
> There are some places it is worse than other statically typed languages, others where it is better.
In most places it is way worse, and I'll find it very hard to find common ground with anyone who disagrees on this.
Feel free to disagree, but I don't find this conversation useful.
How big is your code base? I’ve worked with really large auto generated codes and mypy never took more than a minute or two to run. On more normal sized code bases it’s just a few seconds. It’s also incremental so subsequent runs are faster.
Compare that to a c++ project where it’s not unusual to have 10, 15, 30 min build times.
Or even worse, the time to wait for discovering the problem runtime in prod.
The type hints also help in IDE navigation, where there is zero time to wait, pycharm can go to definition or find references in under a second.
Have you looked at the tools recently? LSP + Pyright is very fast and plugins exist for many editors. Or maybe it's slow on larger codebases or very large files? I don't have that broad an experience yet, but so far it's been very good.
What speed are you looking for in your CI pipline that you are not getting from mypy? Perhaps you are working on a much larger codebase than I am but the app I use it in takes under a minute. And all the unit tests take longer to run.
You're assuming that type hints are by definition static-only. There's no particular reason why type hints can't be both statically and dynamically enforceable, as they are in PHP.
It seems to me like Python needs to get Typescript's capabilities (if Python wants to go further this way of course). It solves all these problems very well, and has no problem with 2 and most of the author's objections.
Typescript does absolutely fantastic type inference, which mypy definitely does not. I think that's a huge advantage for Typescript over mypy, and it's really foundational to the value it provides.
However, even in Typescript it is nontrivial to _annotate_ these complex types, and in most cases it's still possible to do in Python - you just need to commit to using something like dataclasses rather than pretending that what you have is a `dict`.
Basically, to get the most out of static typing in the world of Python, you do have to write more boilerplate than you would in Typescript, and in particular you need to avoid dicts and prefer various sorts of class-based data containers (again, dataclasses, attrs, namedtuples, etc).
You can type Python dicts, see typing.TypedDict. It’s still lacking though, totality is all or nothing until py311, that is to say, TypedDict’s are either Required<T> or Partial<T> in TypeScript parlance. And you don’t get Pick<T>, Omit<T>, etc.
> However, even in Typescript it is nontrivial to _annotate_ these complex types
The only one which stands out to me as non-trivial in TypeScript is thrown exceptions… and even then only because you can only type them in a catch clause, and only with unknown, and they’re invisible to callers. But then this is why people who would care for checked exceptions quite reasonably just use something like a Result type.
But I agree with your conclusion completely, and it’s basically the same one I came to reading the post.
I was curious about how the current state of things compares to TS, as I may soon have a foray back into Python soon, and would be delighted to bring some static types into the mix. Pretty much as I expected, it’ll probably be more effort/less valuable than TS, but more valuable to me than to the author.
I had to come back to Python after working with TypeScript for a while. After going from Python 2.x to TypeScript and being very impressed with static typing, I didn’t want to go back to untyped so I had to get up to speed with the ecosystem. My findings:
— Consider using Pydantic (either its drop-in dataclasses replacement, or its models). It has issues and documentation is lacking in places, but in many use cases it’s a boon when it comes to being confident about data you read and/or pass around.
— Don’t use TypedDict if you foresee needing anything like optional keys, especially with defaults. I went with TypedDict and it came back to bite me later (though was relatively easy to refactor).
— If you use VS Code (which I started after switching to TS), there’re some shenanigans to be aware of. Unlike TypeScript’s elegant approach where you’re in control of installing typing versions, VS Code’s official Python extension will force-install unvetted third-party typings, not caring if they don’t match library versions in your use and not allowing to opt out. In later versions you can unset mypy in Python extension settings, and use a separate Mypy extension that doesn’t do this behind your back and doesn’t send you deep into some rando’s typings when you hit F12 to jump to definition.
— That aside, VS Code can provide development experience is not that far from TypeScript. You can keep a virtualenv nearby and select its Python bin as interpreter in your workspace, giving Mypy access to dependencies (and if VS Code’s interpreter selection GUI resolves symlinks, you can enter interpreter path by hand so that it doesn‘t).
— Sphinx documentation is good at picking up typing hints and works reasonably well with Pydantic’s models, too.
The ability to slowly add typed code to an existing database has been a lifesaver. I always hated JS because of callback hell (and the notions around its community) but since I started working with typescript my mind has been changed. I love it
> If you're using a dynamic language, then _by definition_ the language will not enforce your static hints at runtime.
I don't think this follows by definition. It's common not to check the static hints at runtime, but a dynamic language could choose to treat them as contracts or assertions. Over in Lisp-land, SBCL will enforce static type declarations at runtime [1], contrary to most other Lisp compilers, at least under the default compilation settings (you can force it to skip the checks by compiling with a low "safety" level). If the compiler can prove that a static type declaration always holds, it will omit the runtime check; otherwise it will compile it into a runtime assertion.
> If you're using a dynamic language, then _by definition_ the language will not enforce your static hints at runtime
What definition is that? Raku is a dynamically language with gradual typing which will enforce those types at runtime (if it cannot do so at compile time).
> you can't do this _at all_ in many statically compiled languages
What statically compiled language doesn't provide a way (sometimes clunky) to have heterogeneous values in a dict? You can do that with void* in C, even
> can't fully represent heterogeneous values in a dict, given that you can't do this _at all_ in many statically compiled languages.
This isn't really true. For example, in Java it's Map<K,Object>. For languages without subtyping, existential types are often used. It's quite common to see in infrastructural code, e.g. caching.
Duck typing and heterogeneous dictionaries are (and have been) standard Python.
Adding a type system which doesn’t respect duck typing by trying to access the member even when the types don’t match or which can’t express standard idioms used in Python seems a poor architectural choice.
It’s not saying that Python’s type system is “too much discipline”, but rather that the type system doesn’t encode typical Pythonisms.
I read 0544 and the proposal for protocols fails to support the main benefit of duck typing, as it requires the substituted-for class to be defined as a protocol.
Traditionally, duck typing is used to inject types into a library that isn’t expecting extension at that particular point — eg, substituting a test class for a real class in a data object that normally wouldn’t be a protocol.
I’m not seeing how that PEP addresses that use case.
- - - -
Similarly, the heading on the dictionary says “for a fixed set of keys” — but what if I want a dynamic heterodox dict? Eg, unpacking JSON.
You just end up shoving “any” all over the place. At which point, are the types helping?
Though, protocols do help the dict case for typing:
Your link appears not to work (for me) — can you cite what you believe I have incorrect?
This section seems to agree with me, where it explains why normal classes can’t be subclassed to protocols:
> Now, C is a subtype of Proto, and Proto is a subtype of Base. But C cannot be a subtype of Base (since the latter is not a protocol). This situation would be really weird. In addition, there is an ambiguity about whether attributes of Base should become protocol members of Proto.
I’m not sure why you think it’s “obvious” that you can use a third party library to solve the problem — or why that addresses my complaint that the built-in type system doesn’t work for that.
If anything, the existence of a third party library hints the standard library doesn’t cover the use case.
No offense, but have you actually given it a shot? Duck typing is supported and you can get virtually all use cases of heterogenous dictionaries by using union types and duck typing. The article even touches on this.
edit: To be more precise: The article specifically mentions that accessing invalid members give you a type error, but posits that people will probably not bother and just use 'any' instead. How is that not saying 'too much discipline'?
Yep — I use mixed annotations on my Python code because as the person I’m responding to pointed out, they catch many small type errors. Dataclasses have been awesome.
I’m also generally pro-types, but I think it’s worth having a discussion about this type system in the context of Pythonisms.
- - - - -
If you’re using “any” for most of your types, then it’s not providing value — an untyped statement implicitly has the type “any”.
There’s a reason I brought up the JSON example: loading and manipulating JSON of varying structure is something I do a lot at work.
Sure, but the idea behind gradual typing is that 'Any' should only be temporary. IMO if you're serious about it, stable code should at least pass strict type checking, if not completely forbid even explicit Any.
>There’s a reason I brought up the JSON example: loading and manipulating JSON of varying structure is something I do a lot at work.
Do you have a specific example that you find troublesome? For JSON with a fixed, regular structure (i.e. no subtyping) something like Pydantic works well, otherwise using unions + protocols with custom validation code has covered even my most esoteric use cases so far.
If we're talking about unstable JSON, a recursive union type works well enough in my experience, though defining that type gets a bit ugly if your type checker doesn't directly support recursive types.
Your suggestion to “use a recursive union type” and that being “ugly … if your type checker doesn’t … support recursive types” being required to support varying JSON is my point:
Python types don’t support a common Pythonism used frequently in my work.
In fact, you admit that even fixed JSON can be difficult without a 3rd party library.
You’re lecturing me like I don’t understand types without realizing that you repeated my point about a gap in the current type system.
I think I might not have been clear: this is not a limitation of the type system, but of a specific type checker (mypy), that will hopefully be fixed soon. Both Pyright and Pyre support recursive type aliases right now. Open one of their playgrounds and you'll see that the following, intuitive definition
>In fact, you admit that even fixed JSON can be difficult without a 3rd party library.
Sure, but that's just because the standard library (for now?) doesn't offer deserialization with runtime validation. Java doesn't have that either, if you want to parse JSON you'll have to either roll your own or rely on a third party library.
Don't get me wrong, the situation isn't perfect, but especially if you use the 'right' tooling, you can in my experience already get a perfectly serviceable experience, which to me is indicative that the type system itself is fine and pythonic, it's just the tooling that is lagging a bit behind at the moment.
Your initial statement was that the type system was a poor architectural choice because it doesn't support standard python idioms. To the best of my ability I've tried to communicate that there's an important difference between the type system and the standard library, that one can make it work and that the situation is not worse than most other typed languages.
If you think that's violently agreeing and talking down to you, then there must be some communication barrier that I don't know how to overcome.
A decade or so ago JSON was changed so that JSON text doesn't have to be an object, it can also be a value. 'null' (or say, '5') is therefore legal JSON.
Python's type system is a disappointment, but I don't think this article does a great job of explaining why. My biggest gripes:
1. There is no way to get typing like dataclasses without writing your own mypy plugin. The syntax is simply not expressive enough to do it. This means that if you want to be productive with a library like Pydantic, you also have to add the mypy plugin to your dependencies and add it to your mypy config. Otherwise, no typing.
2. You cannot compose types. There is no way to say "hey this type is the same as this dictionary here, except the keys are all optional." If you're trying to type a RESTful API, your only option is to repeat yourself and carefully keep all your types in sync.
3. To this day, there is still no way to express optional keys. Not "Optional" keys, but keys that can be left out of your dictionary. The closest you can get is this weird hack where you can set `total=False` in your TypedDict, which makes all keys optional.
I really wish Python learned from Typescript, but it's too late at this point. Too many half-baked measures have already made it into the spec.
> There is no way to get typing like dataclasses without writing your own mypy plugin. The syntax is simply not expressive enough to do it. This means that if you want to be productive with a library like Pydantic, you also have to add the mypy plugin to your dependencies and add it to your mypy config. Otherwise, no typing.
> 3. To this day, there is still no way to express optional keys. Not "Optional" keys, but keys that can be left out of your dictionary. The closest you can get is this weird hack where you can set `total=False` in your TypedDict, which makes all keys optional.
> I really wish Python learned from Typescript, but it's too late at this point
Personally, I feel like it's partly just entirely different styles: Python relies on a mix of nominal and duck types, whereas JavaScript is much more focused on structural typing. As a result, there are a lot of things like TypedDict that are absolutely essential for JS to even function but a decent bit less so in idiomatic Python code (keeping a significant amount of dicts around uses much more memory than objects, I recall a PyCon talk explicitly mentioning this too).
Even without NotRequired, you've been able to have some-keys-optional-and-some-keys-required for a long time by putting the required keys in one TypedDict, then adding the optional keys in a subclass with total=False: https://mypy.readthedocs.io/en/stable/more_types.html#mixing...
It's not exactly elegant (hence PEP655), but it works just fine.
I'm a big fan of mypy and static typing as a concrete improvement to Python, but this criticism is so superior to the original article that all I can say is bravo.
Typescript's type system, of course, had and has the advantage of being a clean sheet design with an intermediate compilation step, so the comparison is hardly apples to apples. But there's no denying that I wish frequently that Python could have opted for a more powerful static type system.
Agreed. Also, Mypy's dict inference is too broad, making TypedDict a little less useful. For example, the following code raises a Mypy error even though `foo` conforms to `Foo`:
import typing
class Foo(typing.TypedDict):
a: int
def do_thing(value: Foo) -> None:
return
foo = {"a": 1}
# error: Argument 1 to "do_thing" has incompatible type "Dict[str, int]"; expected "Foo"
do_thing(foo)
This is a misunderstanding of the Python type system. In the Typescript example you are depending on "foo" being structurally compatible to the "Foo" interface.
In the Python case, "foo" and "Foo" might look structurally compatible, but they aren't. "Foo" isn't just a type, it's an object of type "class". You can for example print(Foo), you can't console.log(Foo). "foo" is not an instance of the Foo class. What you are looking for is "dict[str, object]".
TypedDicts fall under structural typing, not nominal typing. So Mypy is cool with this:
import typing
class Foo(typing.TypedDict):
a: int
def do_thing(value: Foo) -> None:
return
foo: Foo = {"a": 1}
do_thing(foo)
My complaint here is that Mypy doesn't go far enough with structural inference. I assume this is because it doesn't support anonymous TypedDicts, whereas TypeScript supports anonymous interfaces/types
If you have a dictionary with heterogenous, optional keys then I believe you have a struct-like class. If you have any constraints on the keys beyond a homogeneous domain and a homogeneous range then you have a struct-like class.
1. It reduces the overhead to use your interface. Your users don't have to import your struct classes, and can just pass in bare dictionaries. If Python's typing was more effective, these dictionaries would also be type checked to make sure you're calling the library's APIs correctly.
2. It deliberately closes off the option of adding state or initialization logic. Classes let you do way more than just encode a struct, and as a codebase grows they have a tendency to become more complex just because they can (like a gas filling its container). This makes APIs harder to maintain and work with. Sometimes, all you need is "raw structs".
While I love the idea behind Python's type hints, they are merely a shadow of the success of TypeScript.
Like the author, I've mostly given up on adding type hints in my Python code. I now only use them when I want to help my IDE find autocomplete suggestions.
Whereas TypeScript was a game changer for JavaScript. I used to hate JavaScript, but somehow TypeScript has become one of my favourite languages! How has the advent of Typing has changed my opinion on these two very similar languages?
- JavaScript without types is a mess, whereas Python comparatively was much better, esp since it does runtime duck type checks.
- Python type hints are much similar to Flow type hints in JS, which I tried, but ditched for the same reasons as Python type hints.
- I was hesitant to try TS's all in approach, cause it was harder to introduce into a project, but after having converted a number of projects to TS, I can see that going all-in is a much better approach than just adding hints as you go.
- TS does checks at many more levels. Eg, if a property is optional or could have different types, it is a syntax error if I don't check the value is valid before use.
- TS does an amazing job of auto-detecting types, so most of the time you don't need to specify types, and it enforces these just as if you declared them.
- TS has reached the critical mass were most popular packages now include type definitions, I very rarely have to add @types/* anymore. This means you get full intelisense on all 3rd packages! I spend a lot less time referring to documentation now!
In hindsight, compiling out types is a great work flow. TypeScript is so good that it has made me enjoy Python less. If there was ever a Python equivalent to TS which reached a critical mass of support, I'd jump all-in in a heartbeat.
This is exactly my experience. I've also found that Python's type annotations for even basic stuff like are way clunkier to write.
For example an optional requires a typing.Optional import or a an ugly "| None" instead of a question mark like TS has. And good luck trying to annotate some complex / nested json, you'll need a bazillion intermediary classes.
There's a subtle difference between TS' question mark and union. The question mark means "this argument is optional", which can be different than "this argument can be undefined".
The following code is valid:
function foo1(value?: number) {}
foo1()
But the following code will raise a type error:
function foo2(value: number | undefined) {}
foo2()
In practice, that rarely becomes problematic. But it's good to know the difference
This conflation of "optional" and "undefined-able" is more obvious in interfaces. This is why TypeScript added the `exactOptionalPropertyTypes` option:
There are many complaints in this discussion about inaccuracies in the post but you've actually captured the essence of the problem--type hints just don't solve the problem well.
My team began using type hints and aggressively applied mypy to new code starting a couple years ago and it _has_ helped immensely. However, we also started doing some work in Typescript around the same time and the difference in developer experience and code quality is pretty clear. Typescript is just a better solution to the problem of adding type information to a dynamic language.
What's especially frustrating is living through the design and early life of asyncio (Guido, you should have adopted the gevent model), then getting excited about typing, only to find it is also not very good. What's _especially_ frustrating is that Typescript existed two years before PEP 484 was published, so the core developers had ample time to seek inspiration elsewhere, and they came up with a sub-standard solution.
Python needs a Typescript analogue, with a compiler that targets vanilla Python.
In my opinion type checking in Python is a lot less necessary, which throws off the cost/benefit calculation. And the cost/benefit ratio also varies with the size of the codebase, team and other things.
Let's emphasis the "gradual" part. The more libraries are carrying type annotations, the more helpful IDEs will be and the easier it is to type-check your own code.
The realities of the browser platform have indeed created a unique environment where many nice languages have appeared that compile to JS. Besides TypeScript there's Elm, ReasonML, ClojureScript, and many more.
"You don’t know if there really is a tool in place to check them."
Every project I interact with that uses type hints runs mypy as part of CI - and I can see that they're setup to do that by looking at the CI configuration (which these days is usually done using GitHub Actions).
I wouldn't add type hints to a project without also configuring mypy for it.
I work on a project that uses type hints but not mypy. To be fair, we used to use mypy, but there was just too much code that it couldn't check (sqlalchemy models, schematics models, some custom models, essentially lots of different data-modelling types!) Maybe it's improved since then, but it just wasn't mature enough at the time.
However, type annotations still make the code so much easier to work with, because they serve as documentation and are understood by the IDE. When I hit "." I actually get correct autocompletion because of the type annotations. When I want to know what a function returns I don't need to dig through multiple levels of function calls.
The article seems to be suggesting that because the type annotations might be wrong that they are worthless, which seems like a ridiculous argument. Any documentation can be wrong, but clearly having documentation that is 99% accurate is better than not having it at all.
I've found that it's pretty easy to slowly enable mypy checking. You can mark modules with "type:ignore" to make mypy ignore them. Then, you can move to marking individual statements with "type:ignore". Finally, you can start marking modules with "mypy: disallow-any-generics" at the top. That requires that functions are annotated with types.
In a large code base, our team has found type annotations and mypy to be quite helpful. If you are writing a quick-and-dirty script, forget about wasting time with type annotations. However, for long-lived and important code, it seems to be worth adding the types.
Another thing I noticed, if your type annotations are hard to write, e.g. lots "Union", "Optional" or deeply nested type expressions, it's probably a sign that your code is poorly designed. Think about what types you actually want to consume and return. Define new containers using dataclasses or attrs to keep the types simple, if required.
Right - I mean we had mypy passing across the whole project, but it just wasn't detecting any issues because so many of our types had to be ignored because mypy didn't understand that the runtime type of a Column field in SQLAlchemy is not Column, and things like that.
Rather than maintain all of the mypy stubs for no benefit, we decided to just scrap mypy.
> You can mark modules with "type:ignore" to make mypy ignore them.
If I added MyPy to the CI/CD checks, some coworkers would flood the code with these to not have to think about their typing. Too bad that some libraries like SQLAlchemy make this mandatory sometimes. I'd like the option to ignore these "type:ignore", but the type system is too immature and has to be overridden sometimes.
Touché. I guess a common pattern I find myself doing for Union types is having if statements checking if something is of a type using reflection. When checking if it is none, it is more elegant.
There are still a lot of libraries without hints, so mypy has to be told to ignore them in the setup.cfg file. Last I checked, sqlalchemy still isn't so good with mypy. So we have it ignored. But mypy is still great overall.
MyPy isn’t the only checker of course, there’s also Microsoft’s Pyright, among others. Pyright is fast and natively integrated in Microsoft’s Pylance VSCode extension, so these days I use Pyright’s type checking in real time even when I can’t be bothered to set up MyPy (no difficult, just not important for, say, <1000 line scripts). I default to strict mode for stuff that’s not too dynamic.
We use pre-commit and you can't even commit until it passes mypy. It can be a bit frustrating sometimes, but overall it has saved us from a lot of issues.
I like the validation scripts being available in a repo so I can run them locally before pushing to CI, but I also often use WIP commits and quick fixups and then rebase before opening the PR, so pre-commit hooks are really annoying to my workflow. I more often than not just do `git commit --no-verify` or simply delete the git hook in `.git`, then just run it myself before pushing. Anyway CI will catch it, so I don't see much value in forcing it on individual machines.
This has been my experience as well, as someone who added a pre-commit hook (black and mypy) to a repo in lieu of adding it to CI (it was a while ago). I came back to the team a year later to find that everyone had simply disabled the hook, and forgone typing/formatting entirely. A day or two of fixing formatting, type hints and linting errors later, the CI pipeline had a new step enforcing all three :)
Our CI pipeline actually just runs `pre-commit run -all` on the repo too. So we know if someone doesn't have it turned on. But I could definitely see taking mypy out of pre-commit, and having it just be a CI step. Rather than waiting the minute on the commit to happen.
Yeah, when you just want to add a checkpoint WIP commit its annoying. Hate having to fight with that. But it prevents people form submitting a PR and then just having CI fail anyways. I could see it being nice to only enforce on CI though. Sucks seeing those red builds because someone didn't run something manually.
I don't see it as a big issue as we just assume PRs with failing builds not to be ready for review and push them back to draft. But whatever works for your team, honestly. In the end what matters is having a CI acting as authority source, the rest is simply workflow related.
Yeah, think its just a difference in mindset. Some teams strive to never have failed builds, while others don't care because its all part of the process. I think we were more in the mindset of you should never push a failing build initially, but have moved more towards its all part of the process.
Yes. The reason type checking is separate from the interpreter is that python is an ecosystem, not a language. Even if you like static typing, 99% of projects benefit from other code that lacks typing. Type hinting puts you in control of where types are enforced while allowing you to use duck-typed code. The cost is that you have an extra step in your build chain - but CI, IDEs and other automation all but negate that cost.
Only facts here. I think most large codebases eventually see type hints drift away from reality as individual contributors are more incentivized to hack in `Any` types to make things compile instead of typing every line properly. This is especially common for handling data objects which come in over the network - often they can have a couple different types but people just type it as one thing for simplicity.
Overall I do think type hints are worth it though, for maybe two benefits. They force you to look at the ridiculous types you are using, like `List[Union[None, str, List[Dict[str, str]]]]` which is the sort of thing that happens in codebases all the time. It adds just enough friction to push people to make explicit dataclasses or simplify function returns, which is good. Secondly they help with tracking functions which return None, which is a pain when following callstacks in big codebases.
My understanding was large python code bases (think Google) have a large problem. Someone makes a code change and suddenly it becomes difficult to find the scope of type errors in their monorepo. That was the driving force IIRC.
That said, I think pytype makes a lot of sense since it infers types from code, which you can edit by hand and then merge back into the python file when your code is stable.
> But as a general rule: You don’t know if there really is a tool in place to check them. You especially don’t know this when you join a new project, but it’s also a little hard to tell afterwards. Did the mypy task in your build pipeline silently break? Maybe mypy is misconfigured? Maybe it spits out errors but does not cause build jobs to actually fail? What if mypy has a bug? What if mypy is not even complete and doesn’t cover all cases?
This seems like an extreme overstatement and a sign of a very weird team environment.
Are you not running your linting and validation locally?
Heck, when dealing with some poorly-typed external code, I run mypy manually to help me figure out what the stupid types I need to assign even are. This is far from ideal, but in lieu of a better solution, it works for me.
I just find the idea that people would be adopting typed Python and then completely ignoring the type-related tooling a bit wild.
And sure, libraries can have bugs, you can open yourself up to big holes with Any... and in cases where I'm super worried about things like that, I'm generally not using Python. But I'd rather use Python with type hints + use the tooling in a sane way then not use the type hints at all.
> Even if the Python runtime did check all the type hints at runtime, then it would still be too late. I don’t want a fancy type exception at runtime. That already exists (most of the time). I want to know about type mismatches in advance.
You should check out Typeguard [1], which lets you add a @type_checked decorator to anything and get runtime type checking that's far better than e.g. a random blowup when you split() on an integer. It does introduce a significant amount of overhead, but it's still useful during development alongside type hints to avoid some of the pitfalls from this article.
In another vein, Any should definitely be used sparingly. For the foo() function outlined in the article, Union[str, int] offers more precision. In fact, you could even argue that using Any is a code smell.
Overall, though, I agree with most of the sentiment in this article. I find myself using type hints more as documentation than anything else, since their true ability to prevent type-related runtime bugs is very limited. Documentation has the same qualms the author outlines, anyway:
> You can not be sure that they are correct. [...] They waste mental energy when reading code, they create new maintenance burdens, and they are potentially deceiving: You cannot trust them.
But they're still slightly better than docstrings...
Type inference has been around (as in "used in programming languages") since the early 1970s.
In the beginning python hit a sweet spot: glue code that wasn't exactly write once, too difficult to do in bash, and too small to go all the way and use C++.
However, it was not without its own design issues.
Do you know if there is a way to do runtime type checking in the whole program with typeguard, beartype or something else? As far as I know you have to go through and add decorators manually. Typeguard had a profiler hook that almost got it right, but is being removed. Ideally I would want to say 'python3 -m typecheck myprogram.py' and it would run typechecking everything in my code (but maybe not in library code).
MyPy is great but it checks "offline" or "at compile time" if you know what I mean. It is like a super linter, and doesn't actually run the program (AFAIK).
What I'm looking for is more like a debugger or profiler. It actually runs the program, and then reports if at any time the type annotations deviate from the actual types. (I guess the general case is too costly - think of typechecking a huge list - but for most cases it should be possible.) There are runtime type checkers for python, but the all require modifying the code, and they don't work properly on the main module.
Type hints work fine. They are hints (that are checkable through tools)
> But as a general rule: You don’t know if there really is a tool in place to check them.
Cool. So you just go and do stupid stuff if no one is looking?
Check it yourself. They're probably there for a reason. If you think they're not needed, then sure, take them out.
> Most of this turned out to be wrong and it threw me off the track during my debugging session.
Here's an idea: Then why don't you fix it. If I see code that's wrong or bad, I go and fix it. It's your responsibility as well
Now maybe try acting like they're anything but type hints (and I'm very glad Guido insisted it is optional and flexible - the last thing the world needed is for Python to get consumed by Java type checking pedantry where the compiler needs you to annotate every single thing)
And every time I do something in Python that would be "impossible" in Java I'm glad I'm not bound by the small-minded type pedantry of Java.
Don't let perfect be the enemy of good. Type hints are usefull especially in large codebases. I find myself using them with minimal effort and they can catch bugs that otherwise wouldn't be catched.
Exactly. The article has some valid criticisms of the weirdness of type hinting, but type hinting is a pretty large paradigm shift added to a 20+ year old language. Yes it will have shortcomings, but in my opinion has made working with large Python codebases actually possible. It’s a good thing.
The way to think about gradual type annotations is not like types in C. You're not saying "this variable is 8 bytes that we're going to interpret as a floating point number." What they do is let you express constraints on the behavior of your program, and then run another program to check whether those constraints have been violated.
Suppose you start with the function
def foo(bar, baz):
if bar > baz > 0:
return bar / baz
return "bar is not bigger"
And this is what you want the function to do, for some reason. But by the time you start writing other code, you've forgotten about the unusual "bar is not bigger" case, and you just assume you're always getting a number back. This could turn into a runtime exception somewhere else in your code because a field in some object somewhere now has a value of the wrong type, but you've never run into it so you don't know about the ticking time bomb.
Now you start adding type annotations to foo:
def foo(bar: float, baz: float) -> float:
...
And when you check the types, mypy yells at you, and you realize you actually need to write this:
At this point mypy starts informing you about all the places you failed to handle the "bar is bigger" return value, and you go and fix them. You keep going until all of the constraints you've put on your code are consistent. You don't have to completely type everything for this to be useful, nor is it particularly important to check this stuff at runtime, because all mypy is doing is telling you that the code you've written satisfies the assumptions you're telling it about.
I love type hints. They make the code so much more readable. I wish they could do more, but knowing what the programmer intended for a variable is huge. Now, when I see code without type hints I think, "Oh man, now I have to dig into everything to know what anything is."
Yeah this is such a huge benefit that’s rarely discussed. It wildly increases the understanding of a code base which makes developers more productive and safe
That's about what I said when the idea was first proposed. Actual type declarations, both enforced and used to guide code generation and optimization, would be fine.
Part of the problem is that, in the minds of many, Python == CPython. PyPy, which is a real compiler, is viewed as "nonstandard". For PyPy, type information, although not in type hint form, would be useful. But type hints, as currently defined, are not. See the commentary on type hints at [1].
This article has a really strange tone. For example, there is this section
"There is an Any type and it renders everything useless". Yeah, so does C with (void*). Just because somebody is abusing a tool doesn't mean the tool is wrong. Any is valid with dealing with a dictionary type that is deserialized from some file. If somebody is just injecting Any into their code because they don't know what they are doing, educate them.
The takeaway that type hints are worse than doing nothing just doesn't comport with my experience where the type hinting has caught very legitimate subtle bugs one code that was infrequently used. I wonder if the author would make the same argument for TSAN, UBSAN, or ASAN? Coverity? Or other annotation systems that help outside of the compiler? These tools are godsends, but do have false positives if their scope is understood.
Not sure what the compiled != easy to use bit comes from anyways. If anything I'd say that part of Nim makes it easier to use, I don't have to worry about whether or not the target has Python installed.
The test for PyCharm inspections only passes when there are no warnings.
Although, I have to admit, we explicitly exclude type warnings because here we have a couple of false positives. So in this respect, it actually agrees with the article.
But then we also do code review and there we are strict about having it all correct.
Yes, I see the argument of the article that the typing in Python is not perfect and you can easily fool it if you want, so you cannot 100% trust the types. But given good standard practice, it will only rarely happen that the type is not as expected and typing helps a lot. And IDE type warnings, or mypy checks still are useful tools and catch bugs for you, just not maybe 100% of all typing bugs but still maybe 80% of them or so.
> Isn’t it better to detect at least some errors than to detect none at all?
> You can not be sure that they are correct. As such, you must always treat them as if they were wrong.
I don't get this argument. Isn't this the case for all other code as well? Most code has not been formally verified to work 100% correct. So you assume always all code is wrong? This doesn't make sense.
> They [type hints] waste mental energy when reading code
How? The author even acknowledges that you could just treat them as code comments if you like. By that argument, all code comments waste mental energy?
> I don't get this argument. Isn't this the case for all other code as well? Most code has not been formally verified to work 100% correct. So you assume always all code is wrong? This doesn't make sense.
It isn't that you have to assume the code is wrong (you always have to assume that code is wrong). It's that even though you write type hints and they pass mypy, you still have to assume that the code has type errors. That is, mypy doesn't rigorously check that the program's types match its annotations. It only approximately checks that.
In (some) other languages, the type checker is 100% sound, so if your program passes type checking, you can be completely sure that the types all match. Opportunities for the code to be wrong are thus limited to mismatches between the types and the desired behaviour.
A lot of people saying author isn't smart/educated enough about the topic, but I kinda agree with him, that they are disappointing. From a "normal" programmers point of view, they are weak and don't really safeguard you well enough out of the box.
Apparently you can install and setup a billion 3rd party libraries in your code editor and a CI pipeline to get a really good workflow, but that just shows that you need to be an expert in order to use these.
I've annotated Python typing with a 20/80 approach.
I annotate strings, integers, floats, simple list and dict.
I don't try to produce complex type specs for lists, dicts, functions.
My company runs some kind of static analysis that checks things.
And I get 80% of the benefit with 20% of the work.
So start small, and don't expect perfection (yet).
Hot take: If you think for a moment about the feature name, "type hints", many (all) of the objections the author has become clear.
It is not called "static typing". It's called "type hinting". Based largely on the name, I wouldn't expect it to error at runtime if you tell me 'hello' is an int. That's just a type hint. I WOULD expect a static analysis to flag is, and mypy indeed does.
Annotating certainly sounds stronger than hinting, but one must dive in and read the PEP to see they in fact do nothing except stash away metadata in a dunder attribute.
I agree. I'm glad that they're optional. One of the main reasons to use Python is to prioritize development speed over performance AND to prioritize read/write-ability over hand jamming mundane syntax. I understand why some who have a background in typed languages might prefer to use Python with type hints, but it should be understood that they aren't very Pythonic.
> I understand why some who have a background in typed languages might prefer to use Python with type hints, but it should be understood that they aren't very Pythonic.
Completely disagree. The Python community has very rapidly adapted mypy because of widespread recognition that yes, type information is extremely helpful for any code bases larger than a few files and/or worked on by more than a few developers. Every major Python library I can think of now has mypy stubs available. If you're going to dismiss them as "not Pythonic" you may as well dismiss anything other than Python 2.7 as "not Pythonic".
Man, the idea that type hints reduce read-ability are crazy to me. It's like if someone told me that they think comments make code less readable.
Like, even if you think that type hints have ugly syntax, prior to them vast most projects just didn't bother with documenting the shape of data, so you had to read source code, guess and experiment. Unless you just don't care about knowing what you're specifically working with (which strikes me as living on the edge), how is it not an ergonomics gain?
> main reasons to use Python is to prioritize development speed
My experience with Python is that it slows development speed to a crawl due to dynamic typing. It’s maybe faster to write the first 1000 lines. But after that it becomes slower and more painful. At least in my experience.
> I understand why some who have a background in typed languages might prefer to use Python with type hints, but it should be understood that they aren't very Pythonic.
Okay, this myth that Pythonic == loosey-goosey duck-type-everything slinging dicts and strings nothing static cause that's too slow, really needs to die.
Pythonic is all about pragmatism and parsimony. If that means not typing anything cause it's a 50 line script, do that. If it means the most efficient way for large dev teams to communicate on sprawling code bases is to use rich types, then do that.
Personally, I use types even in the short scripts, because then I can offload keeping track of what type everything is, instead of having to remember yet another user_dict with whatever keys the producer felt like using at the time. It makes autocomplete faster and I'm less mentally fatigued. Less mental fatigue == more pythonic.
Code is way less readable without types. If I see this:
def create_user(user):
other_function(user)
Then how do I know what `user` is? Is it a dictionary? An object? If it's an object, where's its class definition? To find the answer I need to search for all the code that calls `create_user`, and then code that calls that code, and then code that calls that code, ad nauseam.
Onboarding into a huge, untyped codebase is brutal
Opinions like these are frustrating because, although they make valid points, it comes off as "they didn't do 100% exactly what I want, so I'm not using them at all." With many tools, there is a middle ground between using them for everything, and not using them at all. Python's type hints is one of those tools.
When I'm writing code, and a variable or signature is easy to annotate, I'll annotate it. And guess what? MyPy occasionally catches issues with my types and saves me some work. If a type is very complex, I won't bother spending time annotating. The end result is that I have some code that is annotated and prevents simple errors, and other code that isn't, and I didn't spend much time or effort doing it. To me, that's a clear net win.
Fully agree with this. Type hinting (not checking) is idiomatic to Python. MyPy is a great way to enforce checking. It's great to have the option to use the hints for intellisense only.
Honest question: what's the point of type hinting without type checking?
I must be mistaken, but I always thought "type hinting" was a synonym of "optional type annotations". But if you're not using those annotations for actual type checking at some point, what are they good for?
I view them as similar to python's "private" functions, which are really just functions starting with an underscore. The interpreter will let anyone call them like any other function, but the general rule is don't do it, unless you know what you're doing and are willing to deal with the internals changing.
Python typing is like that. If I say a function takes a List[int], but you know I'm just calling a for loop, you can ignore it and hand me a Set[int]. Maybe it breaks some day, but you're allowed to take that risk, if you have a need.
Do either of these things do anything comments couldn't? Not really. But they're ways of indicating intended semantics without formally documenting your stuff, and when most of what you use the language for is scripts, that's actually pretty helpful. People actually use type hints in a way they didn't with comments, and that's caused a major improvement in code readability. Or at least that's been my experience
You can absolutely look at all this and say python's a crazy, terrible language you never want to touch, but if you've got no choice on language for whatever reason, or you're throwing something together and don't feel like writing `private static final synchronized` forty time, type hints are great.
> I view them as similar to python's "private" functions, which are really just functions starting with an underscore
Good analogy! I wish they were more like "private" class methods, which are prefixed by a double underscore and result in name-mangling: you can still access them from outside if you want, but the code will really look ugly. And you absolutely cannot access them accidentally.
Which is what type checking should be all about, right? Preventing accidental misuse?
> Python typing is like that. If I say a function takes a List[int], but you know I'm just calling a for loop, you can ignore it and hand me a Set[int]. Maybe it breaks some day, but you're allowed to take that risk, if you have a need.
That's what drives me crazy. I come from the statically typed world. This bit of Python's philosophy really clashes with my world view. "These types are just something someone wrote, they may or may not accurately describe the code" seems so wasteful and unhelpful to me...
> That's what drives me crazy. I come from the statically typed world. This bit of Python's philosophy really clashes with my world view.
That’s the kind of the culture shock you get learning dynamic type languages with a static type pov. Type hinting is not really the problem here, but can seem that way because it makes dynamically typed code too superficially similar that it enters the uncanny valley if you treat it like statically typed.
One way that might make this easier is to forget about type hinting entirely at first and learn the language “from scratch”, and add the types back after you get used to write dynamically typed code. Dynamically typed code have its advantage and isn’t bad without type hints (well, at least you have to convince yourself on this, or you’ll never be able to learn a dynamic type language), and the type hints just add back some of the nice things static type provides without compromising dynamic type benefits.
Agreed, but it also clashes with the world view that brought us Typescript, which I also find better than Python's type hinting.
In addition, it makes it harder for me to argue in favor of type annotations with my coworkers. My coworkers come from neither world, static or dynamic; they are learning the ropes. And they just can't see the point. I'm confident I would convince them were this a statically typed language, but with Python I'm lost.
The TypeScript comparison is on an entirely different axis though, the compiled <-> interpreted one. One of the reasons languages go with the type hinting route is actually to avoid a separate compilation step like TypeScript. That may be totally fine for you (again coming from a statically typed language where this is more or less required), but it’s a hill many are willing to die on. It’s all tradeoffs and design decisions.
Agreed it's a different axis, that's why I said "also clashes". I of course think Typescript's choice in this tradeoff was the better one, even though I'm not particularly in love with Typescript either.
At some point everything in proglang is preferences and design decisions, but for me, there are better and worse tradeoffs to make.
The typing (>= python 3.6?) and collections (>= python 3.8?) packages have definitions for a bunch of protocols (basically interfaces for structural typing).
So for that List[int] example, you probably want to take a(n) Iterable[int], Iterator[int], or Collection[int] instead, depending on exactly how you use it.
> don't feel like writing `private static final synchronized` forty time, type hints are great.
We should think about whether there's a reason why we want to be writing `private static final synchronized` over and over again, after looking at the state of the software engineer these days.
It is good for programmers. Programmers think in many different ways, but it seems to me that a sizable portion of programmers think of code in terms types. Type hinting makes it easy to convey type ideas. Where are we going from which type.
I should say that anecdotally I find type hinting very useful when I'm reviewing a PR from a part of the code I'm not intensely familiar with.
when types are not checked and enforced, they get out of date just like comments and docstrings.
i agree with the author, if the hints can't be trusted, even just once, there's no point littering the code with them, and are actually harmful when trying to debug something using faulty hints.
i am a big fan of static typing but not in python. pick a language that was designed around it. you can't make a duck bark.
For libraries, with many consumers of the code, type hinting seems like a good thing. Many times I download a third-party library and waste tons of time looking at sample code. For application code (internal to the service or whatever), it feels less valuable.
They are useful for catching issue before code is committed or deployed. At my work we run mypy as part of our test suite, so failing type checks will block a merge or deploy.
How will they catch issues if they are not checked? The comment I'm replying to insinuated type hinting is just for documentation, and not necessarily meant to be actually checked by the tooling.
You seem to be describing actual type checking, which I understand (though in my opinion, mypy is not a satisfying tool for this).
I wish the tooling outside IDEs worked better. There's obviously some magic going on with tools like PyCharm and VSCode + plugins.
I wish this was the case with command line tools I can plug into the build/deploy pipeline. I know they exist, it's just they are unsatisfying and there are lots of cases where they miss stuff that is trivial to catch in statically typed languages.
As someone who lives in an IDE, I don't understand this. Trying to get all of the functionality of the default IDE, along with the trivially added plugins, to work in a visually digestible and sane way would be a curses nightmare of 30 command line tools. If you made it so they played well together, where a human could interpret what they were seeing, you would have something indistinguishable from an IDE.
It's simply not true. Every statically typed language works like I described. Even Typescript works like this. You can have your IDE plugins, but you definitely don't need them to perform type checks. It's not true this requires "a nightmare of 30 command line tools", you just need one: the type checker (built into the language in most statically typed languages, but sometimes split into a separate tool).
Besides, your IDE doesn't live in the build pipeline (in your CI/CD tool). So you cannot rely on it.
as a reminder for yourself/others who need to read/maintain your code?
(I am often reminded of my perl days, where something I thought idiomatic 3 days ago, is now completely incomprehensible when I just want to make a minor change)
> So is it simply a standardized comment then? I will have a really bad time convincing my team mates, if that's the case.
If your team doesn't understand the value of things that are essentially standardized comments exposed through the IDE (even if it wasn't for validation, which is also available with IDE integration), you have very bad teammates.
As I said in another reply to a comment of yours: maybe so. I cannot do anything about my team mates, but I can push for tools that make the upside of their usage more evident. "Standardized comments", when working with inexperienced/resistant team mates, is not such obviously good tool.
I know because I tried setting it up. It's slow, requires too much babysitting, and fails to catch cases.
In my experience, it just doesn't work cleanly out of the box like in true statically typed languages, and so I get pushback from my coworkers, who simply can't see the point. And I can't blame them.
You get me wrong: my coworkers struggle to see the point of testing at all.
Usually they can be sold on the path of maximum-reward-for-minimum-effort. Hard to enforce/check type hints are not it (they seem like busywork for no actual payoff). With statically typed languages, the ROI would be different: they would either get the thing to compile, or they wouldn't and leave the job.
This sounds drastic but it's really not: outside the realm of purist conversations between fans of programming languages, people just want to do their job and get it over with. If the tooling is convoluted, has too many rules, or they aren't forced to use it ("...or else"), they just won't.
This is a javascript/python shop, by the way. We cannot choose the languages. We can improve the tooling and train the team with better practices though.
> If you/your team want to use a statically typed language, then use one. Python is not it.
This is a bit of a cop out though, isn't it? A response to criticism of a (possibly) flawed language feature cannot really be "well, use another language". How else will the language improve then?
That question hides the assumption that static languages are always an improvement over dynamic ones, when in reality we should think as different species that have adapted to fit into different environments.
> We can improve the tooling and train the team with better practices though.
"Better practices" are what makes you team more productive, not just blindly copying what other people are doing. If your colleagues really believe that automated tests/type checking are not worth the effort, you are not going to convince them by saying "but so-and-so said otherwise". What you can do is ask for their pain points, and see if the tooling can help with it. You can look at your past burn charts and say "look at this bug here, what caused and why did it take so long to fix? Is this the type of problem that would be easier to solve with static analysis of the code? Look at this refactor that we are planning for next quarter, should we try to increase the test coverage to make sure we have more confidence in the changes?"
> That question hides the assumption that static languages are always an improvement over dynamic ones, when in reality we should think as different species that have adapted to fit into different environments.
Let me give you a twofold answer to this.
1. I do think statically typed languages are generally better than dynamic languages for most use cases (with a few exceptions). I don't expect to convince you or anyone else of this. It may even be the case that I'm mistaken about this, but I made my mind after years of experience with both kinds of languages.
2. The alleged hidden assumption is not truly important. One should always criticize flawed features of any language, static or dynamic, and the answer should never be "well, choose another language". How else will languages improve if nobody is working on addressing their pain points?
> "Better practices" are what makes you team more productive, not just blindly copying what other people are doing. If your colleagues really believe that automated tests/type checking are not worth the effort, you are not going to convince them by saying "but so-and-so said otherwise". What you can do is ask for their pain points, and see if the tooling can help with it.
I'm both nodding in agreement and finding it very hard to think of something I said that made you think I disagreed with this.
> generally better than dynamic languages for most use cases
Shouldn't then the exercise be to figure out if these use cases are applicable to your situation?
What is your team and company optimizing for?
> How else will languages improve if nobody is working on addressing their pain points?
"See, at my work we need to travel between two islands as fast and cheap as possible. My team is used to high-speed motorboats, but my experience tells me that hydroplanes are faster. I tried putting wings on my motorboats, but it still wasn't as fast. No, we can not buy hydroplanes. No, my team is not licensed to fly. But if no one acknowledges that motorboats are slower, how will they ever be as fast as a hydroplane?
And no, I haven't really looked into the fuel costs of planes vs boats..."
The problem I see is that you are trying to peg a square in a round role and thinking that the square is at fault for not being flexible enough.
Anyone that has seen the py2 -> py3 debacle will tell you that "let's make python static" is not something that could happen unless you are willing to rewrite the entire ecosystem of libraries and applications, and quite possibly upsetting the majority of current users who were attracted to it in the first place precisely because it is so easy to get started with it.
You can not turn Python into a "fully static" language without changing it so much to the point of making into a different beast. And why should others make all this work to fit into your view of what is "best" when you can just use another language in the first place?
I don't want to turn Python into a statically typed language. I think there's valid criticism to be made about the flaws of its type hinting (as does the article's author), and I cannot help but compare its usefulness to static type checking.
I also cannot choose the language. I'm not in a position to choose languages at my current job; I seldom find myself in that position at any job.
We are going to go back in circles, but as I said in the very first comment: mypy is the tool you want [0]. It won't do everything, but progress has been steady and more and more projects are providing type libraries for it.
But given that your response to the mypy recommendation was to complain (it's slow, it doesn't catch everything) then it makes it difficult to acknowledge the "valid criticism"... as in: people are working on it, what else do you want?
> "We are going to go back in circles, but as I said in the very first comment: mypy is the tool you want [0]"
What makes you think I'm not using mypy, or that I didn't read the article you linked to or follow its guidelines (which are very sensible)?
In your mind, is it the only possibility when someone criticizes a tool that they are "using it wrong" (to paraphrase Steve Jobs)? No other possible reason?
> "But given that your response to the mypy recommendation was to complain (it's slow, it doesn't catch everything) then it makes it difficult to acknowledge the "valid criticism"... as in: people are working on it, what else do you want?"
Given that I didn't say or imply that I don't believe there are people working on improving mypy and Python type hints, "what I want" is merely to support the article's assertion that the current version of type hints Python provides is confusing and not very good.
Please, for the love of all that is holy, don't recommend to me again that I switch to a different language. That's not how this works.
> "what I want" is merely to support the article's assertion that the current version of type hints Python provides is confusing and not very good.
What I am having trouble to grasp is: if you understand that Python is not a statically typed language, and if you claim that you do not want to "turn" Python into such, why do you keep conflating type hinting with type checking?
They are two separate things. That is the nature of the game. Do you have criticisms about limitations from mypy, fine, but then you are not talking about issues with type hinting but merely the type checker tool.
> "What I am having trouble to grasp is [...] why do you keep conflating type hinting with type checking?"
It's not my business to help you with your trouble grasping things, but why do you think I'm conflating type hinting with type checking? I specifically asked in this comments section what type hinting meant if not what I thought, and was told by several people "it's just standardized comments", which I then explained was unsatisfying (and other people agreed with me).
> "you are not talking about issues with type hinting but merely the type checker tool"
How about both?
Let me suggest you a more productive use of your time: address the article's complaints as a top-level comment (which I see you haven't yet). I suppose you're more interested in the article's topic than in correcting my alleged misconceptions?
Because the "issues" with type hinting that you are talking about are only based on what you wished it was, not on what it is or what it could become!
Ask yourself this: what about the python's type hinting story you think could be improved without turning python into a statically typed language? What about the python's type checking story that is missing or broken and that is attributable to the language and not to the tool?
> I suppose you're more interested in the article's topic.
Quite frankly, no. My motivation for the conversation now is mostly to see how long is going to take you to realize that you are merely wishing that your motorboat could fly like a hydroplane. It's not about "misconceptions", it's about you complaining about something not meeting your unfounded expectations .
> it just doesn't work cleanly out of the box like in true statically typed languages
So the response is entirely appropriate. Python is not a statically typed language, of course type hints don't make it behave exactly like a statically typed language.
Fair enough. I can see how my statement was not very clear.
I meant to say this: because Python's type hints don't work as well as true statically typed languages (based on my own experience), this makes them less useful and also makes them feel more like busywork. Because of this, my coworkers (who have no experience with statically typed languages either, and tend to dislike best practices and features unless they see a clear return of value from them) are hard to convince type hints are worth the time to learn and use them.
Also, let me restate we cannot pick the language. We can just make better or worse use of it.
it's a comment which can trigger an action (if you run mypy).
So, unlike a comment, which does nothing, if you run mypy and get some warnings, you can then do something about it (whether it's just # type: ignore or you realise you made a mistake with allocating to variable new_value versus newvalue/new_valu/etc)
(btw, in case someone objects, some comments DO do something, e.g. golang's godoc examples)
Now I'm confused because the root comment I was replying to asserted that type hints are for documenting, not for checking.
If they are for checking, then my other opinion stands: that they are not very good at it (compared to my experience with statically typed languages, even with Typescript!).
They are for documenting and checking. Obviously, if static typing is your only concern, you wouldn't have picked Python to start with, but if you did have a reason to pick Python, you can get a large subset of the value of static typing for correctness and dev-tine information via Python’s support for type hints and the typecheckers (mypy keeps getting pointed to, but it's not the only one) that support it.
Yes, obviously. There are outside factors why Python was chosen. I tend to work with whatever language is needed, and try to make the best of it.
I've tried mypy and pytype, and found both to be unsatisfying and hard to sell to the team (which as I mentioned, don't even like writing tests). If you're thinking "well, that's a culture/professionalism rather than a technical problem", you are absolutely right! But I'm left wondering if a language that was better at static typing would be better because, a- the checking would be mandatory, and b- it would be demonstrably more effective than Python's "optional" checking/hinting. At least it'd remove one variable from the problem. Or maybe not, who knows.
I've found that even when I write code intended to be statically checked from the beginning, mypy often misses stuff. I was enthusiastic about mypy at first and I still use it, but I'm not sure at this point that it is worth it.
Same here. And I come from (and am a fan of) statically typed languages. The type hinting story for Python seems confusing and not completely useful to me.
I just started using type hinting in Python and I love it. Maybe it's your tooling? I'm using neovim with LSP and pyright and the experience is very much like writing code in a Java IDE. As I code, it informs me where there are mistakes or pieces of code that need to be updated, catches typos, reminds me about forgotten imports, etc. It makes refactoring way easier because as you change signatures or object types it flags all the places in your code that need to be updated. Seems like a real game changer to me.
No it's not the tooling, unless by tooling you mean mypy. The issue is that you can write code with type errors that mypy doesn't catch. In what I usually think of as static typing, that would be impossible: type errors are simply not allowed, even if that means the compiler rejects some otherwise-valid code.
I think mypy has to let some potential errors through because otherwise it would spew 1000's of spurious error messages on perfectly good, unannotated legacy code. Something similar happens with the Erlang dialyzer. The same thing applies to tools like Coverity, that aim to find potential bugs in legacy C code.
You could instead imagine a version of mypy that makes no concessions at all to legacy code, and insists that your code be 100% free of type errors, even if that means that some older constructs and styles no longer work. Would that be a good thing? Probably not: we have Haskell for that. Mypy's leaking errors is probably a practical necessity in retrospect. But, I wasn't expecting the leakage, so it surprised and disappointed me when I encountered it. I had thought I was getting something more like Haskell. Mypy still has attractions, but it's less great than I had hoped.
I want automated type checking outside the IDE. I want something that will catch other programmer's mistakes, not just mine. Without enforcing a specific IDE.
LSP support exists for many editors so you are not tied to a specific IDE. It can catch other programmers mistakes if you run it against their code. It doesn't turn Python into a statically typed language, but it's a lot better than nothing.
Sorry, I wasn't clear. I don't want to mess with each individual developers IDE or setup. I want to enforce type checking at the same place we also run mandatory tests: in the automated build pipeline (think CI/CD).
I think this is only going to happen in reality if you're doing funky type hacks in the first place. Mypy definitely has rough edges, but it works perfectly in 95% of every codebase I've seen.
Type hints (with mypy) aren't going to save you from becomming a bad programmer. But they _do_ help you become a better one!
I actually find that mypy is pretty weak and lacking, even within the python ecosystem. pyright seems to do a much better job at the type checking portion of things.
> Despite all that, what I’m hoping for is that someone will come along and tell me that I got it all wrong. “When you do it as follows, the system works: $explanation” Because, you know, I’d really like to have static typing in Python.
I want X. Y (which is designed for somthing other than X) doesn't do X. Thus, I don't like Y.
For me, I'm not writing huge programs in python, I may not need type hints/safety right away, or at all, not that I'm against them in any capacity. The problem I see with opinions like these is the assumption that type checking/hinting/safety are a silver bullet.
Letting perfect be the enemy of the good is not the same thing as doing a cost benefit analysis. In the authors estimation, the costs outweigh the benefits. You don't seem to be considering the cost of type checking at all.
After working in Python without type hinting for quite some time before joining a team that uses it heavily, I was pretty skeptical about how useful it would be give the many rough edges I had read about. After using both approaches for a while I think that type hints are both very flawed and surprisingly useful.
Yes MyPy and type hinting in general has many limitations, rough edges, and occasionally baffling behavior. Despite all that, I think it is helpful for pretty much any Python project. Type hinting does catch a lot of potential issues and the syntax is very simple. I tend to fill in the types even on a first pass at some code since they help so much in tracking various interactions.
I think the bottom line is that something doesn't have to be great in order to be useful.
How is a grype with static typing that it is not enforced at runtime? This comes up like once a week, have these people never used statically typed languages?
> The fact that you can put nonsensical types wherever you want and still get a working program has consequences.
Working is questionable, but nothing in C++ prevents this at runtime either, nothing in JVM prevents this at runtime, static typing is not runtime typing, it should not be, it is not the USP.
> There is an Any type and it renders everything useless
Java has object, C++/C has void*, Go has interface{} - none of these render the type system useless, and Python's type system makes it a lot easier to write correct code than Go's type system or C's type system.
Difference between any and object, void* and interface{} is you can't really do anything with the second. Sure you can store something on an "object" type but you need to cast it back to do anything which discourages usage of it (and add runtime checks, usually)
While in python you can mark something as "any" and continue using it as is.
I would rather my type checker give me the truth than gloss over it. This issue should be kept open (it is) but they should wait until a good solution is found that doesn't gloss over the discrepancy.
I tried mypy when I first started into type checking in Python back when I was trying to do Python in VS Code and I discovered that mypy is a time-sucking anti-productivity disaster that needs to die in a fire.
When I started learning PyCharm though, I discovered that type checking can actually work well in Python and it is not a waste of time at all. So I advise people now that unless you're using PyCharm, do not waste any time on type checking Python. In which case, have at it. I love it. PyCharm is really the only way to do type-checking productively right now.
If you use VSCode and Pylance (Microsoft python language server) you don’t need to setup Mypy, you can enable type checking in user settings and it will use its own engine for type analysis. You have two modes, basic and strict.
I would recommend to try out if you can, the experience is quite good.
I personally find “strict” too strict for the current state of python, but “basic” is already very helpful.
The developer experience is still subpar when compared to Typescript but it’s improving fast (but I mean, typescript with VSCode has one of the best developer experience I’ve ever seen).
Could you elaborate on mypy? I'm not a Python developer (and I really have much more experience with statically typed languages), but if I ever had to maintain a Python codebase, I'd assume that adding mypy and type annotations would be among the first things I'd do, so it's a bit of a surprise to read that.
I love mypy. It's fine, if sometimes a little alarming when you aim it at a previously-untyped codebase. In my case, it turned up plenty of type errors that probably never became a problem in production, but very well could've. It's also a nice thing to put in a CI pipeline to block new code errors.
I spent a lot of time trying to configure pypy in VS Code, time that I didn't have to spend in other languages such as TypeScript, and I never did get it working as well as it does in PyCharm. PyCharm comes already configured. You don't have to do anything for typing. Works out of the box. VS Code is like that with TypeScript but not for Python. That was a couple of years back so maybe it's different today. I hear VS Code's PyLance extension works but I've already made the switch to PyCharm. I might try it out later.
Type hints are a god send in Python if you want to use the language for anything beyond scripting (or single developer code bases).
Without type hints you don't realise how crazy your data structures get in Python. Trice nested dictionaries getting passed around, or pandas running wild across your code base, and on it goes. Instead with type hints we now have insight and can have meaningful discussions on the data structures flowing through our system.
Our team is yet to adopt mypy into our build systems, but hopefully we will introduce it soon.
> and in new versions of your type checker, you have to make sure that they didn’t change their meaning and that they didn’t introduce new flags that you really, really should use.
That's what the --strict flag is for, which the author even quietly uses in a later example without explaining it. Though it may be too noisy to immediately turn it on in some projects.
As for the example of getting something returned from the network that has a mismatching type, there's no way mypy could guess that. You'd have to add an explicit `isinstance(foo, int)` check or use a runtime type checker.
The author does have a point that people might simply use wrong type hints if they don't have a proper type checker set up though. I've seen so much code with `any` instead of `Any`, `[foo]` instead of `list[foo]`, etc. as type hints. Using something like pre-commit and/or editor plugins to run mypy automatically while editing is probably the best. But it's true that even then, without --strict or equivalent, you can too easily satisfy the type checker by simply annotating something as a `dict` or `list`, without explicitly using `Any`.
Still, I don't think this means we shouldn't use type hints at all. It could definitely be more beginner-friendly, but most of the mentioned issues can be addressed with proper configuration.
If you're willing to drop all dynamic stuff and deal with a smaller library ecosystem I would highly recommend trying Haskell. At some point I feel like it's Python 5
I use the type-annotated python at work and at hobby.
I see the point of OP, but I'd say it helps even if it's not perfect or "sound".
Though I agree on its negative impact on the language ergonomics. It removes the joy of writing "scripting language" and as a statically typed language it is far from static-type-native ones.
Now I see it as a tax for compatibility: You enjoy dynamic typing at the beginning of the project, but you have to pay it back once you've grown up to a certain scale.
TypeScript to me feels much more like a native static typed language because it is. Does it help to be a transpiler? Maybe. But I would think that Hejlsberg just has done an astonishingly good work to make it compelling. Python's type annotation is good and completely reasonable, but gradual typing is such a hard problem and being good is probably not enough here.
This is the kind of thing that happens when a language that used to be niche starts gaining more widespread use. People from other languages come along and want the niceties they enjoy to exist in the new language they are learning. The same thing happened to JavaScript; it used to just be a browser language for running scripts, and then it turned into a server side language with a module system and everything else. There were a lot of growing pains involved in that transition.
Now people who use other languages are coming to Python and thinking "Wouldn't it be great if..." and that's how Python the language is going to evolve. But we're going to have to go through this interim phase of awkward adolescence before type annotations mature as a feature.
The thing is, Python is just as compiled as, say Java. Python converts your code to byte code, then executes that bytecode in its VM. These are the same steps Java takes.
So this "whoops" doesn't work for me: "Some language that’s as easy to use as Python and it should be compiled and with good static typing, but it should also not be compiled because then it wouldn’t be as easy to use as Python anymore. Whoops?".
The reason the typing in Python is lax was entirely a choice. Maybe not a good choice, but it certainly has nothing to do with "compiled" vs "interpreted".
That's not really a good definition of compiled. Interpreted vs compiled is very much more a continuum than it was however many years ago when these terms originated - you had compiled ALGOL and interpreted BASIC, and little in between.
Rather, you have to look at it like an attribute based categorization:
Compiled:
- types known at compile time
- limited runtime features like stack traces, introspection
- working closer to primitive data types
Interpreted:
- dynamic, with lots of indirection
- wrapped data types (eg PyObject) vs primitives
- rich runtime with stack traces, introspection, monkey patching
My personal "disappointment" is: it doesn't do anything for the runtime with all the extra work and headache. I.e. it feels like Julia but without the elegance of multi-dispatch and performance.
Correctness in Python is all about unit tests - if you don't bother with them then you're playing with fire (and of course I often "play with fire").
Type hints .... well they help the IDE to autocomplete which is great. They also help you to get the order of parameters in a function/method call correct which is useful if the parameters aren't all the same type!
I converted a medium sized project to type hinting once. The original author wrote no tests and obviously didn't design it to be testable so I was looking for some small way to make it less horrendously unmaintainable. Every test run of the source took hours and there would be some trivial syntax error or whatever right at the end.
Type hints weren't so difficult and I hoped that they would "staunch the bleeding" while I tried to get it into testable shape. In this case they did almost nothing - didn't help me stop a single bug - probably because most types were "str" but that's not the fault of the hinting system as an idea, right? It's just an unprofitable load of work in particular situations.
Static typing fans are always going to be disappointed by Python and it's great because there are other languages for them. For the rest of us we can use them where it turns out to be profitable to us.
Unit tests are, however, by 1000x, more important.
> Correctness in Python is all about unit tests - if you don't bother with them then you're playing with fire (and of course I often "play with fire").
But Python does support type hints, and tools like MyPy do use those type hints to evaluate correctness. It seems to me that Python's type hints are indeed Python's way to check for correctness.
Also, unit tests don't verify correctness. They only serve to check whatever small set of ad-hoc invariant developers put up.
You can have a program with correct type handling that doesn't work but with unit tests you have a program that works and documentation (in the form of the test) about the ways in which the programmer expected it to work.
That's why I find type hinting to be relatively high effort for relatively low benefit and choose to use it only where I feel it makes life better. The optionality suits me. It's a nice to have but that's all.
I never liked how go makes you fix all your types before letting you run a program.
Suppose I wanted to hack something up and run it, just as a new idea? Maybe change a library function that is called in 20 places, but I only want to test it in one call site. Nope! Go forces you to polish up your turd of an idea in all 20 call sites before you can see that your algorithm / model / refactoring is nonsense. If you back out of the idea then all that turd polishing was a waste of time.
The mypy-python workflow, on the other hand, lets me write my nonsense, see that it was a bad idea, then throw it away without eating up more time than I needed to with satisfying all the type checking ceremonial stuff. Much nicer. It’s a linter that follows the “make it work, then make it right, then make it fast” iterative workflow that I’m used to.
Alas, I suppose if I wasn’t an imposter in the programming world, I would be able to dream up my changes and think them through in my head properly before committing them to my text editor. Maybe I wouldn’t even need to do that because my ideas are just great all the time instead of 50/50 between ok and terrible. I’m not smart like that, and it felt like go went out of its way to rub it in.
Maybe I used go wrong. I haven’t touched it in a decade. Is there a “hey ho your types are all out of sync but let’s try and compile anyway” option?
There's a big difference between Python's types and and Go's: in Python they're only there to check correctness while in Go they're there to tell the compiler how to lay out the machine code. If the compiler ignored types in Go it wouldn't just throw an error message at runtime, the behavior would be completely undefined.
I'm sorry, but I'm writing this response without reading all the previous comments, but with almost 600 directly to it and more comments to a comment I hope it's understandable.
However I did read a good portion of them and I'm confused...I thought Python doesn't use types everything is an object according to their documentation.
Now for my 2 cents, if you don't care, don't read it... a lot of the comments aren't based on facts but opinions and a lack of understanding of the actual language, funny that I write that after my previous sentence huh? :) . Every language has goods and bads, don't think as someone who as a Python programmer or C programmer, just a programmer and use the best tool for the task you're trying to complete. If you've been programming for awhile you should have used a dozen different languages at some point. Don't focus on what the language can do but understanding the underlying concepts and then you can easily use the one that makes the most sense for that project.
The author seems to have unreasonable expectations of the python typing implementation, most of which can be fixed with mypyc and stricter flags. They also did not check the concept of gradual typing which is key to why everything is not strict by default.
I too am disappointed a bit by the type hints because they are not expressive enough for me, and some natural python constructs are hard to express. And because mypy has bugs I encounter from time to time.
I find it interesting how we have come full circle on static typing. When I was in University 2009-2013... and years leading up to it, dynamic typing was becoming the hot thing after years of Java/C/C++ dominance. Statically typed languages were looked down upon and talked about as being analogous to "Enterprise", "suit and tie" type soul crushing software development. We had people fulling in Ruby/Rails bandwagon, Java/C/C++ starting to become deprecated from university curriculum in favor of Python... JS starting to eat the front end and HN startups were talking about how they write their entire stack in a flavor of Lisp.
And now, just a few years later- Rust, Go are two of the hottest languages around, TypeScript is eating JavaScript, Python/Ruby have added static typing constructs, C# continues to do well in its niche and even Java of all languages has become somewhat cool again even though being Oracle product.
It's been an amazing transition to watch in action play out in front of my career, I can't even imagine those who have been in the industry for 30-40years who have probably been through a dozen of these cycles.
Yeah they're pretty bad but they are still better than nothing. If you are in the unfortunately position of having to use Python I would recommend using them.
I would also strongly recommend using Pyright (the default checker in VSCode) over MyPy. It's so much better, and the author is really responsive on Github.
But yeah, in general trying to use type hints in Python is like trying to discuss philosophy in a mental asylum.
The whole reason Python and JS surpassed most all other languages is specifically because you can write code fast, without worrying about strict definition of data structures.
The issues with python code bases aren't from a lack of strict typing or type support, but from a lack of a good testing framework.
JS is popular because it's literally the only language web browsers support.
Python is popular because it's very beginner oriented, so it's the first language a lot of people learn. It's modern BASIC.
Neither of those things mean that writing dynamically typed code is a good idea. The idea that you can write dynamically typed code faster isn't even true once you get past a couple of hundred lines.
This concept of "programer is generally dumb, so languages must have features to prevent them from making mistakes", with strong typing falling under those features, is solely an academic exercise that is taught in academia, and perpetuated by people who just wanna fit the mold in FAANG and other big players.
Language adoption directly depends on how quickly people can build stuff in that language, and its an exponential effect, because with speed of development comes more libraries, which in turn allows other people to build stuff that uses those libraries quicker. And the initial speed of development directly depends on the programmer having to manage less things. Static typing, in real world, is often a hinderance because code bases are dynamic, and having to go and refactor code because data definition changed takes time. And if you have competent programmers that write clean code and a good code review and testing framework, static typing gives you no advantage since the time spent fixing issues due to failing tests will be equivalent to spending time fixing issues with failure to compile.
> This concept of "programer is generally dumb, so languages must have features to prevent them from making mistakes", with strong typing falling under those features, is solely an academic exercise that is taught in academia, and perpetuated by people who just wanna fit the mold in FAANG and other big players.
What is the precise definition of static typing? Both the article and many comments in this thread refer to static typing or static type hints in Python. But as I've always understood it, static typing means that types are checked and enforced at compile time. And if python is not compiled, the notion of static typing in python does not make sense. So do I have it wrong? Or is the term ambiguous?
static in this case can be understood to mean simply "prior to (or separate from) runtime". In other words, it's based on what you can check _without_ running the code.
It's worth noting that nearly every statically-typed language currently in existence has two separate "type systems" - the static type system which is the formal type system offered to the programmer, plus a runtime type system enforced, at minimum, by the processor (e.g. you cannot divide by 0) that is ultimately different from the static type system. The 'question' in most cases is "how closely does the runtime type system match the static type system?" In many if not most languages, the answer is "not very".
* fork the language again? We just got out of Python2 hell.
* Docstrings, like we had before they were introduced?
They won’t catch all bugs, but with minimal effort they’ll catch some bugs, and if you make an effort to really dig in, it’ll catch most bugs. To me, to accomplish that overnight on a language with millions of existing lines, that’s a big win.
Idiomatic C and Go programs aren't labeling everything with those types, and the language implementation enforces those types at compile time.
Since types are optional in Python, it's necessarily true that an "Any"-like type is prioritized since it's the default. Moreover, cpython doesn't check anything when it bytecode-compiles.
Any-by-default and no-built-in-checking feels pretty defanged to me.
> Since types are optional in Python, it's necessarily true that an "Any"-like type is prioritized since it's the default. Moreover, cpython doesn't check anything when it bytecode-compiles.
mypy is pretty whiny about types, and sure, if you add Any everywhere then you have just defeated the type checker, but this is no different from putting interface{} everywhere for Go.
> Any-by-default and no-built-in-checking feels pretty defanged to me.
Is running mypy on the code base that much to ask?
I haven't used python's type hints much yet but I used TypeScript a bunch about a year ago and I have the exact same complaints.
Most of the benefit of types goes away unless _everything_ is typed... my libraries, the libraries my libraries use, etc. I should be able to jump around in and out of library code following types and usages thereof in my IDE, making conclusions that are accurate based on what the types say.
When some of your libraries (or indirect dependencies) don't have types, you can no longer ever make any firm conclusions when exploring a complex codebase based on types. I want to change a method in the Widget type, and my IDE says it's used in 4 places. But in reality, that just means 4 or more places, who knows if there are others.
Adding types to a language that doesn't already have them is very hard for this reason. Until many years pass and they're in universal use, they add little value.
I think Typescript nowadays has reached critical mass where most packages at least have a @types/* definition. I can't remember the last time I had to use a package that didn't provide types. I had the types be wrong one time though.
I find it reasonable that Python doesn't enforce the type checking. The owners of each project can choose how much to enforce it and how.
Defining such style rules and validations is necessary with almost every language. A codebase where developers are allowed to use any C++ feature or browser API will become very messy as it grows.
Types help to encourage readable code. If it's hard to add type annotations to a function then the function is too complicated. I've seen this with wildly polymorphic functions that needed dozens of overloads to properly type... which means that engineers need to hold all that complexity in their heads
I have found optional typing to be a great example of an 80:20 trade off. I add type annotations to the easy stuff. I get a huge bang for that. I find that once I go past the “easy” annotations, I get less and less back from that. Between the IDE and my own annotations it catches 95% of trivial type errors.
> Python is a dynamically typed language. By definition, you don’t know the real types of variables at runtime.
I think you mean compile-time and not runtime. There’s not much use in knowing the type at runtime, since it only enables you to print a better error before crashing (“Expected type: x, actual type: y” instead of “AttributeError: 'str' object has no attribute 'append'”).
Types are only truly valuable at compile-time, where they can prevent compilation of invalid code. This is the reason many statically typed languages use type erasure,
which removes type information from the compilation output.
In fact, the type-checking phase can happen even earlier than the compilation phase. This is useful e.g. in an IDE, where you care about type errors but not necessarily about running your program.
I find type hinting useful as an additional level of documentation, especially for complex projects, and it helps my future self understand my own code more quickly.
Also in my experience it can provide static analysis tools more information and provide autocomplete suggestions in a wider range of coding scenarios.
I'm surprised there's no mention of pydantic or other type systems. I started with type hints, and find that the discipline of using types invaluable, especially when coding public API backends for which validation is a strong necessity, whether semi-static (ala mypy) or runtime.
They feel like someone told the Python dev team about Julia's multiple dispatch based on types, and they felt like they had to respond, even though they don't really provide the actual speed and flexibility that motivates their use in Julia.
I agree with everything in this article. The fact that nothing is necessarily checking them makes a codebase full of them too misleading. They become cargo culting; and as the article says their special syntactic status makes them look “official”. Typescript is much better since you can’t avoid them being checked. And that’s said as someone who spent the last decade writing Python, in later years with type hints. There’s also a mypy option that is not on by default but that is absolutely required to avoid misleading false negatives: —-check-untyped-defs
I love the gradual typing approach, and have found it makes codebases better to maintain, even if some libraries like pandas are a total pain.
My biggest gripe is the conflicting warnings between mypy and pyright. Language servers use pyright by default in the editor, but I strongly believe a Python project shouldn't require nodejs as part of its required CI toolchain. So all my projects use mypy in strict mode in CI. Basically this means that mypy is the standard to meet, and that there are persistent warnings in the editor which I have to ignore.
Python is becoming a Turing tarpit language that’s black-holing on its own syntactic complexity. Soon every single ASCII character will have a special syntactic meaning and the syntax itself will become Turing complete in more than one way, similar to C++ that has already collapsed in syntax hell. JavaScript is following the same path by adding every feature from every other programming language. Soon all programming languages will have exactly the same features where everything is possible but nothing is doable or practical.
Landing in a large-ish python project without type annotations is a pain. Author says that he won't use type annotations because they "can be wrong", well, you only rely on variables names and documentation for that, you have even less chance to get that right.
Is that "line" variable a "LineInvoice" or "InvoiceLine" (yes, two different classes) ? Oh, right, both can be used in this place exclusively because one is subset of the other... sigh
Python's base philosophy is duck typing. Whenever I read complaints like this the root cause is almost always trying to turn Python into something that it isn't instead of embracing what it is (if this isn't feasible, then Python is the wrong choice).
Programming in Python worrying about types instead of behavior reminds me of the quote "you can write FORTRAN in any language".
A type checker doesn't make a statically typed language. Also, yes, the program will "run" with failed type assertions.
Compare this with Typescript, where by default the compiler will refuse to compile the code when type errors are encountered. But there are other Typescript compilers that are faster and don't typecheck at all, they just strip the syntax.
If you want the strict behavior, you need to prevent the program from running when typechecking fails. That is entirely possible with mypy.
I didn’t really connect with any of these objections. Python type hints are very permissive and you need to turn them up in strictness to get enforcement. No big deal for me.
But moving between a TS and Python codebase, I just find TS much easier to work with. Maybe it’s that type bindings are more widely defined in libraries (though typeshed has quite a few now). Or maybe it’s something about TS being easier to define type vars / infer types in my IDE? I can’t really put my finger on it.
> It feels really good to see a program compile without warnings or errors. You then know that you got it right.
I don't know about that. So many C/C++ programs have core dumped in my life and needed to be valgrinded to debug memory issues, or went past the bounds of a pointer, or overwrote the stack, or... whatever.
And you still don't get correctness without unit tests.
I'm not saying types aren't valuable, but they're not a crutch to program correctness like people seem to think they are.
In my experience, people with less experience with object oriented programming will end up creating lots of objects only to create types. These objects have no business use case, nor do they aid in any encapsulation. These objects are made just to help typing. Nuts!
Python type hinting is like moving backwards in time because the amount of time devs take to "hint", takes away the main reason for using python, that is faster development. Might as well code in Java at this point.
This has to be a troll comment. The overhead of typing is literally _nothing_ compared to the amount of time it takes to sift through code to figure out
a) what types a function/class accepts as arguments and
b) what types are returned.
If you don't check it then I agree it's useless. However, working with libraries that are typed is an _insane_ productivity boost over working with untyped libraries.
WTF!! If we have to see a codebase littered with TypedDicts just to satisfy type annotations and then have to cast everything to that type - why not just wrap it in a class and call it a day?
When I see garbage like this, I seriously question the productivity improvement brought about by python type annotations. I certainly end up spending an inordinate amount of time unraveling nested unions of types all over now. Considering switching to a team that uses Golang.
Typing encourages a large number of bad coding practices. If you can apply typing to a Python program then you are not taking advantage of the dynamic nature of the language to write simpler and shorter code.
In Python world, unit tests cover what is normally handled by typing.
> Typing encourages a large number of bad coding practices.
I'm sorry, what? What exactly is the bad practice here? Not passing random dicts to my coworkers only to surprise them because the key was userid and not user_id? Or wasting time bugging an overworked dev to ask what their crazy 8 levels of dynamic indirection magic are doing, vs just being able to Cmd-B and step through the code path?
> If you can apply typing to a Python program then you are not taking advantage of the dynamic nature of the language to write simpler and shorter code.
Yeah, no, flat out wrong. You can use typing with dynamicism to do magical things. FastApi works a near miracle, generating serializers, deserializers, openapi docs, automatic api deprecation warnings, not to mention type safety.
Properly typed python is faster to write and safely magical.
The bad practice is treating a dynamically typed language as if it was a statically typed language.
There are shorter more concise ways to write your programs if you are not using static typing.
If you are not writing your Python programs in the Pythonic way then you are not really using Python correctly and you do not get most of the benefits of the language.
Unit tests exist for catching bugs in Python. Type hints are for documentation not for ensuring code correctness.
Yes, there are libraries that take advantage of type hints, through they could of used other annonations and it's valid to use type hints along with those libraries.
But as soon as you are attempting to prove your program doesn't have bugs via typing rather than unit testing. You have gone wrong.
Typed Python is far more bug prone, why? Because structuring your Python code to support typing means writing considerably more lines of code.
It not the type hints themselves but it's the way you will structure your program at a high level to support them.
The more lines of code the more bugs it has. Bugs are well known to be proportion to the number of lines of source code regardless of language used or whether the language is typed or not.
> Typed Python is far more bug prone, why? Because structuring your Python code to support typing means writing considerably more lines of code.
That does not bear out in practice. In my experience, bugs have almost nothing to do with LoC. A short bit of complex, I/O heavy code with lots of state is going to be way more bug prone, and far harder to unit test correctly, than 10x the LoC of purely functional, well-typed, stateless code.
Mutation, parallelism, IO, and complex state space are the four horsemen of the bugpocalypse.
But in fact, I find functional, well-typed, properly abstracted code ends up with less LoC in the codebase, because it's more reusable and composes well.
I wonder who, ever, thought that weak typing python- or raw js- style, was a good idea? Come to think of it, is there a single redeeming quality of weak typing in those languages other than "easier to wreak havoc and quicker to write crazy buggy shit"?
There is probably that stoic school of thought that if you are forced to walk around a construction site without a helmet, it will make you a more thoughtful and better worker, but... come on
As someone who's very used to Typescript, I find these complaints very interesting, because they're often common for Typescript newcomers as well. However, I think they - quite understandably - misunderstand the purpose of these tools, and so get disappointed when the tools don't do what they want or expect.
I think the traditional view of types is that they're about declaring something to do with the memory layout of a particular variable. Maybe not always directly, but I think most people have this kind of intuition that an int is an int, a string is a string, a struct is a struct (and a particular, explicit one at that). In Java with subtyping, things get a bit more complex, but it's usually fine.
That's generally not a good way to approach Typescript/mypy. Typescript, for example, is absolutely unconcerned with whatever value lives inside a particular variable. It's very possible in Typescript to create types that can't even ever exist at runtime (type brands are a good example of this). The compilation process for Typescript is basically just stripping the types away as if they were comments, and leaving the Javascript behind.
What Typescript is is a slightly more complicated linter that requires annotations. The types you put in are basically just labels. When writing Typescript, your goal is to give every type a label, and then run a program that tells you if all of those labels are compatible with each other. If there is a label mismatch (you've used label "number" but you've done operation "push" on it) then the linter will simply tell you you've done something wrong. Moreover, these annotations can be very loose - annotate something as `any` (or `Any` in Python) and the linter will accept anything done to it. This means that you can always break free of the linter and tell it that you know best (which is true - it is probably impossible to statically validate whether a given Javascript program will throw an error or not, so any linter will have some cases where it prohibits a valid program - the programmer requires the power to override the linter).
Viewing Typescript and mypy in this context helps a lot, I find. It explains why casting doesn't throw runtime errors (a cast is simply a lint annotation saying that a particular expression has a particular label). It explains why JSON isn't validated when it's parsed (you're just starting that the data has a particular label/shape - it's up to you to ensure that that's true, but you can do that check however you like, including not at all if you're very confident in the quality of your data).
This raises the question about which process is better: one where types get transformed into real data structures where all code is kept valid at runtime, or one where the types act only as assertions for a linter validating whether the code ought to remain valid. In my experience, a lot of it depends on how valuable you find the dynamism in languages like Python and JavaScript. If it's very important for you, then traditional typechecking approaches are probably not very viable because no static type system can ever cover all correct Python programs. However, levels plus a linter will resolve most programme correctly, and still give the necessary overrides to allow for other options.
So when it comes to the questions in the article, I think the author would do a lot of good to approach Python type hints from the perspective of a linter that requires annotations.
my experience with python's optional type system was that the simple stuff works very well, but when you try to go down a type rabbit hole for more advanced stuff involving generics it becomes really unworkable, inconsistent, and hard to dig yourself out of.
but for little things, like just function signatures, it's great!
Type systems are great for catching representation errors.
However, the important errors that I make are "kind" errors.
Type systems can verify that you're adding two ints, but won't tell you when you add the number of bananas to the number of oranges. Of course, both numbers contribute to the total number of snacks.
I actually enjoy the typing in Python. The only problem is lack of support from third-party libraries. I find myself fighting the typing errors so often by adding: # type: ignore. Especially pandas gives so many errors. Type hints are included but they are often just plain wrong.
I agree treating Type Hints as comments is the way to go. It's a bit nicer to use a library that is fully-hinted than one that is not. But I don't worry about type-hinting my own code, unless it's a particular complicated function signature that other people will be calling.
PyCharm with type-hinting and "inspections" (which include pseudo-static analysis for appropriate type use) have only added to the awesomeness of great code navigation, refactoring, auto-completion, code-formatting, etc.
If you're going to have type-hints, you need pseudo-static analysis.
Mypy checking is actually really good, and you can give it flags to make it super strict. Setting it up in CI is the same work as setting up any other linter. The only fault I had with it was 3rd party libs not providing hints often enough.
I like type hints, but trying to retrofit it into an existing project I’ve found difficult, because you end up with Union[type1, type2, type3] in some places.
How did you get from type hints to black, not to mention poetry? Seems like you're just complaining about there being a mature ecosystem around Python, and the implicit expectation to use it.
And here I thought my explanation at the bottom covered this.
Let's try again:
Python has become a mess. It is no longer a simple productive language to get into like it used to be. Now, you are expected to buy into it's ecosystem, like it is C# or Java.
Well, it is not. It's Python and I am truly sad they are killing it with thousand cuts.
Yes, it's getting more complicated. Yes, there's a whole ecosystem being built around it, and the line between language and ecosystem is blurry. And indeed, there's a hidden expectation for any Python developer to buy into the whole ecosystem.
But that's just a symptom of a mature programming language that's used in a variety of complicated scenarios. You don't have to do it all, and no-one is forcing you. You can always write your one-off script without black, flake8, type hinting, or anything of the sort. If it's just your script then no-one will even know.
On the other hand, if you want your code to be part of the larger ecosystem, then you're expected to write it in a conformant way (e.g. pep8 formatting). And that's a good thing!
I don't think the OP was referring to build pipelines, but rather continuous integration pipelines. Build in the sense: lint, install deps, and run tests.
Look there's a lot of good, long winded comments here. If anyone is like me and scrolls for an easily digestible one: here's the TLDR.
Eventually you'll get fed up with not knowing what type each argument to an API call needs to be. You'll either get on board with python type hints, or you'll graduate to a typed compiled language, or you'll languish in the trough of python productivity.
I disagree diametrically with both the general sentiment of the article, and every major point that was made in it.
Every point in this blog post strikes me as either (1) unaware of the tooling around python typing other than mypy, or (2) a criticism of static-typing-bolted-on-to-a-dynamically-typed-language, rather than Python's hints. Regarding (1), my advise to OP is to try out Pyright, Pydantic, and Typeguard. Pyright, especailly, is amazing and makes the process of working with type hints 2 or 3 times smoother IMO. And, I don't think points that fall under (2) are fair criticisms of type *hints*. They are called hints for a reason.
Otherwise, here's a point-by-point response, either recommending OP checks out tooling, or showing that the point being made is not specific to Python.
> type hints are not binding.
There are projects [0][1] that allow you to enforce type hints at runtime if you so choose.
It's worth mentioning that this is very analogous to how Typescript does it, in that type info is erased completely at runtime.
> Type checking is your job after all, ...[and that] requires maintenance.
There are LSPs like Pyright[2] (pyright specifically is the absolute best, IMO) that report type errors as you code. Again, this is very very similar to typescript.
> There is an Any type and it renders everything useless
I have never seen a static-typing tool that was bolted on to a dynamically typed language, without an `Any` type, including typescript.
> Duck type compatibility of int and float
The author admits that they cannot state why this behavior is problematic, except for saying that it's "ambiguous".
> Most projects need third-party type hints
Again, this is a criticism of all cases where static types are bolted on dynamically typed languages, not Python's implementation specifically.
> Sadly, dataclasses ignore type hints as well
Pydantic[3] is an amazing data parsing library that takes advantage of type hints, and it's interface is a superset of that of dataclasses. What's more, it underpins FastAPI[4], an amazing API-backend framework (with 44K Github stars).
> Type inference and lazy programmers
The argument of this section boils down to using `Any` as a generic argument not being an error by default. This is configurable to be an error both in Pyright[5], and mypy[6].
> Exceptions are not covered [like Java]
I can't find the interview/presentation, but Guido Van Rossum specifically calls out Java's implementation of "exception annotations" as a demonstration of why that is a bad idea, and that it would never happen in Python. I'm not saying Guido's opinion is the absolute truth, but just letting you know that this is an explicit decision, not an unwanted shortcoming.
100% agree - never really understood the movement behind adding types to Python. Type hints are a useless complexity that yield little return. If you want types, use something else other than Python. The whole thesis behind Python is simple is better than complex.
Typechecking allows certain errors to be detected at typecheck time rather than at runtime. I don't personally consider that useless. E.g. A mistake I just made. I'm working in a language I don't know too well right now with strict static typing. The file read function takes a handle. If I pass it a string of the file name, rather than the handle result from file open, it just doesn't compile. In most dynamic languages, that error would not be detected until it executes.
The changes you need to make to a Python program to enable it to be type checked result in more bugs being added than the type checker removes leaving you at a net negative.
Type checking is just a really poor way to ensure program correctness until you start using an exceptionally strongly typed language like Haskell.
You use types in C / C++ / Java because the compiler needs them to work not because it's a good idea.
> The changes you need to make to a Python program to enable it to be type checked result in more bugs being added than the type checker removes leaving you at a net negative.
This claim doesn't make sense to me. Assuming mypy is regularly run as part of CI, why would adding type annotations cause more bugs than leaving it off?
You don't understand what "strongly typed" means... I give you a hint, `"1" == 1` is a type error in Python but "okay" in JavaScript because this one is "weakly typed", you're confusing "dynamic typing vs static typing" with "strongly typed vs weakly typed".
Python is a strongly, dynamically typed language. Most Python type errors could be caught by a static analyzer. If Pythons type hint system wasn’t a disappointment it would help more people catch more bugs when they’re written. Instead Python endlessly throws exceptions at runtime because a programmer got their types wrong.
I find it best to see type hinting as "living documentation". Yeah, you can ignore it, it is Python afterall, but if you want the IDE/tools to have better more helpful hints ... use machine readable hint documentation.
I'd argue if your functions are full of type-assert like guards, _then_ use another language!
On the one hand yeah, I agree with this, but on the other hand you have a lot of enormous software projects written in Python that are decidedly not simple, and would be extremely difficult to completely rewrite in a different language. Type hints do offer some value in this circumstance, by making it easier for disparate groups to modify the same large codebase though static analysis.
My editor now has a fairly deep understanding of my code, and can tell me of all sorts of surprising errors I'm making before I run the code. There have been a few times where I found an error, went into the editor and saw I had missed a error message about that line. Shout out: Using LunarVim with LSP and TreeSitter.
The other thing I'm enjoying is that libraries can use them to make things happen more automatically. I believe it was Typer (the CLI argument parsing library) where if you declare an argument to a function as "files: List[Path]", it understands that the argument will take one or more files, or "-" to mean stdin. If you just say "file: Path", it understands it is a singular file.
I was curious about type hints when they first came out, wasn't really expecting to use them but they seemed cool, but now I'm totally sold.