I'm strictly a "scientific" programmer. I've used multiple languages in 40+ years, but only two lasted more than a decade: Turbo Pascal and Python. As Python has passed the decade mark for me, I sometimes wonder why I'm still using it and what would make me get sick of it.
Some of my reasons for choosing Python in the first place have been mooted. Python is free, and multi-platform, but so is everything else nowadays. I think that battle is over.
For my use, packaging has not been a big obstacle, maybe because I tend to use a relatively small handful of established packages. I don't remember needing a package that I couldn't install with pip.
Easy to learn? Yes and no. I think a beginner can ease their way into Python and get productive, if they start with basic scripting and progressively adding a few language features. On the other hand, some features are simply over the heads of beginners, making the code found in packages practically unreadable. But I've never found any language to be better in this regard.
Fast? Sure, with numpy in its domain of use. Or writing your own C libraries. However, I think a beginner using numpy will run into roadblocks that require help from an experienced programmer, who can visualize what the array operations are doing under the hood. So, writing fast code isn't easier to learn in Python than in any other language.
The editor should have a final option, that turns your code brown if it's just so bad that nobody should ever have to read it, including yourself.
I now use python, but Python is not even half the language Mathematica is for scientific computing.
If only Mathematica was free, I think it would become the defacto language for scientists.
The most important feature of Mathematica is that it is functional and not object-oriented. Mathematics is functional, not object-oriented. So thinking in Mathematica is as easy as thinking in maths. Thinking in python is to force yourself to think in an unnatural way.
I think you're regarding Mathematica from a purely "theory/analytics" point of view, however I think you're missing the fact for a lot of scientific computing you want to interact with other things apart from numerics/analytics etc.. I believe that is really where Python shines and why it's also eating Matlab's lunch (more so than Mathematica's). It's incredibly easy to write a gui to interface with e.g. a scientific instrument do some reasonably fast analytics with numpy and possibly even produce publication ready plots. This is not really possible with Mathematica and while you can do this in Matlab, you need several toolboxes which are expensive, but more over are of extremely varying quality (let's not even talk about the horror of writing GUIs in Matlab). With Python, you have all of that for free, with a single tool (remember most scientists don't want to learn lots of programming languages).
Octave/matlab is a little more enjoyable to use for matrix stuff in the REPL though. The game of “is this in Numpy or Scipy and why won’t it autocomplete” is kind of annoying, IMO.
On the other hand, the non-mathematical functionality of Python of course is much better developed!
I like Mathematica a lot, but a couple of issues limit its usefulness for me. First, I'm an experimental scientist, so a lot of programming is in the lab, automating experiments, running tests, etc. Makers of specialized software tend to choose their battles in terms of the application of their product, justifiably, but this means no single app is general purpose enough to serve me in the office, lab, and at home.
Second, unless an organization is enlightened enough to approve a site license, specialized software tends to create a "silo" of people who can use it, and forces it to be used in a centralized fashion. With Python, I can literally install my tools on every computer that I touch -- in the office, multiple labs, at home, etc. It makes it more likely that a tool will be woven into "how I think."
I can also share things and/or encourage others to try them out with minimal friction. Almost as easy as sharing an Excel spreadsheet within an organization that has purchased a site license. Free means free site license.
The friction of approving a license also discourages people from learning a tool by just giving it a try on some trivial project, or taking it home.
Mathematica (and matlab for that matter) is effectively free at most national labs, and it’s still a tiny market share. That’s because scientific computing is rarely just scientists, but often includes engineers, SWEs, system admins, theorists, grad students, and more, solving lots of problems. If you can’t get wide adoption at a national lab, where are you going to get it?
Similar problem with Julia and Fortran even. They all have their niches and are great at certain problems, but Python (and C/C++) rule the overall landscape. in HEP, for example, it’s even the case that a full batteries-included C++ environment (ROOT) has slowly given way to python.
> The most important feature of Mathematica is that it is functional and not object-oriented.
Python has object-oriented features but there is nothing that requires you to use them; you certainly don't have to express every Python program in terms of classes and methods, the way you do in Java. Doing functional programming in Python is common. Is there a particular aspect of doing functional programming in Python that you find to be a roadblock?
It's more of the fact that all popular scientific libraries in python are written in an object-oriented way. That kind of forces you to write your own code in an OO way. I am guilty of this as I am writing a library myself. But users have to use, say numpy, along with my library. It would be suboptimal to force the user to constantly switch between functional and OO, so now my library is OO.
On the other hand, all the code I wrote in Mathematica during my PhD was functional, because it is just easier to write functional in Mathematica.
> It's more of the fact that all popular scientific libraries in python are written in an object-oriented way. That kind of forces you to write your own code in an OO way.
I'm not sure I see why. The object orientation in libraries like numpy or scipy is to provide the user with fast implementations of the kinds of mathematical objects they will need to do computations. But unless you are defining additional such objects in your own code, there's no reason why your own code needs to be object oriented. You can just write ordinary functional programs that use the mathematical objects the library provides.
For example, say you are doing matrix computations. You are of course going to use the matrix objects provided by the library; but your own code shouldn't need to define any new matrix objects. Your own code can just be straightforward computations using the existing matrix objects in the library.
It doesn't matter how libraries are written, in py you can use in functional way. I don't get what do you mean by forcing you to code in OO? I Had used pandas, scikit-learn, numpy, pytorch in production and I had never wrote them in OO way. Are we from same universe?
I believe this is only the language. Not the set of libraries that come with Mathematica. Without those, one cannot do a lot in terms of scientific computation.
I often like to compare Python and Mathematica to English and Latin.
English is not really a beautiful language - it’s a wild mess of different influences, but it’s easy to get to a basic level of fluency as there are few rules (but many exceptions due to the many influences). And it’s in use by large parts of the world.
On the other hand, Latin is beautiful and pure. It has more rules, but very few exceptions. You can also see how it influenced other languages (hello Jupyter notebooks…) But it’s simply not a very practical language today, as there are few people who use it.
I feel that choosing the language based on how close it is to thinking in maths is putting the cart behind the horse, because in all scientific and applied scientific code I've seen, the actual math models is something like 10-20% of the code, and the majority is all the data management and plumbing around that; so you should pick the language based on how easy and maintainable the majority of the code will be, which is not the 'mathy' part of it.
I beg to differ. If by "math" you mean arithmetic and calculus, then sure. Combinatorics, graph theory; probably most of discrete math is very much object-oriented.
Consider a graph G, with vertices (a,b,...,z). Say, we want to find the shortest path between vertices a and m. A mathematician might write an algorithm called shortest_path(G, a, m) which might call the algorithm breadth_first_search(a) etc. Functions are called on objects to determine their properties. Or to modify them into other objects. This is what you see in math or theoretical CS papers.
In python, in popular libraries, the above will be attained by something like
G.shortest_path(a,b)
which seems to imply as per OO that graph G has a property shortest_path. In mathematics we don't think like this. Because it prevents mathematical abstraction. An abstract algorithm, like say Eigenvalues can be applied to a matrix M or to a graph G. In a functional language, the user would just do Eigenvalue(M) or Eigenvalue(G) as needed.
In python, these would be M.eigenvalues() and G.eigenvalues(), which makes them distinct.
What you're saying is not true of the most popular graph package in Python (networkx). Graphs there are objects with attributes (edges, nodes, etc) and methods (add_edge, etc), but algorithms such as shortest paths are functions as you have called most natural: shortest_path(G, a, b)
It's still object-oriented, though. The graphs themselves are objects, which helps with ergonomics.
There are also functions that operate on graphs, because the alternative would be to stuff perhaps thousands of methods into the graph class -- which would be extremely annoying to maintain as well as abysmally slow.
Since you capitalized it, it looks like a static method on the “G” class. There is nothing OO about static methods - this is purely and organizational construct.
I think it's the superclass A of both G and M that defines the eigenvalue() function. Then when both G(A) and M(A) inherit from A, it captures the mathematical relationship. The subclasses indeed should implement the eigenvalues computations differently, like in real mathematics.
Funcional programming is used in Coq because it captures logical chains, for theorem PROOFS, but not for specific COMPUTATIONS, such as eigenvalues.
I understand that this is possible. But this is not how mathematicians think. If you got together a bunch of mathematicians and asked to invent a programming language to do mathematics, they would never ever invent the concept of a class.
Mathematics (outside of stuff like category theory etc) is simply sets and mappings between sets. Anything more complicated is forcing the user to think in a way that mathematics does not.
> But this is not how mathematicians think. If you got together a bunch of mathematicians and asked to invent a programming language to do mathematics, they would never ever invent the concept of a class.
Hi, mathematician here. Please read the source of SageMath: written by mathematicians, for mathematicians. Therein, you will find thousands of classes. Almost all computer algebra systems, written by mathematicians for mathematicians, have a notion of classes.
> Mathematics (outside of stuff like category theory etc) is simply sets and mappings between sets.
This is an extremely narrow-minded view of mathematics. Mathematicians thrive on abstraction, (category theory is literally mathematics; that's a very strange exception for you to carve out), and if we were to boil everything down to "simply sets and mappings between sets" then we'd be bogged down in utter tedium and nothing would ever get done. Object-oriented programming, specifically class hierarchies and inheritance, are extremely valuable for doing all sorts of math at high levels of abstraction.
Hell. If I were to be especially pedantic, I'd point out that category theory itself is "simply sets and mappings between sets" except that sets aren't quite large enough so it's actually "categories and mapping between categories".
Since the eigen decomposition is specifically a property of the matrix representation of the graph, but other things like shortest path distance between two nodes are naturally enough computed on graphs formed by objects with pointers, perhaps the correct interface is G.adjacency_matrix().eigenvalues().
There are a lot of hierarchies in the families of objects that discrete mathematicians are interested in. Inheritance is extremely natural in such settings.
I think the so much more/hard to even describe is sort of a problem?
There is a lot to it, and in my experience it's a very long road to getting your first problem solved.
There are a lot of very cool pieces but I want it to be something I can pull off the shelf once in a while and be productive rather than something that I need to be an expert in before I can get started.
Another big one that many languages treat as table stakes these days is the whole 'batteries included' thing. Whenever a new language comes out and I have to make my own (or install) core capabilities, it feels like a huge waste of time.
"A lot of batteries included" cannot be repeated enough. This is huge. Some languages you have to write your own. But the flip is also true. Take Node. For function X, there are 100 ill named "add ons" of variable quality and repute. Discoverability is a nightmare. Which-to-use in your mashup leads to paralysis analysis. I use a handful of libraries for some pretty large Python apps, but for so much of the basics, it's all in there. The quality and documentation vary a little, but the similarity/consistency is much better than you find in the wild. Even in Elixir (a much smaller universe than Node/JS), I experience this "15 poorly named packages that do the same thing in different ways".
Agreed. I am a Perl refugee. I was so tired of searching CPAN to find ten packages, but some wouldn't install on Windows, and then I would have to evaluate the survivors. They would cover sixty percent of the problem, maybe seventy, all different coverages.
Now people talk about Python's standard library being "where code goes to die," they want to remove old stuff, makes me sad.
I haven't found that in Elixir... I often find one of two that haven't been updated in a couple of years but still work just fine because the Elixir language itself doesn't change much.
It may depend on what types of libs you're looking for, of course. It's definitely missing a lot of stuff.
I am not a JS person so I dont know, but in the jungle of add ons I wonder why there isn't a project which provides a stdlib like package for JS. Is it because for JS you want to install minimal libs?
Having to piece together a Python installation can be an annoyance for a beginner. I always offer to help them set things up the first time. Since most of my workplace is on Windows, it's easy to have people install WinPython.
This is especially true if they've already tried setting things up, and have made a mess of it. Because WinPython is self contained, it can work on a computer that already has a working or non-working Python installation.
What do you mean by piecing together a python install? Most linux should have one pre-installed or easily available via the package manager. After that venvs and pip seem to be generally enough for most tasks.
Packaging Python applications that are outside of containerized apps can be a headache. You can't assume that the system's Python suits your needs, or even that Python versions in package repositories will, either.
A big pain point still remains when it comes to packaging and distribution.
It's not insurmountable, but if you're used to compiling and/or cross-compiling and shipping binaries, shipping Python apps can be headache inducing. The headaches increase when you're working with Windows or macOS, or using bleeding edge Python versions.
That said, for 99% of Python apps out there, the default tooling should be absolutely fine for packaging and distribution, as should 3rd party solutions like PyInstaller or Nuitka.
Oh of course, packaging python apps for final distribution is a different matter. I was assuming they were talking about the process of setting up a dev environment not deployment.
Oh yeah, 99.999% that's the case for dev environments. The only issue I've seen is that some versions of Linux distros have Python shipping with Tk and cryptography support in separate packages to be installed from repositories, where they might be installed by default on other distros, or are just included with the Python package.
I've been using Python for about 6 years, my last company wrote it's core commercial product exclusively in python. I've never used venvs, but I've yet to run into a situation where virtualenv (maybe with virtualwrappers as sugar), and pip + requirements.txt haven't been able to handle even fairly complex install situations.
My workplace is a Windows shop. I'm multi platform, but by and large the folks using Linux don't need my help. ;-) So when I help people its almost always on Windows.
I think the desire for batteries to be included is a holdover from the time of really really bad (or no) package managers. With a good package manager it doesn't matter if your batteries come included or not. It also provides a way for packages to battle it out external to the language -- compare Python’s `urllib` and `requests`.
The curation of the batteries is the value add. Which grpc package do I want? Which json? Which random number generator? Finding which rust crate is the norm for any given thing, is a task in itself. Then there is the security aspect. Pulling in hundreds of repos with many authors each, is a security risk.
The curation you're relying on is probably not delivering what you hope. If you picked a Rust crate by looking at a SO answer from say 2020, you're probably getting a similar quality of "curation" in Python's standard libraries.
Not because the Python maintainers are intentionally doing a bad job, it's just that their mission isn't to substitute for your research. To them, "Does last year's Python script still work?" is always higher priority than "What's the best way to do it today?".
And what Python's security team decide constitutes a reasonable security fix may conflict pretty badly with what you need.
"Does last year's code still work" is very important to almost all programmers. I wouldn't want to accidentally use a library that gets abandoned or deprecated a few months after I made it an integral part of my software. Being able to mostly stick to the standard lib makes this a lot easier.
That works for Rust's standard library, which won't abandon anything stable (stuff gets deprecated, but that just results in a build warning).
It doesn't make as much sense for Python's standard library where you'd need to be paying attention to goings-on in the community to know if one of the batteries that are included has been judged expired and will be removed.
SO would disable a question of this type for being opinion-based.
I would assume the Python devs review the code for a package before including it in the standard lib, right? That seems like a pretty big deal, well beyond someone just telling you that they tried a package and it worked for them.
Exactly. The single biggest hurdle for me using some open source tools written with newer languages is the ridiculous dependency surface due to a preference for package repositories over curated “batteries included” bundled libraries. I work in an environment that is hyper sensitive to supply chain integrity, which means a lot of languages that have a culture of “use all the third party packages” are out for us.
Give me a batteries included standard library please - much easier to audit and track.
Nearly every place I've worked, the supply chain attack problem has been solved or at least mitigated by using an Artifactory instance with a curated list of packages.
Perhaps the next programming language / ecosystem can remove this problem entirely . Why is it so hard to have access to everything while shipping only what is actually needed?
The need to runtimes shared by multiple applications on a single machine makes this a hard problem to solve, unless it's statically compiled. Of course, Docker renders this moot, and I'm unaware of any languages written in a "Docker-first" manner.
> On the other hand, some features are simply over the heads of beginners, making the code found in packages practically unreadable.
That's too bad. I taught myself Python last year but have kept it pretty "101". That is, I write code simply. Since I am new to Python I don't know if the unreadable code you're describing is because it needs to be (perhaps I am still tooling around in the parking lot of Python) or if it is the result of programmers being too cool for school (ha ha).
Perhaps if the latter I ought to blame the language for giving programmers a basement of esoterica to plunder rather than the programmers themselves.
As obtuse as C could get with pointer arithmetic and the cryptic looking "question-mark-operator", there really wasn't much esoteric stuff in C. Contrast that to my experience with Swift where seemingly every week or so I would come across cryptic code where someone was using a capital T or some bizarre case expression I am unable to mentally parse.
> Turbo Pascal
You suddenly made me miss an ex-girlfriend. Maybe they still code in Turbo Pascal in Portlandia.
There's a fine one just like the ex, Free Pascal, waiting at https://wiki.freepascal.org/ to be checked out. Now the dream can become reality, but do take into account that satisfaction can depend on being realistic.
I think one big factor is easy entry but almost no limit:
Like BASIC back in the day it's a language that allows newcomers to do things long before they are comfortable with the added syntax requirements of C based languages
Despite that the sky is the limit, almost all problems a developer will encounter can be tackled with python. Yes for many problems there are languages that are better suited for the task, but in most cases a project does never even reach a state where that will matter.
And it's installed by default in many Linux distributions.
But I think this may be evidence for the argument that scientists should be paired with research software engineers who take the science codes and turn them into usuable libraries. Scientists don't necessarily need to produce codes that others use, simply ideas that can be translated into useful libraries. Certainly some scientists can do this themselves, but not all can nor should be expected to.
Some projects are distributed on PyPI as source code, and sometimes that source code has C modules that need to be compiled. The same can be true with packages that ship with Rust modules.
Both might require you to have compiler toolchains installed.
It's rare, but I've run into packages that don't install easily with pip, due to not having binaries for a specific Python version + platform (something like this one where the combinations of Python+platform+MacOS version are limited, for example - https://pypi.org/project/virgil-crypto-lib/#files ).
Also there are probably tons of "packages" which were never published officially to pypi but exist on github, that people have made public but decided not to maintain.
Similar camp. Two reasons I enjoy Python, both related to documentation:
1. docstrings
2. keyword arguments
Docstrings mean I can do REPL experiments without the additional friction of opening a browser to RTFM. Keyword arguments reduce how much I have to lookup the semantics of a function's signature.
Python does a great job at putting together powerful scripts quickly. But I have a really hard time opening up a big Python codebase and making sense of it quickly. No static types can make data pretty hard for me to follow. This may just be something I need to improve at, but I have a distinctly better feeling working with unfamiliar code with static types. Python makes me feel I’m at the will of the documenter to be informed, whereas the types self-document in many cases in other langs.
I think this is actually a big difference between SWEs and programmers who are scientists/data scientists/analysts etc. first. In my experience, people with more formal education in programming find type systems to be very helpful, while people from other backgrounds, find them confusing and an nuisance.
I don’t think you need a “formal education” to see the benefits of strong typing. Whether someone is attracted to it or not probably has more to do with their specific use case.
Strongly typed languages is simply a trade off where you get peace of mind by paying for it with extra time and effort spent. For some people that’s a no-brainer and for some people that’s just something that gets in the way.
My background is mostly in sysops and monitoring, and for me types are a life-saver because I value stability and predictability over almost anything else. If my job was something more similar to “ship new stuff fast” I’m not sure that would be the case.
I agree it's a trade off but for me the ratio of effort spent/effort saved in lack of bugs and peace of minds is like 1/1000. Such a tiny bit of effort for huge payback in effort/mental energy saved.
> If my job was something more similar to “ship new stuff fast”
Ironically, the slowest moving codebase I have ever seen was written in Python. It was impossible to "ship new stuff fast" without breaking things. I think Python works up to a certain size/complexity level, then it completely falls apart.
Yes; I've come to realize the yarn "Internal quality and productivity rise and fall together" is more true than I ever thought. And with python, (and IME most dynamically typed languages), the internal quality has far more ways to go awry.
1. Python is very strongly typed, every single item has a clearly defined type. That said, it is dynamically typed, meaning that the variables are not a "containers" for predefined types, but simply labels on objects.
2. What brings benefits is not static typing itself, but static analysis which is mandatory in most statically typed languages (i.e. the program won't compile/interpret if the analysis breaks). With tools like mypy Python has a perfectly capable static analysis - even without any type annotations, although they massively improve it - it's just not mandatory and the program will happily run even if it fails.
Just anecdotal, but I dropped out after a year or so of CS and I value types more than the vast majority of developers I've ever worked with, most of whom have much more formal education than I do.
You may well be right. Personally, I dislike writing python because it lacks static types and all too often I end up triggering runtime exceptions, especially when integrating with third-party code. Do you suppose scientists/etc are more tolerant of runtime errors, at least while writing the code?
In my experience, the biggest issue in numerical/scientific code is to get correct results, these cases will hardly every be caught by typing. Moreover in many scientific programs you deal with well defined inputs, i.e. the chances of e.g. calling a function with an incompatible type are quite low. The most common error I have encountered is a mismatch in dimensionality/shape of input arrays (which I believe isn't straight forward to type for either, in particular if you want the shapes to be flexible, but all inputs to have the same shape).
Wemake [1] is much more comprehensive, although you might find it overwhelming at first. It includes a lot of useful flake8 rules that tackle complexity, enforce consistency and best practices and much much more.
It's honestly not very good compared to a "real" type system (platform). I've used it about a year ago for a couple of projects, and it was painful and ultimately not very useful exercise.
I've used Mypy since 2016 on big as well as small codebases, and it has been extremely useful for me. It caught numerous bugs at typecheck time. The benefits of better readability and e.g. better jump to definition and autocomplete are harder to quantify, but subjectively they feel substantial. If you are gradually annotating a big codebase, it does require a "critical mass" before types start to pay off, I agree that adding them later can be painful.
Mypy's type system is quite advanced compared to some statically typed language like C and Go; it has generics, union types, inference, and it can prove properties of types from control flow. I work mostly in Rust, Haskell, and Python, and I rarely find Mypy limiting.
You do have to embrace types though; if you have a dictly typed program where everything is a Dict[str, Any], then putting that in annotations isn’t going to be very helpful; converting the dicts to named tuples or dataclasses is.
As an alternative perspective, I've found mypy to be pretty poor in comparison to other rich type schemes. Particularly in comparison to the Javascript/Typescript ecosystem, it really feels like taking a huge step backwards.
Partly that's because the ecosystem support isn't there - lots of libraries don't have types, or have types that behave oddly, or even require their own plugin for mypy. I suspect that's going to slowly change over time, but I feel like the infrastructure doesn't feel as strong as it did in the early days of Typescript, and the gradual change feels even more gradual.
Partly it's just that the syntax is painful as soon as you want to do anything remotely complex. For example, generics are defined in different ways depending on what you're making generic, and the TypeVar system is usable, but ugly and very unintuitive. There's also missing syntax for things like inline type assertions, which aren't great, but are often useful for mixing static and dynamic code together.
And I think partly it's also that Python has a much higher level of dynamism in the type system - lots of "clever" things are possible in Python that just wouldn't be possible in the same way in Javascript - which means that mypy has a much more difficult time in trying to describe the idiomatic Python patterns that people are actually using. In Typescript, figuring out the correct types feels like an extension of writing natural Javascript code and understanding the flow of data through the program. With mypy, it feels like I'm restricted to writing the subset of Java-like code that conforms to the type checker.
It's a disappointing feeling, because I like static types, and I had hoped it would help solve the problem of large codebases in Python. But in my last project it became so painful to use, and felt like it was adding so little (and preventing so few actual issues), that I kind of regretted pushing for it.
I hope a lot of these problems are just teething issues, and that over time it'll get better. But right now I would be very cautious about using it in production.
In practice, that's very rarely used, partly because inheritance is so uncommon in Javascript programs. However, for a long time, Javascript lacked anything like the __getattr__/__setattr__ methods, it has no real operator overloading, and decorators are more limited and generally rare outside of certain ecosystems. These sorts of metaprogramming techniques that are very normal in Python are more rare in Javascript, and when they are present, they're generally easier to type.
> You do have to embrace types though; if you have a dictly typed program where everything is a Dict[str, Any], then putting that in annotations isn’t going to be very helpful; converting the dicts to named tuples or dataclasses is.
Or even just TypedDict, which often works without any changes besides annotations.
While I agree that type hinting by its very own nature feels a bit bolted on, I vastly prefer going into code bases which include type hinting. I personally always add type hinting to the code I write, as I actually consider it quite useful.
I remember mypy being slow and buggy. I remember one mypy upgrade broke all our builds because they changed some type of resolution thing. IIRC after some outcry they backtracked and started providing some migration path.
The other thing which rubbed me the wrong way was that the python was happy to run the code with completely wrong type hints.
I guess I went into it with wrong expectations, even though it says right in the name - it's "type hints". The whole experience felt more like a formalized documentation with partially working optional verification (which can't really be relied upon).
I've also had a mixed experience with mypy. Take a look at using pyright for static type checking instead, it's worked quite well for me.
But do write type hints. I recently got thrown into a large-ish project where neither types nor docs where used. Trying to figure out wth a parameter was supposed to be wasn't a pleasant experience for a newcomer. In addition to improving DX, I also believe it's alot more effective in the long run.
I saw how these guys were developing: write code, run code, deal with the runtime crashes they encounter, then run code some more and deal with other unexpected runtime crashes. It would have been a lot faster and more stable if they'd just used type hints and static type checking, as their IDE could've easily found many of these bugs for them immediately.
Type hints are basically for documentation and metadata. You also find a bunch of third party libraries and frameworks, like pydantic, fastapi, etc. that makes use of them.
> Type hints are basically for documentation and metadata.
They are also for static checking.
If you choose to run the code despite that failing (which you can, because there is no execution dependency on type-checking) that’s a choice, but its kind of odd to complain that Python lets you do that.
I'm not the original poster and not complaining about it. I once worked at a shop that ran mypy checks as a precommit hook and never really found it terribly useful, but to each their own.
I don't follow. Python's static types can be gradually introduced to an untyped code base. Sure, they may be unhelpful if you don't use them everywhere. But how can they be painful? It's not like they prevent your code from being compiled?
Having a Scala code base not compile because someone went all-in on type-level programming, and now simple changes require an understanding of category theory and the associated compiler flags .. that's real pain.
python's type-documentations are useful but sometimes they are just wrong which makes it impossible to actually trust in them.
> Having a Scala code base not compile because someone went all-in on type-level programming, and now simple changes require an understanding of category theory and the associated compiler flags .. that's real pain.
Well, the same can happen in python if someone goes crazy and uses a lot of reflection / dynamic changes (i.e. overwriting builtin methods etc.). In both cases it sucks and should have been prevented by other people, but at least in the case with Scala you still have a compiler that can help you "unroll" those changes because it tells you when you screw up. In python you can only have tests or pray.
Oh wow you should definitely give this a shot again. Type hints in Python save the entire language for me.
People do have bad habits of cramming everything into dictionaries but if you do some type hinting and use data classes heavily you’ll really have a good experience.
It's a far cry from an actually typed language. I like Python and deal with its code bases for a living, but frankly it has gotten too big for its britches.
Sorry but this is a garbage take. Typing negates the need for writing useless tests that just verify the correct value types are passed - it does not negate the need for writing functional tests.
It has typescript built in so I can very quickly make a script.ts file anywhere and run it in the CLI with `deno run script.ts`. Works flawlessly and I get access to TS's amazing type system without any build step or setup or even any additional files
for any data-intensive scripts I still sorely miss pandas though
What's the story in Deno with the standard lib and filesystem interaction? Because those are the two vital parts of scripting for me. I dabbled in Node scripting for a bit and ditched it because I didn't feel it was a viable option compared to Bash or Python.
It's all built-in. No extra dependencies or even imports necessary. Relative and absolute file paths work as expected. You can save and read any files freely
You can import synchronously, asynchronously, as a stream, as a string, etc. All the functionality you'd expect is built-in with the Deno module (e.g. `Deno.readTextFile('./my/path.json')`) or even with module imports
When reading large projects written in C I find myself surprisingly little helped by static types. Almost everything is structs, pointer to structs, or structs with pointer to more structs.
I could imagine that if developers choose to use less complex objects and less indirection then types in such large projects would be more useful in explaining the data. It just hasn't been my experience so far.
I feel exactly the opposite, compared to Python at least. It's usually easy to find the definition of the structs, and then you can quickly make sense of what a given function can and cannot do, and what it as access to.
With Python I never know, since something might have dynamically added or changed some methods or fields along the way. I almost always end up sprinkling dir() all over just to figure out what exactly is going on.
I think that's because it's C, where you have to use void* occasionally, structs are often anonymous to avoid compile-time increases from including another header file, and there's a hellscape of pointers involved in doing anything significant. If you picked up a modern language with static typing, like Rust, Go, Kotlin, or even C++/Java, you might see some significant readability benefits.
Imagine how much worse those structs with pointers to structs would be without static typing though. This feels like a complaint against C, not a complaint against static types.
It is a comment about how large projects, at least from my experience, seem to use complex data structures that takes several indirection to follow.
In both C and python I have also see something akin to a mini language inside the large projects, where understanding the code becomes almost impossible without documentation. In C, both macros and pointers to structs with more pointers can do a lot of heavy lifting to hide every detail of how something is being done and just leave the intention. Great if one want to see a high level concept and creating new features, but terrible if you want to know where that one bit of information is being stored and how to inject your own code into the core parts of the project. Similar, large projects in python tend to use meta, monkey patching and other dark magic patterns to really hide the low level details while making it easy to create new features.
I think that's a take in the category of 'you don't want types, you want better names'.
Static types are very useful for compilers but looking at a function and seeing int -> int -> int -> string -> bool -> int, says very little about the semantics of a program. It's always the names and documentations that tell human beings how to make sense of a program.
When we put things in record types, the sense-making value isn't in the static analysis but in the fact that our vague collection of parameters now has a proper name that indicates what it's all about.
> Static types are very useful for compilers but looking at a function and seeing int -> int -> int -> string -> bool -> int, says very little about the semantics of a program.
Sure, but on the other hand a type of APIProxyEvent -> LambaContext -> APIGatewayProxyResponse says quite a bit more about the semantics.
Unless a function is highly abstract, int -> int -> int -> string -> bool -> int is probably an underspecific type signature.
EDIT: To be clear, I generally find that the thesis “you don’t want types, you want better names” comes from assuming bad types, and suggesting replacing them with good names. And, for casual inspection, yes, good names may be superior to bad types. On the other hand, I can’t statically check good names, I can statically check types (good types, or bad-because-underspecific types, but good types, as well as telling me more as a reader, will also statically catch more possible errors.) Ultimately, what I want is good types and good names,
It's not so much assuming bad types as saying that names and types properly used serve different needs. For example, if you have types defined in your program, you could literally replace them with some random characters and the type-checking would be equally good, for the compiler, even if you don't undestand a thing. The value of types really is in the structure they represent, and enforcing certain constraints, not in any human understanding of the program. You could even have badly structured types with decent labels.
When you're using types as a means to check names you're likely to misuse types. Synonyms are a good example of this, where people will make so many types each type only ever occurs once, instead of having a well named variable of a more generic type.
> It's not so much assuming bad types as saying that names and types properly used serve different needs.
Different but overlapping needs, yes.
> For example, if you have types defined in your program, you could literally replace them with some random characters and the type-checking would be equally good, for the compiler, even if you don't undestand a thing
Sure, but for human consumption you want good names of types, not just good logic of types, just like you want good names of variables.
But, while you can (without types) overload names of variables with the human information that would be in names of types, this is bith less ergonomic that separating the two kinds of names, and doesn’t support type logic the way types that are both well-structured and well-named do.
> The value of types really is in the structure they represent, and enforcing certain constraints, not in any human understanding of the program.
I could not agree more strongly with this, it is no more true of types than it is of the rest of code: yes, with any part of code the logic is was matters functionally, but humans maintain the code, so if its not written (both names and structure, within the variations that produce correct behavior) for human understanding, its less useful, and potentially useless.
> When you're using types as a means to check names you're likely to misuse types.
I don’t know what this is referring to. If there is a statically verifiable feature, it is a real type constraint.
> When you're using types as a means to check names you're likely to misuse types. Synonyms are a good example of this, where people will make so many types each type only ever occurs once, instead of having a well named variable of a more generic type.
Synonyms/aliases (at least in languages I am familiar with) are explicitly not types, but alternate names for types. They aren’t checkable, only the underlying type is. They are useful in much the same way as named constants.
The other day I had to make sense of some Python code that used numpy heavily. It was an absolute disaster to figure out what the code was actually doing because of Python's lack of proper typing.
Take for example, the numpy.dot function. This is from the documentation:
> numpy.dot(a, b, out=None)#
> If both a and b are 1-D arrays, it is inner product of vectors (without complex conjugation).
> If both a and b are 2-D arrays, it is matrix multiplication, but using matmul or a @ b is preferred.
> If either a or b is 0-D (scalar), it is equivalent to multiply and using numpy.multiply(a, b) or a * b is preferred.
> If a is an N-D array and b is a 1-D array, it is a sum product over the last axis of a and b.
> If a is an N-D array and b is an M-D array (where M>=2), it is a sum product over the last axis of a and the second-to-last axis of b:
So to figure out what this call to dot actually does and what it's output type is I need to know the actual types of 2 inputs. Of course, those inputs are the results of other function calls, that are also dependent on the types of multiple inputs. When reading code you have to manually keep track of each type as you read through it, you have to look up every function to see what it does and what it can return under what conditions.
Programming languages are not for computers, they are there for humans to understand what is going on. Python absolutely fails at this because python code is almost totally unreadable. It is at best a write-only programming language.
> Static types are very useful for compilers but looking at a function and seeing int -> int -> int -> string -> bool -> int, says very little about the semantics of a program. It's always the names and documentations that tell human beings how to make sense of a program.
Sure, agreed, but if a program contains poorly named types then it's a good bet that if types were not mandatory, those authors who could not be bothered with properly naming their types would be equally incompetent at naming their variables, parameters and functions.
IOW, if author competence is important for typed programming to be readable, then it is even more important for untyped languages!
One thing about math heavy fields is that their core primitives are well rounded and concise, you can write complex math formula because you have a mental map of domains and projections.
On the other hand every business comes up with its own little world.. and suddenly you're in the dark. Here the need for strong and static typing helps.
I've been thinking (and researching) heavily in preparation for a new (offline, desktop) app I want to build. The app has a lot of data wrangling, and probably a decent amount of ML. Python seems like the logical choice, but...
I've written a lot of Python, taught classes in it, deployed production code... and I still feel a semi-conscious urge to reach for something else whenever I contemplate starting a new project. Something about its approach, syntax, common idioms, always feels just a tad clunkier to me than I'd prefer. The whitespace makes it hard to paste lines into my repl, and (probably the biggest thing) comprehensions ARE NOT SUFFICIENT replacements for easy anonymous inline functions (JS () => , or Ruby's blocks).
I like dynamic languages, I really like the advances in tooling with type checking and LSP support. I like dynamic notebooks (either inline in VS Code #%%, or straight Jupyter), the massive package ecosystem (but obviously hate the actual packaging tools), cool tools like Rich... but it just doesn't make me as happy to use as other languages.
I'm still trying to articulate why. Maybe I was just ruined by learning Ruby first.
Unless you have a clear specification for what you are going to build (as in, you have most of your data schemas and processes sketched out), stick with Python and benefit from the development velocity as you figure things out.
I have embraced this workflow completely. I used to be more concerned about which language, but now I find I much more useful to just start immediately in Python. I spend most of the time working out the kinks and edge cases, instead of memory management or other logistics. Maybe ~75% of the time, that's it, no need for further improvement.
Recently I chose to rewrite several thousands of lines of Python in Go, because we needed more speed and improved concurrency. Already having a working program and tests in Python was great. After figuring out a few Go-isms, it was a quick couple of days to port it all for big improvements.
End to end tests mostly. Those were kept static throughout the process as a baseline and for benchmarking. That let us have simple metrics like "test 1 had a 100x speed up" and "test 3 took a way too long to finish before the rewrite."
Any unit tests got rewritten along with the test's relevant code. Now that we are sticking with this Go based solution, adding more unit and regression feels more reasonable now.
This works especially well if you test the python version as you go, then run the same python tests against the C++ replacement. The up front bother of pybind or similar is minor relative to not having to recompile the C++ to iterate on test scripts.
The same thing can be said about Lisp, Ruby, or JS/TS. Start a REPL, shape things as you would clay.
What Python gives you, and what's not easy to get elsewhere, is access to really high performance numerical code, wrapped into quite ergonomic interfaces: Numpy, Scipy, Pytorch, etc. Many other things choose to make Python the glue language, like above, or to embed it (from Gimp to Blender).
This makes Python stand out, despite its shortcomings (which are well-known), and make it an even more reasonable interface language for whatever next big thing. Network effect in action.
Notice the rewrite most often will never come when you do this because it will probably not be needed. And if it comes to it that you do need a rewrite, celebrate the success of your project!
+1 for the Elixir recommendation. Whenever I'm looking for ergonomics, I try to look for the functional languages (Haskell, Elixir, Ocaml, Idris, etc.).
And since you mentioned dynamic languages and Ruby, I think it might be a really good match for you!
I recently discovered it. It's pretty amazing for how incredibly easy it is to build distributed applications with it, that is the part which impressed me the most. It handles concurrency well for sure but same program seamlessly across multiple computers is just awesome. I know it's basically just taking this functionality from being built on top of erlang already but it's still very impressive. Plus like crystal if you already know ruby, having tried to base it's syntax on it, it's an easy thing to pick up if you can grasp the functional aspects too.
>> and (probably the biggest thing) comprehensions ARE NOT SUFFICIENT replacements for easy anonymous inline functions (JS () => , or Ruby's blocks).
I'm confused by this statement. As I understand it, list comprehensions exist to transform data. Inline functions in JS would be equal to lambdas in python, I.e. anonymous functions. So what differences between JS and python do you think is missing?
> declaration of a whole separate function that now lives physically far from its use
Why are you doing that? Just declare the (named) function right there where you’re gonna use it. If it’s any endorsement CPython does it all the time.
> I personally hate the map and filter functions and their goofy list-like types.
You mean iterators? There’s nothing else they could possibly return. Map and filter are lazy and work with arbitrary iterables even ones that never end or have no possible way to define bind.
> Why are you doing that? Just declare the (named) function right there where you’re gonna use it. If it’s any endorsement CPython does it all the time.
I think a lot of people have no idea nested functions exist in python...
Completely agree. Once I discovered Java Streams it became so much more frustrating for me to write Python code that manipulated collections in anything beyond the most trivial of operations.
Whenever I want a lamba that's multiline, I usually just declare a nested function. Usually pretty clear, and reusable, and same scope rules as a lamba
It sounds like you're incredibly picky. I moved away from Ruby to Python years ago for 99.9% of my professional work. Ruby to me is write once read nowhere. "Professional" developers invent all sorts of creative DSLs to make code impossible to use. Rails is insufficient for a project of modest complexity owing once again to it's 9 levels of hell worth of DSLs. If you follow the prescribed formulas exactly, to the letter, and never stray from the DSLs given in Rails, and Ruby code in general, is fantastic. Once you start to stray you end up in hell.
Python has it's faults. Most are handled by a good editor. I only switch from Python when I need something lower level. With Python a company gains:
1. Simple language with relatively low ramp-up time
2. Reasonably fast with proper tuning. I've been on projects processing impressive numbers of RPS built entirely in Python.
3. Comprehensions. You claim they are not sufficient. With them I rarely need to lambda anything. When I do I have `functools` to do any other work.
4. A massive ecosystem of libraries, resources, etc.
A company loses:
1. Projects become difficult to understand at a certain size. This is not properly addressed by even the proper use of MyPy.
2. Somewhat more involved testing process compared to it's contemporary - Ruby.
To compare apples-to-apples you can even look at the interpreters themselves. Python's mainline interpreter has been able to run CIRCLES around Ruby's for years now. I can understand using something like Go for systems level work, but there are few languages that give you so much for very little investment. Even dropbox was run entirely on Python for LONG time before it became too unruly. That's INCREDIBLE given the simplicity of the language.
What you use on your personal time is irrelevant. Your complaints apparently are not experienced by the swathes of companies switching from almost every language under the sun to Python. It's here to stay. At least until Go's ecosystem catches up to Python's. My belief is Go will be the only language to unseat Python at this point. Python will still be around as a "perl-replacement" for basically ever as far as I am concerned.
This is usually the case with people who use and like Python, which can be a stubborn take. And I can’t quite put my finger on why Python seems to draw out these sentiments. They usually act like there's some objective reason when the reason is usually because they like Python and simply don't want to learn or use anything else. The concluding paragraph of this article confirms this.
> Most criticisms I see leveled at Python are still completely unfounded. Many times the criticism can be addressed by using the language in a different way.
Is that true? Functional programming missing, slow, poor module system and REPL, weird scoping, environment and distribution issues, the nightmare that was 2.x to 3.x, tacking on features a la C++, etc.
> So be it; if I’m not working in one of those areas, then Python is still probably the best fit for me. I used to hear that Python wasn’t the best at any one thing, but it was second best at most things. I agreed with that line of reasoning for a long time, but these days Python is as good as any of its peers for many things, and it’s still quite effective in many areas where it might not objectively be the “best” fit.
This is just weird, especially if one doesn't know or use other modern languages. F#, Rust, Elixir, Ruby, Racket, OCaml, etc.
In my opinion, unless Python is being used for scripting, machine learning, or scientific programming, then it's the wrong tool. And I think other languages can even compete with the scripting and scientific programming aspects.
I suspect they mean that Rails gave Ruby its 15 mins. (and why the lucky stiff!) I, personally have no issues with Ruby (I was a very early adopter coming from Perl, C, C++, Python), but find it to be "Python with extra steps" more or less.
I'm a relatively inexperienced developer compared to a lot of people on HN, so I suppose take my perspective with a grain of salt. For context, I started with JS, spent some time with C because I wanted the history/perspective (didn't have a practical need for it, though), then got into bash for purely practical reasons. I also have a bit of experience with C#, primarily with Unity, and C++, because ultimately I'm looking to build audio plugins and applications, but at this point I've done little more than take a tour of those two languages. A year ago I began writing Python because I inherited a lot of code from a former coworker. At this point I'd consider it my main language.
I've heard people echo the "Python is the second-best language for anything" sentiment a lot. A bit broad and a touch pithy, but over all I'm inclined to agree. It's a language clearly (and explicitly, as said by Guido) designed to be readable, and I think that more than anything else is why I have a case of the warm fuzzies for it. Compare even a small thing, like f strings in Python vs string interpolation in JS -- f strings are simply less noisy and therefore easier to read. Decorators, when used well, make it a fantastic language with which to write libraries that a user can grok quickly. Etc.
If I could design my perfect language, it would be Python, just with clear variable declarations via a keyword such as `var`, optional true static typing and consts, and better tooling -- though, I have to say, after diving in and really learning the ins and outs of venv, poetry, all that stuff, the "Python's tooling is a disaster" meme is IMO a touch overblown; I felt that way as well at one point and in hindsight it was due to me feeling overwhelmed, as opposed to any objective evaluation of the tooling; that said, there is definitely room for improvement.
I myself am incredibly interested in properly learning C++, Rust, Go, and improving my JS skills, but overall it never quite feels like the juice is worth the squeeze, because Python really does just feel like "the second best language for anything." And since my interests are pretty wide -- everything from web dev to scripting to desktop apps to audio (well, Python is perhaps the worst language there), most of the time it just seems more practical to figure out a way to do the thing I want to do in Python, especially because there's usually a library for it already. That said, I'm a solo developer at a non-tech-related-industry job, so I'm not rubbing up against many of the same frictions I see discussed so often here on HN.
As I said, though, I'm still pretty green and am aware that I know just enough to be dangerous, so I'm very open to hear why anyone disagrees with this viewpoint.
> If I could design my perfect language, it would be Python, just with clear variable declarations via a keyword such as `var`, optional true static typing and consts, and better tooling
I highly recommend taking a look at F#. It is functional first, but it is multi paradigm in that it supports imperative and OOP programming just as well. You can write code in F# effectively how you’d write code in Python, if you’d like. F# is indentation sensitive, has sane scoping, the dotnet CLI tooling, modules, pattern matching, easy and regular async, scripting, notebooks, static typing with type inference (meaning you don’t need to type annotate everything), etc.
I would argue that of everything Python does, F# does it better and more. So if we accept that Python is the second best language at everything, then F# becomes a rather interesting language to consider.
Thank you! I've always wondered if/when I'd have a reason to check out F#, so now I do! :)
Also, after I wrote the above comment, I checked out Nim, and it in many ways is exactly what I was describing. And Cython as well, even for all its (thankfully continuously reducing) weirdness.
But seriously. I like how fast I can start getting results with it. Sometimes I don't even need to create a file, I simply do "python3 -c ..." or use it as an interactive shell
Python has only one really serious drawback for the kind of tool it is and its not remotely its own fault: its not a first class resident on mobile.
In a different universe its sweet easygoing manner would have created enormous value by enabling all sorts of casual programmers to have some control over this most ubiquitus of computing devices.
Alas we dont live in an era where tech aims to empower users.
NB: There are a number of projects that allow some of this (kivy, beeware) but they need to swim hard against the current.
Python is weak on front-end web and mobile, but I suppose its impossible to expect Python to be great at everything. For the number of niches it can fill, I think Python is still ahead.
maybe thats precisely because the frontend has turned into a monoculture. having multiple competing stacks with different pedigrees helps innovation. one example I can think of is the erlang/elixir/phoenix/liveview stack that revisits old patterns and gives them new life.
who knows, maybe wasm and projects like pyscript will stirr things up
I know thats a big part of the reason I wish it was accessible natively and frictionless in Python. I'd like to not learn a whole new inferior language to make a decent UI.
> the issue is slow startup time due to module imports being dynamic and slow
yes that is still the case e.g with kivy based apps (and apk size too, as you need to ship the entire stack). but such issues could be helped if python was to be promoted to "first class citizen". apparently, at least on the android side, the possibility was considered and rejected early on
in any case, whether python or something else, what is missing is easy programmability on mobile. current devices feel more suffocating than Microsoft Windows desktops ever were. you can also see it in the open source app collections in f-droid which I would say is on the underwhelming side.
Most developers need to have at least two languages at their fingertips, so none of them needs to be perfect. What is important is to a spanning set for all the use cases that are relevant to you.
I think if we switch from endless arguments about best languages to the optimal pairs of languages we'd book huge gains (and less online noise :-)
Off the top of my head, it lacks decent support for functional programming (like multi-line lambdas - pretty basic functionality even for a non-functional-oriented language!), properly supported static typing, proper multithreading, decent performance (in the main implementation)...
> like multi-line lambdas - pretty basic functionality even for a non-functional-oriented language!
Lambdas with one or more statements instead of a single expression (the actual limitation: “multi-line” is a misnomer) are only an issue because python is a strongly statement-oriented imperative language, and not expression-oriented like most functional and some other languages.
> Functions are first-class objects. If you want a multi-line lambda thats just a function.
Yes, lambdas are just anonymous functions that are restricted to a single expression, no statements. This is a problem for programming in functional style, because python idiomatically uses statements a lot, and doing a named definition for a single use function is more verbose and less readable in mant cases.
You can define a function anywhere, including in the scope of another function. So it doesn't need to be floating somewhere far away from where it's used. (This is the idiomatic way to write decorators, for example.)
Why do your anonymous functions have to live far from where they're used? What's the non-aesthetic difference between this:
def func(g_func, a, b):
g_func(a, b)
def scope():
c = "c"
def anon(a, b):
print(a, b, c)
print(id(anon))
return func(anon, "a", "b")
and this:
def func(g_func, a, b):
g_func(a, b)
def scope():
# NOTE: The `|a, b|: (...)` syntax is made up
c = "c"
return func(
|a, b|: (
print(a, b, c)
),
"a", "b"
)
?
Note that the `anon` function in `scope` is only compiled once. You can verify this by copying the first version to a file named `temp.py` and then running `python -m dis temp.py`. You can also run `scope()` twice in a row and see that the same object ID is printed both times.
That said, for purely aesthetic reasons, I would like it if Python had some kind of syntax like version 2, but that's tricky with significant whitespace, and I've never found the lack of such syntax to be particularly limiting in Python.
You can delete references to any object in Python, including functions:
>>> def f(): None
...
>>> del f
but it's pointless to try to manually manage memory in Python in most cases. If you're processing huge amounts of data, there are probably better approaches than using `del` and/or `gc.collect()` to keep memory usage down (like batching and/or using subprocesses).
Whenever I think I should switch to faster, more efficient language (C#/Go/Rust/...), I am repeatedly blown away by what Python lets me do, with an incredibly ecosystem of packages, and all the ergonomics of incremental development from a REPL.
Case in point is my current project: building a parser for PDF bank statements from all UK banks. I can match a 10page bank statement against a library of 100 templates, and then extract its data in a standardised format, in around 50ms per statement.
That's broadly comparable to the time for load the source file from cloud storage.
And that 50ms is processing pages sequentially, using a single core on my little laptop. Plenty of scope to parallelise if I wanted to.
The reason why I'm still using Python, to borrow a slogan from Bruce Eckel long ago, "Python fits [my] brain".
50ms doesn't sound especially fast. If you had a million of them to process thats 4 hours, maybe an hour with some parallelism? A 100x speedup from going native then becomes quite welcome. Python has its place, but the fact you even think the tens of ms domain is "fast" or even fast enough on modern CPUs shows the real strength of Python, which is most people dont actually care at all about performance. Thats not to say its performant though. Just that no one cares anymore. Some rando boss is happy to wait 4 hours for your script to chug through a million statements, because no one actually told them it can take 2.5 minutes in a native language. If they did maybe the boss would suddenly speak about their dream to not only process the data but have a near real-time BI panel instead of a batch report so they can react within one business day. The issue with Python is missed value, not the value it can deliver.
Why limit parallelism to 4x? Spin up a ton of lambdas and get it done in 10 seconds.
You're also forgetting that 50ms time to load each file from cloud storage. It's still 4 hours and 2.5 minutes with your native code compared to 8 hours with Python. Suddenly not such a massive improvement.
Even then I don't see the issue with it taking 4 hours to do a million of them. Do you need to do a million per hour? Is it even likely they'll need to do a million of them total?
Do bank statements even come in frequently enough to do a realtime dashboard with? I get my bank statement every 30 days.
How much longer does it take to develop the native version? How much longer does it take to modify when a bug is found or a bank changes their statement layout? How much more do you have to pay a native code engineer compared to a Python dev and how easy are they going to be to replace eventually?
One dimension to consider here is cost of compute.
Going from 8 hours to 4 hours is a 50% reduction in the time to compute and we're assuming this is occurring on the same relative hardware/instance size.
At scale, that could translate into hundreds of thousands of dollars in savings.
But your points are relevant - as with anything related to development, "it depends" rules the day. There isn't a clear cut "x is objectively better than y" in general.
> The issue with Python is missed value, not the value it can deliver.
I'm very aware of the business value here! The limiting factor in these types of "messy real-world data" problems is the developer time to get 100+ templates right on all the different variations encountered in the wild. I can iterate on each template extremely effectively in a Jupyter notebook REPL, and immediately rerun a sample of 100 statements for that bank in a few seconds.
While the total corpus of statements I have access to is actually around a million, no one cares how quick processing them all are if the extraction isn't reliable enough!
Exactly. The time spent developing a solution for your problem in pretty much any other language is going to cost magnitudes more than processing millions of documents in python. And that’s ignoring the ongoing maintenance where the hackability of teasing data out of PDFs in ipython is going to top any other system.
I have a soft spot for the work you’re doing since I’ve spent a good portion of my life now extracting data from PDFs and can appreciate the joys of the process more than most.
> If you had a million of them to process thats 4 hours
If you had 20 bank accounts sending you monthly statements for 100 years you'd have 24,000 of them. 50ms each would take you 20 minutes to process the 100 year backlog.
If you have a million of them to process then you're a bank or similar type of institution that can devote resources to this, either computational or developer time to optimize things. A 128 core c6a.32xlarge would turn the runtime from 4 hours into 1.9 minutes and cost $5 to run for an hour.
I agree with your reasoning and your insight about missed value is valuable, but this is not a Python problem. It's a scaling problem.
You can solve scaling problems in many different ways with different costs. Any reasonably sized company has a Kubernetes cluster, just launch 10 instances and you're down to 24 minutes instead of 4 hours. You could also buy faster hardware. You might increase single-node performance by rewriting in a performant language (this won't help much if I/O is the bottleneck though, which often is the case). Talking to your data supplier to check if they can send the data in a machine-readable format is probably also something to consider. Or you might conclude that any effort is not worthwhile at all. Who knows?
Scaling problems are not be avoided as a negative thing, they're beautiful and ought to be celebrated. What's not beautiful is letting the thought of potential scaling problems lead you to make decisions that optimize for long-term results instead of short-terms results.
A laptop generates about 27 grams of CO2 per hour. So going from 4 hours to 2.5 minutes would save about 0.1 kg of CO2 each time the app is ran. So for a company of 1,000 people working 48 weeks per year that's almost 5 tons of CO2 that could be saved.
Ah well, 5 tons of CO2 is a fair trade off for being able to use Python, right?
I understand where you're coming from, and I do care about the climate as much as the next person, but this comparison seems downright silly. Basically, for your example to work every single person have to run the same script every week and wait for 4 hours. Are there even that many bank statements? And then, if you, your spouse, and a child take a single flight coast to coast, round trip, that is 4.5 tons right there.
My problem with Python, especially when reading someone else's code, or my own old bad code, is that I find that Python doesn't fit [my] brain.
The lack of typing (in my experience you always end up with a bunch of libraries that don't have type hints) means I spend most of my time just figuring out what I'm supposed to pass to a function (I hate the django docs. Give me something like docs.rs where I can quickly scan what functions something implements, and see what they accept and what they return instead of forcing me to read an 800 page novel that STILL doesn't give you that basic info), and writing tests and trying to guard against passing the wrong thing around.
It's gotten a lot better with type hints and typeshed, but I agree, not knowing what type an arg should be and hoping duck typing works out isn't great.
In my own code, I always use typehints and write good docstrings (nearly) everywhere.
Visual Studio Code's type checker saves so much time and has improved my code quality to no end. It is especially powerful on polymorphic inputs, making sure the different code paths operate on the input type you expect.
I started enforcing type hinting in my personal projects. 99% of the time it's easy and useful to add. When I can't figure out how to make a mypy error go away within 30 seconds, I just disable it for that line. I feel this makes for very good 80-20 value/effort tradeoff.
I will say that more than one bug has been found after I disabled the single line hint though! Always surprising to me how powerful types can be.
I do not see the benefit of type hints. Good docstrings (and naming of the function and arguments) are superior, at least for the human.
Type hints are too much clutter.
There’s a certain program size from which they start to make sense and further still the benefits are obvious - you have a computer to help solve puzzles for you, so why not use it if the puzzle is ‘will this peg fit into that hole?’ when you have thousands of pegs and holes.
And so you end up writing climb_the_wooden_ladder(l: WoodenLadder) plus climb_the_aluminium_ladder(l: AluminiumLadder) or press all into an OO hierarchy instead of
climb(thing):
'''Climb a thing with steps'''
# prevent run time crash:
if not hasattr(thing, 'getSteps'):
hitDeveloper()
Docstrings are great until you change your mind about something fundamental.
Once you make that big scary change you might have to rerun your unit tests dozens of times before you've found and fixed all of the bugs that your change caused.
A static type checker, plus a bit of foresight and diligence, will give you a much more complete view of the carnage up front, so that you have a good idea about the cost of making that change while it's still cheap to decide differently.
And the best part is that with python you can only bother with the type hints where you need them (e.g. at the boundaries between teams)
I agree with you there. The experience I was thinking of here involved taking a codebase which wasn't modular enough and making it modular. So the "big scary change" involved changes to the module boundaries (which are the only thing I bother with type hints for anyhow).
Since I'm programming in Python, I experienced invalid code only in very few cases, more or less just where the function expected an iterable and I gave a string instead of a list of strings (typing errors are well prevented by the colour hints e.g. neovim gives with the right plugin).
And some libraries sould be better documented.
But the compile time error messages of C or C++ or Delphi most of the time do not deliver a shorter feedback loop, because they are generally ugly and hard to understand.
And you should write tests in all languages, so static typing isn't a timesaver in that respect, too.
>Case in point is my current project: building a parser for PDF bank statements from all UK banks. I can match a 10page bank statement against a library of 100 templates, and then extract its data in a standardised format, in around 50ms per statement.
I'm interested in management of domestic documents and similar institutional correspondence, and I'd love to hear more about how you approached and solved your problems with the PDF bank statements.
I have lots of bank statements ... and other documents like utility bills, rates invoices, etc ... and I'd love to just be able to feed them to a script and automatically tease out salient details.
Plenty of languages have a huge package ecosystem. Your problem is that you don’t know them. I’m not saying the distinction is helpful for your situation, but your problem is “I have too much time invested in learning Pythons ecosystem” and not “other languages don’t have all the packages I need.” For REPL that’s another mindset. You use a REPL to experiment and prototype, I use unit tests. You use python because you already learned it. Your brain has been fit to python, not the other way around. You can fit it to other languages if you choose. I am currently learning my 7th “use in production” language.
You're partly right. However Python's sweet spot for me is when writing code to process complex, messy, poorly understood data. Using a Jupyter notebook lets me keep partially processed inputs as live objects, examine and change their in-memory representations, and quickly prototype the next stage of processing. It feels like sculpting with playdoh.
Once that's working, the code moves from the Jupyter notebook into a proper code module with unit tests. Then the emphasis is on building the production system rather than manipulating the data.
I get what you mean with unit tests. Just for me it feels further away from the data, which is where most the complexity in my business domain lies.
I have Python analytics capturing HTTP Headers, processing them, and finally storing them in less than 8ms on a single core dev instance. Python is plenty fast for many things.
Also, I love the syntax. It reads like a sentence.
More precisely, modern CPUs are so much faster than network or file I/O that if your program has to wait on I/O, the CPU is going to sit idle most of the time anyway, so trying to optimize CPU time is pointless. The GP's use case is an example: he's capturing HTTP headers, so the bottleneck is network I/O, not CPU. So Python for his use case is, as he says, plenty fast.
For use cases where CPU is the bottleneck, such as numeric simulations, Python users reach for something like numpy, where all the CPU intensive stuff is being done in C extensions, not Python bytecode.
The person you’re replying to is saying that Python meets their needs. Judging by what they’ve said about their use case, I doubt it’s a meaningful affirm to their overall pipeline.
Your counterpoint is that it’d be slower if it were running on a Pentium 4. Do you honestly think that they don’t know this?
Yes, this is precisely the point. Things are at a point where Python’s benefits are worth more than what another language would bring in speed improvements. I imagine that there would be less of a benefit to busses if we had to Flintstone them everywhere instead using a combustion engine. That’s not the current reality though.
When building a large system, the choice of language is always non-binary. You do not have to commit to one and only one language for the entire system. Python offers a great stating point for most cases.
When you scale, certain modules will start becoming the bottlenecks. At this point one can just re-write the given module as a sidecar and Python by design is easy to bind external libraries.
I have a completely opposite reaction, one of the things I hate the most with Python is its list-incomprehensions. They quickly become hard to read and very imperative, instead of declarative. Something like stuff.filter(it.unprocessed).map(cleanup(it)) conveys the intention behind each statement a hundred times better. And list comprehensions becomes impossible to reason about once they involve flatmapping or other relative simple functional expressions.
I agree very much so. List comprehensions and can be nice and concise and readable in many situations, but due to the whole "one true way" thing in Python, they're used in every situation, even when a map/reduce/filter etc would be better. Although since there are no pipes, I sorta get it.
Yep. What bothers me is when you start looking at list comprehension is that you don't know if it's filter(), map(), or a combination. You have to read the list comprehension till the end to understand.
Definite agreement. I started a long time ago with Python2, and what attracted me was how simple and readable the syntax was.
With Python3 there is a move towards "idiomatic" Python code, which to me means moving away from clear and concise to the old programmer trap/readability nightmare of "look how many things I can get it to do with one line of code!"
Let's not even talk about those insane people who use nested list comprehensions.
Not sure if it would make my top-5 Python-hate list (1-4 would probably be dedicated to packaging and distribution), but list comprehensions definitely feel like a misstep that contributes little value to the language.
You can easily replace this with various forms of map/filter in many other popular languages. Assuming cleanUp is a single-argument function/static method:
These are also easier to read and write than the Python version, as you can just follow the dataflow from top to bottom (or left to right if it fits into a single line). In Python, your eyes have to jump around to find where to the next bit of processing happens, not just in comprehensions but also with map/filter/list/len and co which are freestanding functions for some reason.
Sure you can do that and I use things like this all the time in other languages, but the python example was very basic. There are other cases that are sometimes harder to reproduce e.g. when iterating over dictionaries to create a list, when you have tuples etc.
All Python list comprehensions do is [(expression involving foo's) for foo0, …, fooN in (iterable) if (expression involving foo's that should return a boolean-ish value)]. Nothing more, nothing less. Nothing that can’t be expressed by filter-then-map.
So, given the dictionary {"a": 1, "b": 4, "c": 2}, if you want ["b=4", "c=2"], you can do this in Python:
d = {"a": 1, "b": 4, "c": 2}
l = [f"{k}={v}" for k, v in d.items() if v % 2 == 0] # replace [] with () if you want a generator instead of a list
Or this in C#:
var d = new Dictionary<string, int> { {"a", 1}, {"b", 4}, {"c", 2} };
var l = d
.Where(pair => pair.Value % 2 == 0)
.Select(pair => $"{pair.Key}={pair.Value}")
.ToList(); // remove this if you want an IEnumerable<string> instead of a List<string>
The Python version has tuple unpacking, whereas the C# version iterates over KeyValuePair<TKey,TValue>. Slightly more typing, and perhaps slightly less ideal if you’re dealing with tuples without field names, but still pretty readable. The C# solution is also more obvious about the execution order.
And if you want to sort the output by the dictionary value?
l = [f"{k}={v}" for k, v in sorted(d.items(), key=lambda kv: kv[1]) if v % 2 == 0]
# 5 3 2 1 2a 4 <- Execution order
var l = d // Execution order: 1
.OrderBy(pair => pair.Value) // 2
.Where(pair => pair.Value % 2 == 0) // 3
.Select(pair => $"{pair.Key}={pair.Value}") // 4
.ToList(); // 5
I find the non-Python versions easier to read, since they show the exact data flow in order (instead of Python’s [3 for 1 if 2] order) and they allow more advanced scenarios.
Another effective technique for making readable, maintainable code is to break the list comprehension out into a generator (as a closure):
This example…:
a = …
vs = [
v(x)
for b in a.f()
for x in b
if x and c1(x)
and not c2(x)
]
…becomes:
a = …
def each_v():
for b in a.f():
for x in b:
if not c1(x): # comment
continue
if c2(x): # comment
continue
if x:
yield v(x)
vs = set(each_v())
Line orienting each conditional makes commenting much easier. The collection used for vs is clear. The formatting is less susceptible to being churned up by black. You get to give the thing a name, which helps.
Additionally I find that it's easier to add complexity to the loops; once you want to start doing nontrivial things in a list comprehension, you often need to rewrite it into a loop first anyway.
This is basically selling point for comprehensions. You can't add complexity to them so any comprehension you see fits a few common loop patterns you can grok immediately.
If you see an actual for loop in python it means "pay attention something odd is probably going on."
The compact gets rid of all the null values returned when unprocessed is false.
Interestingly I wasn't exactly sure what the python code would do, which is why I used the compact. In the python code, does 'stuff' end up only having the result of the cleanup function for items that are unprocessed? Or will it have all of the items, and the ones where unprocessed returns false just end up being returned unmodified?
My Ruby version returns the processed subset of items that were unprocessed. The python version was not clear to me, since I don't know what happens if s.unprocessed returns false.
So basically, it sounds like we were both confused by the same parts of the language we are unfamiliar with... what happens to elements where s.unprocessed is false? It was clear to me that in Ruby it returns a nil, but I have no idea in Python... what happens?
Nothing, literally. This is map+filter optimized (or a for+if+append but using dedicated opcodes for efficiency) so you don’t have to do multiple passes and/or create temporary lists with superfluous elements.
I like list comprehensions now after having used the language for a while, but I will say they were the hardest for me to grok when I was first learning the language. even now it can be difficult to follow what's going on if you have nested ones.
I love list comprehensions but my brain starts melding with nested list comprehensions. The syntax is not obvious and I have to Google it and figure it out again 100% of the time. The JavaScript .map() method is much easier to understand.
I may be the outlier here, but I have a very hard time comprehending list comprehensions. They've always seemed like a SQL query compressed into a single line (or two if you're lucky!): readable when it's simple, Lovecraftian when it's not. I've always found the functional pipeline style much more readable and much easier to explain to others, especially with IDE type hints.
Python comprehensions in particular are unfortunate because the flow of the query, like in SQL, is not linear - the projection part is written first, but happens last. Thus, reading a non-trivial query requires moving back and forth to resolve the references.
Compare that to C# LINQ, which is also a kind of sequence comprehension. That one not only has linear syntax where the data flow in the query is strictly left to right, but it's explicitly defined by mapping it to functional pipeline style. Thus, it ends up being a strict improvement on the latter.
(Another example of "good" sequence comprehensions in this sense is XQuery FLWOR)
> they’ve always seemed like a SQL query compressed into a single line (or two if you’re lucky!):
If its more than a very simple expression, with one simple for, and no if or very simple if, I try to use at least (more, with parens, if necessary): one line for the result expression, one line for each for clause, and one line for each if clause.
But, while the boundary is subjective, it doesn’t take much before functional pipeline style is clearer (Python lambdas being more limited and syntacticaly cumbersome, functional pipeline style isn’t as clean in Python as it is in other languages, though.)
IMHO the key to legible comprehensions is to use them only to express composition and to put the logic into verbosely named functions.
Chained pipelines (`foo.filter(...).map(...)`) are fine to me, certainly more verbose then comprehensions for better & worse, but I get lost in pipelines expressed as nested functions (`map(filter(a, ...), ...)`).
h1(items) + rest of my code...
I might see what are you using for but I am not sure if concenating implicitly (or with line breaks / spaces) is a good design decision for a language.
your example has also multiply statements. The question is more important: Which syntax to a more generally better readability. (as this is solvable in most languages)
No, it is built from pure expressions. This means that our list is defined in one block (everything within the [ ] tokens) and immutability is enforced by the compiler.
It's the ecosystem. All languages have libraries for bottom-up parsing. But few are as well-documented, feature-full, and easy to use as SLY: https://sly.readthedocs.io/en/latest/ Many languages have libraries for ANSI color support and pretty printing, but no other language has Rich: https://rich.readthedocs.io/en/stable/index.html While other languages may have rudimentary datetime support, Python has (multiple!) libraries for converting between Hebrew, Hijri, Coptic, and Armenian dates.
A group of researchers published a paper on this topic, "Energy Efficiency across Programming Languages" [0]. They have tables listing the relative energy usage and memory consumption.
It's just one paper of course, but the energy results weren't too surprising. C was the baseline. C++, Rust, and Java close behind. Languages like C# and Go near the middle, and then Python, Ruby, and Perl at the bottom for most energy used.
I only skimmed it, but it looks like they forgot to account for the extra energy used to run the coffee pot, for the programmers using less productive languages.
This got a reasonable amount of attention, but it’s not particularly relevant. The example solutions would never be written in pure interpreted Python - you would import numpy and use the numeric functions from there, effectively using C rather than Python.
You seem to favor human brain cycles over CPU cycles. Conversely, time spent on memory management and pointer arithmetic is time that could’ve been used to add value to the world.
Most code isn’t fire and forget though. Bugs occur, requirements change. The human cycles are a perpetual investment. And Python is simply very, very maintainable.
I do agree that there is a point where the scale tips. I wonder where that is, and what kind of application we’re talking about then. Many applications are at their basic level old corner shop crud, really.
> Well for code that gets executed billions of times spending more human cycles seems a fair trade off.
Do you feel the same way about (e.g.) modern mechanized farming compared to classic farming that used human labor (generally unfree and unpaid labor, such as serfs or slaves)?
Comparing manual labor to industrialized farming is comparing (paper) accounting books to programming.
Whether you add gps based differential seeding to just doing it by ‘feel’ but still with machines is the difference of programmer brain cycles. Efficiency is not all obviously and there are different efficiencies one can find relevant. Why is this controversial?
You're advocating for more programmer work and less machine work. The situations are exactly the same. Just because programming is "brain work" rather than "muscle work" doesn't somehow make it not work.
Not a true dichotomy though. I'm more productive in Rust then I am in Python, and I'm not currently spending time managing memory or doing pointer arithmetic (you can end up doing some of that where it's inescapable, eg cyclic data structures & embedded devices/interfaces that requires manipulating memory). Though YMMV as far as productivity goes. There are of course many languages that occupy a similar niche, this is just the one I happen to favor.
On the other hand, when you spend more programmer hours on less productive languages, those programmers end up having a larger energy and natural resources footprint with the increased amounts of money paid out.
99% of programming projects aren't "web scale" and the development costs far outweigh the operating hardware costs.
Why hasn't anybody created a more efficient version of Python? Slap some static types on that baby, get rid of the heavy runtime, and you've got a good start toward something special.
There are more efficient interpreters and compilers, problem is they don't work with the existing libraries. If you mean another language, there is Lua for scripting purposes and Julia for math stuff. JavaScript also exists, which despite its fame it is actually MUCH faster than Python, and you can even run it in a runtime such as bun.js and get even more out of it.
IMO the biggest problem is that the Python libraries ecosystem is just too good to miss out on. I absolutely hate Python but even I have to admit it is much easier to do some stuff in Python compared to other languages simply because of the libraries available to do basically anything you could dream of.
Python performance, both single core and concurrent, seem to be its primary Achilles heel. However, Python is unmatched for developer speed in compute heavy AI / ML since it uses C, etc., under the hood. I don't really like Python but it's impossible to avoid in data science because you don't have to re-invent the wheel.
Several languages have persisted beyond their natural shelf life due to their library ecosystems. For a period of time, CPAN was one of the best arguments for using Perl. More recently, Java's ecosystem is also a pretty good reason to use Java right now, even if you don't like the language.
Languages seem to thrive early due to initial ease of use (Python, particularly, but also relative uptake of Go vs Rust), but can wane due to a thousand cuts (or sigils in Perl's case).
The OP's comment is that while Python's native interpreter is slow, Python code as a whole can still be fast since people will delegate the hot spots to non-Python libraries optimized for speed.
To refute that, you linked to John Carmack quoting Horace He saying that Python's native interpreter does CPU-based math slower than a GPU. But...that's why people writing Python use libraries to delegate math-intensive work to a GPU, which is the OP's point.
If one is forced into writing code that needs to be fast in a language other than Python, then I'm not sure I understand why Python gets to claim any credit?
Slight nitpick: the credit that was being claimed is that Python allows fast developer speed while not sacrificing overall runtime performance.
The code that is written in a language other than Python is often not written by the developer writing the Python code.
As an example, I've been fiddling with inferring automatic data extraction rules for data in scraped webpages.
The meat of the algorithm and its heuristics is in Python. The heavy lifting of parsing HTML and evaluating CSS selectors is done by a C parser that I did not write. For me, prototyping the heuristics in C would have been quite a bit slower.
I generally respect Carmack, but it's clear he wasn't considering Jax or even numpy, but just python flops. Jax code running on TPUs often reaches order of magnitude of the hardware's capabilities.
But would this "subset of python" be actually useful?
Once you break compatibility stuff doesn't work… and a fast python that can't run my software is less useful than a slow python that runs the software.
Probably doesn't have to be a subset. Call back to cpython for the really dynamic madness, much like a jit tiers back out of machine code when assumptions don't hold.
> Why hasn’t anybody created a more efficient version of Python? Slap some static types on that baby, get rid of the heavy runtime, and you’ve got a good start toward something special.
The biggest strength of Python is an ecosystem built on a blend of python code and native extensions, leveraging Python’s strength as a glue language. Dumping the runtime while remaining compatible with the extension ecosystem is quite a challenge, as is adding static typing and taking full advantage of it while also maintaining compatibility with existing ecosystem of code which is a mix of untyped and typed in the existing (optional) type system.
Which is probably why most “more efficient” Python efforts don’t try to do both of those things. Some did, or at least replacing the runtime, but most of them fizzled, not because they didn’t offer benefits, but because burning access to the existing ecosystem wasn’t worth those benefits. OTOH, what we see now is efforts to improve the base interpreter while maintaining compatibility, and opt-in compilation tools that can be used within Python, with a subset of Python or a statically-typed Python-like language that builds to extensions for the Python runtime. Examples include mypyc, Cython, Numba, etc.
Reimplement a large enough, commonly used subset of python stdlib using this dialect and we may be in the business of writing cross platform apps (perhaps start with android and Ubuntu/Gnome)
You may be interested in https://en.wikipedia.org/wiki/Cython which was a fork of Pyrex which dates back to the very early noughties. It is basically as you describe and even has a way to "bypass the GIL" in extension modules. There is also an easy way to create a "compiled script" using `--embed`. This is essentially a gradually typed system [1] like the Common Lisp `declare` facility. Cython even has a warning system to tell you what you forgot to declare, and an annotated HTML generator to highlight "more Python C API heavy" portions of your code.
Personally, I think going all the way to Nim [2] is more satisfying than a "gradually typed" system, though.
There are several "close to python" languages like this, actually, I remember one popping up on HN semi-recently.
If you expand into just "kind of Python-like", you have languages like Nim that approximately fill the hole of "efficient Python".
My impression, though, is a lot of the popular frameworks/libraries for Python lean heavily on runtime reflection in Python (e.g. Django). So people end up stuck to the full Python interpreter/runtime.
Nim has impressive compile-time reflection which with its syntax macro system can (some of the time) get you similar powers to Python's dynamic reflection. E.g., https://github.com/c-blake/cligen instead of `autocommand` or `click` or `argh` or you name the automatic CLI generator, only with full C-like speed for your programs and a very low effort FFI to C. People have done web frameworks, too, but I do not know them well enough to compare to Django. Perhaps someone else could follow-up.
Compile times are not the 25-50 ms of a Python interpreter startup, but they are similar to the 250-500 ms of a Go or D compile, at least for small light on metaprogramming loops kinds of programs, and with, say, the TinyC tcc backend.
Of course, big frameworks take a long time to import (or to compile). So, your comparison mileage may vary. And once "ready to go" and finished a Nim program can start up in sub-millisecond time like any C program (esp. if statically linked).
I consider Ruby to be quite similar to Python. However, it's kind of like Python's weird brother, with optional parentheses for calls without arguments, auto-returning the result of the last expression, and some odd-ball syntaxes. It manages to be even less efficient, though.
As someone who probably has more experience with Ruby than you do, and probably less with Python, I'd characterize the relationship as inverted -- Ruby is more consistent than Python.
In Ruby, parens are optional for all methods, and only required if you need to force non-default precedence.
Returning the value of the last expression seems reasonable to me (just like a UNIX shell).
In Python, I'm bothered by the little inconsistencies. Why is str.len (or length, or size, or count, etc) not a method? Why len(str) instead?
And Python has plenty of oddball syntax: significant whitespace, triple quotes?? You get used to it of course.
Ruby and Python are about equally efficient -- if you're optimizing for code efficiency, both are bad choices. In many domains, that's not a critical metric, so we talk about developer efficiency. I'll argue that developers are more efficient when they are more happy, and for me that selects Ruby by a mile over Python.
But I won't pretend that it's more than a matter of taste. :)
I feel very similarly about Python vs. Ruby inconsistencies! I remember starting a job and used python for the first time coming from Ruby, and being extremely disappointed in the inconsistencies in the standard library and the sheer… lack of features in the Stdlib compared to Ruby. I’ve yet to use a language with such a magnificent stdlib and documentation. I main python nowadays, and I find the ecosystem and typing to be better than Ruby, but Ruby is the best language I’ve ever used for many reasons
> And Python has plenty of oddball syntax: significant whitespace
Yes. Especially this. The whitespace thing in and of itself has kept me from ever developing a fondness for Python. Clearly there are plenty of people who aren't bothered by it. De gustibus.
Significant whitespace is probably my favorite feature of Python, which might have to do with the German keyboard layout making it very cumbersome to enter curly braces.
Maybe not oddball syntax, but rather syntax that doesn't make intuitive sense. With Python, only nested list comprehensions come to my mind that are a bit tricky to read. With Ruby, I often felt lost reading code and it was sometimes hard to figure out the name of a thing or search for a specific syntax, e.g. with a single character like @ - especially in the official reference documentation. But maybe it's on me for not learning Ruby properly first.
They are both dynamic, OO scripting languages, whose main implementation has a GIL, but beyond that, they aren’t all that similar.
> However, it’s kind of like Python’s weird brother,
Methods-with-self-in-args-list isn’t the “weird” one?
> with optional parentheses for calls without arguments
Parens are optional in Ruby on method calls and method definitions except where necessary to resolve ambiguity (including in the newer special single-line method definition syntax, its not a special syntax option for no-arg methods
> auto-returning the result of the last expression
Ruby is basically an expression-oriented language [0], everything returns a value, and a sequence of expressions returns the last expression in the sequence. Explicit “return” lets you short-circuit that. Python is a statement-oriented language, and statements don’t return things except an explicit return tells the function its in to return something. Both are common kinds of languages, just very different. Neither is “weird”.
> Methods-with-self-in-args-list isn’t the “weird” one?
I agree that it's uncommon, but I do like the clarity it adds compared to something like "this" which magically refers to the current instance. Hasn't C++ just added something like it with "deducing this"?
Because it honestly doesn't matter most of the time.
In the majority of use cases, your runtime is dominated by I/O, and for the remaining use-cases, you either have low-level functions written in other languages wrapped in Python (numpy, etc.) or genuinely have a case Python is a terrible fit for (eg. low-level graphics programming or embedded).
Why bother making a new variant language with limitation and no real benefit?
Look, Carmack is a genius in his own corner, but he is taking that quote vastly out of context. The actual linked article[1] is quite fascinating, and does point to overhead costs as being a potential bottleneck, but that specific quote is more to do with GPUs being faster than CPUs rather than anything about Python in particular.
More specifically, overhead (Python + pyTorch in this case) is often a bottleneck when tensors are quite small comparatively. It also claims that overhead largely doesn't scale with problem size, so the overhead would only matter when running very small tensor operations with pyTorch with a very tight latency requirement. This is... rare in practice, but it does happen to occur, then sure, that's a good reason to not use Python as-is!
You’ve posted this multiple times in this thread, and not once has it been relevant to the point being made. You are sticking your fingers in your ears and deferring to a contextless tweet by a celebrity.
His code has had 16.6ms to execute since before a lot of people here had been born. Of course Python is hopeless in his domain. It’s creator and development team will be the first to admit this.
In any case, if your program is waiting on network or file I/O, who cares whether the CPU could have executed one FLOP's worth of bytecode or 9.75 million FLOPs worth of native instructions in the meantime?
It's trivial to prove that this is true for most software. Luckily modern OS's are able to measure and report various performance stats for processes.
You can open some software such as htop right now, and it will show how much CPU time each process on your system has actually used. On my system the vast majority of processes spend the majority of their time doing nothing.
Is it true for all software? Of course not! Something like my compositor for example spends a lot of time doing software compositing which is fairly expensive, and it shows quite clearly in the stats that this is true.
I would add high-volume parsing / text processing to the list of bad fits for Python, although I'm not sure if there are native extensions for the different use cases?
Quite possibly; do you specifically mean NLP work? I'll admit, it's not something I work in myself, spaCy seems to be the go-to high-performance NLP library, and does appear to use C under the hood, but I couldn't say how it performs compared to other languages.
I had SAX-style parsing of XML and XSL transformation as concrete use cases in mind, because that happened to be what I worked with. I believe I went with Node.js at the time, which had a library that was much easier to work with than what was common for Python. Although, I mostly used Microsoft's or Saxon's XSLT processors overall in that job.
Another use case was parsing proprietary, text-based file formats with ancient encodings. I believe I did use Python for that as there wasn't that much data to convert anyway and it just worked.
Go and Python have a quite different set of strengths. Python heavily leverages operator overloading and other metaprogramming features that Go does not support (well).
If you pick up Go expecting it to be a "more efficient Python", you will be sorely disappointed.
Hard to judge. In Python it's very easy to create nice porcelain APIs for powerful frameworks because you can hide complex behaviors behind operators. Imagine Pandas or Django without overloaded access operators.
I didn't mean how beautiful the api looks like… In my experience authors of go libraries are somewhat noobs on linux/terminal stuff and their libraries fail to take things into account.
Generally I’d expect less efficient languages to be more user friendly (well, a language which has neither advantage is just bad, why is anybody using it?). So, hypothetically these companies should get away with spending fewer programmer-hours on things. They won’t have to run the coffee pots, and air conditioners as late.
I wish that the pgadmin team hadn't rewritten pgAdmin4 in Python. The GUI performance is abysmal. I can see it lagging when typing SQL text in the window. The experience is really worse than the worst Java IDE I happened to use in the early 2000. The previous version was written in C++ and worked much better.
So while Python might be good for some stuff, please, please do think twice before choosing it for GUIs or other kinds of apps that should be snappy.
> So while Python might be good for some stuff, please, please do think twice before choosing it for GUIs
The GUI for pgAdmin4 isn’t in Python. Between 3 and 4 pgAdmin didn’t just change from C++ to (largely) Python with otherwise similar architecture, it changed from a classical desktop app to a web app with a Python backend (so you could run it on machine close, in network terms, to your db server, to which your workstations would have only an HTTP, not a pg protocol, connection, without maintaining a separate code base for that and desktop use.)
As a person who has written Rust code professionally for 4, here's why I returned to python: batteries included.
That's it. Having cargo download 1000s of package from who knows who for basic functionality is unacceptable. Rust's features, while absolutely phenomenal to work with, just aren't worth having insane npm-style dependency graphs and all the risks they entail. In comparison, Python's built-in modules are a godsend.
So all in all, I think you should continue to stick with python.
I still really like Python for scripts and small programs, but I’m always going to use Rust now for CLI’s or anything larger, I get the problem you have with it, but the trade off is worth it for me being able refactor quickly and with confidence as the program grows.
I think python vs rust is always going to be a tradeoff either way. I just wish we had the best of both worlds in one language. Rust's base language with python's extensive library. Perhaps a new language will emerge that can satisfy both requirement now that Rust has a somewhat stable "safe" FFI. One can only wish.
I worked with Python for quite a few years but at 2013 it became too painful for "DevOps". The more I was thinking about how things could/should be the more frustrated I was. Python and bash were the "inspiration" that pushed me to start working on Next Generation Shell.
Particularly annoying in Python is running external programs (yes, I know there are libraries) and manipulating structured data (list comprehension is not aligned with how I think, which is map() and filter() and lambdas not limited to single expression).
I’ve started using Python more for fun lately (coming from C and C#). I wish more languages used white space as things are so much cleaner - the odd messed up return statement / parallax error isn’t worth all of the curly brace noise imo.
If you have a loop inside another loop and say and if statement, it can be hard to tell where the end is:
I'm in the IF
What am I in?
vs:
I'm in the if
}
}
}
What am I in?
I'm sure it's just personal preference and I don't like it. I don't think significant whitespace is sort of a universal good that maybe it seemed like when it was proposed. It now seems more problematic to me.
> If you have a loop inside another loop and say and if statement, it can be hard to tell where the end is:
In either case, the aligned visible lines above (presuming reasonable block size) will be what clarifies that, assuming correct indentation (which is necessarily the case with syntactic indentation) and the brackets just add (in the example) three additional lines of separation making that harder.
I think it can become non-obvious in 10-15 lines. It is a trade-off, not a universal advantage. I suppose an IDE could cleverly draw little lines exactly when needed. One of the nice things about Python however is no IDE is required. This is one misgiving I have about C#, it is great but an IDE is nearly required.
I continue to reach for Python first for my personal scripting, especially gluing together existing libraries.
Building tools multiple people will be using regularly? Might be worth speeding up at least the slow parts with another language. (I don't know it well enough for it to by my first choice here).
Also: I appreciate typed languages more when working with a team. I'm not familiar with the options Python offers to keep teams on the same page.
all the reasons i switched away from python (ranked most important to least)
1. packaging python is awful. Turning a large python code base into a deb is awful. I mean really, how many _different_ setuptools or pip do you really need?
2. not strictly typed. Really impedes large code refactors
3. not concurrent. The concurrency primitives are a joke compared to any other language. The GIL kills everything.
4. slow. Python gains all it's performance by doing the important parts in C
python is not a good fit for anything I do these days
Good decision to move away. Tools are supposed to help, not get in the way. Though types are there, opt-in, and who cares if C modules make it fast if the end result is fast?
As of late, my "language" of choice is a mashup of python and nim. I use python for the ecosystem and nim for parts that need to run fast. It's the easiest interoperability I've ever worked with.
Wow. Why did you say 'still'? I use python a more than decade, with other languages as well (go, typescript). And I even don't think to abandon it. I do so many things with it, that it would be nonsense.
Yes, the drawbacks are here (async is not as good as in Go, typing I would say is rigid, packaging is not perfect at all). But I have a lot of projects where it just doesn't matter.
How popular would Python be without the scientific users and the machine learning ecosystem?
I have the feeling that its user base is pretty lopsided, but I might underestimate its importance in other areas, like teaching programming. I'd love to see a breakdown of its popularity by field.
This. Somebody should actually trace the milestones that led to the current state as it not by any means obvious why it happened. At some point a winner-takes-all dynamic obscured some direct competitors like R or julia but prior to that all bets were open. It could also have been c++, since libraries like armadillo and eigen make it easy to write script-like code. Or even java that is everywhere in enterprize
Julia came too late to the scene. When I first started using Julia 0.3, there was already Theano for Python, and Tensorflow was in the making. R for some reason is not as easily navigable as Python. The slowness of compiling Armadillo and Eigen makes it a no-go for iterating on new ideas. Maybe it should be asked instead why Octave didn't take hold, as Matlab is popular among engineering communities and the syntax is also simple enough to understand.
Very. Unless you're a diehard JavaScript head, it's still the language of choice for smallish projects like a Discord bot, or for any kind of sophisticated script.
Google / YouTube no longer use Python “for product”, at least, not in the serving path. Dropbox has also been moving away and towards Golang.
Instagram (Cinder), Google (Unladen Swallow) and Dropbox (Pyston) have also all experimented with heavy engineering investment trying to improve Python performance and only one of them (Cinder) has outcompeted the “just rewrite it in a faster language” strategy.
I think it's a great pattern to start with Python and do something once performance becomes a problem. You can afford it when you're successful thanks to the code you wrote in Python, or it won't be a problem anyway.
P used for something in places in parallel of hundreds of (several exotic, rarely used) technologies is not really predicting how much P was used without scientific and ML uses. What if P is mostly used for scientific and ML purposes in those places, or as an experiment, study, not necessarily being a backbone? Or with other secondary agenda, overstated, etc. The mere fact that P is used in a huge amalgam of A,B,C,... tells very little alone concerning the question. Almost nothing.
It's not solely used for scientific and machine learning at any of the aforementioned companies. It is the (or was) chosen by a number of technology companies who have the resources and knowledge to select the correct languages to address one or more of their existing problems. That is evidence of use and use case.
I would argue moving away from Python at global scale doesn't even negate my point. All products change with scale.
It's also the primary language used for everything regarding VFX.
Think Pixar, ILM, WETA, Disney etc. it's mostly inhouse tools so nobody ever sees them but the code bases are huge and are growing as fast as the demand for more and more high quality VFX from streaming services and studios.
Now that you say VFX, Python is indeed heavily used in this industry. It's used in internal tooling but many of the open-sourced projects under the Academy Software Foundation (ASWF) umbrella also have a Python interface. Python is also popular among color scientists, of which many work in VFX it seems. By the way, the industry also makes heavy use of Qt, also in combination with Python, to quickly build production tools with a GUI.
I really struggle with this because I'm a C++/Java dev who is 4/10 with Python. I dont understand why anyone would choose a single threaded language but its just everywhere I can't avoid it.
I'm C#/Java dev who works with Python and ML devs. I'm amazed how fast they can put things together and solve problems. We've embedded Jupyter to our product recently, and what I can tell, that's the way. Even the integration team started to use it to glue things together. I'm becoming a huge fan, it does the job.
If your problems need threads and raw CPU, you’re doing well, since that obviously isn’t Python’s niche. The thing is, most problems aren’t like that and if some parts are, Python probably has a C extension for it, so it doesn’t matter nearly as much as you think.
Python is pretty decent for small to medium sized stuff where performance isn't a big issue (setup stuff, simple backends, ...). A lot of stuff falls into that category.
I especially enjoy the types. Coming from Typescript and being quite a fan of the "dynamic language with optional typing" pattern it was quite cool to find that Python just has that.
Sadly quite a few pieces of software are likely going to stick with Python 3.7, so we are probably going to have to stick with the old syntax (explicit imports from typing and uppercase types) for a long time.
I had a team recently who were getting really frustrated with Node. They are systems people, integrators that sort of thing. I have been careful about pushing my own prejudices onto them. I made a throwaway comment about how x problem would be so easy in Python and one of the senior guys asked to try it. I showed them Flask about a month ago and they are over-the-moon. The next day they were showing what they had built!
Now they have plans for a whole set of simple webapps. It's like a new lease of life!
It would not run on a $5 VPS from DO, because to deploy it required more memory (two gigabytes I think... mēh).
Mailman 1 ran in 40K. It was written in Perl. Clunky, not flash, and functional.
Mailman 3 is written in Python using Python tooling - Django I believe.
Clunky, flash, and functional. And costed my $5 a month extra.
I do not hate Python, I forgive the language for people misusing it. But it is a scripting language. That is what it is good for. Calling other systems. Not building the tight loops
It’s easy (and common) to point to Instagram et al as examples of people using Python and even Django at scale. Beyond those big hitters though, Python is obviously a very popular language, and Django is a very popular web framework. The people making these tooling decisions aren’t just noobs. Python and Django obviously meet the functional and performance characteristics desired by a lot of people.
I have Django projects, some of which I am going to assume are more complex than Mailman, using far less than two gigabytes in production. Not through any extreme optimisation efforts. It’s just…what’s happened. In my experience, the resource footprint you experienced with Mailman is not representative.
Notwithstanding the fact that we’ve seen amazing performance improvements in Python as recently ad 6(?) months ago, if speed were my primary concern, I am obviously not picking Python. This doesn’t mean that it should be relegated to the frankly at this point outdated bucket of “scripting languages”. The language’s features, performance profile, and package ecosystem all surpass these use cases. The conventional wisdom, which I completely agree with, is that these days things are good enough that speed need not be a primary concern for most people most of the time. Sure, oldschool Mailman was fast, but did anyone want to or know how to actually work on it anymore? Evidently not.
We ported a client-deployed command line program from Python to Go two years ago and couldn't be happier. It's solved all our problems around shipping code to customers in different environments. It greatly reduced our support workload.
However, we still use Python for APIs and Web Apps. No language is better than another, some are just better for one solution more than another.
After 10 years of nonstop use, I have stopped using Python almost completely. The main reason is that working with Go makes my life much easier when it comes to building distributed stuff on AWS, which is my main focus at work. I miss Python's development speed and how easy it makes it to test things, and I still consider it the king of modern monolithic apps.
Python is the modern perl. It connect useful and fast things build in C in a more easier more expressive but way slower package.
That’s his quality but also the biggest problem. When you have something that cross the boundary python/c it get painful as you have two language to jungle with the handling of packaging, compilation.
For serverless with AWS Lambda, ironically enough, Python is the fattest runtime, and the language that makes more sense, since lambdas are small functions.
Python is a good language and I use it a lot. But I can only say Python2 was fun but Python3 is serious. I could live with the quirks and shortcomings of Python2 and still enjoy programming. But even after a decade of history I still find many of the things unenjoyable that Python3 decided to introduce. I know why they did it, it's not an intellectual complaint. It just describes the feeling of starting a new Python program. Used to be great fun. Nowadays, merely a technological compromise.
For most web-based things, frontend and backend, Typescript is where the zeitgeist is these days. Better types, web native, more performant, better async support, better library ecosystem, new cutting edge tech projects, etc. Python's forte and niche is in ML and scientific computing. For quick scripts, either are fine.
I was mainly asking what do people use today instead of Python. Things that Python was usually used for, but now people use something else? Hence the whole "I STILL use Python after all this time" kind of thing.
Good post. I think this is the mature and somewhat inevitable lens through which to assess which programming language you're using. I have been using primarily Python for the last ~4 years or so. It's probably the most boring language I have ever used (it probably doesn't help that I've used Scala, Haskell, Rust, etc).
Obviously, some languages are better than others for specific use-cases and one should think through the strengths and weaknesses of each before beginning a new project (assuming you expect the project to have some longevity). Similarly, it definitely is possible to make the wrong choice about which language to use. But even as a PL "enthusiast", I've learned over ~15 years of leading successful projects and teams, that evaluating languages is always team/org/business-dependent and the considerations that factor into those contexts are distinctly different from the kind of considerations that I, as a PL enthusiast (and I suspect many others in the HN crowd) would prioritize. You have to consider things like:
1. What stage in the business in, i.e. early stage where you may simply want to get something, anything working and it's viable to do so in whatever language will get you there fastest (warts and all, tech debt be damned, etc) vs the critical juncture where you're beyond finding product-market fit and you now have a business that is working and you need to increasingly focus on reliability, scalability, etc?
2. What does the pool of engineering labor in the market you have access to look like? For example, it may be the case that you'll have difficulty, for one reason or another, finding developers who work with language X, but could easily find many who can work with Y (of course, good devs should theoretically be able to ramp on any language, but whether you'd want them to do depends on how crucial using that particular language is to your venture, whether you can afford the ramp-up time, etc.
3. Also somewhat related to the quality of engineers you have access to, the community/packages/ecosystem surrounding the language may matter a lot to such a degree that even if some other language were a better fit from a purely technical perspective, it may leave you to have to fend for yourself in ways that your team simply isn't up for.
.. and many more considerations.
Beyond all of this, I resonate with the author's orientation of wanting the language to just get out of the way. Sometimes you're looking to geek out on a stimulating and interesting language. Other times, that's the last thing you're thinking about. Bonus points if the language you find most compelling is also the one that, for reasons like the ones mentioned above, is the one that makes the most sense for you and your team. All in all, the point is that assessing a language in isolation, without any regard for what you're actually trying to accomplish, what your team looks like, etc. doesn't make much sense.
Side note: I do wish more Substack authors would allow comments even from non-paid subscribers. I could understand restricting it if you already have an established paid subscriber base and a chatty comments section. But if comments are non-existent, it seems to me like the author is potentially losing out on valuable feedback/commentary by restricting commenting to paid subscribers.
I wish I could just stick with a language forever and thoroughly master it. What did Bruce Lee say about it? He didn’t fear the man that practiced 1000 different kicks. He feared the man that practiced the same kick 1000 times. Something like that. The industry fetishizes the constant stream of new languages and frameworks, primarily based on nothing more than the fear of “falling behind”. It’s dumb, and essays such as the “blub programmer” feeds into it.
There is nothing abstract about a kick however. A programming language is always an abstraction, with different languages optimizing for different things. Understanding more than one is helpful imo.
The “Bruce Lee” of programming would definitely use assembler and deeply understand computer architecture at a fundamental level. The idiom of Bruce Lee is someone who is hyper passionate about their craft.
"I wish I could just stick with a language and thoroughly master it."
Assembler, C and shell script have been and will be around for many years.
As an end user, i.e., someone who runs programs more than she reads/writes/edits them, I find Python's startup time is just too slow. I have a folder full of tiny, fast shell scripts and C programs that I rely on year after year. Replacing this with a folder full of Python scripts is difficult to envision.
Assembler, C and shell script will still be ubiquitous and foundational no matter what subsequent programming language hype comes over the internet. Boring stuff that works. I am typing this comment from a program written in C.
"The industry" (cf. the end user) seems to have endless time to burn because the stuff it promotes through hype always presents a massive timekill to those who embrace it. A conspiracy theorist might imagine that the industry is trying to keep itself busy and maintain peoples' attention. Regardless of intent, that seems to be the end result.
It is like that hardware company that wants people to keep buying new computers every few years, and tries to prevent/discourage anyone from repairing and retaining them.
> It’s dumb, and essays such as the “blub programmer” feeds into it
I don't think the Blub essay is about learning a constant stream of new languages. I mean, the author of the essay doesn't seem to be constantly learning new languages.
The Blub essay, whatever its flaws may be, is about people being unaware of languages and features more advanced than their tool of choice, and finding them strange. Or in other words, that it's easy to see how poorer tools are more limited than the ones you're using, but very hard to understand how better tools are better, or even acknowledge they are better at all.
The one thing that scares me about that sentiment, which otherwise I mostly share, is that "forever languages" tend to become kitchen sink languages. Typically to their detriment.
You can learn a language just to explore new ways of thinking and eventually return back to your kitchen sink language.
I think it's worth learning a lisp, a functional language, a logic pl, or something from some other paradigm you don't typically use. Maybe do a side project in it. The lessons you learn will surely make you a better programmer overall even if you don't maintain your mastery of it
If you’re creating the language, i totally see the value but If you’re just a user of a given language then there’s always going to be more value to be extracted from advancing your knowledge of computation instead of any one lang.
If your business is trying to solve hard problem X and your best idea is to use this tricky advanced language feature and then apply these compiler flags… then you’re going to lose to the person who knows of a programming paradigm (e.g. logic programming) unlocking a simpler solution or a data structure unlocking a more performant solution.
That sounds a bit abstract. To make it more concrete - imagine a problem where you need to apply a function in concurrent windows over a time series but you’ve never experienced any of the array programming languages.
As a language that celebrated its 50th birthday this year, C is already a long-lived language. It's not terribly far behind the first non-research compiled language, Fortran, which celebrated its 65th birthday this year.
Both are, for all practical purposes, immortal languages at this point. You correctly point out that they aren't guaranteed to survive the next technological mass extinction event, but I think that might just be a truism.
Nothing really stops you from doing that, as long as you're still willing to learn other languages, libraries, paradigms and whatnot that are needed to get stuff done.
I consult as an X "expert", but have worked, and do work, extensively with tools outside of X's scope.
Been using Python for over 15 years, and although there are many programming languages that I find way more attractive (Lisps and functional programming languages), I keep sticking with Python, because you can learn it very quickly, you have a vast amount of resources and a large community that‘ll help you, and there is nothing you cannot do with it: From Webdev to data analysis to machine learning, there‘s a module for almost everything you need to solve.
Some of my reasons for choosing Python in the first place have been mooted. Python is free, and multi-platform, but so is everything else nowadays. I think that battle is over.
For my use, packaging has not been a big obstacle, maybe because I tend to use a relatively small handful of established packages. I don't remember needing a package that I couldn't install with pip.
Easy to learn? Yes and no. I think a beginner can ease their way into Python and get productive, if they start with basic scripting and progressively adding a few language features. On the other hand, some features are simply over the heads of beginners, making the code found in packages practically unreadable. But I've never found any language to be better in this regard.
Fast? Sure, with numpy in its domain of use. Or writing your own C libraries. However, I think a beginner using numpy will run into roadblocks that require help from an experienced programmer, who can visualize what the array operations are doing under the hood. So, writing fast code isn't easier to learn in Python than in any other language.
The editor should have a final option, that turns your code brown if it's just so bad that nobody should ever have to read it, including yourself.