Hacker News new | past | comments | ask | show | jobs | submit login
Sunsetting Python 2 (python.org)
1616 points by azizsaya on Sept 9, 2019 | hide | past | favorite | 704 comments

Python 2 to 3 (at least by 3.3 or so) was one of the easiest transitions I've ever done. There's a library ("six") to help, and in almost all cases you can write 2-and-3 compatible code, which means you can go piece-by-piece. (Unless your manager makes drive-by commits of py2-only code, months after you all agreed that all new code should be py3-compatible, and then leaves town for a multi-week vacation...)

Dependencies? I helped upgrade a couple third-party modules, too. That's part of the job. If you choose to use a dependency, you're vouching for it. That means if the current version of the language no longer supports your favorite library, you need to fix the library, or find a new one. If we switched from phillips to torx, and your favorite tool brand didn't make torx drivers yet, you've got to either convince them to start, or switch brands. Professionals don't get to use this as an excuse to badmouth torx, and stick with phillips.

If people spent half as much energy upgrading as complaining, this would have gotten done 5 years ago.

Apple took only 3 years to go from first announcing a new CPU architecture to releasing an x86-only OS. 2 years later, Rosetta stopped working, so there was no way to run old apps on current hardware at all. That's got to be one of the advantages of proprietary systems. It's amazing what you can accomplish when you have no choice. People complained a little but they got it done.

By experience, people with large Python projects often overblown the difficulty of porting in their head. Unless you have a very rare irreplaceable dependency or some terrible C extension, porting is easy.

It's tedious yes. Boring even. But most projects get away with 2 weeks of investment. And yes, it pays back. Python 3 is a vastly superior language when it's about introducing less bugs or debugging existing ones. It's just not stuff what had be sold to the public. People want to hear about better perf or fancy features, not better error messages or banned logical errors.

The hardest code base I saw ported was the Twisted project. It's a good counter example: it was long, hard, and required very gifted people (thanks Hawkowl !).

But most projects are not like that. They are full of dumb operational logic that can be ported 80% by 2to3, and a bit of manual fix.

The fact is, people love to complain, so you will hear massively people with that particular example from the movie industry, or this guy who coded this Fortran extension that was in such a tough situation. Well guess what, that's not what most migrations are about.

Most migrations are Django/flask websites, math teachers exercises, physicists scripts, sysadmin tools, etc. Straightforward stuff. I should know, I've been moving from industry to industry for 10 years, all my clients are different, but most of them end up with similar stuff in their repo because Pareto is a thing.

Nonetheless, the screams from the first ones scared the later.

I think it's less about the difficulty of manual changes and more about the cost to verify that all of the changes actually work.

Dynamic typing on the language side, and a less than perfect test suite on the user side are not a good combination for large projects facing a project wide migration.

+1 to that. Add on top of it that by design, a lot of Python coders are not professional programmers, and you get yourself a better model of why migrating was so long. People coded an untested unstructured project that worked on their machine. It was enough to get the job done, and was a big reason they choose this language in the first place. But migrating that is not fun. Plus they didn't understand their own system, so touching it on such large scale seemed overwhelming, and hence the "it's too hard".

It's about risk. Python is a scripting language. That means that it's often auxiliary to a primary programming language. Your project might be in C++ but you use Python to package it or generate some assets, etc. Do you really care what language your auxiliary scripts are written in? No. Are you going to spend any amount of time converting them to Python 3 syntax? No. Your boss doesn't care, and you don't care. It would only end badly.

Scripting is only one use of python, it's also a math language that is used for implementing servers, machine learning and statistical analytics, data processing pipelines, and more. Numpy/Pandas allows fast math operations because it makes python the interface for operations implemented on c-arrays underneath the surface. For many many people, python is the primary programming language. The ability to use the same language for scripting as for your primary application is part of the charm of python.

The existence of the cloud means that you can scale wide instead of scaling tall, which makes developer time and complexity management more valuable than hardware efficiency. Python is optimized for primary applications in exactly that kind of environment, where the performance characteristics of something like C++ don't make up for the clumsiness of the language.

> Unless you have a very rare irreplaceable dependency or some terrible C extension, porting is easy.

In my very brief experience with python over the last few weeks this is very common. Half of our dependencies were abandoned before python 3 existed, when mercurial shuts off the hg we'll even lose the source to some of them. Python 3 get's the blame but the real problem is that the company has ignored maintenance for decades, it's entirely there own fault. There's no "business value" in maintenance until the whole lot needs to be rewritten.

Another underlying cause is package tools like pip, they make it too easy to take on dependencies with zero thought given to their maintenance windows and management think the can just ignore upgrading/replacing them regularly.

> when mercurial shuts off the hg we'll even lose the source to some of them.

Are you perhaps thinking of when Atlassian removes the Mercurial repositories hosted on Bitbucket?

Mercurial isn't going anywhere or shutting down anything; it's open-source software with some very large users and committed developers.

(The majority of engineers at Facebook all work in one enormous Mercurial repo. Facebook naturally employs a few people to work full-time on Mercurial, and I don't think they're the only ones.)

Atlassian's decision about Bitbucket is regrettable. Especially regrettable is to actually delete repos for so many open-source projects that may not have active maintainers (rather than keep them online but read-only.) To my mind it marks a stain on their reputation that should make anyone think twice about relying on Atlassian for years to come.

But you should be able to avoid losing the source to any of your own dependencies. Between now and May 2020, go through all your dependencies and make sure you make your own clone of all the source repos.

Yeah we have this particular problem, a Python program written in Python2 as late as 2014 for RHEL6, using Cython extensions. It compiles and packages fine for RHEL7, but when I run it on it just crashes upon receving a UDP packet from the network.

The original programmer is long gone, and I while I am a semi-competent Python scripter and a slightly above average C programmer, I'm not looking forward to having to dig deep into this code, and patch it up to keep running.

> By experience, people with large Python projects often overblown the difficulty of porting in their head. ... most projects get away with 2 weeks of investment.

Where did you come up with this "2 weeks" estimate? That has not at all been my experience.

What has been your experience?

> What has been your experience [converting Python 2 to 3]?

Many months.

Step zero is education. Your team has been programming in Python 2. You need to make sure they know the differences in the environments, how to use create cross-compatible code, and how to use six. They also need to setup a second dev enthronement, and become comfortable switching between 2 and 3 regularly.

Step 1/2 is prioritizing. Determine how important is the move to Python 3, and what other features and dev work will you have to sacrifice to make it happen. While having a nice plan in place may provide some level of comfort to management, you can be sure it will thrown out, amended, extended, and/or ignored throughout the project. Upgrading Python dev environments brings no near or mid term value to the company. So expect developers to continue their current work-load while also attending to this tech debt.

Step one is getting the libraries into shape. This is easy if your project only relies on actively developed libraries with good teams behind them, but nearly impossible for abandoned libraries with no new commits in the last few years. Sure in retrospect, it was a bad idea to use these libraries, but at the time, they were incredibly powerful or popular. Generally, for abandoned libraries, it's easier to find a newer Python 3 library than work on fixing someone else's code. But this may mean rewriting large parts of an app, and may alter functionality, so constant communication with product mangers and a flexible approach is necessary.

Second step is working on your own code. One or two modules is no problem. But more than a dozen takes time. Scripts like 2to3 are not helpful, you need to use tools like six and modernize.

Third step is testing. If you have a good team, then you should already have good tests with known coverage. In that case, you should make sure your coverage matches, and take a very hard look at the code that is not covered. If your tests cover less than say 60% of code, you need to invest quite a bit of time of either testing directly, or building many more tests.

Step 4 is partial rollout. There will inevitably be issues you didn't think about or catch, so you need to plan the roll-out carefully and either split traffic or at the very least be able to quickly revert.

Step 5 is to watch carefully for customer complaints. Any complaint may or may not be related to the switch from 2 to 3, and you need someone on your team who stays on top of that, and can communicate the issue to the correct module owner.

Step 6 is deciding when to drop support for Python 2 altogether, as there will have to be a time where you have the two environments running side-by-side while you're testing. After all of this work, you'd think this part would be easy, but in every organization some folks will be wary about dropping support for something that already works.

Obviously, this isn't just a linear progression, you can do some of these in parallel, and will likely have to take a step back a few times. My back-of-the napkin, experience based, non-scientific estimate suggests that a 100k LOC project can be managed by two developers in two-three weeks, but a 1M LOC project (including libraries) is roughly 5-8X that much work.

The entire science stack is "terrible C extensions". This is also the kind of tight-budgeted stuff that needs relatively rare Python/C developers, not just your average overpaid Python web developer.

I'm not saying we would've cured cancer if people didn't have to deal with this bullshit, but it's definitely a setback.

The science stack is also the one that starting migrating the earliest. Hell, numpy started supporting Python 3 eight years ago, while five years ago people were still complaining so much they got the EOL delayed. Not saying there are not some specific things that were late or never happened, but again, they are about a minority of projects. The thing is this minority is making all the noise.

Given the reach and popularity of Python, of course you will always find plenty of testimonies saying they suffered. Again, it's important to remember you can't make everybody happy, especially if your job is important. If porting is easy for 90% of projects, you did a great work, period. 10% of millions of users having it hard is still going to raise a lot of voices against you by the sheer power of numbers, but you did ok.

There is also dishonesty. Most people reporting this problem or that problem didn't actually encounter it. They are reporting somebody else experience they heard about because they wanted to make a point. Or vaguely tried something and ran away after they saw 10 minutes of fiddling didn't solve it.

In 10 years going from company to company, I met only 2 projects that were hard to port, but much, much more complainers. When I looked at what was really going on, the fact was just that they were afraid to port, and so just repeated all the stuff other people told them would go wrong. Or they tried running a few scripts for 5 minutes on Python 3 and gave up seeing too many error messages.

I know there are honest situations in the lot. But again, the most honest people are not making noise, because they just ported their code and noticed it wasn't the hardship they've been told.

> The science stack is also the one that starting migrating the earliest.

...but the last to actually finish migrating. In fact, it's still ongoing.

> Given the reach and popularity of Python, of course you will always find plenty of testimonies saying they suffered

Everyone suffered. Everyone had to deal with Python2 versus Python3 bullshit. It's not just about migrating some codebase.

Everyone had to use an inferior version of Python, whether it was because library choice was limited (Python3) or because the language was intentionally left without feature backports (Python2). There was a huge amount of plain "brokenness" in the ecosystem and it was a huge waste of everyone's time.

> Everyone suffered. Everyone had to deal with Python2 versus Python3 bullshit. It's not just about migrating some codebase.

Updating software version is part of the job. Like creating tests, writing documentation, training newbies and dealing with customers. It's not hard just because we don't like to do it. Porting to Python 3 was not hard for most people. They just really, really didn't want to do it.

I get it. I didn't want to add that on my plate too. Like I didn't want to migrate from my Centos 7, I didn't want to move from mysql to postgres and I didn't want to learn the entire setup of Webpack. 3 times. But "suffering" is a big word that has no place for the vast majority of projects.

Like I said, it is not just about updating software. You can't realistically update all your dependencies by yourself.

You are stuck with a tough choice: Do I start out with Python3 and tons of broken packages? Do I limit myself to Python2 and face a costly migration later on? Do I run the extra cost of supporting both? This the choice you had face for the better part of ten years of migration. Perhaps it's not obvious that all Python-based software was worse for it, but that's what happened.

I'm not talking about some web backend service where most of what you do is trivial stuff. You can write and re-write that in almost anything, it doesn't matter.

Which common parts of the science stack are still partially two-only?

Conda ships on 3.7, and I'd consider it the definitive "scientific" stack.

You're right, most packages appear to be Python3 now. Nevertheless, a lot of client code has been written for the Python2 stack and it's still in widespread use. That stuff won't get migrated soon, if ever. Security doesn't really matter there anyway.

> If we switched from phillips to torx, and your favorite tool brand didn't make torx drivers yet, you've got to either convince them to start, or switch brands.

This is the best analogy I have yet read. Thanks - you nailed an argument I have had at multiple gigs/clients way to often.

> Professionals don't get to use this as an excuse to badmouth torx, and stick with phillips.

D'accord. Exactly. I had very fruitful discussions with professional builders about tools, tool brands and their respective strengths and weaknesses - but never has a professional decried Bosch Pro vs Makita vs DeWalt. One has ones favorites - and that is fine. But badmouthing - I have never encountered.

You've never encountered badmouthing from a pro builder about other tool brands? How many did you talk to? Builders make the Ford v Chevy debates look minimal in comparison.

I think he means more 'craftsman' vs. 'builder' in mentioning professionals.

I don't think there's a time I can remember when Norm Abram (probably now one of the greatest or at least most prominent carpenters of a generation) ever mentioned the particular tool brands he was using.

And looking back on earlier seasons of This Old House, he was hand-nailing thousands of nails a day (nowadays he and everyone else often uses pneumatic tools for this).

I'm sure he prefers a certain brand, but unlike popular YouTube personalities and more 'celebrity' builders, he isn't a brand ambassador for Craftsman, DeWalt, Milwaukee, Ryobi, etc.

But yeah, when you talk about 'builders' (the more common carpenters you encounter on most job sites), it seems like most of them will die on a sword defending whatever brand(s) they've sunk a fortune into.

I'm a DeWalt guy, mostly because I invested in a 20V Max drill a long time ago and have a ton of batteries for all my electric tools now. But I used to use Makita before switching systems. I wouldn't call myself a master carpenter by any means, but like AvE, I find good and bad in almost all the 'top tier' lines from tool manufacturers.

Every "craftsman" I know is fanatical about the tools they . use, be they carpenters, chefs, artists, or programmers. Also, let's just call a craftsman what they are - experienced builders.

Weird. I'd say the attitude exhibited by De Niro's character Sam in Ronin is more common. They're just tools.

You want it written in C for a 16bit bargain basement ALU? I can do that. You want two implementations, one in Pascal and the other in PHP? Not a problem. You want my advice? No? That's fine too, the advice is expensive and who knows after all these years if it's even worth anything.

Ever seen anything built by a craftsman as opposed to a chippy :-)

At my first job we used to build some hydraulic models in hard wood that where works of art.

Our wood shop was so good they made a museum quality piece of furniture for our retiring boss and it was better than the best Chippendale furniture when new.

Having used quite a few different brands over years,I'd say that most of the more expensive ones are just good enough.All the nitty gritty details why Makita does a better job than HILTI or that Milwaukee will last forever are mainly personal experiences.

With most categories of gear, there's a sharp quality threshold that hits below a given price range, and once you are above it, you need not go further to do professionally useful work. It's been a helpful rule of thumb for me.

Personally, I'm biased towards Milwaukee's RED Fuel cordless stuff, because I used a cordless hammer drill of theirs at work and it lasted hours, which signaled to me that if something like a hammer drill going through 1-1/2 foot concrete continuously can last a long time, then a normal drill or an impact driver probably won't die on me while working.

I do like the grips and ergonomics of some brands more than others.

I used to work for a plumbing company,which was divided into two divisions: mechanical ( heavy pipework, industrial heating and any other big stuff) and the rest(household heating, general plumbing,etc.) The mechanical division swore by Milwaukee.They had it all. The rest of us had anything from Makita to Hitachi. Still, something thst got stuck in my memory: a colleague was using high end Hilti power drill to drill holes in ceiling.He did hundreds of holes a day.The drill, despite its superior design kept failing because of concrete dust falling into the drill.Hilti failed to address this when asked how to deal with this,so the guy ended up putting a plastic pipe cover on his drill to protect its internals from getting all the dust.. Worked pretty well.

No, never. I need to invest in power tools myself so I’ve been asking a bunch of coworkers who use their personal power tools at work. The response I hear from everybody, without fail, is “Well, there are a lot of good brands.” I don’t even hear people defending their own choices.

The one piece of concrete advice I got was “Whatever brand you get, go for the 20V, not the 12V”. Last week I saw that same guy at work showing off his 12V driver. So I’m pretty sure nothing matters, and I’ll probably just get the brand with the prettiest color.

Well it’s a shitty analogy because a swap to torx is basically a single API change.

The switch to py3 is more akin to using a new building material. Easy to start with once you know the differences, but it’s a bitch to retrofit if you depend on a method of construction incompatible with it.

“Go change the dependency” is a naive ideal. Most non-tech businesses have no budget for that kind of tech debt work.

Most non-tech businesses have no budget for that kind of tech debt work.

That's fine. I'm sure there are some that also don't have a budget for fire insurance, or a security system, or whatever. For anybody who wants to gamble with no fire insurance, or not fixing tech debt, that's a conscious choice they get to make. Now they have to be responsible for the outcome(s) of their choices.

Using python2 is in no way similar to going without fire insurance. There are internal-only apps that will run fine for decades to come with no risk to the business on py27.

There are many maintenance mode projects where the switch to 3 represents absolutely no upside and is all downside risk. Not everyone is developing web apps in constant flux that are exposed to the Internet. That is a tiny fraction of python’s use cases.

> I'm sure there are some that also don't have a budget for fire insurance, or a security system, or whatever. For anybody who wants to gamble with no fire insurance, or not fixing tech debt, that's a conscious choice they get to make.

A security system and fire insurance is at most around $2000/month, and it's a cost that is understood and often required by law. Diverting a $100k+/year employees to spend their time on tech debt is quite a bit more expensive.

Companies rightly try to minimize time devoted to tech debt.

Actually most of them do money, however in these type of companies tech people aren't listed to as much,as maybe sales or finance are. It just depends on who's running the circus on the day: if it's a guy with short term goals only- you'll never get a budget,but the ones with longer visios would most likely listen.

There's not budgeting for fire insurance or security systems, and then there's deliberately building your workplace next door to an asylum for pyromaniacs and across the street from a gas station.

If you built software using Python 2 at any point in the past 5 years, you have no right to complain about the added technical debt of updating to Python 3. But not wanting to incur that technical debt in the first place would also have been a good reason not to use Python 2 in the first place; whether that means adopting Python 3 early or simply using a completely different language.

"Hey boss, we're gonna use this hip new language that will break everything for no good reason a few years down the line."

Congratulations. You're fired.

"Hey Joe, why did our customer database get leaked and published on a dark-net site?"

"Hey boss, we're using an outdated programming language for which security updates are no longer provided, because we were told there's no budget for fixing technical debt."

"Congratulations, you're fired."

Same outcome either way... might as well make some effort to do the right thing along the way...

You’re confusing runtimes with the language. Py27 as a language has no security flaws that would allow a database to get leaked.

People will continue to use py27 going forward with likely a pay-for-support runtime because that’s cheaper and less risky than changing a million lines of python code.

I would assume it’s the dependent libraries in your 27 project that no longer get support, not the language itself.

Nah, you're not fired. That's why you CYA and get it in writing. It's the manager who decided that they didn't want to do a rewrite, it's not your problem they didn't listen to your advice.

Like it or not, there are a lot of teams that choose to incur technical debt to avoid what is seen as needless work on something that is working fine at the moment.

You're missing the point. If decision makers saw it coming that Python will just break in two, they would have never allowed its use. Other "serious" programming languages rarely if ever break, and when they do, it's usually minor.

Python2 will almost certainly still remain in widespread use and it will still get security updates through alternative distribution channels. It's not that much work fixing such relatively rare issues versus migrating a major codebase.

It's not "broken"... Python3 is just the evolution of the language. Languages change and evolve, some more and some less than others. Sure, Java or COBOL maybe have done a better job of not breaking backwards compatibility, but Python 3 isn't exactly Perl 6 here.

But hey, if some 3rd party group effectively forks the language and maintains a Python2 branch after the official EOL date, good on them. But I'd argue that using that is risky in the same way that using a Pale Moon or Waterfox is (arguably) riskier than using mainstream Firefox.

> Python 3 isn't exactly Perl 6 here.

For what it’s worth, Perl 6 can load Perl 5 libraries: https://github.com/niner/Inline-Perl5

> It's not "broken"... Python3 is just the evolution of the language.

Python3 broke compatibility for spurious reasons. The language didn't change that much, it could have kept compatibility with minor concessions.

> Java or COBOL maybe have done a better job of not breaking backwards compatibility...

Literally every single other major language has done a better job at it. That's a big part of why they are major languages. Also, most of these are statically typed, so any minor breaking change is far less risky to pull off.

> ...but Python 3 isn't exactly Perl 6 here.

Well, maybe if Perl6 didn't happen Perl wouldn't be irrelevant today.

> Literally every single other major language has done a better job at it.

C++'s upgrade woes make Python 3 look downright trivial in comparison.

What are you talking about? C++ has had very few breaking changes in the language.

If you're talking about ABI breaking, that's a different story. Python breaks the ABI in every major release as well. That generally requires a recompile, not a major code migration.

IceWeasel FTW.

Security through obscurity, baby...

> “Go change the dependency” is a naive ideal. Most non-tech businesses have no budget for that kind of tech debt work.

Every non-tech business that maintains technical infrastructure finds itself doing that kind of tech-debt work regularly (if, often, chronically behind the vendors recommended schedule) for every platform they use Py2 to Py3 may (or may not, really) have been a bit more painful than a (multistep, because those types of shops never do it timely enough that a single-version-bump makes sense) .NET framework, Visual Studio, SQL Server, and Windows Server upgrade (and, yeah, those often happen together as multistep upgrades in slow enterprise shops), but it's not a fundamentally different kind of problem (and replacing dependencies that didn't make the same upgrade, often because they were abandoned years before the firm made it's upgrades, is often a part of that.)

> Most non-tech businesses have no budget for that kind of tech debt work.

The thing about debt is that ignoring it doesn't make it go away magically. If business relies on tech for crucial functions, it's only a matter of time before debt comes due.

>>”Go change the dependency” is a naive ideal. Most non-tech businesses have no budget for that kind of tech debt work

If you don’t push your clients/employer to have room in their budgets for these types of things, you are the one being naive.

One of the easiest compared to what? Certainly not a Python point release. A couple of points:

1. Python core dev pretended Python 3 was good and ready by, like, 3.1. It wasn't.

2. While your problem may have been painless (I'm glad), that doesn't mean that everyone who complained was just complaining.

(I use Python 3 as the default now, but as someone intimately involved with an async IO library at the time it came out, I maintain that a) the transition was botched b) core dev did not listen to any of the problems people pointed out for _years_.)

To give you an idea of what botched means: several people, names withheld to protect the guilty, had to wade in 6 months worth of shit to get a 2.x release that made TLS vaguely OK, in, like, 2014?

> 1. Python core dev pretended Python 3 was good and ready by, like, 3.1. It wasn't.

IIRC the messaging from Guido and Python core dev circa PyCon 2008 and 2009 was that it was good enough for people to start thinking about how they might migrate. This resulted in experiments with different approaches to migrations, maintenance of versions for Python 2 and 3, etc, but at that time there wasn't a clear end date and no rush to migrate. It seems like a lot of people misinterpreted what was actually said. Similar misinterpretations seem to happen within the wider internet community around the GIL and perf considerations. That cliff is finally here ten years later after a lot of experience with migrations.

Python 3.1 came out 10 years ago. 3.2 was released in 2011. What does that mean about the complaints since, let’s just say 2015 to be fair?

I heard several people saying that 3.6 was the first Py3 worth transitioning to. Largely for async (I think async/await came to live then). There is this take on performance: https://hackernoon.com/which-is-the-fastest-version-of-pytho...

For me, having run a slew of large Python code bases, 3.3 was the first 'useful' version to upgrade to as it re-enabled u"string" syntax. Without that, making 2/3 compatible code was hard.

3.5 fixed % formatting for bytes, further making 2/3 transitions easier (or harder, depending on your use/abuse of strings vs bytes).

3.6 added the first 'exciting' new feature: f"formatted string {literals}" if you don't have an asynchronous type project that can make good use of async/await.

I do think "worth transitioning" is the final group of people who have the most reason to hold off. If new features are what made it worth transitioning, then it probably was 3.6 with async/await. If you relied a lot of unicode and already had a mature codebase, you'd wait too. If it was a new project you'd probably start with 3 much earlier because of unicode.

Ordered dicts and f-strings are also very sweet!

I expect sametmax was referring to the new feature in Python 3.6 that plain old `dict` now preserves the insertion order of its elements. See docs here: https://docs.python.org/3/library/stdtypes.html#dict.values

(Or in language-lawyer terms: CPython 3.6 introduced this behavior and documented it as an implementation detail, and for Python 3.7+ it's guaranteed a feature of the language proper.)

And optional type specifiers. This was the key for me.

The Py3 string handling was beyond frustrating prior to 3.6. For what I do, Unicode complexity is not required. Having to stuff it into everything was more trouble than it was worth.

The ubiquity of emoji alone mean that Unicode is everyone's problem in 2010+, and ignoring it won't make it go away. It's Python 2.x where dealing with Unicode (which is everywhere) is far too complex, and more trouble than it is worth. The "complexity" of Python 3 string handling is worth it, and no worse (even < 3.6) than any other modern programming language, and possibly easier than some by making clear runtime errors where accidents are easiest to make. (Though arguably, perhaps I'm biased by having done plenty of Unicode work in other languages that Python 3 did seem so familiar and easy from the start for my Unicode needs. Your mileage of course varies, we all have different backgrounds.)

Python 3's Unicode handling is uniquely bad. I haven't heard of any other language where you can obtain magic strings that crash the program if you try to print them:

    % python3.7 -c "import sys; print(sys.argv[1])" "$(echo -e '\xff')"           
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 0: surrogates not allowed

Oh, that's a fun example. ("Fun", anyway.)

The key surprising thing that's going on here is this clever hack (clever, but a hack):

> In Python, file names, command line arguments, and environment variables are represented using the string type. On some systems, decoding these strings to and from bytes is necessary before passing them to the operating system. Python uses the file system encoding to perform this conversion ... > > On some systems, conversion using the file system encoding may fail. In this case, Python uses the surrogateescape encoding error handler, which means that undecodable bytes are replaced by a Unicode character U+DCxx on decoding, and these are again translated to the original byte on encoding.


This is meant as a way of fudging the fact that (a) for UI purposes, you want to treat filenames as text strings; (b) your Linux filenames are probably all encoded as UTF-8 (or your locale encoding); (c) but they might not be -- they could be arbitrary bytes, except only NUL; (d) and if they are, you really want to not munge the name when you go back and try to operate on the file.

The fudge is that filenames get decoded as (by default) UTF-8... but if invalid, the offending bytes get stuffed into the UTF-16 surrogate space. Then filesystem APIs encode as UTF-8, except they look for that surrogate hack and turn those to the original bytes, so it all round-trips.

It goes pretty wrong if you try to hand such a hacked-up string to something that just expects to encode normal real Unicode with UTF-8, though. That's what's happening in your example.

The magic words are `os.fsdecode` -- that's how you get back bytes round-trip clean from that hack.

what an excellent writeup of why modern languages have something like 4 stringy types: bytes, unicode, os string, path string. you could go further if you want to talk to other computers.

That seems more like a difference of opinion on how to handle a bad string than an example of python 3 being bad. I'd argue failing silently is worse, as in programs outside of your toy example you might end up storing badly encoded data etc. Broken data is way worse to deal with than a crashing app

Nothing I do in my ordinary routine full-time engineering work for the past decade has required or does require anything outside the ASCII character set or, alternatively, plain bytes. The use cases my users present to me does not include emoji or non-ASCII characters. If those arise, naturally, one would select tools to address those needs. I don't know the future. In a year or two, my routine work may, of course, change.

> Python core dev pretended Python 3 was good and ready by, like, 3.1. It wasn't.

I would expect any stable release of software to be "good and ready." Can you explain what was wrong with Python 3.0 and 3.1?

> core dev did not listen to any of the problems people pointed out for _years_

What problems were those?

A major, breaking issue[1] was how Python 3 handled missing LC_* variables. By default, a Python script would simply crash with any non-ASCII input or output. It was fixed[2] in 3.5[3].

[1]: https://bugs.python.org/issue19977

[2]: https://www.python.org/dev/peps/pep-0540/

[3]: https://docs.python.org/release/3.5.0/whatsnew/3.5.html

There were very few new features, only breaking changes. With the exception of Unicode defaults which few yet understood.

Strings in python3 were terrible for memory well into mid-3. Raymond Hettinger is a core dev and he recently stated on Twitter that he didn’t feel py3 was a clear upgrade win until this most current release.

Raymond has been consistent in saying Python 3.6.1 was the release where Python 3 became better than Python 2. That was March 21, 2017, and we're now in Python 3.7.4.

I appreciate the general point that that's far more recent than Python 3s initial release but it's also a large exaggeration to say "until this most current release".

Also to understand why it took so long is because Python 2 kept improving after Python 3s release. Python 2.7 was a great release and many further features from Python 3 got backported in it's 2.7.x point release.

Sorry, I misremembered. 3.8 is the first one he said is a clear winner: https://twitter.com/raymondh/status/1167218640095043586

2017 is a lot more recent than 2008 (Py 3.0).

> 1. Python core dev pretended Python 3 was good and ready by, like, 3.1. It wasn't.

The post you're responding to starts verbatim with `Python 2 to 3 (at least by 3.3 or so)` and your reply is about Python 3.1, released 3 years earlier. The amount of changes was quite big.

You've got to let go of these grudges. Time has past, enough time that you shouldn't be complaining about these things here, because all you're doing is giving the Python 2 "out of my cold dead hands" crew more ammo.

Just let it go...

> python 2 to 3 (at least by 3.3 or so) was one of the easiest transitions I've ever done.

Good for you. Some of us had code bases of considerable size and complexity though.

The fact is that for working software on the python platform, this upgrade represented work that had to be done that for a legacy app that was still chugging... little benefit. If you already coded around the python 2 limitations for Unicode eg, then python 3 was not a help at that point for something already in service. It was just more cost for no benefit.

> If people spent half as much energy upgrading as complaining, this would have gotten done 5 years ago.

These aren’t exchangeable. Why do people on the internet think bitching (either constructive or not) is some valuable currency? Often times the people complaining had no means or position to of the work. 4 promotions later I sure as hell wasn’t going to participate in that 2 to 3 mess on a project produced years ago, but I could still opine in the situation. I also had little incentive to fund it.

Your analogies are bizarre. No idea what you’re trying to convey with philips v torx but I’m going to wager it’s explanatory power in this case is shit anyway. I can still demolish it, having actually worked in manufacturing there were times we told a supplier (ie Python) to fuck off and piss up a rope because what they were proposing was not compatible with our existing tooling and it would be too costly to convert for little benefit to us.

I respect apples prowess in the consumer space, but there’s a reason you don’t see their products regularly put into industrial roles where your timeline is more than 5 years. Apple products are disposable, many applications in industry are expected to last. Python is a general purpose programming language (or at least billed itself as such). Your comparison is poor.

I found some Unicode bugs in porting—and I get continued free security updates. Seems fair to me, and certainly not “no benefit”. I can keep on 2.7 as long as I like; nobody’s forcing me to port. Compare the situation with Java!

I like Python. But it loses in the upgrade comparison to Java. Existing Java code almost always just keeps working with new compilers and JVMs. And new JVMs tend to increase performance of your code for relatively little upgrade effort.

In my uninformed opinion not including a python 2 interpreter in python 3 was the first mistake

> Compare the situation with Java!

Isn’t that comparable to java? Oracle gives you paid security updates on version 8. You can keep using the previous version without security updates or migrate to OpenJDK 8, which have their own security group. More options, seems better to me.

Your argument ignores that 2.7 was held onto for a decade. Legacy applications did not hold back adoption.

Which still seems like a short period of time, considering timelines in engineering are to support a version for 60 years.

>considering timelines in engineering are to support a version for 60 years.

In the vast minority of cases.

Not in most electrical engineering (except power plants), nor in computer engineering. 60 years ago was 1959. What software/computer project from that time is still running?

Even moving outside of the electrical domain, how many physical products outside of civil engineering is expected to last that long. I certainly can't expect support for my car for longer than 25 years.

If there's a difference it's not time scale necessarily, it's that CPAs are better at amortizing support/maintenance costs across decades. CPAs have amazingly detailed depreciation charts for non-software engineering lifecycles. If you build a civil engineering project, you figure out asset depreciation versus maintenance schedules, and you budget accordingly.

I've yet to see a CPA adequately depreciate software assets, anywhere. Maybe we should help them out by building better depreciation schedules. There seems to be a lot of CPAs that don't believe software depreciates over time, and maybe that's the largest disconnect in labeling it "tech debt", because accountants hear that term and think they can ignore it on a balance sheet they don't care about, but "tech depreciation" might actually scare them straight (until they find the tax advantages in calling it that).

They really need our help :-)

In a previous job that was mostly in an engineering/manufacturing department, but with a lot of Perl/Python automation scripts, we had an internal conference. One of the keynote talks was when not to automate using SW. It went into the cost of maintaining SW over the long term - including the fact that authors leave, and people who understand their scripts require higher salaries. Most people who write these scripts are not hired for SW roles, so their replacements likely cannot debug/extend.

Classic cars can still get parts after 25 years, but a lot of times its custom machine shop stuff. There is a market for it. Really there's also a market for restoring old computers (look at The 8-Bit Guy, LGR, The Living Computer Museum (Seattle) and all those other people who restore old hardware for fun and education.

But I get your point, those are special cases for preservation. For mainline things, especially with today's processes for continual integration, dependency checking and advance build tooling, dependency rot is something that should be accounted for in all software project plannings. If your dependencies are a few months out of date and you don't have the time to update them and re-run tests (people write tests right?) things are just going to hurt more and more later.

You can keep running Python 2 just like you keep your COBOL systems running, nobody's gonna stop you except common sense.

IBM does quite a good job of making sure that COBOL is supported long term with all needed updates for many many years to come. Python 2 is not in that position.

You mean, IBM does quite a good job of making sure IBM COBOL is supported long term. They are maintaining their compiler, which is exactly what PSF is doing. They are maintaining their interpreter, which is Python 3.

The GP post missed the fundamental difference between keeping COBOL running and keeping Python 2 running. Python 2 was also PSF's interpreter. IBM handling a COBOL upgrade like PSF handling the 2-3 transition would be unacceptable.



These are two very different kinds of institutions!

People who need COBOL support from IBM are paying a lot of money. Giant piles of money can get you many kinds of help that people won't volunteer to do for free... among them, maintaining ancient software in amber.

If you need Python 2 support and you are willing (and able) to pay the kind of money that IBM's customers pay for COBOL support, you'll be OK. For a start, Red Hat (aka also IBM!) shipped Python 2 in RHEL 8, which means they'll be supporting it until 2029 at the earliest.

To circle back to the original point, no COBOL committee would break commonly running COBOL programs like the Python 2-3 transition. Using COBOL in an example with Python is just wrong. Maybe the break is justified, maybe it isn't, but some languages do a lot of work to make sure things continue to work.

Would PSF exponentially increasing support and maintenance costs for Python 2 into multimillion dollar contracts and bundling over-margined hardware in with the bundle to make it more of an IBM-like transition help?

Yep, someone has to pay in one way or another like Red Hat customers on 7, but to say Python has near the life cycle of COBOL is just disingenuous. Old COBOL still runs, but Python 2 programs will not. It really shows what the achievement languages like COBOL, RPG, and Fortran are in terms of longevity and migration.

> but Python 2 programs will not

Er, why not? It's not like there's some kill switch in Python 2 that will make it stop working after January 1st, 2020. If it works now, then it'll still work, you're just not guaranteed fixes anymore. At least, not for free. As stated in the article, paid support options exist from several vendors.

Any software in the modern era that isn't upgraded for new OSes or security patched is dead.

Ok, sure. To you, maybe. But that's a far cry from saying they won't run anymore.

You can still run 1960's code today for both COBOL and Fortran - I suspect some Fortan 2 oddities might not work I am thinking of the sense commands

Probably, but the care the Fortran folks take to not break anything but still evolve a modern language is amazing.

Right, but in engineering it's kind of expected to get support for a version for at least 60 years. Software engineering is just really weird in that it moves so fast and nobody seems to care to break things.

They (in engineering general) aren't expected support something free though. Putting it another way: You can get you support with Python2, if you pay.

What free things are supported for 60 years in engineering applications?

Name one.

Fairly ancient Fortran compiles OK in open source compilers. Not quite 60 years old, yet, but it'll be there soon.

It's already >65 years old. The 65th birthday was this summer 2019.

We sell welding systems to weld stainless steal, copper pipes, etc. We always give a warranty of 24 month, guarantee paid support for 10 years, and support older machines only if possible. I am not sure which industry you are talking about, but 60 years is the exception in my experience.

I think this is where paid support comes into picture. Volenteers can keep improving Python, business that cant/wont upgrade can pay someone to "handle it", and consultants can make money. Everyone is happy.

The FAQ does point to vendors that plan to offer long time support/conversion paid support plans.

You have a dependency rot problem. Are you missing unit tests? Because having a ton of unit tests can reduce dependency rot, breakage and overall make engineering upgrades just a lot easier to deal with. They don't catch everything of course, but they can catch a lot.

If you haven't put in the priority to update your Py2 to Py3 apps by now, I really think your shop has the wrong priorities.

It's not just about Py2/3. Dependency rot is one of the worst form of technical debt. It often shows broken CI, broken security scanning, lots of generally broken processes that will just keep hurting a team further and further down the line.

I don't really agree with much of what the person you replied to said, but certain domains have had their hands tied and I also can't imagine them working as you describe.

CG/Visual Effects industry is still firmly using Python 2. Only in 2020 are they taking the first step to transition to Py3 [1]. Users and studios are held back because the Python runtime is used inside major applications; Nuke, Houdini, Maya as well as libraries and APIs. None of them have released a version that runs Python 3 yet.

The reasons for delaying it (mentioned in a footnote on that page) makes sense to me. Previous years were focusing on coordinating updates to GCC, Boost, C++14, Qt, and waiting on Python bindings for Qt.

Also, I've worked at a couple studios many people have probably heard of and none of them have unit tests covering much of their code. The focus is on tools that facilitate in-house artists where responsiveness to needs are valued over architecture and completeness. Requirements change for each project and previous requirements are often sacrificed (until a new project needs them in a few years).

I'm itching to move to Python3, but even for standalone tools I've felt it better to choose a completely different language (or Python2) instead trying to mix Python2 and 3 because having them co-exist creates more headaches in managing the environments, dependencies, and coding styles.

[1] https://vfxplatform.com

> If you haven't put in the priority to update your Py2 to Py3 apps by now, I really think your shop has the wrong priorities.

I am not sure how you could possibly know this. You have no idea what else they were working on instead.

Because no matter what you're working on, you won't get very far without eating, sleeping and at least once a decade putting in some maintenance work.

My overriding motto as an engineer is "If it ain't broke, don't fix it." My python 2.7 component has run flawlessly for years now, with virtually no need to update. I just haven't had to worry about it. We decided to update to py3 a few weeks ago, and this component is now having sporadic hiccups. Like, it runs fine 95% of the day, then all of a sudden a timer callback in a Tornado ioloop just stops running for 30 seconds after working fine for 10 hours or something. This disables the entire trading system. It's hard to explain to anyone, myself included, what I gained by updating a perfectly working system to something that now shuts production a few times a day. Now my time is diverted from doing actually useful tasks to fixing code that wasn't broken to begin with.

The tone of your writing is rude, unpersuasive, and lowers the level of discourse on the forum.

That’s like, your opinion, man.

I didn’t think it was the most amazing comment either, however, it was intended more to illustrate how things are rather than how they ought to be, I think some interpreted as a strong opinion in favor of the circumstances which was not intended. Based on how it scored (somewhat surprisingly) it clearly resonated with more than a few.

You've managed to say their analogy is bad in two paragraphs without even explaining why it's a bad analogy. You even admit you don't really understand what they were trying to say. Poor explanation, or poor understanding?

If an analogy requires more explanation than the original concept what purpose does it serve?

I think in this case at best the analogy grossly oversimplifies the issue. If it really were just a “screwdrivers” problem as explained then the 2 to 3 migration would have been mostly trouble free and would have happened. Clearly it did not go that way so that analogy can not possibly be appropriate.

[not speaking on behalf of any employer, opinions my own]

I think many people underestimate the challenge that the 2 to 3 migration presents for large enterprises. The core issue is that even though the migration for any given module is normally really easy, the total effort required to migrate is still essentially O(n) in module count/file count, because even with current tooling you still need to have an engineer look at every module to do the change safely. Even if it only takes ~5 minutes per module to make the changes and validate that it works correctly, this becomes a giant undertaking when you have tens of thousands of files to migrate.

The fact that it takes a long time also creates other problems. Your business isn't going to hit "pause" on other development, so there will be changes constantly introduced into modules you've already "swept". It's going to be hard to make sure 100% of your engineers and code reviewers are knowledgeable about the specific requirements to make sure the code works in both 2 and 3, so you would really like some automated safeguards to make sure they don't introduce anything that won't work in 3. Pylint helps with this, but won't catch everything. Unit tests are obviously essential, but:

1. Even a well-tested project won't have tests that cover 100% of code paths and behavior.

2. You're stuck running the tests on both python2 and python3 for the duration of the migration, which doubles the resource (compute, memory, etc.) cost of your Python CI and regression testing infrastructure for the duration of the migration.

I see far too many commenters attributing the delayed migration to laziness or complacency. In reality, most big companies have passionate Python advocates who really want to be on Python 3, but the scale of the problem and the lack of tooling to tackle it with a sub-O(n) amount of effort make the overall project risky and expensive for the business.

> The core issue is that even though the migration for any given module is normally really easy, the total effort required to migrate is still essentially O(n) in module count/file count, because even with current tooling you still need to have an engineer look at every module to do the change safely.

Compiler errors sure would be handy here :^)

Those are probably the biggest missing piece. Compilers and static analysis tools make it possible for relatively small teams to migrate between compiler versions for C++, Java, etc. It's often still a huge amount of work, but it allows a lot of the work to be automated, and doesn't require an engineer to manually inspect every module.

As one of the complainers, my time wasn't ever going to go into fixing third party libraries. Instead, I spent my time moving to languages that had more respect for their invested users.

Also, your screwdriver analogy makes no sense. The choice here is not about "your favorite tool brand". Mixing and matching tool brands is straightforward. The more appropriate analogy with screwdriver heads would be to imagine that you now need to change out all your hand tools, manufacturing lines, assembly robots, and equipment from multiple vendors from phillips to torx all at once. Sure, you can do some prep work, but you've got to coordinate a cutover at some point, in sync with tools that you have no control over or have to recreate from scratch. All for zero benefit in the end, other than the hope that they don't make the same mistake again with Python 4.

I am not sure what you want to say with your comment? That you move to another language as soon as there are breaking changes because you are invested as a user and therefore dissapointed? While this is a valid reason, it's just the way languages work. Some prioritize getting the language right and other backwards-compatibility. If your argument is categorical (languages that make breaking changes are bad) then indeed, python is not the language for your use-case (which is fine btw!). Java never made breaking changes, but it forced a massive limitation in the design-space for some language features (generics for example) for which it still pays a heavy price years later.

Also I think you're too harsh on the analogy, most of the time there's a way to write python2 and 3 compatible code, so which means the transition can go smoothly.

This is a bit and oversimplified, but it also seems a bit weird, to me, to respond to a situation of "man it's going to be really annoying to rewrite all this code" with "You know what sounds like a better option? Rewriting all this code!"

I think a fairer interpretation of the situation is "Well, we have to rewrite and test all this code anyway, so the barrier to picking another language to go forward with is quite low right now."

That's a corollary to this, for sure, particularly if there's another language that would be more appropriate. That said, the OP seemed to be moving on primarily out of annoyance. Assuming that's the case and Python is the best language for the project (keeping in mind things like "I can hire the same quality people who WANT to work in this language), I stand by my (still slightly tongue in cheek) position.

A question comes to my mind on the desires of developers to work on Python projects. Would it be fair to say Rust or Go might have more desirability these days? The WANT question makes me think this is a bit more than simple annoyance but provides an excuse to travel to greener pastures.

It’s a bit less absurd when the person gave the reason that they feared a similar breaking python release in the near future. This might still be incorrect of them, but to me anyhow it’s not strange or silly reaction.

>Instead, I spent my time moving to languages that had more respect for their invested users.

I’m not sure what more they could have done. Python 2 was released almost two decades ago, with an original planned sunsetting that got pushed back to 2020. That’s plenty of time and respect for their users.

>All for zero benefit in the end, other than the hope that they don't make the same mistake again with Python 4.

You’re free to stick with python 2 for as long as you’d like. You’re free to fork the project and continue develop it. To say there is no benefit to new and evolving languages is odd and short-sighted.

> I’m not sure what more they could have done.

Not broken 2? None of the breaking changes in 3 were necessary.

the first part of this captures how my company has handled python 2/3. there’s probably 1-2 hundred thousand lines of python 2 that works well and is largely in maintenance mode. if we were going to change it, it would be a piecemeal step by step replacement to a different language, not upgrade to python 3. going from 2 to 3 in sufficiently large code bases is already a language rewrite. the company has moved on from python

the zero benefit is where you show that you didn't even try to think about it. missed opportunity to stay silent.

(Unless your manager makes drive-by commits of py2-only code, months after you all agreed that all new code should be py3-compatible, and then leaves town for a multi-week vacation...)

That seems like a very specific example...

I had exactly _zero_ blocking issues. The only relevant thing for me is that I keep reaching for StringIO and urlparse (and friends) and they're used differently now.

_Everything_ I relied upon, every single dependency in my projects, gradually migrated over the course of a year and nothing broke.

Furthermore, I now have asyncio, aiohttp and friends to play with, and it's been pretty good.

If you were writing major libraries that needed to work on both 2 and 3 like many of us were, I suspect your opinion would be different. It was never a case of 'oh we'll just switch to 3 now' because that's not at all how it worked.

> If people spent half as much energy upgrading as complaining, this would have gotten done 5 years ago.

Yeah, right. Look at the creator of Python himself. Took about 3 years to migrate just his employer, and that's a top-tier software shop [1].

I've heard many people talk like you over the years, always being dismissive of the cost to others in python 2 vs 3. The complaints against the migration were not unfounded.

1: https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one...

Let me put it this way: at $dayjob, where 'IT' isn't particularly specialized, we just celebrated a switch to SQL Server from a homegrown database written in the early 80ies.

I wager that in many cases, fixing something that isn't broken never reaches the top of the TODO.

I was pretty confused by the change to str object, until I sat down to properly think it through. I wrote an SO response in case others were snagged on it. https://stackoverflow.com/questions/5471158/typeerror-str-do...

> Python 2 to 3 (at least by 3.3 or so) was one of the easiest transitions I've ever done.

I suppose you don't use Java.

> (Unless your manager makes drive-by commits of py2-only code, months after you all agreed that all new code should be py3-compatible, and then leaves town for a multi-week vacation...)

Isn't what code review and CI/CD is for?

Isn't force-merging without waiting for code review or test results what managers are for?

Sounds like you should get a new manager.

As a manager one should be extra careful to be _seen_ eating your vegetables, flossing your teeth and getting your code reviewed.

(What you do in private is a different matter. In some companies office politics can require some skullduggery.)

It was just a joke, but we're talking about a manager who "makes drive-by commits of py2-only code, months after you all agreed that all new code should be py3-compatible, and then leaves town for a multi-week vacation..."

Totally disagree. There's rarely a good reason to break backwards compatibility. It's beyond rude to choose to break compatibility and then force all your users to fix it.

The Apple comparison is a bad one since Apple controlled the whole thing.

How well would that work on MacOS?

you are comparing an open source language to the apple os.

Yes, and? GP was very clear that this is a comparison to show how quickly it can be done when it's proprietary and not open source.

What surprises me about this is that the documentation for Python 2 does not explicitly say that the language version is about to be unsupported, see e.g. https://docs.python.org/2/library/zipfile.html

Contrast this with the Postgres website, which tells me I'm browsing old docs (because Google still offers old links), see e.g. https://www.postgresql.org/docs/9.2/tutorial-window.html

Are there plans to add a banner to Python 2's docs? I think it would be quite instructive.

I cannot agree enough.

As a casual python user, it is quite common to stumble in some issue, after somme googling land in a doc page, try my solution, and find it does not work. Then googling after the failed solution, noticing that the previous doc page was describing a Python 2 feature that do not exists (or works different) in Python 3.

Me three! It happens all too often that an example Python script fails in one, then succeeds in the other. I'm also a casual Python user, trying to ramp up because of its obvious popularity and utility. I wish that the basic syntactical changes could have been designed to be more backward-compatible. But, eventually I suppose, python2 will just be a footnote in history.

I'm afraid it will forever remembered as a cautionary tale of a bad transition.

Same situation here. Trying to jump on this bandwagon and finding it's a slithering target.

I just wish they'd named it something different. If there's neither forward nor backward compatibility, I'd argue they're not the same language and should not have the same name.

Most languages have had breaking changes at some point in their history. Even the language known as "Perl 5" had them in almost every version.

In the real world, I've found Perl scripts written in 1999 (and poorly - as they were written by me in my teens) will still run in 2019. It's basically the policy that even language mistakes won't be undone:


This issue is part of why Haskell is so hard to learn. Haskell's package database Hackage keeps all old deprecated incompatible versions of packages, with documentation pages all well indexed by Google, with higher page rank because they are longer lived than the current package version, with no indication that they are deprecated and incompatible with current Haskell deployments.

It really shouldn't be beyond the wits of the hackage developers to include a normative URL in old docs...

This is also the problem haskell stack solves

"Haskell. The cause of, and solution to, all of life's problems."

"Stack" is the name of a product; it wasn't a generic reference to "the Haskell stack".

Try Hoogle.

Hoogle has its own problems...

I keep forgetting how have python one liner of a http server. And always when Google the python2 version will appear and always to play with the search term to get it.

python3 -m http.server

100% agree. I think every Python 2 doc page should have a big banner on top mentioning the EOL status of Python 2 and linking to the coresponding Python 3 page (if there is one). Googling "python + keyword" far too often still takes you to the Python 2 docs.

> I think every Python 2 doc page should have a big banner on top mentioning the EOL status of Python 2 and linking to the coresponding Python 3 page (if there is one).

I'd support that if every Python 3 page had a link saying "if you're still a sorry SOB stuck on Python 2, here's the link to the related library." It's just as bad for Python 2 developers, it's roughly a 50/50 shot of whether you get docs for 2 or 3.

I'm not sure this is the right place, but maybe open an issue here?


Or email the mailing list for more help?


I was about to file a bug, but they found out it already exists, just hasn't been merged into the 2.x tree.

- https://github.com/python/cpython/commit/46ed90dd014010703c7... - https://github.com/python/cpython/pull/13638 - https://github.com/python/cpython/blob/master/Doc/README.rst...

Python 2.7 isn't yet unsupported, so it doesn't compare to the deprecated Postgres version you linked to. Click through to the oldest _supported_ Postgres version, and it isn't clear when _it_ will lose support - no banners at all - , just like the currently supported Py 2.7.

That's true.

And because of changes in PostgreSQL versioning, people don't release that the oldest support version (9.3) is rather old.

PostgreSQL 9.6 was released in 2016, 10 in 2017, and 11 in 2018.

But PostgreSQL 9.3 was released way back in 2013...not quite as old as Python 2.7 but close.

Django has a similar banner at the top if you (perhaps via Google) get to a page about a feature in a deprecated version. So it's not like this is uncommon in the Python world. Hopefully they get that fixed soon.

HL7.org has a banner on every version of the FHIR spec with a different message depending on the version. http://hl7.org/fhir/

I would discourage this, as it will lead to banner blindness. Only pop a banner up if you want the user to read and react to it. Otherwise, they'll just get used to the banner being there and will often times not read it at all.

It seems to work pretty well for everyone. FHIR developers are keenly aware of the version of the spec they are (currently) targeting.

Also they are color coded.

In 2015, there was no way I could have moved to Python 3. There were too many libraries I depended on that hadn't ported yet.

In 2019, I feel pretty confident about using Python 3, having used it exclusively for about 18 months now.

For my personal use case at least, this timeline worked out well for me. Hopefully it works out for most everyone. I can't imagine they made this decision without at least some data backing it up.

Based on your timeline, a four year gap from "not yet ready to migrate" to "we won't fix security vulnerabilities" is very short.

Python is an open source project I've used and contributed nothing to, so I don't have the right to be a back seat driver. Were it a commercial project and I was a customer, I would be quite upset.

Python 2's EOL was first announced in 2008, it was extended in 2014. That's more than a decade of forewarning that this was coming down the line. A decade of Python2 receiving security fixes.

Were I a commercial customer, and had been told to switch to the new version ten years ago, then it's on me if I still have started the migration yet.

It's interesting seeing who the holdouts are, though. Less than a year ago, the CEO of Sentry (the crash reporter) locked a Github thread with essentially "we'll get there when we get there, sometime in or after 2020." But then, as late as 2016 their stance was pretty much that they would never migrate, so I suppose it's not surprising that they aren't super prepared for it:



I mean, even supervisord is migrated at this point, and it was a long-time laggard.

Calibre is another well known case, and they don't plan to migrate.

Actually, I can't wait to remove it from my hard drive on Jan 1, since it's one of the most poorly designed apps I've ever used. It's really not surprising at all that they're not able to migrate it, assuming the quality of the app is indicative of the quality of the code. I wouldn't be surprised if the same was true of a lot of applications that are refusing to migrate. (I'll probably have to find an alternative to ebook-convert, the command line tool that's the only part of Calibre I use, or maybe I'll rewrite it in Python 3 myself.)

The only good part of calibre is the `ebook-convert` commandline tool that has a stupid-simple interface (polar opposite of the gui) and handles all kinds of formats that pandoc doesn't.

The gui, Calibre proper, is a total nightmare. Functional, but my god is it esoteric and ugly. I've heard people actually use that gui as their primary ebook reading software and I just can't fathom how they find that tolerable.


I dislike Calibre main GUI as well. It looks a bit dated and all, but really convenient for downloading books from Userfiction forums like Spacebattles via the Fanficfare plugin. Also good for simple editing of ebook metadata. Built-in ebook-viewer is highly configurable with custom keybindings and themable with custom CSS.This is my setup


I use Calibre as my primary e-book reader. I don't see many alternatives on Linux.

I basically use Calibre ebook viewer for its configurability and customization. If you want a good ebook reader, give Foliate (https://github.com/johnfactotum/foliate) a try. That and Bookworm (https://github.com/babluboy/bookworm) are the best standalone ebook readers I've seen in Linux.

Other than having a simpler layout I don't really see either as better than calibres reader. What's better other than looks?

If that's the case, I'm glad to see things are changing. Up until very recently, there seem to have been no plans to switch: https://bugs.launchpad.net/calibre/+bug/1714107

> No, it doesn't [need to convert to Python 3]. I am perfectly capable of maintaining python 2 myself. Far less work than migrating the entire calibre codebase.

> I am perfectly capable of maintaining python 2 myself.

That's a very strange quote. I want to have this much self-confidence.

Calibre is, in fact, migrating. There are at this point dozens of commits over the past few months to that goal. Not sure what changed his mind.

Reality? You can maintain an absurd, unintuitive, dated UI by yourself fairly easily, but even someone who pushes out two minor updates per week is going to be taking on a little too much in my opinion if they believe they can also support a fork of a major language.

And make a gnome core app with GTK3? That would pretty cool :)

His responses leave an incredibly sour taste in my mouth. Not only is he obviously short sighted with his "Python 2 is not the past and Python 3 is not the future" but all his responses just sound like someone who got up on the wrong side of the bed. Every morning.

You couldn't switch to Python 3 until the middle part of this decade unless the stars aligned with your dependencies, the library support wasn't there. And the first releases of Python 3 were glorified betas, the first "usable" version of Python 3 is often considered to be version 3.3 released in 2012.

But you could easily write Python 3 compatible code so that upgrading once your dependacies were ready was easy. You also could have contributed to your dependancies to help them become Python 3 compatible.

I often hear this, but I just don't see this working in practice. A very similar example I have experience with, I've often worked with people who tried to write code that would be "compatible with Windows" (I primary work in a Linux environment). It makes the code more difficult to read and as soon as you try and run it in Windows it fails spectacularly.

You can write code that's compatible with both Python 2 and 3, but I don't think you can expect to be using Python 2 exclusively and write code that works well in Python 3 unless you're constantly running it with Python 3. Especially, if you don't have much firsthand Python 3 experience. This is much easier if your code is isolated from those dependencies and you write tests that run in both Python 2 and 3, but that's quite a bit more of a commitment than you seem to be implying.

There are a ton of open source projects that have done python 2/3 simultaneously in practice.

I get your point though about not being able to run full integration tests due to dependencies. Still, if you were making a good effort to keep the code Python 3 compliant, it would make upgrading much easier.

Also, my other suggestion- contributing to Python 3 compatibility for modules you depend on! That will give you Python 3 experience :)

Quick: run a shell command, split up the reply by line, run a regex on each line, and save the results to a yaml file— now, does what you wrote work correctly on both Python 2.7 and Python 3.3+? Yeah that's what I thought.

It's still hard to get bilingual Python correct today, and it was considerably harder before 3.3.

Okay, I did it.

    import subprocess
    import re
    import yaml

    output = subprocess.check_output(['ls', '-l'], universal_newlines=True)
    matching_lines = []

    for line in output.splitlines():
        if re.search(r'total', line):

    with open('matching_lines.yaml', 'w') as f:
Python 3.7: https://repl.it/repls/PotableDamagedAutocad

Python 2.7: https://repl.it/repls/OrchidDirtyNotification

Nice. Yeah my life got a lot easier when I learned about `universal_newlines` changing the subprocess return type under Python 3.

The much worse case of this that I dealt with a few years ago was reading in and modifying XML files, and then saving the modified XML contents as strings in a yaml file (a keyed cache). The input XML files (which I didn't control) were not consistent about having the encoding marked, and that really made things a mess for ElementTree.

A taste: https://github.com/ros-infrastructure/rosdistro/search?q=utf...

This program would crash with a UnicodeEncodeError upon encountering any file with a non-latin character in its name on Windows with Python 3.5 or earlier.

I know that's not the version of Python you were targeting, and I know you didn't write this program to be cross-platform, but I wrote a similar program and I was repeatedly surprised by those sorts of problems. I don't entirely agree with mikepurvis, but I do feel his pain.

No one is arguing that upgrading would be trivial; the argument is that it's necessary work that could have been started 10 years ago. Also, if you run your unit tests on Python 2 and 3, you'll get pretty good pretty fast at writing 2-3 compatible code.

The comment I was specifically replying to asserted that "you could easily write Python 3 compatible code", and could have been doing so 10 years ago.

This is simply untrue— there are a dozen small stumbling blocks that mean that writing bilingual code has a real cost. And it's not as easy as just having the right CI setup, especially if it's ten years ago and most of your dependencies haven't migrated.

Ah, I see the confusion. In this case, "easily" is relative. The assertion "you could easily write Python 3 compatible code" was in response to the (implied?) assertion "you can't do anything to migrate unless your dependencies are migrated". Perhaps "simply" would have been a better word choice than "easily". To your point, writing 2-3 compatible code has a real cost, but to the GP's point, it's the obvious solution to the "can't do anything until deps are migrated" problem.

> To your point, writing 2-3 compatible code has a real cost, but to the GP's point, it's the obvious solution to the "can't do anything until deps are migrated" problem

Q: Ten years ago how would you have explained to your manager that you needed your team to start writing 2-3 compatible code, which obviously takes considerably more time and effort than sticking with 2 compatible code?

It doesn't take considerably more time.

At this point, all the code I write is 2/3 compatible, and the things that annoy me are all py3 only features. Backwards compatibility stuff is pretty easy. Most of the easy problems are easy, and six has solved all the hard ones for like 7 years.

And I've been writing some of my code 2/3 compatible since 3.4.

> But you could easily write Python 3 compatible code so that upgrading once your dependacies were ready was easy.

This sounds a bit like telling someone, oh, you could've easily started driving an EV, so that once charging stations were actually out there, you'd have been ready...

> This sounds a bit like telling someone, oh, you could've easily started driving an EV, so that once charging stations were actually out there, you'd have been ready...

While car analogies are notoriously bad, it's much more like "I'm going to buy a plug-in hybrid and use gasoline until charging stations are widely available". You can write code that works in Python 2 that takes zero effort to run in Python 3. I (and many others) have been doing this for years. Just look at the number of packages on PyPI that run, unmodified, on 2 and 3.

[Edit: gas -> gasoline]

I try to write compatible code myself too, and the only reason it's "zero effort" to run in Python 3 is that I've already invested said effort when making it compatible. Getting strings and paths and stdio to be cross-compatible and working correctly in both 2 and 3 can be quite painful in my experience. Which was kind of my point with the vehicle analogy. It requires a significant investment that you're ignoring, and lack of enough underlying support to justify it during that time.

Driving an EV without charging stations is (nearly) useless. You can still run Python 2-3 code with a Python 2 interpreter with no significant consequence.

EV drivers don't need charging stations at all if they have an open plug at home or at work. The metaphor still sort of works that Python developers had plenty of time where maybe they didn't have enough charging stations on the wide open highway roads of Production to use Python 3 there, but certainly could have been happy to use an EV to commute from home (hobby projects) to the work parking lot (side projects, automation projects, new projects) and back while they waited for those "charging stations" to be built.

I don't want to debate the amount of shoehorning necessary to validate the metaphor. We all understand concretely the issue at hand, the metaphor is only obfuscating at this point.

I feel like there were 3 steps:

- 2008-2012 Python 3 becoming usable (byte formatting, six library)

- 2012-2016 Libraries (Django, etc) becoming compatible

- 2016-today Applications (Trac, Ansible, Chrome[1], etc) becoming compatible.

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=942720

2012 was 7 years ago.

well what if your best guess was that py2 would never actually be deprecated. Certainly in the early tens I had serious doubts.

The thing is, a four year gap is very large from the perspective of "should we drop everything else and prioritize porting our library to python3 right now".

If the deadline wasn't 2020 but 2024, then you wouldn't get more time, simply we'd be at "not yet ready to migrate" state right now, as major libraries would not have switched yet.

If the only way your organization can prioritize this is either “drop everything and update” or “completely ignore and do no work to prepare,” then it sounds like Python versioning is not your biggest problem.

I get this, though: porting code affects all parts of the code base, so it generates git conflicts all the time. It's way more of a pain to do than routine feature development or refactors. It gets difficult to have part of the team work on features while another group works on the port to a new version, because that generates a ton of conflicts. It's easier to get the entire team working on the port.

I had to manage a mature codebase upgrading Ruby versions, and it was painful. And that is fairly mild by comparison; not the big jump from Python 2 to 3.

Unfortunately, I think this type of thing actually is the case for many organizations. Trying to explain the business case for paying off technical debt to a non-technical person can be very difficult when "right now everything works."

I've had to fight developers over this.

A lot of devs like to not worry about new things.

I can really understand that… but I believe that such organizations will experience significantly more problems than just Python upgrades. (I work for one.)

> porting our library

Library, as is most open source libraries out there, sure.

For a big company-internal Python codebase that also has to wait for all the major libraries to migrate first, that time frame can quickly shrink to 1-2 years, which is very little.

You're waiting for other people to do stuff, and for many of them 1-2 years is the timeframe when they are likely to start working on it. From that perspective 1-2 years is not "too short", it's the "we intentionally chose this as the optimal time". That doesn't really depend on when the deadline is; if you postpone the deadline, then they postpone the migration - as pretty much every library did back in 2014 when the deadline was postponed by 5(!) years.

Beggars can't be choosers; if your company wants or needs major libraries to have migrated 4-5 years before the deadline, then your company has to participate in making that migration happen. Or accept that it's going to be done later than you'd like, because the needs and motivation of these library maintainers are quite different from the motivation of Python core maintainers.

This x 1000.

“The free libraries we chose to depend upon haven’t migrated.”

Do you have programmers?

“Of course.”

Put them to work.

“We have other priorities.”

Well then. Your choice, your outcome.

This is dumb. You're blaming people who probably don't get to call the shots. The managers who do have the ability to make different decisions are, themselves, often under the gun to meet other commitments.

Sure, they will have to cope. One of the ways people cope is by accepting security risks and continuing to run with old versions. Another is to decide that Python isn't for them, and move resources toward better platforms with commercial support for older code.

The fact is, that until a few years ago there were too many libraries that were not compatible for most organizations to make the switch. So, effectively the window for those organizations has only been a few years.

> You're blaming people who probably don't get to call the shots.

I think he's blaming organizations, not engineers.

I also think organizational dysfunction is a problem that isn't a technical problem that the Python project has the capability or duty to solve.

> You're blaming people who probably don't get to call the shots.

No one is doing that.

OTOH, it's not PSFs job to protect developers from bad management.

There is also the third choice - you don't migrate and keep doing your job (developing features) on the legacy code base. Things like certificates will have ad-hoc solutions pushed whether you like it or not, and the language will keep trundling along.

If there wind up being major security issues, there will be little choice but for someone to take up the mantle and fix them. Because companies aren't going to switch, and when you've got a botnet ravaging the internet they can't flip a switch and do a big-bang rewrite, someone will have to push out a patch for them.

This isn't the end of Python 2, it's just the Python team washing their hands of it. Ask PHP or Cobol users how their attempts to kill legacy codebases worked out for them.

This whole endeavor was complete folly from the Python foundation to begin with. A big-bang rewrite because they didn't like the syntax, and they didn't even fix the GIL while they were at it.

This attitude is ridiculous.

I'm supposed to go and tell all the various projects that I depend on that they have to change their upgrade timescales to suit mine? Even if I had enough time to help them all meet that kind of timescale, why would they agree to fit their changes into my required deadline?

Maintainers are under a lot of pressure. Having users come along and say "we'll add 50 developers to your project if you agree to ship v3 compliance by next month" is not reducing that pressure, it's adding to it.

More programmers != better results or faster delivery. Your solution is just going to create more problems.

You fork the library. You fix the incompatibility issues. You issue PRs back to the mainline to help them adopt the same fixes. At some point in the future you can switch back to the main repo if they've migrated, or you can consider that library abandoned and continue to maintain your own or find an alternate.

At no point do you have to wait on anyone else, you choose to.

You can do that, but then they won't accept your solution, so you just spend time developing a parallel system that you need to support yourself until the end of time. Been there, done that.

If there is significant demand for solving the problem your fork solves, others will migrate to it. This is made harder when the main solution gets to squeak along for 10 extra years on a version of the language that should no longer get first-party support. Of course there is going to be less demand for that Py3 solution when there is no immediate consequence for continuing on Py2. Killing security update support for 2 will boost the demand for libraries using 3, and make it more likely that your fork gets merged or becomes the defacto standard.

The Python 2->3 window is an example of where being too nice to too many users harms the project. Once Python 2 is dead, people will choose between being the sole maintainer of a Python 3 fork of some Python 2 library versus a being a sole maintainer of their entire Python 2 stack from the language layer on up.

How are they supposed to migrate if I can't openly host my solution because it was developed on company time? Merging back into the original project is one thing, but making your solution available some other way is a whole other story.

This is the reality of having lots of open source dependencies. Important changes/bugs may exist, and the maintainers are under no obligation to fix them to suit your business needs. If you want them fixed, you contribute to the code base (or fork it) and fix it.

Still better than my last programming job where the vendor of a vital piece of HW had provided a C++ API (also vital), but refused to provide dlls that would work in anything newer than VS2010. At least with open source you can do something about it.

You shouldn't take "Do you have programmers?" too literally.

Programmers ~= Resources ~= Money

=> If you want your dependencies to be upgraded in a reasonable time frame, try to contribute to financing the maintainers (or support them in a manner _they_ find suitable).

The time frame is 12 years though, since the sunset was originally announced in 2008.

A lot of libraries refused to migrate before Python 3.5(?) came out which was ~4 years ago. IIRC earlier Python 3 versions had performance regressions compared to Python 2.

This. For most organizations, the effective window has only been 2-3 years.

Personally, I would guess that many organizations will be running in Python 2 for at least parts of their codebase come 2020.

> Based on your timeline, a four year gap from "not yet ready to migrate" to "we won't fix security vulnerabilities" is very short.

Is it though? Especially since as late as 2014, everyone was under the assumption that in 2015 security support would end. We were already trying to migrate long before that. The five year extension seems pretty reasonable to me.

It was longer than Apple’s move to 64 bit apps on the iPhone.

Actually, depending on what action you pin dates to, it was twice as long.

One other thing - you can’t run non-64 bit apps anymore. You can still run Python 2.

That's a pretty low bar. I still think that move by Apple was pretty terrible. Especially given that it's 100% impossible to downgrade iOS (your last point).

And it’s entirely impossible to run 32 bit apps on newer Apple ARM chips. There is always a cost to keeping old APIs or in Apple’s case, old silicon.

It's actually longer than android has been alive...

But it's not commercial, and you had four years of guidance.

I feel like this the python community has been the epitome of class regarding communication during a transition.

In 2014, things were not quite ready so they extended it 6 more years. If you were upset by this you really need to get out of software development.

Windows 10 Enterprise LTSB 2015 (the original long term support version of Windows 10 Enterprise) doesn't support installing .Net 4.7 at all.

There are an increasing number of applications which require .Net 4.7, we found one early this year.

Promising "10 years of support" for a product is great, but if/when other vendors decide to undercut this, you can still be SoL :(

I'm pretty sure there will be more than one commercial offer for Python2 support in the future.

This makes me wonder, are there any software frameworks and languages that are specifically built for multi decade use?

Where they only release security updates and no breaking changes?

I think there are plenty of programming languages (maybe even most of them?) which take backward compatibility as a very serious goal. Obviously there's a balance between "don't break user space!" and fixing major problems (e.g., C's gets()), and most of the major ones seem to lean toward the former.

At the company I work at, we've got a large amount of Fortran code from the '70s that works fine with the latest and greatest Fortran compilers (barring deprecation warnings which help guide refactoring efforts). Same with C/C++ from the '80s.

Common Lisp code tends to be very stable long term. Occasionally people dig out some old code from the 60s or from really old books and papers and it will just run on modern systems. Maybe with slight modifications since the standard was finalized in 1994 and there were some changes to the way scope works in common lisp compared to say the older mac lisp, but anything written since that is written in portable standard compliant common lisp is pretty much guaranteed to continue working while there are still people willing to maintain common lisp compilers for whatever hardware exists in the future. It's actually extremely comforting to be able to come back to a project after not touching it in a few years and have every test pass even if I upgraded all the dependencies(as long as I picked the right dependencies that is :), lispers tend to be pretty conservative about breaking changes in mature and popular libraries.

"Built for", probably not. "That guarantees support for", some Windows releases and Microsoft stacks ended up doing more or less that, be it by accident or strategy. Probably some mainframe-related stuff too, being the only sort of thing around before the late '80s and possibly still in use.

There is also the case of Oracle, which defines support as "you can read our knowledge base and we will answer a phone before telling you it can't be done", meaning their support is literally for life (of the company). And they still manage to screw it up by deleting old KM articles...

Everything else in IT goes towards shorter and shorter support windows, because the field is always in flux and change (or rather churn) accelerates every year.

Anything based on real standards - C, POSIX/SUS shell, SQL, SGML, and others - is made without half-life and generational churn.

Many compiled languages might have breaking changes at the language level but are still compatible at the dll/binary level. Java 1.0 and C# 1.0 code can still be compiled and run on modern JVMs and CLRs, and rust 2015 and 2018 work together as well.

C18 is basically backwards compatible with C89. That's 30 years. The only function removed from stdlib is gets(), which was deprecated in C99.

Excel spreadsheets, which contain an astonishing amount of business logic, tend to be backwards-compatible far longer than the OSes that Excel itself runs on.

Food for thought, anyway.

I would have said Java, but then they decided to shove modules down everyone's throat.

Wouldn't Java 1.0 code still work in Java $LATEST even with modules? As i understand it the question was about keeping stuff working, not refusing to add new stuff.

1.1 is what would still work. There was not much Java 1 code in existence. Java 1.4 to 5 was annoying and the last release to introduce a lot of source code in compatibility. Which was sometime in 2004. Modules is mostly figuring out what switches to set on startup, not so much refuses to work.

I would say the Perl but then the Perl 6 fiasco made Python 3 look like a model citizen for upgrading code.

Perl5 is not being EOL'ed in favour of Perl6. When the time does come around for Perl5, I would guess that only a small fraction would migrate to Perl6.

> I can't imagine they made this decision without at least some data backing it up.

Mind that there is a feedback loop between data of current usage and an eol announcement.

If they announce an EOL close by this creates pressure to migrate.

In my mind, I kinda hope that data they used to back this decision up was asking "Hey, all you other volunteer Python language devs? How many of you want to work on the old Python2 codebase still? How many of you would prefer to be working on Python3? OK, that's just about unanimous. How about we all get drunk on Newyears and shut it down on Jan 1st?"

You can still get Microsoft Windows XP support today. If you're prepared to pay enough.

I wonder how many of the people who're upset by this have ever or ever worked for companies that've ever sent anything more than pocket change to the people Python?

Personal use can be quite different to business use cases. For many companies the conversion can be quite horrific, they might be quite happy to pay for support for older versions.

There are companies offering extended support beyond EOL for version 2.

I meant my personal business use case.

>In 2015, there was no way I could have moved to Python 3. There were too many libraries I depended on that hadn't ported yet.

You could work on porting them or be part of that effort?

Some people have full-time jobs and other obligations already that don't include time for this kind of effort.

Your employer should be pitching in your effort. If they're not, and they're choosing to be a bad citizen, then on their own heads be it.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact