
Where are we in the Python 3 transition? - joeyespo
http://www.snarky.ca/the-stages-of-the-python-3-transition
======
jacquesm
The python 2.x to 3.x change will be studied for many years. How to completely
rob an ecosystem of its momentum in one fell swoop. I still don't understand
why 3 couldn't have had a mode switch defaulting to 2.x behavior. The whole
debacle would have been avoided and all the ugly details would have been
hidden behind the language implementation. It would have definitely been more
work and you might even call it an ugly hack (and Ed DeCastro would likely
have thrown a fit if one had asked his opinion) but it would have at least
made the transition instant and seamless.

The present mess will - if not resolved in a much more drastic fashion - cause
more and more teams to switch away from python. It's a real waste of the
potential that python had (and still has).

I never even bothered porting my code, I'm 'stuck' on 2.7 and anything new
gets written in another language. Even bloody PHP, the language everybody
loves to hate on, gets deprecation and backwards compatibility better than
python.

~~~
nostrademons
That's the same approach that IE took for many years - IE7 embedded the IE6
rendering engine, IE8 embedded both the IE7 & IE6 rendering engines, etc. You
could switch between them with a meta tag, or with developer tools.

It didn't stop IE from being robbed of momentum.

I think this is more a case study of what happens when fundamental assumptions
that a technology ecosystem is based upon change. Things like string &
collection types, async mechanisms, interface/protocol definitions, package
managers, programming paradigms, developer preferences, and hardware
performance characteristics are foundational to a language's definition. After
all, the network effect that binds a language ecosystem together is all based
upon being able to share code between multiple unrelated developers.

In Python's case, unicode happened. A lot of old Python code was written on
the assumption that everything was a sequence of ASCII bytes; it was just
plain broken in the presence of unicode characters. There's little you can do
here other than accept that much of your ecosystem's strength is gone anyway
and build for the future.

I suspect we'll see similar problems crop up in other languages in the near
future; in particular, promisification in ES7 is a big threat to the existing
Node ecosystem, which is all based upon callbacks. A language that jumps on
the promise bandwagon early (Rust? Go? Elixir?) could rob Node of much of its
momentum. Similarly, if Android & iOS win out over the web, that could lead to
a big resurgence in Java or rise of Swift over server-side technologies like
Node or Go.

~~~
choosername
so python is the IE of programming languages? yikes ;)

~~~
nostrademons
Python 2 is, yep. And Java, and C++, and PHP, and all those other languages
that were super popular at the turn of the millenium but now feel really
dated.

Actually, C++ has an even bigger claim to the title, because it now has
"modern" versions that are much better, more standards-compliant, and have
language features borrowed from newer languages...but a number of enterprises
are stuck with no-exceptions, no-STL, no-move-semantics subsets because that's
how all the existing code is written.

~~~
blub
The IE metaphor is terrible for programming languages :)

I don't know about PHP, but C++ (and I suspect Java too) is backwards
compatible and doesn't belong in the same class with Python. One can still
compile older code and start using new features as they see fit, there is no
hard break.

------
sitkack
I moved last month and I have been using Python since 1.5.2

What prevents me from recommending it to everyone I pass on the street is that
the PyPy 3.x line is back on 3.2.5 which makes it not compatible with many
libs which have a 3.3 minimum, which is most. Yes, I give to pypy.org
[http://pypy.org/py3donate.html](http://pypy.org/py3donate.html)

Recommendations to folks

    
    
      * switch your daily interpreter to Python3
      * use tox, https://pypi.python.org/pypi/tox
      * use six, https://pypi.python.org/pypi/six
      * when you go native, go cffi, https://pypi.python.org/pypi/cffi

~~~
eiopa
This kills it for me too. I rely on pypy, and I just can't see myself
sacrificing my productivity by using Six.

Also, Python 3 neglected to fix one of the most annoying things about the
language - default arg value

    
    
      def foo(x=[]):
          x.append(1)
          print x
      
      foo() # 1
      foo() # 1 1
    

Why is this still busted??

~~~
throwaway91919
Can you (or someone) clarify what problem you're talking about? This seems to
work as expected for me:

    
    
        def foo(x=[]):
            sum = 0
            for a in x: sum += a
            return sum
        print foo([3,5])  # 8
        print foo(x=[3]) # 3
        print foo([]) # 0
        print foo() # 0
    

Is it the fact that kwargs are required to have defaults? Or what?

~~~
saltylicorice
The problem is that the default argument is mutable.

    
    
      def f(a=[]):
        a.append('v')
        print a
    
      f(['ok']) # ['ok', 'v']
      f()       # ['v']
      f()       # ['v', 'v']

~~~
throwaway91919
Ah, thanks.

------
rdtsc
> While some people feel stuck in Python 2 at work and are "depressed" over
> it, others have reached the point of having transitioned their projects and
> accepted Python 3, both at work and in personal projects.

It is not as simple.

I would be simpler if it was 2006 again. Python would have less competition on
being an easy to use language, batteries included, with good enough
performance (SMP wasn't as popular then). Java had a bad perception -- too
bloated, enterprise-y, Javascript on the server wasn't happening much. Go
wasn't there. Scala not as popular. In that context, people would eventually
have switched to Python 3 even if it took some pain (security maintenance,
some newer features etc), because GIL wasn't a big deal and it sure wasn't
going to be Perl again (maybe Ruby for webdev?).

The situation is more difficult today -- Python has more serious competition.
Once a team has found the time/money/excitement to switch to Python 3, it is a
high chance they'll start looking and evaluating other
languages/platforms/libraries (and I speak as a person who loves Python and
used it professionally for 12+ years). It is now competing with Go, Elixir,
Javascript, Scala, Julia, Clojure, even Rust, Java, and C++14. No matter how
cool Python 3 is and what new features it brings, it will have a harder battle
to fight just because of the current environment.

~~~
criddell
> It is now competing with Go, Elixir, Javascript, Scala, Julia, Clojure, even
> Rust, Java, and C++14.

That would have happened no matter what. It's not really a bad thing, is it? I
can't see how something better coming along is anything but great news.

------
captainmuon
If someone would make a Python 2.8 (or call it Python 5=2+3 if you like), I
would pay for it. It must run basically every python 2.7 (or 2.6) program
without changes. It should backport as many features as possible. The meaning
of "str" and string literals (bytes or unicode) should be configurable on a
per-file basis. It should have a print statement. Finally, it should be able
to use all the binary modules from 2.7 (maybe after recompilation).

Currently at our work (in physics research), I have the feeling that people
are giving up on Python. It's not worth the trouble to port to python 3 -
especially because we can't switch all code at once; we'd have to support both
versions for years. C++ which is also heavily used here is getting much nicer,
with proper tooling it is even more pleasant to work in (code completion etc).
We are mostly using Python for a) configuration files and b) quick and dirty
scripts. For these cases, Python 2.7 will be good enough forever. If someone
does something more complex, they'd probably use Python 3 - but they treat it
like it's a completely different language rather than a new version of Python
2.

~~~
jamesdutc
You'd have to pay a lot.

Jokes of mine like
[https://github.com/dutc/rwatch](https://github.com/dutc/rwatch) and
[https://github.com/dutc/didyoumean](https://github.com/dutc/didyoumean) and
[https://bitbucket.org/dutc/astexpr](https://bitbucket.org/dutc/astexpr) are
examples of adding major features to the CPython interpreter at run-time. It
works fairly well in practice. It ends up looking like a `__future__`-import
pragma that has interpreter lifetime scope rather than file scope.

`__future__` imports cannot be undone within the same file -- after all,
they're pragmas scoped at the file level, not actual imports -- and it would
make sense for these backported features to behave similarly, though at the
scope of the interpreter's lifetime. That might not do what you want.

As you'd expect, the most superficial of Python 3's new features (syntax
changes, `raise from`, &c.) would be easiest to backport. The bulk of the
difficulty would be in merging those changes into Python 2.7. Making the
result hot-loadable would be just as easy as in `dutc-rwatch`, though being
able to toggle features on-and-off independently would require combinatoric
effort with the naïve approach. You'd need to do something smarter.

All in all, it's wholly possible to do what you want, and you probably have
the skill to even do it yourself.

I'd happily do the work myself, but only for lots of money. Unfortunately, the
burden of both certifying the resulting interpreter as correct and of
maintaining this fork would be large.

And then when you're on-boarding some fresh new post-grad you've just hired
into your research group:

"Oh, we use Python 2.8 here. That's what we call Python 2.7 with an
unsupported collection of modules that mutate the currently running
interpreter to backport features from Python 3. Don't worry; it only breaks in
ways that no one outside of this office will ever have a chance of ever
understanding or helping you debug."

~~~
jamesdutc
Oh, I should also mention that I found a way to embed Python interpreters
within themselves. I have a couple of different ways to do it.

Here's one:
[https://gist.github.com/dutc/eba9b2f7980f400f6287](https://gist.github.com/dutc/eba9b2f7980f400f6287)

Here's another:
[https://gist.github.com/dutc/2866d969d5e9209d501a](https://gist.github.com/dutc/2866d969d5e9209d501a)

I've given, like, a million conference talks about this.

Better than creating some horrible Python 2.8 hybrid would be funding (or
convincing me to volunteer to do) the work of completing this bridge. This
involves writing 2/3 and 3/2 PyObject shims and figuring out some GIL and GC
issues.

Then, within a given host interpreter, you'd be able to execute modules within
a guest interpreter of whatever Python version you want.

With the shims and the bridge work, you'd be able to interact with these
objects seamlessly from host-to-guest and guest-to-host.

------
wh-uws
As much flak as ruby has gotten for its performance issues and "magic" (read:
heavy preference for use of meta programming) ...

One thing I think the ruby core team has done a great job of in my opinion is
always having some awesome new features in the next major release that make
you say "hey thats awesome" and make you want to upgrade despite any headache
it might cause and the headaches are generally minimal.

Its been interesting to see python so publicly and painfully fail at that.

A whole new language version release that as far as I can tell only changed
internal plumbing and went so far as to make you have rewrite print
statements?

I just don't understand how they could think people would want to move to
that.

~~~
lyschoening
Maybe Python 2 to Python 3.0 didn't introduce many "awesome" features, but 3.4
and 3.5 have been great.

~~~
AnkhMorporkian
asyncio and pathlib are absolutely killer features for me. I never much cared
about Python 3 until asyncio came on the field. As soon as I saw some example
code Guido had written, I started using tulip, and have been a willing
participant on the wild ride it has taken.

~~~
rspeer
Pathlib lets me down a bit -- I find myself having to str() the Path objects
and lose their benefits, because the rest of the standard library doesn't use
pathlib.

But asyncio is great. It shows how much better asynchronous programming can be
when you have support from the language itself. I can't imagine going back to
a byzantine system like Twisted even if they did manage to finish their Python
3 port.

------
AstroChimpHam
I'm a long time Python user and I still tend to go for 2.7 over 3. When
writing packages, I'll add support for 3 as an afterthought, but that's it.

The main reason for this and the main reason I prefer Python to other
languages is the excellent and complete ecosystem of libraries. I've never
seen a single Python library I wanted to use in the last 8 years that didn't
have support for Python 2.7. I still run into those that don't support Python
3 well. So, using Python 3 means being terrified that I run into a case later
on where I need to use some package that only supports 2.7 and I get stuck.
With all of the other potential problems to run into while coding, why would I
want to add that extra one, especially for something work-related?

I imagine this will continue being an issue until some major packages stop
supporting Python 2.7 entirely and force people to make a choice.

------
joseph8th
I was in denial stage until we picked Python 3 for a project at work. Then I
went into anger and bargaining with the other dev on the project. Heh. _blush_

Being forced to come to terms with Python 3 has made me see the light, though.
I still have a host of personal projects in Python 2, but anything new I write
is in 3.

For me, it was concurrent.futures that pushed me into acceptance stage.

~~~
macavity23
My experience was similar. An even bigger push in my case was the async stuff
in 3.5.

It's going to be a little while before pypy supports 3.5, but once that
happens, Python will be very well placed. Something like Tornado + pypy +
py3.5 looks a lot like the app server stack of the future IMHO.

It has been a massive PITA, and probably could have been better handled, but
hindsight is easy. The future looks bright.

------
foobar2357
The Python team is blind to the huge number of users that will never switch to
Python 3. My industry (a large one) is currently standardized on RHEL 6 (and
the CentOS equivalent), which uses Python 2.6! I'm not generally able to
install arbitrary software or demand upgrades without a lot of beurocracy. I
was looking forward to the day when we move to RHEL 7 with Python 2.7.

I've started to view the lack of changes in 2.7 as a feature. I would value
the stability and lack of breaking changes. Knowing that Python 2.7 is on a
limited timeline, and that the Python 3 folks (users and developers) are
trying to kill off 2.7, I can't think of any reason not to switch to another
language.

In short, my field is nowhere near "acceptance" of Python 3, and the "anger"
stage will just make us (me at least) leave Python behind. I will never want
Python 3.x, and I don't even want Python 2.8. I wish people would recognize
that stability is a virtue for some of us...

~~~
mkesper
The investment in porting your code from Python2 to Python3 should be many
times smaller than porting to an entirely new language ecosystem.

~~~
dimva
Yea, but what's to stop Python 4 from doing the same thing as Python 3?

It's clear from the tone of this article that the python core devs haven't
learned the value of backwards compatibility even after their huge python 3
failure. The author makes it seem like the problem preventing python 3
adoption was python users' unwillingness to accept the inevitable, rather than
the poor migration path provided (especially initially).

In a massive, old production codebase, upgrading your platform is incredibly
risky, even if no code changes are necessary. But when you have to make code
changes in a dynamic language...

The thing is, not all systems have tests. Not all systems even have known,
defined behavior. Upgrading these things can be a multi-month, very risky
project.

Given that python 3's backwards incompatible changes were made for purely
cosmetic reasons and there are few compelling reasons to upgrade, I totally
get the anger people feel. I work in startups with relatively new codebases
and good test coverage, but I can empathize with other types of organizations.
I'm shocked that python devs don't, even after 7 years.

~~~
msellout
That is an incorrect characterization of the core development team.

[https://opensource.com/life/14/9/why-python-4-wont-be-
python...](https://opensource.com/life/14/9/why-python-4-wont-be-python-3)

------
51109
From LearnPythonTheHardWay.org:

> A programmer may try to get you to install Python 3 and learn that. Say,
> "When all of the Python code on your computer is Python 3, then I'll try to
> learn it." That should keep them busy for about 10 years. I repeat, do not
> use Python 3. Python 3 is not used very much, and if you learn Python 2 you
> can easily learn Python 3 when you need it. If you learn Python 3 then
> you'll still have to learn Python 2 to get anything done. Just learn Python
> 2 and ignore people saying Python 3 is the future.

Besides, judging from the number of DARPA-sponsored projects that require
Python 2.7, the future is not going anywhere soon.

~~~
swozey
I'm so tired of seeing people act like LPTHW is some gospel. I tried using it
to learn Python and absolutely hated it. It scared me away from Python.
Especially the "don't ever use 3, use 2" thing.

And if you're going to assume that government entities use of a language is
what is going to drive our market than you better stop using Python and hop
back on to the ColdFusion and Fortran train.

~~~
51109
It's not an act. LPTHW is my Bible. I retake the courses every Sunday. Saying
that I should hop on the Fortran train for stating that even advanced fearless
Python users opt for 2.7 is heretics!

------
craigds
Personally, I can't wait to switch to python 3. I'm glad most of the community
is finally getting behind it.

The cogs move slowly. We've only just upgraded our huge codebase to 2.7, but I
expect we'll be on some release of 3.x in about two years. Almost all of our
big list of dependencies now support 3.3+, so there is little reason to not
upgrade.

------
TheCowboy
One frustrating part of attempting to transition is that not all libraries
make it clear which Python versions are supported. Usually it's Python 2-only
code that isn't clearly labeled as such, but it happens with Python 3 code as
well. Sometimes a library doesn't work under specific versions of Python 3.
The result is that library research takes longer than it would.

Please include this in the README for any project, even if it seems obvious.

------
Walkman
> I have yet to hear from someone who has used Python 3 that they think it is
> a worse language than Python 2

the maker of the most succesful Python projects: Armin Ronacher

~~~
mianos
Just ask him about unicode for LOLs.

------
JustSomeNobody
Use 3. Unless you have a good reason to use 2, then use 2.

The world doesn't have to be perfect. Go make something.

------
cdnsteve
Can someone explain aiohttp server vs wsgi? It seems wsgi is the old standard
but now you can run a server right in python, kind of like node. Are people
running aiohttp instead of wsgi?

It seems like aiohttp can do everything that Flask can do. Are more people
using this?

~~~
RodericDay
I want to know too. I did a little toy project with websockets in aiohttp and
it was extremely enjoyable.

It'd be reassuring to know that other people see it as "the future" too, so I
could invest into it more heavily.

------
jeffdavis
I think one fundamental miscalculation went like this: early adopters will
start writing new code in python3, and it will gradually become more normal
for new code to be python3, and eventually it will snowball and python2 will
become legacy.

The problem is that early adopters have a choice: python3, or a completely
different language. And there are a lot of good languages to choose from, and
early adopters are a more radical by nature.

That's why breaking backwards compatibility is almost always worse than it
looks like on paper: the early adopters you are counting on to get you through
the transition are also the most likely to leave your platform completely.

------
BuckRogers
What I think people figured out is you can port Python2 code to PyPy4. Then
for all the testing involved, you remove any potential CPU performance worries
and the GIL in one fell swoop.

Those two perks are worth more than any piddly features of Python3, which were
already resolved in the ecosystem. I ported to PyPy instead of Python3 and
gained those advantages, but can always port to Python3 when/if it ever gains
traction. But I'm just as likely to migrate new projects to something else
than Python3.

I view this blog as a long-form denial letter to himself.

------
someguydave
I like the fact that the 'python core' devs have moved on to doing something
other than pointlessly fiddling with something that's reasonably good, ie
python 2.7

As for 'lack of support', what are folks expecting support for? If there's a
known security issue, somebody will write a patch.

------
beej71
Those of us running Arch Linux have to check the rearview mirror to see the
transition... back in 2010 when "python" started defaulting to Python3.

It wasn't the smoothest thing, but the world didn't end, either. Lots can be
learned from it if you're thinking of making the move.

------
tonetheman
I am on CentOS and honestly for most of our applications there is just no
compelling reason to change. We can do everything with 2 that we need.

I have seen some async stuff that looks interesting in 3 but not enough to
make me change.

------
mrfusion
I've always wondered why they couldn't package 2 and 3 in the same binary and
let the user have a switch for which to use?

~~~
takeda
They don't need to. You can install all versions (at least tested 2.4 - 3.5)
side by side without any conflicts, which makes the whole dispute which
language version to use so silly.

------
upofadown
>With five years left until people will need to pay for Python 2 support, ...

Is this some sort of thing?

~~~
kemayo
It's just the author pointing out that the EOL for official support for Python
2 is 2020. After then, the Python dev team won't be providing any (free)
support, so people who want to keep using it will have to pay for it in some
way -- whether by investing time in maintaining a fork, or by paying someone
else to do so.

The absence of a "Python 2.8" fork suggests that the idea of forking hasn't
been popular enough to happen organically without someone putting money in,
after all.

~~~
someguydave
>The absence of a "Python 2.8" fork suggests that the idea of forking hasn't
been popular enough to happen organically without someone putting money in,
after all.

Or it's evidence that python 2.7 works just fine for many people.

------
x1024
Wait, Python 3 has existed since 2008 and the transition is at 20%? How is
that any form of success?

~~~
craigds
He said that 20% have reached "Acceptance", the last of the 5 stages, and said
that most people are in stage 4 (Depression). That'd put overall progress at
more like 80%.

This article also wasn't actually talking about _adoption_ of python 3, just
applying the stages of grief to how people _feel_ about it. IOW, adoption of
python 3 might still be worse than 20%, but 20% have accepted that they will
need to adopt it soon.

Anecdotally as a full time python guy, I'd agree with the OP. The number of
people interested in Python 3 has been growing a bit every year, and it seems
opposition to adopting it at all has finally mostly dwindled :)

~~~
claystu
According to the python community survey of 2014, 80% of users are still using
Python 2. Of course, that was two years ago, but 2014 was still six years
after Python 3 was released.
([https://wiki.python.org/moin/2.x-vs-3.x-survey](https://wiki.python.org/moin/2.x-vs-3.x-survey))

If the python developers were any normal business, they would have abandoned
Python 3 and returned back to developing what their customers obviously
prefer; the problem here is that the python developers essentially concluded
that because users get python for free, they aren't customers at all and the
python developers can do whatever they want and their users will eventually be
forced to fall into line.

Except they won't...

Python users are refusing to migrate to Python 3 as a community. It's a shame
that the python developers have decided to try to show their customers who is
boss because it won't work--the developers will lose. Many people will
continue with 2.7 forever and fork the language in 2020 like Perl 5 day. The
rest, like me, have dumped python and moved on to greener pastures.

~~~
x1024
Yep, it's true. My company started a major python2.7 project in 2015.

~~~
BuckRogers
Pretty smart bet. You have production-ready PyPy to migrate to, and can always
port to 3.x if it ever takes over (looks like never).

------
Walkman
Extended the maintenance timeline from 5 to 10 years is not helping the
transition at all.

~~~
someguydave
The extended maintenance timeline is an implicit admission that nobody wants
to hop on their wagon.

------
PhantomGremlin
There was a Simpsons episode[1] back in 1991 in which Homer, after eating a
poisonous fugu fish, must deal with the five stages. It's an all time classic:

    
    
       Dr. Julius Hibbert: Now, a little death anxiety is
       normal. You can expect to go through five stages.
       The first is denial.
    
       Homer Simpson: No way, because I'm not dying!
    
       Dr. Julius Hibbert: Second is anger.
    
       Homer Simpson: [furiously] Why you little... !
    
       Dr. Julius Hibbert: After that comes fear.
    
       Homer Simpson: [worried] What's after fear?
       What's after fear?
    
       Dr. Julius Hibbert: Bargaining.
    
       Homer Simpson: Doc, you gotta get me outta this.
       I'll make it worth your while.
    
       Dr. Julius Hibbert: Finally acceptance.
    
       Homer Simpson: Well, we all gotta go sometime.
    
       Dr. Julius Hibbert: Mr. Simpson, your progress
       astounds me.
    

If only this Python transition were so simple.

[1]
[https://en.wikipedia.org/wiki/One_Fish,_Two_Fish,_Blowfish,_...](https://en.wikipedia.org/wiki/One_Fish,_Two_Fish,_Blowfish,_Blue_Fish)

