Hacker News new | past | comments | ask | show | jobs | submit login
Python Versions Used in Commercial Projects, 2016 Edition (semaphoreci.com)
160 points by markoa on Nov 11, 2016 | hide | past | favorite | 167 comments

Maybe unjustified, but I'm always kind of suspicious of a post which is largely focused on raw data (like this one) and which presents only pie-chart visualsations [0].

Yes, Python has a pretty dirty history, with many people choosing to stick to the Python 2.7 that they knew and loved. And yes, commercial software tends to move waaay slower than the wider community (many banks are still running COBOL). If you're focused on making money and pleasing clients then "it worked for us before" is always going to be the strongest argument.

Major players in the Python eco system have pledged to move away from Python 2 [1], and if we had non pie-chart visualisations, I'm sure we'd see huge trends towards Python 3 in the last year. Even slow-moving corporations are starting to use Python3. Yes, MacOS defaulting to Python2 is still a problem, but Ubuntu switching to default Python3 is already a huge step to get companies to move forwards.

[0] http://www.businessinsider.com/pie-charts-are-the-worst-2013...

[1] https://python3statement.github.io/

It's not just because we "know and love" Python2. We also have immense legacy Python2 codebases that it would be time-consuming and risky to upgrade to Python3.

And we have other business priorities we have to pursue. As long as Python2 is viable. Which it still is and probably will be for quite a while. (Even once official support for it is dropped, there will undoubtedly be large enterprises that rely on it and will continue to maintain forks.)

This is why the Python community gave 15 years to migrate. This is why we created many, many tools (2to3, six, Python-future, mypy) to help with it. And this is why Guido is always saying 2.7 users are important and need respect.

Still. 15 years. It's not a last minute change. Python has been incredible in maintaining the 2.7 around, giving people help and time to change. I know NO other tech that did that. The JS community breaks his toys every other week. PHP just jumped a version to avoid it. No one is migrating to Perl 6. Python is actually the only one that pulled this off, and have been very, very careful to give you the margin to do it.

Everything needs to evolve, and one day you will have to do it. Don't to it in 2020, or it will be way more costly.

And yes, you have many advantages in having Python 3, mainly ease of dev, debug and maintenance. This is not as hard as it seems. Took me a week alone to convert a huge system. Don't feel overwhelmed by the task, it's really not the end of the world.

> Took me a week alone to convert a huge system.

Can confirm. Porting even huge code bases has become much easier lately thanks to recent changes in Python 3 (unicode literals!) and Python-Future. "from six import u" and many other hacks are no longer necessary.

Try it, and there's a huge friendly community there to help you :-)

And when you have almost a million lines of code dating back 10 years; about 3% test coverage; and an executive team breathing down your neck to implement the features that should have been prioritized 5 years ago by the "leadership" they bought the company from?

When you have 3 engineers, one of whom is working full time on bugfixes and preventing everything from catching fire and one of whom is working full time on applying the second of 8 major versions' worth of upgrades to your framework (along with third-party dependencies that have to be version-compatible) just to catch up to a version that's still maintained with security patches?

When every framework/library upgrade, let alone language version upgrade exposes subtle backwards-incompatibilities with your external services, so you have to upgrade those too, and then spend 2 weeks understanding and solving the resulting forwards-incompatibilities?

Just because it was easy for you and your codebase doesn't mean it's going to be easy for everyone else.

I'm not saying we'll never doing it. I'm sure as hell not saying we (being the engineering team) don't want to do it.

I'm saying we haven't done it yet because we can't do it at all until we complete the other upgrades. We can't do it at all without spending a week or three months or however long finding or writing replacements for crappy (and in a few cases not so crappy) abandoned third-party packages that aren't and never will be supported under Python3. We can't do it until we can spare 3 months from our timeline to exhaustively click-through test the damn thing and verify we haven't broken anything.

And saying we had 15 years to migrate is bullshit. Our major dependencies haven't been stable on Python3 for more than about 3 years now.

By the way, did I mention before that we have other business priorities? Python2 is still supported. It'll continue to be supported for a few more years. Why should we spend valuable time converting to Python3 now, when the other things we're working on could be the difference between whether or not we're still around in a few more years?

No, it won't be more costly in 2020. It'll be less costly. We'll have a larger engineering team because we solved more business problems and customer problems, so the business earned more money and could afford to hire more people. Or we'll already be out of business and it will have had nothing to do with which version of Python we were using.

Here Python 2 vs 3 is not your problem here. You problem is that the project is in very bad shape. Anything that disrupt it will cause you huge problems. Python migration just happen to be the wake up call.

No tech can be asked to be calibrated to compensate 20 years of bad project management.

I'm not blaming the tech. I like Python3 and start new projects in it when I can.

Yes, our code base is a giant fucking disaster. I wouldn't even begin to try to deny that.

I'm also pretty certain that there are others in the same boat to a greater or lesser degree. I've seen too many codebases that were the product of starting with a team of one or two and a mandate to move quickly. It's impressively easy to dig yourself into this sort of hole, and it's impressively common among startups in my experience.

I'd imagine that large enterprises are even more likely to have even larger legacy codebases without full test coverage. They may have more resources to throw at the problem but are also likely to have more political obstacles and even higher standards for verification that they haven't introduced new bugs.

But the GP was insisting that there's no reason not to have upgraded yet other than sheer stubbornness and unwillingness to learn new things, or even laziness on the part of the engineers. That's incorrect and insulting.


Do not underestimate COBOL. The latest version is from 2014. https://en.wikipedia.org/wiki/COBOL#COBOL_2014

How many commercial projects use COBOL 2014 though? I know Fortran 77 is still considered newfangled in some circles, even though Fortran 90, 95, 2003, 2008, 2015 exist. Same with C90, C++98, etc.

Net-new COBOL code is being written every day in the Enterprise world. I work on a team that's actively updating and maintaining a large COBOL codebase. The default standard that most people write to is COBOL-85.

Just because the language is old doesn't mean that it can't be the right tool for the right job. For batch processing-type tasks, there nothing better than COBOL on the mainframe. By default, COBOL doesn't allow for you to dynamically allocate memory. That makes it incredibly easy for the compiler and system to optimize the compiled code and make it run super fast.

COBOL is great for what it was designed for: processing data. If you want to read data from a file/database, process it in some way, and then save that data back to the database/another file, you can't do much better than what COBOL offers.

I'd love to know what "super fast" means. Are we talking about ETL jobs that would take hours? We run our batch file processing jobs in Python. It's not the fastest. But in a world where we can bring up more instances/containers to help process the work faster, I'd love to know how much faster COBOL is at these jobs that it'd be interesting to consider against the alternatives.

I code in Fortran 77/90 an average of 3 hours a day and I have a lot of fun! For science, the language is just vanishing and you can focus on the equations with at the end a crazy fast code.

The other 3 hours are in Python, Go and JS&cie.

Also, we (Ubuntu) definitely won't maintain or ship Python 2.7 indefinitely; I'm keeping a keen eye on how the ecosystem moves to decide when that happens.

I was under the impression Ubuntu is re-shipping the upstream debian python2x and 3x packages. Are your maintainers packaging 2x from scratch? If so...why?

Well, our maintainers (primarily Matthias) package both for Debian and Ubuntu, so they diverge only in the sense that the cadences and versioning are not completely aligned.

But regardless of the packaging effort, the real overhead is tracking security for the [overlapping] LTS releases.

You'll keep ship Python 2.7 for more or less time than Perl 5?

Hah, that's a trick question. Probably for less, given how deeply Debian tooling depends on perl5.

Also, for the Python ecosystem closed source projects are less relevant than the open parts of the ecosystem. And the open side has rather clearly decided which versions will go forward.

On a data level: I wouldn't be surprised if most of theese projects are either not actually new (ie. only added to their CI system, codebase is older) or is related/interacts/integrated to/into a Python 2 codebase, which arguably isn't a from-scratch project, either.

As a data point: Google still uses python 2 and they used to employ Guido van Rossum.

Doesn't his current employer also use python 2?

Yes. He has to maintain Python 2.7 as his day job at Dropbox.

And isn't he working on Piston, high performance JIT Python, which is python2.7 compatible?

AFAIK he's mostly working on mypy/typing at Dropbox.

Google is entering dinosaur territory. Big and impactful, but as a consequence, slow to change and laden with decades of preexisting infrastructure. I wouldn't expect them to be using Python 3.

If they put numbers on it, they'd be telling us how many customers they had? They might - understandably - not want to do that...

Google Trends chart Python 2 VS Python 3:


Wouldn't call 28% "largely ignored", rather "still lagging behind". Thing is, 3.6 looks like the real deal breaker here, what Python 3.0 should've been. With asyncio, compact dicts and scandir I've gotten to the point where I can almost justify to my employer why, if everything is still working, we should move to Python 3.6 ("all will be faster" -- I know I'm kind oversimplifying here, but my boss pretty much doesn't understand software and just wants results/money).

There's also the sentiment of "I don't know if [insert module here] will be supported", which has become mostly baseless fear, but people still think Py3 support is lacking (when it's not![0]).

[0] https://python3wos.appspot.com/ -- also, take notice how 9 of this guys come from here [1]

[1] http://mozbase.readthedocs.io/en/latest/

Edit: when I posted this comment, the link was titled "Python 3 largely ignored" (not the article per se, but the submitted link had been titled that way). It has been changed, but this was a bit of important context for my comment.

I'm still waiting for supervisor, beanstalkc, and cloudsigma.

The setup.py for Supervisor noted support for Python 3 starting in May of 2014: https://github.com/Supervisor/supervisor/commit/ad5cd0849a5f....

Here are the blocking issues for official support: https://github.com/Supervisor/supervisor/labels/python%203

I've been running it under Python 3 for a while with no problems.

Not sure why supervisor matters, it's just a program. It's the same thing as Homebrew running Ruby - totally doesn't matter what version they're using.

It does if you're using their API

Of course, specific projects might not be covered, but I've seen a lot of people go "ok, just to be safe I'll use 2.7 in case down the road I get to need anything that's Py2 specific".

In your case you know exactly what your needs are, and that they're unsupported in Python 3, but I'm sure that many of those 70% Py2 projects could run just as well in Py3.


Can I ask why you're using supervisor instead of systemd at this point?

Can't speak for gp, but we're using supervisor inside Docker containers as lightweight init process. In our case the main advantage of Docker containers is ease of installation and consistent environment - so we don't always use just one process per container.

Systemd on the other hand is just something that is being forced down our throats. :-/

Well, have a look here http://py3readiness.org without having to figure where 9 of those come from ;)

Some research on most of the projects which don't:

supervisor: Application supervisor, not a widely used library - it's probably in the top 360 since people install it using pip instead of the distro's package manager

carbon, graphite-web and diamond: Components of the time series DB Graphite, see supervisor

Thrift: Apparently there's a Python 3 pull request? [1]

ansible: Difficult story since they have to support old systems which only support Python 2, so no point in migrating to Python 2 until those are no longer in use (that includes Python 2.4!).

This is a really unique use case for Python so it's hard to compare it to the rest. The part of Ansible which runs on the controller has been ported already [2].

google-apputils: There are Debian packages for Python 3, so probably missing the PyPi metadata [3]. Appears to be unmaintained.

aspen: Niche web framework. They seem to be working on Python 3 support [4].

newrelic_plugin_agent: see supervisord

python-cjson: Unmaintained drop-in JSON library that can be replaced with something else with no effort [5].

python-openid: There's a Python 3 fork which looks nicer than the original [6].

m2crypto: Porting effort underway, but mostly an issue for existing cde - there's no reason to use it for new projects since python-cryptography is better in all regards

python-ldap: Issue for many enterprise Django projects (this is the one library in the list that I personally use), but there's a Python 3 fork so it's fine [7].

INITools: Abandoned, it used to be pip dependency so that's probably why it's in this list (?).

marionette: see supervisord

pathtools: moved to stdlib in Python 3.

So, nothing major left in the top 360.

All of the huge blockers like Twisted and Paramiko are now green and I migrated all my stuff over.

The Fedora porting project has some great insight into how it's going in the real world:


OpenStack is interesting, too: https://wiki.openstack.org/wiki/Python3

In general, no major libraries left, but lots of grunt work that needs to be done.

Some stuff not on the list, that I encountered:

python-dajax: Turns out it's a bad idea to put frontend logic in your backend code, but that doesn't make the dependency go away (great opportunity to finally undo that mistake in my code)

wxPython: They decided to do a full rewrite instead of a port so it takes some time. Qt, I guess?

[1]: https://github.com/apache/thrift/pull/213

[2]: https://docs.ansible.com/ansible/dev_guide/developing_module...

[3]: https://packages.debian.org/sid/python/python3-google-apputi...

[4]: https://github.com/whit537/mongs/issues/17

[5]: http://artem.krylysov.com/blog/2015/09/29/benchmark-python-j...

[6]: https://github.com/necaris/python3-openid

[7]: https://github.com/pyldap/pyldap/

Python 3 is omnipresent in my work right now and it is so nice not to have to worry about, for example, unicode issues.

The only thing I'm dealing with which is still not Python 3 compatible is AWS Lambda, which only offers 2.7 right now. Supposedly Amazon is working on 3.x but... come on.

Same here.

Haven't had a single Unicode bug in production since we switched to Python 3. Also, chained exceptions (extremely valuable in walled-off production environment).

Type annotations actually work and have prevented a number of bugs.

The asyncio syntax is really nice with Tornado (don't use aiohtto, use Tornado! it's much more mature)

Python 3 doesn't remove unicode issues, although granted you probably run into fewer issues when defaulting to unicode over a bytestring.

Python 3 doesn't remove all unicode issues, it's not magic. But the pain is 95% gone compared with Python 2. I write Python for a living, and my life has been a lot better since we switched.

Because unicode is default and the byte-type is single-byte adressable (rather than 1 character in <encoding>) it makes it more straightforward. It's not perfect, but it helps a lot.

It doesn't? I've ran into UnicodeDecodeError exceptions literally every time I've had to deal with external input in Python 2. It was a nightmare, and it made me hate Python.

With Python 3, that simply has never happened to me. Never.

I don't know why. I don't know what the change, what the secret sauce was. And what's even better: I don't have to care. Python now simply works for me.

So yes, Python did remove Unicode issues.

I'm slightly moving backwards. Previously I just make Python 3 scripts with 0% retrocompatibility because I thought that Python 3 was going to happen soon and it was futile to try to resist. Now that my programs and scripts are getting used more frequently at work and outside of work I'm making more Python 2-3 compatible code because often people just try to use python myscript.py (that might just call any interpreter), or they use myscript.py in an development environment that only uses Python 2.

It's a bit sad to not be able to use some things that I really love about Python 3 but it makes me happier to see other people enjoying my work, so I'm going to bear with it for now.

The only time we're writing new Python 2.7 code is when we're working with Twisted (and they are vigorously migrating to Python 3). All other new code is Python 3.5 based and we're not looking back at all.

Using asyncio has been a delight for a lot of the small, quick web services we need to write (though not a replacement for Twisted). Looking forward to seeing the asyncio infrastructure grow.

Same at my place, 2.x is banned for all purposes.

Ansible only works with 2.x, so that is perhaps the only exception I would make... (if you are an ansible shop).

But ansible scripts tend to be like 5 lines, so it is not the same as the scale you would write a webapp or something in Python3...

For Ansible, it's important to differentiate between the master and client side code.

Client side code has to run on everything starting with CentOS 6, so they have to stick with Python 2 for a long time.

For the Ansible master (which runs on the management machine), Python 3 porting work is already underway.

We're a Python shop but all of our Ansible work is just YAML. Ansible is a program more than a library so it's almost irrelevant what version of Python they use. Now, of course it's highly annoying to have to bootstrap python2 on Ubuntu Xenial but that's really not relevant to what version of Python a group uses for regular development.

We use Python 3.5 and are VERY happy about it.

Came here to say this. I've default to Python 3 for all new projects (our new stuff is primarily on Ubuntu 16.04 anyway), but we use AWS Lambda a lot and it's still stuck at 2.7 for now.

They're also only Node 4.3.2. It seems that they need quite a while to test new versions, or they have little man power and are constrained to keep working with what they know it worked.


That's why Python 2.7 will be around and maintain dominance for a while.

When the latest stable releases of the primary environments are defaulting to something you have to have very good reasons why not to use the default (even when it's easy to change the default).

Administrators (in the old school not the new "everything in containers") don't like using non-defaults.

The default for Debian [0] and Ubuntu [1] is Python3. The `/usr/bin/python` symlink will be Python2 forever, since Python itself advises it. PEP 394[2]:

> for the time being, all distributions should ensure that python refers to the same target as python2.

[0] https://www.debian.org/doc/packaging-manuals/python-policy/c... [1] https://blueprints.launchpad.net/ubuntu/+spec/core-1311-pyth... [2] https://www.python.org/dev/peps/pep-0394/

Am I incorrect in recalling that ArchLinux decided against this?

edit: seems to be the case[0]

[0]: https://wiki.archlinux.org/index.php/python#Python_2

Read the next line in PEP 394 ;)

> however, end users should be aware that python refers to python3 on at least Arch Linux (that change is what prompted the creation of this PEP), so python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3.

Yes, which is why it is a staple of PKGBUILDs for Python software to do

  sed -i s/python/python3/ /all/the/things

Before upstream CPython had any defined policy on the python symlink.

All major distributions have completed their switch to Python 3 or ship it already.

This is Canonical's stance:

> All Ubuntu/Canonical driven development should be targeting Python 3 right now, and all new code should be Python 3-only.

Regardless, nowadays many people deploy Python applications as stand-alone bundles. We're happily running Python 3.5 on CentOS 6 and it took like ten minutes to deploy (pyenv!). This lets us use a rock-solid base system with an up-to-date application stack, which makes our old school admins happy.

Hmm, ok so you suggest all of a sudden switching to explicitly calling Python 2 with python2, like it is now for Python 3? Python 3 is present and working by default but we just decide, by convention, that Python 2 = python and Python 3 is python3. For obvious backwards and horizontal compatibility.

  $ lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 16.04.1 LTS
  Release:        16.04
  Codename:       xenial

  $python3 --version
  Python 3.5.2
Edit: Just to check I started a clean 16.04 (on DO) and did the commands:

  # lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 16.04.1 LTS
  Release:        16.04
  Codename:       xenial

  # python --version
  The program 'python' can be found in the following packages:
   * python-minimal
   * python3
  Try: apt install <selected package>

  # python3 --version
  Python 3.5.2
So, no python 2 by default but python 3 and you call it using "python3" as adviced in the pep qznc talks about.

Afaik, it has nothing to do with Ubuntu or any other Linux distro specifically, but this is rather a choice from python itself. See PEP394 [1]:

"for the time being, all distributions should ensure that python refers to the same target as python2."

I read above as the python community itself made "python" the default for python 2.7 and "python3" for python 3.x. Nevertheless, an unfortunate choice at this time. I understand the reasoning at the time of writing (March 2011) but now this should be reconsidered.

    [1] https://www.python.org/dev/peps/pep-0394/

We actually hit an unfortunate consequence of this with some scripts being called in python 2 mode while we were working in a python 3 virtual environment.

Hmm... you really use the system Python installation for development? Python is not pathologically backwards compatible, what happens when the next OS version use a newer Python 2 version?

You don't simply upgrade your production machines to the next OS version with fingers crossed. You do some testing and validation first.

Say, you use containers and so your application is isolated from the host system. When the next version of your host system comes with a new version of Docker, you have to do similar testing and validation anyway.

Personally? I don't, I use Python 3 and virtualenvs but then everything I write in Python is for myself to do things for me.

If I was writing software that had to be used in as many places as possible or sold, I'd absolutely strongly consider using 2.7.

OS vendors have similar issues with other open source projects. They tend to ship versions which are years out of date.

I understand the desire to have stable releases. But... if you're not shipping the most recent stable version of something, and instead shipping something that's 4-5 years old... it's frustrating for users and developers alike.

Former Python developer here. The problem was two-fold: Python 3 broke backwards- and forward-compatibility (in a practical sense: there were workarounds, but libraries didn't necessarily use 'em, and Python is nothing if not a glue language); and for the longest time there really wasn't a good reason to switch. I think it wasn't until last year that it really started to make sense to go with Python 3 — and by then I'd already switched to Go.

Python and Go aren't really mutually exclusive though. Some things will better be done in Python, others in Go.

Tried go and fairly quickly headed back to Python. Kept running into annoyances that slowed development. Not being able to get a NULL back from a database without a third-party library was the final straw (go 1.5, might have changed since then).

I understand there's obvious use cases for Go, but an amazing amount of what we do is taking data from place 'A' (web, socket, file), sending it to place 'B' (another web, socket, file), and then returning a result. Go doesn't buy you much that asyncio won't.

> I understand there's obvious use cases for Go, but an amazing amount of what we do is taking data from place 'A' (web, socket, file), sending it to place 'B' (another web, socket, file), and then returning a result. Go doesn't buy you much that asyncio won't.

Static typing. Oh Lord is it nice.

When I wrote Python, I never quite knew if there'd be a runtime error (in my code or a library). With Go, there are a lot fewer of those.

static typing has become possible with python3, though i don't use it for my personal projects.

So do you use 'Go' as your glue language now? or still use python for that?

Yup, professionally it's all-Go, all the time. But I've always been gluing libraries, not programs, together. I think I'd still reach for Go over Python for gluing programs together, but I've not done it enough to be completely certain.

Are you using Go exclusively?

All the jobs I see are Ruby + Go or Python + Go, which makes it pointless to learn Go if you don't care about Ruby or Python.

> Are you using Go exclusively?

Pretty much, in my career. Sure, i know plenty of languages, but my day job is slinking Go.

I honestly don't want to program in Python any more. I never thought I'd say that, but it's true.

I'd guess a lot of the X + Go jobs are porting from X to Go.

This appears to be a common story. We ported a bunch of Python 2.7 to Go. The improvements in speed and memory consumption are outstanding.

But we will be upgrading a large code-base from 2.7 to 3.x. For things like web data aggregation and parsing, it is hard to beat Python; the tooling in that space is really good.

Definitely looking forward to some unicode pain alleviation from the rewrite.

The utter pointlessness of the Python 3 changes pretty much killed the language for me. print-as-function helped pretty much nobody, and I'm seriously starting to think i18n is a mistake.

What date does 1/2-3 mean?

I've had raids cancelled when we agreed on 19:45, but showed up an hour late because he didn't realize that nobody plans raids in GMT.

Är det inte störande när folk börjar prata ett främmande språk mitt i sin post? Å it ens Google Translate hjälp nä dialekt kom i bildn. And as a non-native English speaker, I'd also much rather have the original English version of whatever you're doing, than the half-baked translation that's missing tons of features that are always "coming soon".

No, Unicode-as-default didn't cause this, but the switch shows a switch in culture towards making i18n easy.

>...and I'm seriously starting to think i18n is a mistake.

You don't see value in meeting customers in their own language and culture?

I don't see value in half-assing it, which still costs you resources, but also annoys those users (including me!) when you get it wrong or disable features because you haven't translated them yet. And there are enough cultures/languages that doing it properly isn't a viable strategy.

Users generally update by default, if thousands upon thousands didn't it's because someone else fucked up.

I find it funny how a small group of people thinks they know better than the outer community, to the point that they feel like they should have a say in what thousands of businesses use to run successful code.

More than this, I would argue that most people using Python 3 are those new to the language. This is only from personal experience, so it's really just anecdote.

PS: In a kind of jesty side-note, I know the general argument is that "python 2 was broken", but really, how broken can something be when thousands of businesses depend on it and more than that, choose to keep using it when a "better" alternative comes about?

My company has been writing a significant portion of their commercial code in Python since version 1.5 and (almost) everything new in Python 3.5.

Coming from a country with an extended alphabet, I've always been aware of encoding issues, especially before almost everyone settled on using UTF. So the treatment in Python 3 (and equally in Go) is a game changer for me and many other language users out there. I'm not saying that, say, American developers don't need to support other alphabets, just that it's less pressing since they don't tend to deal with them on a daily basis.

So for this feature alone, I'm all in on Python 3.

I don't follow the Python community closely, so I might be totally off the mark here, but my general impression is that it was shipped prematurely. From what I've read, in more recent years a lot of work has gone into easing the transition... But it seems a bit late. As an outside I wonder: did they fail to put in a reasonable effort at all in providing a smooth transition from the start, or did they try and failed to account to the realities of our world?

Recklessly breaking backwards compatibility without a smooth migration plan is hostile to developers. Although it's not a programming language, this is something that I think React has gotten right with their current versioning scheme [0].

[0] https://facebook.github.io/react/blog/2016/02/19/new-version...

Python 3 was shipped with the expectation that the migration would be long. Python 3.0 was the proof-of-concept release, basically, and it wasn't really until 3.3 that momentum started picking up around porting. 3.5 and soon 3.6 are beginning to offer large, attractive new features people want and can't get on 2.7, which further incentivizes the move.

But in order to make it happen, and to provide the long lead time for migration, 3.0 had to get out the door when it did. So it wasn't "premature".

There's a smooth upgrade path via importing from `__future__`, but that doesn't help if you've been ignoring non-text data in your strings. The automatic 2to3 translators don't know what de/encoding you want to use.

Upgrading to 3 doesn't cause bugs, it reveals bugs that many people would rather brush under the rug.

Python really should learn from PHP7.

Yes, it breaks a lot of things, but it's totally worth it for the performance gains.

Py3k solves problems partially like Unicode or async/await but it's a non-issue for skilled python2 developers.

People like incentives to upgrade. Period.

Lots of incentives for us:

- clean Unicode support which prevents mistakes which used to bite us in production

- native async syntax, compatible with Tornado

- asynchronous generators (yield from a coroutine!)

- pathlib and os.scandir

- type annotations

- matrix multiplication

- chained exceptions

- faster dicts, ordered by default

So many useful features, and with the latest Python 3 releases porting has become really easy thanks to improved compatibility with the old syntax.

They are all pluses, but most of those didn't exist till the last couple years. So effectively the first 6 or so years of Python 3 existing had much less incentive.

I think the flip around - ~30% adoption of a newest version at 1-2 years in - especially one that breaks backwards compatibility isn't bad.

I've started numerous python projects at work over the last year all on 2.7. I think we just started the last one. We're finally ready to jump to 3.

Indeed, and ordered dicts come from Py 3.6, which is in beta now and expected for December.

If async and this had been in Py3 from the start, they could've done a far better argument saying "hey, we broke all this stuff, but you get asyinc I/O and faster dicts" (a note of faster dicts: since many classes and language mechanisms use dicts, it should have an encompassing effect on all of python performance -- maybe not 30% or 40% faster, but a few % all around, which is kind of what micro benchmarks are showing with 3.6 betas)

All of these incentives you listed are also doable in Py2k.

Native Async syntax won't be doable in Python 2.

We have gevent, tornado, twisted, pyuv.

This is spot on.

What if I am not running into any significant Unicode issues for the past 6 years?? (insignificant == stuff I can fix in 10 minutes). What if async/await has been achieved through other libraries (e.g. GEvent, threadpools, whatever)... What is the incentive to switch a large (working) codebase?

The code is just easier to maintain and debug. Unicode (which is a HUGE deal for a french like me) is not the only thing that has been fixed in Python 3.

A lot of bugs disappear (the API is unified, division is true by default, imports are absolute), or are easier to detect (clear exception if you compare incompatible types, bytes are very explicit).

Many duplicate ways of doing things are removed (modules are renamed, builtins have been cleaned up), the syntax is terser (no more object in class, no more from __future__, super() has a shorthand, no more # coding, not more u'').

It's easier to write lower memory code (most things are generators by default, yield from allow to delegate to sub generator).

Creating clean library is also much easier. Now I'm playing with type hints, and yes, it does help to avoid bug. "raise from" makes error reporting much better. No relative imports make organizing code much easier.

Basically everytime I go back to Python 2, it feels like a pain. So many little things I didn't notice, take for granted, but that adds up quickly.

The worst being of course to code a 2/3 compatible code and to have to do things like: https://github.com/Tygs/ww/blob/master/src/ww/types.py. But I do it only for libs that I want to use specifically to help me migrating project (ww wraps everything the same way in 2 and 3, making migrating easier).

I look forward for 3.6 to introduce f-strings, which I will adopt asap because it's so convenient.

I expend considerable effort that I am not obliged to expend to use Python 3 whenever possible. I'm sure that I'm not unique.

Right now in my life the major things stopping me from using python 3 for all the things are.

  * Ansible
  * Two large in-house developed apps taken over from previous devs
  * Debian Wheezy
And as you can guess, at least one of those will hopefully disappear soon.

Other than that I try to use python 3 as much as possible. I know a lot of the workaround for getting pyvenv working on various distros for example.

I also know about a lot of alternative libs like ldap3, dnspython3 and more.

On 1st point, yes, you need Python 2 for Ansible (and Fabric) but it lives nicely side by side with Python 3, so your apps–the not-legacy ones at least–can use Python 3 no problem.

I think about Python 2 as Ansible's dependency, not something I have to use because it's there.

> I expend considerable effort that I am not obliged to expend to use Python 3 whenever possible.

Isn't that kind of a bad sign, though? Should it really require 'considerable effort' to use the latest release of a fairly widely adopted language?

It's a good sign in the sense that people want to move forward.

But yes, it's a bad sign in the sense that more effort is required than with Python 2.

It's getting better all the time though. Most of the effort is because of a local Mac OS development environment, legacy Debian Wheezy systems that are hard to replace, but also legacy Ubuntu systems that are hard to replace.

We use Ansible (system 2.7) but our code is written in 3.5. I'm not sure what you're talking about since never the twain shall meet. It's BAD to use the system Python for any of your internal production code (use a virtualenv!).

I do use venv, I have one for each version of ansible. 1.x, 2.x and dev.

But what do you mean when you say your system uses 3.5? Yes of course the system python can be 3, or 2 simultaneously but ansible still requires python 2 to run.

And I haven't tried this but I'd assume lib ansible can't be used by python 3. Maybe you can confirm.

this headline is a bit much. 1/3rd of new projects use Python 3! A lot of people are excited about the language now that a lot of the warts have been dealt with and language compat is higher than ever.

Though all it takes is one dependency to throw a wrench in the plans, the major projects are Py3 now.

Though porting large python 2 projects is still a huuuuuge pain. This is more a result of python 2 badness than python 3. But a lot of work.

I’m not sure why but the Mac has not installed a "/usr/bin/python3" so I continue to avoid 3.x migration (aside from minor things like "__future__" imports for division and print_function).

I don’t want to add a dependency for end users when my current code works fine with the built-in Python at 2.x. Nor do I want to bloat the size of downloads to embed something like an entire interpreter binary. (Swift has the same issue; there is currently no way to refer to it externally, as it is in flux; I do not want to embed loads of Swift libraries in every build so I will wait to migrate until they have stabilized binaries and made them available from a common root.)

I blame in part, that the default install for OS's like Ubuntu and Mac is Python 2.7. If you are starting python development, and want to support more systems, and you know the default is 2.7, then that's what you will target.

What does "default install" mean?

/usr/bin/python will always be Python 2. See https://www.python.org/dev/peps/pep-0394/

Python 2 isn't installed by default on Ubuntu cloud images since 16.04.

> for the time being, all distributions should ensure that python refers to the same target as python2 .

> however, end users should be aware that python refers to python3 on at least Arch Linux...

That clearly means he will recommend /usr/bin/python migrates to python3 some day, and that today developers can already not count on it being python2.

No. It means that Arch did something that all the other distributions thought was so obvious that it didn't even need to be specified, and then they had to write a PEP to explain what is going on, write down what was considered to not be needed to be written down, and what users can no longer assume as a consequence.

Beyond the abstract, goes goes on to explain that, no distros shouldn't do what Arch did (like you said), that developers must strop trusting that python points to python2, and that he will change this PEP when the context changes.

That's as antagonistic to "python will always point to python2" as you can get.

Python 3 is the default in Arch too.

Python 3 is the default in 16.04 LTS.

More explicitly, a standard installation of Ubuntu only depends on (and leaves installed) Python 3:

   $ lxc launch ubuntu:16.10 yakkety
   $ lxc exec yakkety python || echo no py2
   no py2
Python 2 is still available via a package, and in the main repository, which is what we track to deliver security (and critical bugfix) updates for. But we definitely see the importance of moving the ecosystem along -- despite the historical complexity of the transition, and the challenge in getting people to move from their comfort zone.

I just posted that my default Python is 2 and I'm on 16.04.1, but I suppose the upgrade from 15.10 perhaps left the default as-is?

I think the fact that python points to python 2 and python3 points to python 3 does not mean one is default and the other is not. One would think so, yes, but this is a bit of difficult case, for historic reasons. Both versions will need to be present because many things depend on 2 still (ufw for one I believe), python 2 was just there first and hence earned the badge "python". To now switch to explicitly calling it python2 and calling python 3 python would break many things probably.

On a clean 16.04 install, python 3 is present, python 2 is not, however, you still need to call python 3 with "python3" [0]

[0] https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#Python_3

Edit: See qznc's remark on the pep, apparently it's even advised the keep "python" pointing to python 2.

I suppose the upgrade from 15.10 perhaps left the default as-is?

Yes. If you do a clean install of 16.04 then you won't python2 installed at all.

Anecdotally, we had a simple events API running Python 2.7/Flask/uwsgi/nginx that needed to do quick I/O operations on a growing # of HTTP requests. To increase throughout, we experimented with Python 3 for its async/await style concurrency. It didn't seem to help much and we ended up just rewriting that API in Go and see way better throughout-to-resource ratios by shipping 5mb binaries to Docker & Kubernetes.

Point is, we're a hybrid shop that does all first pass services in Python 2.7 and then move them to Go when they become suffiently trafficked and/or critical.

My colleagues at academia (scientific computing) mostly default to Python 2. Personally I'll switch to Python 3 when approximately over 50% of my colleagues use it or when some mind blowing new feature or library is introduced only for Python 3. Makes things simpler this way. Also, I don't currently have enough spare time to port all my code.

Don't be so scared of porting your code. On my current work, we have a few thousand lines of python (spanning different services and supporting libraries) throughout in ~18 months and we were blocked by pika (a module for communication with AMQP servers).

When pika got a python3 compatible version, we ran out of excuses to not use python3, but still my colleague was worried that "it would take too long to port things, and our-code-is-working-so-why-bother". I decided to try it anyway, and it took me less than a lazy Friday afternoon to run 2to3 on all the packages and get the tests to pass.

Given that your work is with scientific computing and (presumably?) you have control over the deployment environments, I'd really recommend that you avoid do the switch sooner than later.

> Don't be so scared of porting your code.

I stuck with that line, because maybe his colleagues are also waiting for "50% of colleagues using Py3" and it's just a deadlock until some few brave enough starts to port and tip the balance.

Can confirm. Ported our internal stuff and it was much easier and faster than expected.

I find this really surprising as all of the scientific libraries that I know about support python 3.

Python 3 also has a load of improvements that would make scientific programming faster.

Perhaps instead of thinking about not having spare time to port your code, that you're saving time for future you by not creating more work to port over when you end up really needing that mind blowing new library :)

Python 3 is the default teaching language for the RaspberryPi. The BBC micro:bit runs MicroPython - a reimplementation of Python 3 for microcontrollers.

Teachers and kids are already ahead of the game.

When today's kids graduate they'll view Python2.7 in perhaps the same way I view Delphi or VB6 ("you're still using that..?" etc,).

And then Codecademy go and use Python 2, sigh

I've recently started using python a lot more, it was a natural progression when I moved from using a windows dev machine to a mac book. One day I needed to do something quick, and python was the quickest way to do it. I typed python into my terminal, saw 2.7, and built from that. I had no incentive to look into upgrading, I had everything I needed to accomplish what I wanted. My good experience then has grown to the point where it's probably the second most common tool in my toolbag, but it's still an "I need to do xyz, and I don't care how it looks I just want to see if this will work". So I still stick with the defaults. I only think about using the latest greatest tools if I think the code I'm writing has long term potential.

From my personal opinion just porting a project from 2->3 we will get there. Python 3.6 offers to much to let is pass.

Ecosystem has moved far enough its just a matter of time when big frameworks etc. deprecate python 2.7 and this mostly happen around the real EOL of it.

I'm actually surprised (pleasantly) to see that adoption is that high. I would have guessed at less than 10%.

I suppose the sample set (projects using a tool like Semaphore) is likely to be biased towards more forwards-looking teams, but still.

Where I work we're just waiting on Ansible. Everything else we write and use works with Python 3, except the tool we use to deploy it all. Frustrating.

Why not use both? I'm shipping python 3 software using Ansible and it works great.

Using version 2.2?

Ansible is just a program - I'm not sure where the confusion lies. You can use Chef (Ruby) to deploy Python apps after all ... the version of Python used by Ansible is totally irrelevant to the interpreter you use for your codebase.

As far as I remember, if your local virtual environment is Python 3 Ansible doesn't work (might not even install).

well, you can install ansible outside local virtualenv. It's a single program like Firefox, so probably you won't need different versions and could be installed as such.

Just like neokya said, you just create a Python 2 virtualenv for deployment and a Python 3 one for development.

Interesting data-point, but what is the trend? We already know 2.7 is a victim of its own success, right?

What would a project that supports both Python2 and Python3 show up as in these charts?

How many of the projects support both?

As languages go, Python 3 is pretty successful. It unfortunately shares a name with the language it is a variant of so people tend to compare the popularity of the two. The popularity of Python 3 should be judged against all similar languages, not just the insanely popular Python 2.

Commercial projects are the laggards. I'd expect them to be the last to move over. More interesting I think is the progress in distributions. This matters because commercial projects use distributions too, and will need to follow them in the end.

Sentry is also still Python 2.7, thanks to Armin and his friends:


looks to me they have maybe changed their stance? See commits. https://github.com/getsentry/sentry/tree/master/src/sentry/m...

Let this (and Perl 6) be a lesson to language designers: do not make backward-incompatible changes, ever. It cripples adoption. Java managed this much better by being able to mark APIs as "deprecated".

I don't take this lesson. Java also managed by adding new features sorely desired. PHP 4 to 5 was great. PHP 6 is a story much like Python 3, but the PHP community had the good sense to abort it instead of forcing everyone along and PHP 7 will probably not take long to move to. Perl 6 can run Perl 5 code, not sure of your point there, I think the issue with Perl 6 is just how big it is. The only feature that intrigues me anymore there is making grammar-parsing as easy as regexes.

Python 3 had no carrots, as others mention it only seems like now it's got enough going for it to be worthwhile. That's the lesson even Microsoft forgets and relearns continuously: if you're going to push out a major upgrade, you better have nice carrots attached. The stick only approach doesn't work when there is competition, and Python has plenty of competition, lots of it just with itself.

> Python 3 had no carrots,... The stick only approach doesn't work...

I don't use Python for any serious development these days, but this is my impression as well. Python 3's promise was "we'll break all of your code in exchange for a few minor things some people think are improvements." When users balked, the devs decided to force the change by brow-beating and shaming users, with predictable results.

Perl 6 had lots of carrots, but was so different that the only plausible portability story was to run existing code in an embedded Perl 5 interpreter with some kind of interoperability layer. It's a new language, still looking for its niche. Ruby devs were so used to breaking changes that they considered porting to 1.9 to be more of the usual. Meanwhile, C and C++ happily coexist, despite major divergence by C++ and minor incompatible changes by both languages.

You got a point I agree with but a bad example since Perl 6 is a new language. (That the name don't reflect that is still a heated discussion.)

(And Perl 5 code can declare which of the newer features it will support. Excellent backwards compatibility.)

Then again, continuing to add backward-compatible changes to Perl 5 would have made it an even bigger pile of mud than it already is. Its refusal to make backward-incompatible changes is both the reason why Perl 5 was so ubiquitous and why it's so very ugly. So I can certainly understand the desire to start afresh with 6, in spite of the costs in terms of adoption.

It does seem like take-up has been low so far, but it's probably still too early to call whether it will turn out to have been the right choice.

I'm less familiar with Python, so can't really comment on there, but it does seem that Python 2.7 had fewer egregious faults than Perl 5.

There are quite a few backwards incompatible extensions for Perl 5. But you must declare what new features to use.

Then there is Devel::Declare etc with extensible syntax... :-)


I think the main change (accurate handling of strings potentially encoded with multiple bytes) hit a lot of languages from that time in bad ways. I think it's something that more or less had to be done since the alternatives are worse, but we haven't seen a similar issue pop up in a long time. I don't think the lesson applies in many other places.

Certainly open to my mind being changed, though!

I actually don't know why it's been such a big pain with Python 3.

Ruby 1.9 also fixed the accurate handling of strings potentially encoded with multiple bytes.

Fast forward to today, we didn't have such a turmoil like with Python and now, not even Ruby 1.9 is supported by the official devteam anymore. And there is nobody holding out on 1.8 anymore.

Compare that to Python 2 which will still be supported until 2020!

One big difference between Ruby and Python (and one of the big reasons that I think python3 failed to take off) is that the Ruby community has been much better at making sure that their major third party libraries are in sync with the core language. I don't remember the exact timeline, but basically when Ruby 1.9 came out you could more or less instantly start using it with RoR so there was no reason to wait with experimenting with it.

When python 3.0 was released both django and numpy where still years away from getting official python 3 support into their releases so no one bothered to invest too much time into it, and this caused a major loss of momentum that is only today starting to slowly return.

To some extent, I think it's reasonable. PHP had the same issue (old libraries assuming single byte strings) and the way they handled it is pretty awful.

Essentially all string manipulation functions also have an equivalent function with the same name prefixed with "mb_", which works for multibyte strings.

The practical effect is that people use the default string functions and then things break when multibyte strings start showing up! It is backward compatible, but it's certainly ugly and that's quite a tradeoff.

Python2 is stable. This is actually a major advantage in some applications.

Is python's upgrade difficulties from 2.7 to 3.x due to being interpreted rather than compiled? I feel like the types of problems seen in a non compatible upgrade are exactly what I consider easy problems in Java. Look for the compilation issues, fix them, done.

In Java I am largely talking about library upgrade issues because Java made the decision to remain language backwards compatible. Something that has it's costs but also some real benefits. See Linux and it's promises to never break user space.

I think this shows just how good Python 2 is ;)

How do they define "commercial" ? How do they define "new" ?

My office has moved new projects to Go. We were using twisted web server which still isn't p3 compatible. It also doesn't help that Python is basically single-threaded.

Stop spreading misinformation. Python is fine for multithreading despite the GIL. It's simply that there's no free lunch -- no one strategy for parallelism is best for all programs.

Flamebait title, and doesn't match the article. 30% adoption, while not exactly a big success, is nothing to sniff at.

Title should be changed to "Python Versions Used in Commercial Projects, 2016 Edition". The data pertains only to Semaphore's commercial CI customers.

The only thing I haven't liked when comparing 2 to 3, is how 3 is even slower.

CPython is so slow, they really should start taking performance more seriously.

Taking a guess here: Maybe all the people who are concerned about performance have moved to PyPy?

That or using NumPy for any arithmetic bottlenecks.

But the new dict and a few other optimizations in 3.6 will make attribute lookups pretty close to C++ speed if not the same.

They don't support 3 :(

What, PyPy? Their homepage states 3.3.5 compatibility: http://pypy.org/

And they're skipping 3.4 going straight to 3.5

Oh, I am sure they didn't last time I checked. Thank you.

I'm glad that python 2 is unpopular with cool kids. Otherwise they might be inserting broken feature every minor revision.

Of course it is. Python is full of language snobs who insist their language is beautiful and the quality is great. They insist there's no reason to change, except for some reason, to go.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact