
Python 2, Python 3, Debian and Porting - neokya
https://lists.debian.org/debian-devel-announce/2015/04/msg00005.html
======
ak217
Shameless plug of a modern Python 2/3 porting tool that I wrote:
[https://github.com/kislyuk/eight](https://github.com/kislyuk/eight), which
builds upon the excellent work in
[https://pypi.python.org/pypi/future](https://pypi.python.org/pypi/future).

~~~
aidos
That looks like a really good solution - and an ideal approach as we get to
keep writing in Python 3. Are there any large caveats?

~~~
ak217
There is still a delta between how unicode behaves on 2 and how str behaves on
3, for example. In general, you'll be fine as long as you have a test suite
that runs in CI for both 2 and 3 - to catch incompatibilities early.

------
rspeer
When it comes to finding which packages need to be ported:

I've been disappointed by the "Python 3 Wall of Superpowers" and the fact that
they've basically never updated their package list. (They update the Python 3
readiness, but you'd never find out if a popular package created in the last 4
years was incompatible, or if packages on the list have become unpopular.)

Today I found py3readiness.org, which is much more relevant and up-to-date. It
has a better call to action, too.

[http://py3readiness.org/](http://py3readiness.org/)

~~~
neokya
OP and maintainer of the above site here.

Thanks for noticing py3readiness.org

Yes, it's up-to-date and takes advantage of
[https://github.com/brettcannon/caniusepython3](https://github.com/brettcannon/caniusepython3)

Also, you can help to keep it updated
[https://github.com/chhantyal/py3readiness](https://github.com/chhantyal/py3readiness)

~~~
jordanb
Both lists should replace python-ldap with ldap3. The former is abandon-ware
and will never (probably) be upgraded. The latter already works great with
python 3 (and is a better library all-around).

~~~
rspeer
There are more packages like that, in fact:

* Python-MySQL is abandoned, but it's been superseded by python-mysql-connector.

* oauth2 is kind of superseded by oauthlib.

* pathtools can probably be replaced with Python 3's pathlib.

* ipaddr can be replaced with iptools, which has the additional advantage that you can contribute to it without approval from Google's lawyers.

~~~
neokya
We know about that. Many of the packages are overriden if there are drop-in
replacements like PIL -> Pillow.

In above list, these are not drop-in replacements and it will require lots of
changes in your application code if you want to use them. One idea is to show
alternative packages
[https://github.com/chhantyal/py3readiness/issues/9](https://github.com/chhantyal/py3readiness/issues/9)

------
diminoten
Am I correct to assume this would be a _fantastic_ place to start contributing
to open source projects by helping convert these Debian tools over to Python
3?

~~~
k2enemy
Yes, as noted in the message:

    
    
       If you're interested in this effort, please email me. This is a really good new
       contributor task, so if anyone's asked you how they could get involved with
       Debian, you should send them to us!

~~~
paultag
Yep! My email is paultag@debian.org - sign up for the porting list @
[https://lists.alioth.debian.org/mailman/listinfo/py3porters-...](https://lists.alioth.debian.org/mailman/listinfo/py3porters-
devel)

Yes, it's hilarious that version of Mailmain needs to be ported to Python 3.

~~~
kerkeslager
Paul, I already emailed you but I'm responding here because I have a feeling
your response will be useful to other people interested in helping: could you
give some direction for people like me who know Python 2 and/or 3 and want to
help, but don't know the first thing about the structure of Debian packages or
how they are developed?

For example: I'm looking at ./main/f/flask-wtf/flask-wtf_0.10.2-1.json and I'm
not even sure what I'm looking at. The contents of that file are:

    
    
        {"has_python3_module": false, "canidate": true, "trove_python3": true, "has_python2_module": true}
    

What's all that mean?

On my machine `apt-cache search flask-wtf` finds a package called `python-
flaskext-wtf`. Digging deeper with `apt-cache showpkg` I find the homepage
[http://packages.python.org/Flask-WTF/](http://packages.python.org/Flask-
WTF/), which in turn gets me to the code [https://github.com/lepture/flask-
wtf](https://github.com/lepture/flask-wtf). I haven't updated the code to
Python 3, but let's assume for a second that I have: now what? How do I tie
this all together? Do I just send you a patch for this code? Is this even the
right code?

If you point me in the right direction I'll be happy to write up a tutorial to
help other folks contribute.

And as a side note: I really like Hy.

~~~
Buetol
To know what every key means, you can look at the script:
[https://github.com/paultag/debian-meta-
archive/blob/master/s...](https://github.com/paultag/debian-meta-
archive/blob/master/scripts/python3/main.py)

 _has_python2_module_ : is a python2 package

 _has_python3_module_ : is a python3 package

 _trove_python3_ : Is is declared as python3 ready in Pypi ?

 _canidate_ : Python 3 module already there and Python 3 compatible on Pypi

I published it as a csv to better understand the situation:
[https://github.com/mdamien/mdamien.github.io/blob/master/sta...](https://github.com/mdamien/mdamien.github.io/blob/master/status.csv)

~~~
mjcohen
Is "canidate" a misspelling of "candidate"?

------
grumbel
> We can all soon look forward to the day where we no longer have to play
> Unicode whack-a-mole

I don't expect Unicode trouble to disappear with Python3, quite the opposite,
it's trivially to insert ticking Unicode time bombs into Python3 code,
especially in things surrounding shell scripting. Something as trivial as
this:

    
    
        print(os.listdir())
    

Will explode when it runs into filenames that aren't encodable by Unicode
(i.e. perfectly legal filenames in Linux). Linux allows raw binary in a lot of
places (filenames, arguments, environment variables) where Python expects
Unicode.

~~~
eliben
Are you implying that by using Python 3 incorrectly you can run into unicode
problems? I agree with that.

However, there are ways to do this safely. At the boundaries of your code you
have to deal with the unicode/bytes question, in any language out there. The
good thing about Python 3 is that once that boundary is crossed, unicode and
bytes are nicely separated.

~~~
grumbel
> Are you implying that by using Python 3 incorrectly you can run into unicode
> problems?

The problem is that `print(some_string)` is now incorrect, even so it looks
perfectly harmless. So far I haven't seen any "best practice" guide on how to
fix that. `sys.stdout.write(os.fsdecode(some_string)` works, but is pretty
ugly. Handling all filenames as bytes instead of strings works as well, but
again, that gets ugly quick. Cleanest way seems to be:

    
    
        PYTHONIOENCODING=utf-8:surrogateescape python3 somescript.py
    

But should I depend on users setting `PYTHONIOENCODING`? Should I do that in
my scripts? It feels all very rough and causes a lot of issues that you really
shouldn't have to deal with in the first place.

> The good thing about Python 3 is that once that boundary is crossed, unicode
> and bytes are nicely separated.

Not exactly "nicely separated", the unencodable bytes get squished as
surrogates into the Unicode strings waiting to cause trouble later on.

~~~
the_mitsuhiko
JFTR. In Python 3.4 and later I think, stdout has a surrogateescape error
handler by default. Not that it helps much, because this will fall apart if
you write to other things. The story with this apparently is now so confusing
that I am not sure what the correct way to do it on Python 3 is.

From my experience most people don't care anyways. In some of my libraries the
Python 3 version just refuses to work unless the encoding is utf-8 to avoid
accidental failures down the line which nobody can debug.

~~~
eeZi
Unfortunately, that has not made it into Python 3.4.

------
deckiedan
As someone who uses a lot of Python scripting for sys-admin automation stuff,
is there anyone here on HN who could ask apple to start distributing a python3
binary with the OS by default?

Otherwise we're going to end up with a similar situation as BASH and other
utils in a few years time, where OSX comes with Python2.7.x, and everything
else in the world is Python3...

(Although, by 2020, apple will probably have deprecated OSX in favour of iOS,
and have javascript as the only blessed scripting language...)

~~~
bkeroack

      $ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
      $ brew install python3 pypy3
    

(to be less pithy, it's super easy to write code that will run on both 2.7 and
3.x -- avoid print-as-statement and use format strings instead of '%' for
interpolation and that will get you 95% of the way there--probably 99-100% of
the way for simple automation scripts)

~~~
burntsushi
As the maintainer of a Python project that is 2.7 and 3.3+ compatible on the
same code base, there is _nothing_ super easy about it. It's horrific
actually. Enough so that I'll never do it again. It goes far beyond just print
functions. Libraries like `six` help, but the subtle differences in how
Unicode handling has changed makes it a real challenge.

~~~
eeZi
A nice read: [http://lucumr.pocoo.org/2013/5/21/porting-to-
python-3-redux/](http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/)

~~~
burntsushi
I've read all the relevant material. That doesn't mean it isn't painful. The
same author of that blog post wrote another blog post complaining about how
Unicode is handled in Python 3.

------
chimeracoder
> Python 2 is scheduled to be EOL'd upstream officially and for good in 2020.
> We're in 2015 now (wow, that went quickly), and keeping our release cadence
> up (3 years a pop) puts Stretch up in 2018, and Buster in 2021.

Am I missing something? Did Debian decide to extend their release cycles?

Jessie is due to be released next week, which is 2 years since Wheezy (May
2013). In fact, the last several years have all been 2-year cycles, not 3. The
only three-year cycle was Woody → Sarge (2002-2005), and AFAIK, that delay was
one of the reasons that Ubuntu had room to enter the market. Since this put
pressure on Debian to make their development more rapid, I'd be surprised if
they decided to move in the opposite direction.

~~~
paultag
Yes, everyone's making fun of me for not saying 2 years rather then 3.

Also, our releases aren't timed, they're based on bug count, so the timing of
the releases wasn't even the main point (and doesn't really matter) :)

I also typoed Python 3 as Python 2 in one of the lines, so, uh, just switch
those two numbers. I'm pretty sure it works like that.

Either way, getting Buster off Python 2 means the transition will be more bug
free, since we'll have a bit of slack time.

------
FreakyT
Nice, I'm glad to see that Python 3 adoption is finally picking up steam!
Python 2 is nice, but it has really started to overstay its welcome, I think.

------
rkuska
We (Fedora) are step ahead.

Currently we are waiting for a last few packages to be ported (anaconda is
already running on python3 and ready for Fedora 23 release). Ironically the
hardest part isn't porting (we've sent a huge amount of patches to upstream
projects) but getting upstreams to accept the python3 as supported interpreter
("what is this python3 thing?" yes, this is a real quote).

I am happy to see a new distros joinning a party, gl!

------
afarrell
If you want to switch back and forth between python2 and python3, one option
is to install miniconda (like Anaconda, but with just the package manager) as
your Python distribution and create a python2 environment (conda create
--name=myproject-py2 python=2.7 && source activate myproject-py2) that you can
then pip install the python2-only libraries into. Then, when your libraries
support Python3, create a new Python3 environment and install them into that
(conda env export > environment.yml && conda create --name=myproject-py3
python=3.4 && conda env update --name=myproject-py3 --file=environment.yml &&
source activate myproject-py3)

See [http://conda.io](http://conda.io)

Disclosure: conda is open-source, but I work for company that sponsors most of
the development.

~~~
Foxboron
I believe virtualenv (with/without virtualenvwrapper) is most widely used. Not
sure if there are any advantages/disadvantages compared to conda tho.

[https://virtualenv.pypa.io/en/latest/](https://virtualenv.pypa.io/en/latest/)
[https://virtualenvwrapper.readthedocs.org/en/latest/](https://virtualenvwrapper.readthedocs.org/en/latest/)

~~~
a3n
One disadvantage of conda/anaconda is that you can't, or are discouraged from
using virtualenv with conda/anaconda.

I use Anaconda on Win7, like it a lot, but I haven't figured out how to use,
say, the python 2 that comes with anaconda alongside the python 2 that comes
from python.org. I've looked around, haven't really understood what I should
be looking at for that. But whatever it is, it isn't virtualenv if you use
Anaconda.

~~~
bkcooper
_One disadvantage of conda /anaconda is that you can't, or are discouraged
from using virtualenv with conda/anaconda._

True, but conda has its own virtual environments that work quite nicely. You
can make a new one with conda create and switch between them with
activate/deactivate. You even get the same sort of interaction with pip (for
the packages you want that aren't in conda or binstar), and I've been able to
use e.g. Emacs packages that expect to work with virtualenv and have (mostly)
had success using conda's envs instead.

~~~
a3n
Yep, and your answer is the sort of answer I see all around, including on
their forum. I haven't figured out the doc.

How would you use conda's environments to switch between Andconda's python 2
and python.org's python 2?

~~~
ngoldbaum
Why would you want to do that? If you're using anaconda or miniconda, you
should be using Continuum Analytics' builds of python, not python.org python.

~~~
a3n
Because I want to use Anaconda and its infrastructure, but skeptical people in
my circle would want to see that whatever was built will run on the version of
python that they would download (which would not be Anaconda).

"Why would you want to ..." answers are generally pretty frustrating. It's a
big universe with a very large possible set of individual circumstances.

Can this be done?

~~~
ngoldbaum
If you're building conda packages then the only possible consumer is other
users of miniconda or anaconda.

I'm not sure "whatever was built" refers to in this case. Some kind of app? Of
course you can use anaconda for normal library or application development,
since there's functionally no difference between Continuum Analytics' python
builds and the python.org python build.

~~~
a3n
I want to run the following, using either of Anaconda or python.org python 2,
on the same machine depending on the whim of the day:

    
    
      C:> python -c "print 'hello'"
      hello

------
syllogism
My biggest peeve writing Python 3 is pretty trivial...

I cannot get used to typing print()

Every time I go to debug a line, I get a syntax error, and have to go back and
change it. And I type a lot of debug lines!

I need to find a vim extension to handle this, but so far it just makes me
frustrated whenever I try to type Python 3.

~~~
rquirk
Have you tried using the logging module? Might be overkill for really small
scripts as it requires a few lines to set up, but even the tiniest tool ends
up requiring print-debugging at some point. With logging it works the same in
2/3, you can leave all the debugging statements in, and if a library also uses
it then you get additional debug output from that too. Add a "debug" command
line flag to pass when something goes wrong down the road (to change the
default log level), and you don't have to add/remove statements for debugging.

For a vim extension - I use snipMate
[http://www.vim.org/scripts/script.php?script_id=2540](http://www.vim.org/scripts/script.php?script_id=2540)
It's a bit on the dead side but works great as-is. ifmain followed by the
expander key (shift-tab in my case) expands to the if __name__ ... idiom. You
could add your own "p" snippet that expanded to print() in python code.

------
bayesianhorse
Interesting side node: Blender (an open source graphics suite) switched to
Python 3 in like 2008 (so far as I can determine).

The reasoning probably was "Hey, the rest of the Python community would
probably leave us in the dust, if we don't..."

------
krylon
Oh man.

I've been meaning to check out Python 3 for a couple of years now, but so far
I did not have a compelling reason. Maybe it's time to start anyway.

~~~
Estragon
Yeah, reading through this thread, I haven't seen a single reason why I should
care about Python 3.

~~~
task_queue
Because Python 2 will be phased out in 5 years.

~~~
BuckRogers
By who? The core developer team? Or by PyPy and Pyston? The people that matter
(the users) aren't phasing out 2.x in 5 years. That much is clear.

------
NeoAbdn
Someone asked why not swap to Elixir or Rust, but why not actually swap to the
sysadmins dream of Ruby?

~~~
task_queue
I don't know a sysadmin on this planet that dreams of using Ruby as systems
scripting language. That sounds like hell.

Again, its the trade off between a relatively simple port and adopting a new
language with different standards, technical difficulties and culture.

------
kristjanvalur
Or just fork python 2 and keep on trucking there in happy land.

------
stefantalpalaru
> Booo, Python 2!

It would be better for everybody if we'd treat Python 2 and Python 3 like
separate languages. Then we can reason about the sanity of spending
development time to port existing and working code from one language to
another.

~~~
Walkman
The problem is they are really really similar and you can write code that runs
on Python2 and Python 3 at the same time.

------
winter_blue
This seems like a good time and opportunity for Debian to try their hand at
one of the recent languages that have popped up like Rust, Elixir, etc.

~~~
task_queue
You're being downvoted because writing Debian's system infrastructure in a
language whose spec isn't finalized or is not well known is a Bad Idea when
other options exist.

Python 2 -> Python 3 is a trivial port compared to adopting a new language and
the technical/cultural baggage that come with it.

------
upofadown
I am getting really tired of this "if Python3 is to live then Python2 must
die" stuff...

Perhaps Debian should avoid the war entirely and switch to another scripting
language.

~~~
Karunamon
Python2 is a ticking time bomb - it's officialy EOL'ed in 2020, all new
development is happening on 3.x

It's like the jump between Ruby 1.8 and 2.0 - there is no reason to be doing
anything in 1.8 in 2015.

~~~
Yuioup
I predict that Python 2 will be forked by a drop-in replacement by 2020.

~~~
lmm
I'll bet you $50 that no non-maintenance fork of python2 will be shipped by
default in any of the top 5 linux distributions before 2020.

~~~
BuckRogers
Only because the core dev team is making it their priority to move distros
over to 3. It won't make any difference, the strength of Python is in the
library support, not the distro support. 2.x support by libraries will last a
very long time, nothing worth its salt will be dropping it. By the time they
do, Python3 and wherever it is by that point will be unrecognizable to the
original release of Python 3.0.

