
How we rolled out one of the largest Python 3 migrations - ddeville
https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one-of-the-largest-python-3-migrations-ever/
======
mr_toad
> However, rather than use the native toolchains directly, such as Xcode for
> macOS, we delegated the creation of platform-compliant binaries to py2exe
> for Windows, py2app for macOS, and bbfreeze for Linux.

I wish the authors of more Python tools would deploy standalone applications.
I do not like having to maintain various sets of Python installers/package
managers (because every Python tool seems to use a different installer).
Especially on cloud servers that often lack a whole set of dependencies that
Python developers just seem to take for granted.

I’m not a Python developer. I don’t have the time or the inclination to
repackage various tools and untangle dependencies.

Given the choice between trying to figure out how to get multiple Python tools
to behave together, or using another tool, I’ll almost always choose an
alternative.

~~~
cjhanks
This is possible and actually pretty trivial.

You install all 3rd party dependencies into some directory. The command line
entry point is then a simple BASH script which sets the PYTHONPATH to the
appropriate installation location and then does the appropriate exec call.

You then have a functionally portable python installation.

~~~
lugg
Pretty trivial?

Really?

I think as soon as you start writing all caps trivial has gone right out the
window.

Trivial would be application dependencies managed by the system package
manager.

I'm at the point where I won't touch a python app or library that can't be
installed via Pacman. It's just not worth my time.

And... python is still the only thing on arch that gives me the shits
everytime it upgrades.

~~~
throwawaymath
This exchange is a good demonstration that the word "trivial" has lost its
meaning in the same way "literal" has. Much like I usually hear someone use
term "literally" for figurative emphasis, these days I mostly hear the word
"trivial" used to describe something which is clearly nontrivial.

Math textbooks have been doing this for decades, but it's leaked into common
parlance with online discussion.

~~~
CogitoCogito
I think the best translation of "trivial" when used amongst mathematicians is
that it is something you should be able to figure out with your current
knowledge without too much difficulty (though it might require an hour of
thought). Said in another way, you don't need to learn/develop new tools or
techniques for something that is trivial.

Of course this is not when most people think when they hear the word so really
it's a term of art that should probably be avoided when talking to non-
mathematicians.

~~~
Ntrails
> I think the best translation of "trivial" when used amongst mathematicians
> is that it is something you should be able to figure out with your current
> knowledge without too much difficulty

As i read through my old uni maths notes there are often wild leaps from a to
e along with a little scrawl saying "trivially" or "obviously". They may have
been true once, but god dammit 21 year old me was a knobber

------
thijsvandien
Interesting write-up, though it leaves me terrified how a relatively small,
more or less single purpose application like the Dropbox client has over 1
million lines of code in Python alone.

~~~
panic
The size of a codebase is less about its function and more about the number of
people adding code to it.

~~~
thijsvandien
What would you consider the main cause(s) of that?

~~~
panic
The job of a programmer is to write code, so that’s what they do. I doubt
there are very many people at Dropbox whose job it is to remove code!

~~~
maltalex
That’s an extremely narrow definition of a programmer’s job.

I think that in terms of lines if code, my contribution to many projects was
net negative.

------
makecheck
It’s funny, after Mojave was released recently I thought it might finally have
a python3 _installed_ , even if it’s not the default. Nope, still Python 2.7.

This is good for me since I deployed a Python-dependent app under the
assumption that the system Python would be stable and reliable. It allows
relatively complex things to be achieved with a tiny download package.

I’ve been prepared to adopt Python 3 for awhile but it just isn’t necessary
when using system defaults.

~~~
Longhanks
It'll be very interesting to see what happens with the next macOS when Python
2 will be EOL (which is roughly 3 months after its release). The upgrade to
Python 3 is long overdue.

~~~
CJefferson
Python being EOL sounds scary but actually won't matter to Apple. They already
apply custom patches, they can carry on running python 2.7 forever, with minor
bug fixes where really required.

~~~
guitarbill
> they can carry on running python 2.7 forever

would they do this? does anything macOS internal depend on python 2.7?

~~~
varenc
the `xattr` terminal command relies on the system installed Python, though
this seems to be the only example.

If you look at the source in `/usr/bin/xattr` it does some work to deal with
different versions of Python. All the work ultimately gets handled by the
xattr module preinstalled with the system Python. This module has Apple's
copyright in it and is different than the `xattr` module on pypi.

Wonder how this Python one-off in macOS came to be.

------
madrox
As a python engineer still living in 2.7, it's great to see major codebases
making the move. I know I have a similar experience coming in my future, and I
appreciate hearing what seems like more or less a success story come from it.

------
wiz21c
FTA :

>>> On the surface, the application would more closely resemble what the
platform expects, while behind various libraries, teams would have more
flexibility to use their choice of programming language or tooling.

I'm always fascinated by how the implementation of the core principles of an
application is dictated by factors alien to it, such as OS, company
organisation, etc. Therefore, the job of coding is often a small part compared
to the amounts of trivialities, project management decisions, customer's
ideas, corporate policies, etc. Although my soul is a coder's one, I always
realize how much coding is just a small part of what I call application
development.

------
drej
Here's a nice overview of how Facebook migrated their codebase to Python 3.
While it's different in nature (server side vs. client side), it's rather
interesting.

[https://www.youtube.com/watch?v=H4SS9yVWJYA](https://www.youtube.com/watch?v=H4SS9yVWJYA)

------
vemv
I hope this kills (or helps killing) the 100+ thread count I have always seen
in macOS. It surpasses any other thread count from far more
important/sophisticated processes.

I'd say that's a waste (if not abuse) of the system's resources and scheduling
system.

~~~
lathiat
I wouldn't be that worried about 100 threads, when you have 10s of thousands
you can think about worrying.

------
sandGorgon
Is there any sample code that illustrates this kind of python embedding versus
one of the freezer scripts ?

We used to build desktop software using pyqt and freeze it. I wonder how that
entire toolchain looks like in this new way.

~~~
jnwatson
See
[https://docs.python.org/3.7/extending/embedding.html](https://docs.python.org/3.7/extending/embedding.html).

The idea behind embedding is you might have a Python shell in a larger app.
But you can also use it to tightly control the execution of the interpreter.

------
yieFiseh
But their Debian packages still depend on Python 2.

------
chrisseaton
What happened to their new implementation of Python they were building? I know
it was cancelled, but not really why.

~~~
phonon
Probably because it was 2.7 only, and only 10% faster on Dropbox's code.

[https://blog.pyston.org/2017/01/31/pyston-0-6-1-released-
and...](https://blog.pyston.org/2017/01/31/pyston-0-6-1-released-and-future-
plans/)

~~~
chrisseaton
I'm surprised 10% wasn't enough - 10% bottom-line improvement in programming
language implementation is normally massive. Twitter is singing from the roof-
tops about 10% improvement in Java performance from the new Graal JIT
compiler.

~~~
phonon
PyPy is already way better than that, and has a Python 3.5 implementation.

~~~
pmart123
I'm by no means an expert on the subject, but isn't there an advantage to
having a JIT compiler use LLVM?

On another note, I believe one thing that has been problematic for pypy
adoption is that it does not automatically work with C extensions or Cython,
and generally if someone already had performance issues with CPython, they
would have written some C/Cython extensions?

~~~
orf
PyPy is far more than just 'python with a JIT'.

It's a tool that allows you to write an entire interpreter in RPython (a
subset of Python) and then have it build a native binary with a _free_ jit
compiler included, with the specifics of your language encoded within. The
reference implementation for this project is a Python interpreter.

It's seriously, seriously cool.

------
j7ake
Anyone curious to estimate the cost for Dropbox to migrate to Python 3?

~~~
xchaotic
and also calculating the cost savings if any. On a technical level the whole
exercise seems to lead to a slightly more elegant, consistent code base, but
it's still a long way from earning or saving any actual money.

------
dwheeler
If you can't get it directly (I can't), there is an archive available at
[https://web.archive.org/web/20180925210033/https://blogs.dro...](https://web.archive.org/web/20180925210033/https://blogs.dropbox.com/tech/2018/09/how-
we-rolled-out-one-of-the-largest-python-3-migrations-ever/)

------
d__k
A guide to migrating to Python 3 (with pleasure :)

[https://github.com/arogozhnikov/python3_with_pleasure](https://github.com/arogozhnikov/python3_with_pleasure)

------
Born_Again
Great write-up and those two graphs are interesting. It's cool to learn how
different companies treat their beta users. I wish this article touched upon
more of the technically difficulties with switching from Python 2 to 3 too.

A̵l̵s̵o̵ ̵a̵ ̵s̵m̵a̵l̵l̵ ̵h̵e̵a̵d̵s̵-̵u̵p̵:̵ ̵a̵t̵ ̵t̵h̵e̵ ̵e̵n̵d̵ ̵o̵f̵
̵t̵h̵e̵ ̵f̵i̵r̵s̵t̵ ̵p̵a̵r̵a̵g̵r̵a̵p̵h̵,̵ ̵"̵v̵e̵n̵e̵r̵a̵b̵l̵e̵"̵
̵s̵h̵o̵u̵l̵d̵ ̵b̵e̵ ̵"̵v̵u̵l̵n̵e̵r̵a̵b̵l̵e̵"̵ ̵(̵u̵n̵l̵e̵s̵s̵ ̵y̵o̵u̵
̵m̵e̵a̵n̵ ̵w̵e̵ ̵s̵h̵o̵u̵l̵d̵ ̵l̵o̵o̵k̵ ̵b̵a̵c̵k̵ ̵a̵t̵ ̵t̵h̵e̵ ̵s̵a̵c̵r̵e̵d̵
̵p̵y̵w̵i̵n̵3̵2̵ ̵l̵i̵b̵r̵a̵r̵y̵ ̵w̵i̵t̵h̵ ̵h̵o̵n̵o̵r̵)̵.̵

~~~
pzone
Venerable is correct here.

~~~
Born_Again
You're right, I haven't seen it used outside of a religious context before.

------
peter_retief
I hope the people who told me that python wasn't good for large rollouts read
this

------
amanzi
They forgot step 1 - hire the creator of Python... But seriously, that's an
interesting post.

~~~
amelius
Guido van Rossem recently stepped down from leading the Python project.

[https://news.ycombinator.com/item?id=17515492](https://news.ycombinator.com/item?id=17515492)

~~~
vgy7ujm
Phew

------
rawoke083600
I would have thought that Go would be perfect for this sort of system-level-
like programming ?

~~~
crdoconnor
Go code is about ~30% more verbose than equivalent python.

~~~
melling
How much faster is Go? Memory usage?

It has to be less demanding on your laptop battery, for example.

~~~
oofabz
Performance depends very much on what you are doing. Native Python code is
much slower than Go, but Python code execution is not the bottleneck in many
Python programs. NumPy may be faster than Go and disk IO is the same speed in
each.

