
Mercurial’s journey to and reflections on Python 3 - ngoldbaum
https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-reflections-on-python-3/
======
ploxiln
I've been involved in multiple non-trivial libraries and frameworks that
supported both python2 and python3 for many years with the same codebase ...
and it really wasn't anything like this. The python3 "adaptation" effort for
mercurial was just bungled by multiple terrible decisions.

First was the idea that normal feature contributors should not see any b"" or
any sign of python3 support for the first couple years of the effort. Huge
mistake. You need some b"".

But you don't need all b"" everywhere. That was the second huge mistake. Don't
just convert every natural string in the whole codebase to b"". The natural
string type is the right type in many places, both for python2 (bytes-like)
and python3 (unicode-like). The helpers for converting kwargs keys to/from
bytes is a sign that you are way off track. This guy got really hung up on the
fact that the python2 natural string type is bytes-like, and tryied to force
explicit bytes everywhere (dict keys, http headers, etc) and was really
tilting at windmills for most of these past 5 years.

Yes, you pretty much had to wait for python-3.4 to be released and for
python-2.6 to be mostly retired in favor of python-2.7. Then, starting in
early 2014, it was pretty straightforward to make a clean codebase compatible
with python-2.7 and python-3.4+, and I saw it done for Tornado, paramiko, and
a few other smaller projects.

~~~
pdonis
_> The natural string type is the right type in many places_

For many programs, yes. Not for a revision control system that needs to be
sure it's working with the exact binary data that's stored in the repository.
Repository data is bytes, not Unicode.

I think this article is an excellent illustration of the Python developers'
failure to properly recognize this use case in the 2 to 3 transition.

~~~
jharsman
I was an early adopter of Mercurial and the teams insistence that file names
were byte strings was the cause of lots of bugs when it came to Unicode
support.

For example, when I converted our existing Subversion repository to Mercurial
I had to rename a couple of files that had non ASCII characters in their names
because Mercurial couldn't handle it. At least on Windows file names would
either be broken in Explorer or in the command line.

In fact I just checked and it is STILL broken in Mercurial 4.8.2 which I
happened to have installed on my work laptop with Windows. Any file with non
ASCII characters in the name is shown as garbled in the command line interface
on Windows.

I remember some mailing list post way back when where mpm said that it was
very important that hg was 8-bit clean since a Makefile might contain some
random string of bytes that indicated a file and for that Makefile to work the
file in question had to have the exact same string of bytes for a name. Of
course, if file names are just strings of bytes instead of text, you can't
display them, or send them over the internet to a machine with another file
name encoding or do hardly anything useful with them. So basic functionality
still seems to be broken to support unix systems with non-ascii filenames that
aren't in UTF-8.

~~~
pdonis
_> the teams insistence that file names were byte strings was the cause of
lots of bugs when it came to Unicode support_

File names are a different problem because Windows and Unix treat them
differently: Unix treats them as bytes and Windows treats them as Unicode. So
there is _no_ single data model that will work for any language.

~~~
hsivonen
The Rust standard library has a solution for this that actually works: On
Unix-like systems file paths are sequences of bytes and most of the time the
bytes are UTF-8. On Windows, they are WTF-8, so the API users sees a sequence
of bytes and most of the time they match UTF-8.

This means that there's more overhead on Windows, but it's much better to
normalize what the application programmer sees across POSIX and NT while still
roundtripping all paths for both than to make the code unit size difference
the application programmer's problem like the C++ file system API does.

~~~
pdonis
_> On Windows, they are WTF-8_

Seems like an apt acronym for Windows... :-)

On a more serious note, Python seems to have done something fairly similar
with the pathlib standard library module.

------
fireattack
I understand author's reasoning in the context of a transition, but as a "non-
Latin" language user, defaulting str to unicode literals is the best change in
Python 3. Coming from C#, I never get used to Python 2's approach. It's a pain
in the ass working with non-Latin characters in Py2 starting from simply
output in console, especially on Windows.

>assuming the world is Unicode is flat out wrong

True, but Py2's approach makes lots of developers assume the world is Latin-1.
I see _way_ too many examples of things broken on a Chinese locale
environment, including Python's official IDLE ([1]).

[1] [https://bugs.python.org/issue15809](https://bugs.python.org/issue15809)
(Summary of this bug: in 2.x IDLE, an explicit unicode literal used to still
be encoded using system's ANSI encoding instead of, well, unicode.)

~~~
int_19h
The most amusing quote in the entire article is this (emphasis mine):

> This ground rule meant that a mass insertion of b'' prefixes everywhere was
> not desirable, as _that would require developers to think about whether a
> type was a bytes or str_ , a distinction they didn't have to worry about on
> Python 2 because we practically never used the Unicode-based string type in
> Mercurial.

Requiring developers to think which one it should be is, of course, the whole
point of the changes in Python 3 - and it's what produces better apps that are
more aware of i18n issues in general and Unicode in particular.

And the complaint doesn't even make sense if taken at face value - if all
strings in Mercurial are byte strings, then what is there to think about? just
use b'' throughout, no need to worry about anything else. Of course, the devil
is in the details, which is reflected by the word "practically" in that
sentence - this kinda implies that there _are_ places where Unicode strings
are used. At which point you _do_ want the developers to think about bytes vs
Unicode.

So the real complaint is that Python switched the _defaults_ in a way that
made bytes-centric code more complicated - because it has to be explicit now,
instead of the Python 2 world, where bytes was the default, and Unicode had to
be requested explicitly. Which, of course, is the right change for the vast
majority of code out there, that operates on higher level of abstraction,
where "all strings are Unicode by default" is a perfectly reasonable
assumption to force.

~~~
sfink
> And the complaint doesn't even make sense if taken at face value - if all
> strings in Mercurial are byte strings, then what is there to think about?
> just use b'' throughout, no need to worry about anything else.

The article directly answers that question. Many, many things in the standard
library now only accept unicode strings, not byte strings. So a wholesale
change to b'' everywhere breaks lots of stuff.

> So the real complaint is that Python switched the defaults in a way that
> made bytes-centric code more complicated - because it has to be explicit
> now, instead of the Python 2 world, where bytes was the default, and Unicode
> had to be requested explicitly.

Once again, the article directly states that the default is not the problem.
The lack of escape hatches is. Paths are not unicode strings, and pretending
they are does not work. Using bytes when you need bytes works only until you
need to call a library function that only accepts strings.

~~~
WorldMaker
Paths _are_ Unicode strings on Windows. Yes, POSIX adds a lot more spice to
the mix, but if the intent is a cross-platform tool, then Unicode is a
reasonable lowest-common-denominator assumption for filenames in 2020.

~~~
Conan_Kudo
Paths are Unicode strings everywhere _but_ Unix/Linux. And I would even argue
that this is a broken aspect of POSIX today. We _should_ make Unicode the
baseline for paths in POSIX-compliant systems, but there's probably too much
hand-wringing for that to ever happen.

------
jmilloy
> in Mercurial's code base, most of our string types are binary by design: use
> of a Unicode based str for representing data is flat out wrong for our use
> case.

I feel like this is the essence of the article: specific constraints/choices
of Mecurial made their port to Python 3 difficult. Working with early Python 3
certainly did not help. But there seems to have been some stubbornness here
mixed with a lot of retroactive justification.

> One was that the added b characters would cause a lot of lines to grow
> beyond our length limits and we'd have to reformat code.

This is almost ridiculous. You are going to write a JIT partial 2to3 instead
of just increasing your length limits and/or using an autoformatter? (Of
course, it turns out they eventually did do that... after a bit more
stubborness regarding the autoformatter.)

> So I'm not sure six would have saved enough effort to justify the baggage of
> integrating a 3rd party package into Mercurial.

Couldn't this have been a very occasional copy and paste, instead of a
downstream dependency?
[six]([https://six.readthedocs.io/](https://six.readthedocs.io/)) "consists of
only one Python file, so it is painless to copy into a project."

> Initially, Python 3 had a rather cavalier attitude towards backwards and
> forwards compatibility.

Yes, can't disagree. Early adopters who attempted to write 2- and 3-
compatible code suffered the most.

------
intrepidhero
> _Matt knew that it would be years before the Python 3 port was either
> necessary or resulted in a meaningful return on investment (the value
> proposition of Python 3 has always been weak to Mercurial because Python 3
> doesn 't demonstrate a compelling advantage over Python 2 for our use case).
> What Matt was trying to do was minimize the externalized costs that a Python
> 3 port would inflict on the project. He correctly recognized that
> maintaining the existing product and supporting existing users was more
> important than a long-term bet in its infancy._

Having just done transitions on a number of much smaller projects I had the
same thought. Changes to string handling tripped me up and the changes to
relative imports took some thinking. But the biggest frustration was the
nagging question: _Why am I doing this?_

edit: missing word

~~~
libria
> Why am [I] doing this?

Lack of security updates past 2019 forced our hand. Did you find a way around
that?

~~~
hsivonen
There's a project for keeping Python 2 alive:
[https://github.com/naftaliharris/tauthon](https://github.com/naftaliharris/tauthon)

It's particularly uncool that Guido brought up the prospect of lawyers
([https://github.com/naftaliharris/tauthon/issues/47#issuecomm...](https://github.com/naftaliharris/tauthon/issues/47#issuecomment-266470996))
to force it not to be called Python and opposed to letting people who care
about keeping Python 2 alive evolve it as "Python 2". (I know he has the legal
right to insist on the name change. Still uncool.)

~~~
mjw1007
I think it makes a great deal of sense for the Python core team to say "we're
finished with Python 2 and want nothing more to do with it".

But I'm very disappointed that the Python Software Foundation isn't explictly
supporting people who want to keep Python 2 compiling and running on modern
systems. I think that would be well within their remit to "promote, protect,
and advance the Python programming language".

This is particularly so because Python is widely used for scientific purposes,
and being able to reproduce old results is valuable.

Even before Python 3.0 appeared, I came across scientists saying "I prefer to
stick with Fortran because new Python versions break old code too frequently".

~~~
int_19h
PSF does not object to people who keep Python 2 compiling and running, such as
ActiveState ([https://www.activestate.com/company/press/press-
releases/act...](https://www.activestate.com/company/press/press-
releases/activestate-offers-extended-support-for-python-2-beyond-eol/)).

This case is different, because it's a project that uses the Python name, but
actively adds _features_ to the language. This is the classic example of brand
confusion - someone might try to use it, find something to complain about, and
PSF's reputation suffers as the result. They also get support overhead from
the users of the fork (even if all they do is tell them to go away, that is
still triage time that could be spent on other issues).

~~~
mjw1007
"Does not object" is better than nothing, but I think it would be better if
the PSF actively helped to coordinate this work (again, without bothering the
core team). As far as I'm concerned, this is exactly the sort of thing that
the PSF exists for.

------
makecheck
It’s funny, on the Mac one becomes used to constant changes, rewriting damn
near everything just to stand still. Yet I designed my Mac app long ago to
depend on the system “Python 2” (bound to C++), because it seemed that both
the installation itself and the Python language and libraries were very
stable. Looking back, this turned out to be sustainable for a remarkably long
time, as “Python 2” really did evolve only additively and there was almost no
reason to even touch 15-year-old code that was relying on Python 2. For the
Mac platform especially, this reliability is unheard of.

More amazing to me is that in Catalina, the release famous for breaking just
about everything else, “Python 2” is still there and works as it always has!
Of course, Apple did announce that it will be ripped out in the next release.
:)

~~~
pfranz
I think this weird thing happened with Python 2. I believe Python 2.6
(Oct-2008) was the last "feature release" and 2.7 (Jul-2010) was intended as a
bridge. So since 2008, 2.x users have been shielded from most all of the
normal churn of any widely used language that's in active development.

What I don't think people realize is that not only are you expected to move to
3.x, but you'll have to keep up or fall behind with new 3.x releases. During
that same period (since 2008) 3.x has had 9 big releases. Of course that 2.x
stability was done with the assumption you'd move to 3.x and isn't sustainable
for PSF indefinitely.

------
adontz
I have never seen such rejection in Django community, despite real problems,
like with WSGI design, handling I/O and thus working with bytes a lot.

Every huge task, like porting from Python 2 to Python 3 or any other huge task
is either everybody's task or just a small group's one. And since latter seems
more reasonable to not interfere with ongoing development, former is the only
way I have seen such tasks to succeed.

Artificial rules to create comfort for one group at the expense of another
group, like the following

>> This ground rule meant that a mass insertion of b'' prefixes everywhere was
not desirable, as that would require developers to think about whether a type
was a bytes or str, a distinction they didn't have to worry about on Python 2
because we practically never used the Unicode-based string type in Mercurial.

sound pretty much wrong to me.

If there is a pain, it should become everybody's pain, or otherwise people
will simply burn out and hate own work, like the author did. There is no way
porting to Python 3 can be harder than porting to Rust. Rust is statically
typed and not garbage collected. Everyone would have to think if they need
string or array of bytes anyway, but also, who owns them.

Overall, described situation looks like management issue and not a technical
one to me.

Edit: typos.

~~~
roca
> There is no way porting to Python 3 can be harder than porting to Rust. Rust
> is statically typed and not garbage collected. Everyone would have to think
> if they need string or array of bytes anyway, but also, who owns them.

The Rust compiler statically checks those decisions, while in Python issues
with string types will only be caught at run-time, so everywhere your test
suite has missing coverage, porting is likely to introduce regressions. That
is one way in which a Rust port would be easier.

~~~
hinkley
I used to switch unit tests from jasmine 1.3 to mocha because jasmine is kind
of a mess, and jasmine 1.3 tests look too much like they should still work in
jasmine 2.0, except some of the corner cases on equivalence of objects are
wrong. So some of your tests would go red with no code change, but others
would be green and stay green even when the code no longer functions properly.
Like cutting the wires to your smoke detector.

It would take quite a bit of change in a language for a port to be safer than
an upgrade, but it's not completely impossible.

------
war1025
We are on the brink of completing the transition to python3 at my work.

The end result of this is that I just spent a good chunk of last week
reviewing a pull request with 70,000 lines of changes, which was one of the
final in a series of ~10k line pull requests that came in through the fall.

All of this was the heroic effort of one of my coworkers who had the
unenviable task of combing through our entire codebase to determine "This is
unicode. This is bytes. Here is an api boundary where we need to encode /
decode." etc.

It was a nightmare of effort that I'm glad to have behind us.

~~~
d0mine
Something is wrong if there is no third type: the "natural" string (bytes on
Python 2, unicode on Python 3).

~~~
quietbritishjim
Surely any "natural" string would be better represented as unicode in Python
2? What is an example that wouldn't be?

~~~
war1025
I believe what they meant was that for many strings it really shouldn't matter
if they were bytes or unicode. They would perform their function correctly
either way. That's completely true, but you do still have to go through and
find the cases where that doesn't work.

------
michaelhoffman
The biggest problem with the Python 2 to Python 3 transition was not that
breaking changes were made. It’s that breaking changes were made in a way such
that you could not easily have code that worked both on Python 2 and Python 3.

It took years before the advent of six, Python 3 u’’ literals, and modernize.
The author discusses this at length.

~~~
choppaface
Another big problem is there was no significant incentive to adopt Python 3.
That’s why it took so long for large projects to transition. In comparison,
during the last decade, C++ went from dodgy C++11 toy projects to all new code
being written in modern C++. The modern feature set is that good.

~~~
Jasper_
C++ doesn't mandate you switch from std::cout to fmt in order to use lambdas.
If they did that, I think we'd see a lot less modern C++.

~~~
choppaface
That’s a find-and-replace fix that can be addressed reliably. A relatively
smaller problem versus moving off of boost.

The compiler support for C++11 (and especially inconsistencies in Debian
packages, compiled flags, etc) was a very painful issue for several years. But
auto is that useful ...

~~~
Jasper_
Right, moving to std::cout to fmt could be as simple as a find-and-replace
fix. That the C++ committee could have inflicted this minimal pain on their
users, but chose not to do it, shows some amount of concern for backwards
compatibility and old codebases. By comparison, Python 3 changed the entire
text model and dropped the mic, and waited for 8 years to start to pick the
pieces back up.

~~~
choppaface
I guess I don’t understand your argument that “Python 3 changed the entire
text model and dropped the mic.” format strings are optional; the old %
operator still works fine. The change to unicode is dramatic, but personally I
haven’t run into major problems. I’ve had unit tests break because of it, but
that’s why one has unit tests. I’ve also worked on a very large python webapp
that underwent painful internationalization, and in that case we ended up
using unicode strings everywhere anyways.

~~~
richardwhiuk
The % didn't use to work fine. .iteritems() was made for no good reason.

Python 3 could have required all strings began with u" or b", but they didn't
- they did something which encouraged breakage.

------
nemothekid
> _Perhaps my least favorite feature of Python 3 is its insistence that the
> world is Unicode. [..] However, the approach of assuming the world is
> Unicode is flat out wrong and has significant implications for systems level
> applications (like version control tools)._

Isn't this more a problem with Python not easily differentiating between
String and Byte types? Both Go and Rust ("""systems""" level languages) have
decided that "utf-8 ought to be enough for anybody" and that seems to be a
good decision.

~~~
Jasper_
Yes, but that insistence that Bytes and Unicode are two different things that
Shall Not Be Mixed was mostly a Python 3-ism. Python 2 had different types but
you could be sloppy and it would kinda work out.

There was this assumption that Unicode code points were the correct single
unit to talk about Unicode. You iterate over code points, you talk about
string lengths in terms of code points, you slice in terms of code points.
Much like the infamy of 16-bit Unicode, this is an assumption that has kinda
gotten worse over time. Now we can and do want to talk about bytes, code
points, and newer sets like extended grapheme clusters. I think this is
probably the big failing of Python 3's Unicode model. Making a string type
operate on extended grapheme clusters might fix it, but we'd be in for the
same sort of pain, and the flexibility of "everything is bytes, we can iterate
over it differently" of Go and Rust is much nicer in comparison.

The second thing was this assumption that everything remotely looking like
text was Unicode, despite this maybe not being true. HTTP has parts that look
like plain text, like "GET" and "POST" and the headers like "Content-Type:
text/html". But the correct way to view this as ASCII bytes, and no other
encoding makes sense; binary data intermixed with "plain text" definitely
happens, and the need to pick and choose between either Unicode or Bytes
caused major damage in the standard library which still persists to this day
-- some parts definitely chose the _wrong_ side. Take a look at the craziness
in the "zipfile" module for one other example. It's probably fixed now, but
back then, I basically had to rewrite it from scratch in one of my other
projects.

They eventually relented and added back a lot of the conveniences to blur the
line between bytes and unicode again, like adding the % formatting operator
for bytes, which I think shows that their insistence on separating the two
didn't really pan out in practice. And yet, migration is still a pain.

~~~
int_19h
> Python 2 had different types but you could be sloppy and it would kinda work
> out.

It would "kinda work out", if your Unicode strings were ASCII in practice, and
only then. Because whenever a Unicode and a non-Unicode string had to be
combined, it used ASCII as the default encoding to converge them.

Which is to say, it only worked out for English input, and even then only
until the point where you hit a foreign name, or something like "naïve". Then
you'd suddenly get an exception - and it happened not at the point where the
offending input was generated, but at the point where two strings happened to
be combined.

This was a horrible state of affairs for basically everybody except the
English speakers, because there was a lot of Python code out there that was
written against and tested solely on inputs that wouldn't break it like that.

Intermixing binary data with text can be represented just fine in a type
system where the two are different. For your HTTP example, the obvious answer
is that the values that are fundamentally binary, like the method name or the
headers, should be bytes, while the parts that have a known encoding should be
str - there's nothing there that requires actually _mixing_ them in a single
value. In those very rare cases where you genuinely do have something like
Unicode followed by binary followed by Unicode in a _single_ value, that is
trivially represented by a (str, bytes, str) tuple.

The problem with the Python stdlib isn't that bytes and Unicode are distinct.
It's that it's overly strict about only accepting Unicode in some places where
bytes should be legal, too. This is orthogonal to them being separate types.

~~~
otabdeveloper4
> Because whenever a Unicode and a non-Unicode string had to be combined, it
> used ASCII as the default encoding to converge them.

They could have just changed the default encoding to utf8. (For those too lazy
to configure their Python properly.)

There, problem solved - and no need for a breaking Python 3.

~~~
int_19h
It would still be a mess any time you have to deal with byte strings that
aren't UTF-8. The problem is with the implicit conversion itself - it
shouldn't happen, because there's no way to properly guess the encoding. But
there was no way to get rid of it without breaking things.

~~~
otabdeveloper4
> But there was no way to get rid of it without breaking things.

Even such a breaking change would be a molehill compared to the mountain of
breaking changes in Python 3.

Point is, they had one job, and they failed.

~~~
int_19h
That change was at the heart of the breaking changes around strings in Python
3. If the conversions remained implicit, most people would probably have never
even noticed that string literals default to Unicode, or that some library
functions now require Unicode strings.

------
rossdavidh
Having worked in python for about a decade, first in python2 and lately in
python3, and having seen projects convert, I find this article baffling. I
found Six to work pretty well, and where it didn't it wasn't hard to change.

I think the core error here was in NOT doing what he calls a "flag day"
conversion. Sometimes it is easier to do something quickly, than to live with
it happening slowly. I've done "flag day" conversions, and they were pretty
painless, if stressful at the time.

~~~
CJefferson
Mercurial still can't have a "flag day" as Macs are still distributed with
Python 2, and not 3. Therefore it would make Mercurial significantly worse for
mac users if it didn't support Python 2.

~~~
krupan
I love mercurial to death, but let's be honest, how many Mac/mercurial users
are there? Very few, I would guess. Now, how many of those users _don 't_
install a version of python not included with the OS? I'd guess we are getting
pretty darn close to zero there.

~~~
CJefferson
I work with 2 such people (academics, prefer mercurial, use R rather than
Python so I can't imagine would have had any reason to install Python 3). Lots
of people use version control without being "developers", so not needing a
more up to date Python (or, at just happy with Python 2).

------
ufov2
The approach of doing the transition slowly over many years maybe was a
mistake here, and another thing making it harder seems to have been little
support from the top of the project.

I ported two projects with ~200000 Python-SLOC (about the same size as
Mercurial according to sloccount) back in the early 3.x days. Doing this via
more or less flag-day conversions within a few months, converting the
codebases first to 2to3-able subset, and as a second step later on dropping
2to3 via common dialect of Python 2/3 with six, was not very painful in the
end.

~~~
sfink
Did you have a large ecosystem of third party extension modules that also had
to obey the flag day?

------
epage
I can't believe their leadership de-prioritized the port until the last minute
when they have an ecosystem on top of them that also has to port. I feel that
was irresponsible.

The project lead said to not push `b""` on people. That was a mistake imo that
led them down a very frustrating rabbit hole (transformers, `pycompat`) that
probably greatly extended their port time. One reason given is to not confuse
devs with those details but they are critical details and ones you can't avoid
with Rust. This inconsistency makes me wonder if the post is mostly
misdirected frustration. A lot of it centers on.

I agree about the early python3 releases making it harder. I don't remember
what the python leadership's intent was but i think I actually agree with what
they did, now. Over my career, I've come to appreciate starting with the ideal
and working backwards. This let's you learn what is needed rather than wasting
time on speculation (planning or dev) or making a more crippled product.

I can understand frustrations with bugs / differences in python versions. I
ran into that a lot just within `2.7.*`

In my mind, the most notable complaint is the stdlib's mixed efforts in
supporting str or bytes. I feel "batteries included" maked this harder. They
had to port a lot. Not everything can get the same level of scrutiny,
especially from domain experts that represent a variety of use cases. They
also can't break compat. If they weren't battries included, the porting
efforts would be more directed, pull in the right people, and you can fix
things later if you get it wrong.

What I find interesting is how different our experiences are that lead to the
same place. My frustrations with python are rooted in build tools and
packaging and have been loving Rust.

EDIT: I'm also surprised at the hostility towards distribution packagers.
Instead of working with them to find mutually valid solutions, the express
frustration at distributions and cripple themselves in not allowing third-
party dependencies.

~~~
Conan_Kudo
> EDIT: I'm also surprised at the hostility towards distribution packagers.
> Instead of working with them to find mutually valid solutions, the express
> frustration at distributions and cripple themselves in not allowing third-
> party dependencies.

These days, it's "cool" to hate your downstreams (y'know, bite the hands that
feed you and all that).

Seriously though, as one of those "distribution packagers" (Fedora, Mageia,
OpenMandriva, and openSUSE!), it sucks that I encounter this more and more
often. I try to be somewhat involved in the projects I package and contribute
where I can, be it code, advice, or anything in between. Ten years ago, people
were generally friendly to me. These days? It's rare to get a thank-you.
Usually I get grumbles and anger for _daring_ to ship it in a distro package.
I've even had a couple of patches rejected that fix real bugs simply because
they were discovered as part of my packaging and testing something because it
doesn't happen on the dev's machine in his virtualenv on his Mac...

------
peatmoss
I think my takeaway lesson is that it’s very hard to introduce large, breaking
changes to a language and not alienate a large proportion of existing users. I
don’t know that there’s a right way.

I look at Perl, which was a juggernaut when I first used Python, and
announcements of Perl 6 certainly didn’t help Perl’s slide. Often cited is the
fact that Perl 6 is a _totally_ different language unrelated by anything but
creator and name. The Perl brand was not enough to carry the bulk of Perl
users from Perl 5 to Perl 6. Perl 6 is now called Raku, which probably better
reflects the magnitude of the change.

On the other hand Python 3 is a small but still significant departure from
Python 2. If they’d called Python 3 something else, we’d probably be griping
about how superficially different from Python 2 it was without bringing
substantially new ideas.

Oddly my feeling is that Racket, in its departure from mainline Scheme,
largely did retain its core audience, but that may have been a feature of its
usage in academia.

Fast forward to last year when a prominent Racket architect announced “Racket
2” which would completely change the syntax of the language. Prominent
community members reacted negatively, due to fears of Perl 6’s fate. But now
they’ve decided to simply call the new research language Rhombus and have
reiterated plans to continue supporting Racket. I went from feeling very
negative to the change to being okay with the direction.

I’m not sure there are lessons to draw, other than noting than version bumping
versus making a new language with a new name can be bad for entirely different
reasons.

~~~
edflsafoiewq
I think that's the wrong takeaway, especially since it kind of absolves Python
of doing anything wrong. There were many examples of ways that this was unduly
hard just because of how poorly the transition was designed for.

> While hindsight is 20/20, many of the issues with Python 3 were obvious at
> the time and could have been mitigated had the language maintainers been
> more accommodating - and dare I say empathetic - to its users.

~~~
mikepurvis
The author's suggestion of permitting a certain set of "from __past__" imports
seems especially astute. This would have made it much more possible much
earlier to have a single large codebase running natively on the Python 3
interpreter, but with modules (especially leaf modules) at varying degrees of
ported-ness.

In contrast, the original porting guidance for module authors was actually to
maintain the Python 2 source as the master copy, and use 2to3 to transform it
for running tests or cutting a Python 3 release. How is a transition ever
supposed to happen if the new hotness is perpetually a second class citizen?

~~~
masklinn
> The author's suggestion of permitting a certain set of "from __past__"
> imports seems especially astute.

Hell that's essentially what ended up happening over time, as past features
got reintroduced (the `u` prefix, bytes.__mod__, `callable` being
reintroduced, `range` being improved, …), as well as the serious
Python3-ification of the Python2 branch that was 2.7.

~~~
mikepurvis
I feel like "from __past__" would have allowed projects to get to the point
where they were "on Python 3" much sooner, though.

The mentality would have been "we're on Python 3, and we have a long tail of
cleanup to do excising from-past out of our codebase" rather than "we and our
users are all still on Python 2 but we thiiiiink our code is mostly using
constructs that are Python 3 safe. We get a green CI checkmark, anyway, but
who really knows."

------
reggieband
In the past I have been a vocal advocate for the way the transition from
Python 2 to Python 3 was handled. However, it should be said I use Python
primarily as scripting glue, e.g. for build scripts and automation tasks. I
have never worked on a "large" Python code base nor did I have to migrate
anything. Almost everything I had written in Python 2 was just naturally
replaced by newer scripts in the due course of time.

I also remember my first forays into Python 3 and the annoyance I had at some
of the decisions. I recall when they relented on the % operator for string
interpolation and I agree it was a poor initial choice to leave it out. I
totally agree with the author that Python 3 could have made some subtle
changes earlier on to help those with massive codebases.

And I still feel it was the right move. Somehow Python is even more relevant
today than it was when this painful process began. While some may say that
popularity is despite missteps I actually believe the general slow and
cautious push forward is one of the primary reasons Python continues to
succeed. There is a balance between completely abandoning old users (e.g. Perl
5 to Perl 6) and keeping every historical wart (e.g. C++). IMO, the Python
community found a middle ground and made it work.

~~~
lizmat
Re: There is a balance between completely abandoning old users (e.g. Perl 5 to
Perl 6)

Please note that Perl 6 has been renamed to Raku
([https://raku.org](https://raku.org) using the #rakulang tag on social
media).

In the original design of Perl 6, a source-level compatibility layer ("use
v5") was envisioned that should allow Perl 5 source code to run inside of Perl
6. So the _plan_ was to actually __not __abandon old users.

In my opinion, this failed for two reasons:

1\. Most of Perl 5 actually depends on XS code, the hastily devised and not
very well thought out interface to C code of Perl 5. Being able to run Perl 5
source code in Perl 6 doesn't bring you much, unless you have a complete stack
free of XS. Although some people tried to achieve that (with many PurePerl
initiatives), this really never materialized.

2\. Then when the Inline::Perl5 module came along, allowing seamless
integration of a Perl 5 interpreter inside Perl 6, using Perl 5 modules inside
of Perl 6 as if they were Perl 6 modules, it basically nailed the coffin in
which the "use v5" initiative found itself already in.

And now they're considered different languages after the rename to Raku,
dividing already limited resources. I guess that's the way of life.

~~~
reggieband
I think my wording "abandoning" was more inflammatory than I would have liked.
And I didn't want to call out or target Perl 6 / Raku. What I meant to convey
was that the language team behind Perl 6 (as it was known before the
rebranding) made a decision that it would be a new language and not an
evolution of the existing language. It was the first example in my mind, one
most people would recognize, that anchored one side of the continuum I was
describing. I assume there are better examples (or worse offenders) but I
don't know of any off hand.

------
mikl
There’s no doubt that the 2 -> 3 transition was rough for the Python
community. I personally stopped using Python as my go-to language in the early
Python 3, since writing Python 2 code felt stupid since it was outdated the
moment you wrote it, and 3 wasn’t really well supported by the community and
tooling yet.

On the other hand, Python adoption has really taken off since Python 3.5-ish.
Python has never been more popular.

So while you may wonder what might have been, had the transition been
smoother, it’s hard to argue that Python 3 is a failure. All’s well that ends
well, I guess?

Although it’s sad that Guido felt the need to step down. It’ll be interesting
to see where Python goes this decade, now the transition is over and there’s a
wealth of possibilities in front of it.

I expect there’ll be a lot of people looking to replace JavaScript with Python
once you can run it in the browser with WASM.

~~~
roca
"All's well that ends well" neglects the costs to the community of bad
decisions. It also encourages people to think that those decisions must not
have been very bad, and not learn from those mistakes.

You can see that at work in the responses here. "And I still feel it was the
right move. Somehow Python is even more relevant today than it was when this
painful process began." I.e. success is thought to justify every decision made
along the way.

I see this fallacy at work in Linux too. "Linux is successful, therefore
haphazard CI and using email to track bugs and patches must be a fine way to
operate".

~~~
mrr54
Haphazard CI and using email to track bugs and patches _is_ a fine way to
operate.

------
zmmmmm
It mostly makes me question the wisdom of implementing such a tool in Python
in the first place - if you want low level access to raw underlying
representations etc, using a super high level scripting language seems like a
"wrong tool for the job" scenario. I am sure on the other hand they got a lot
of productivity benefits from doing that, which is great, but having taken
that tradeoff I don't think it is fair to "sour" on a language when you
clearly applied it out of it's domain and then encountered problems due to
that.

~~~
Tanooki_Mario
You can't really say this after the language has supported the feature as a
design decision for 15 years and then removes it. Part of the popularity of
python is it made things easier for systems programmers. Unless you want us to
write everything in C/C++ again?

------
souprock
Not what you want to hear about a version control system: "Python is a dynamic
language and there are tons of invariants that aren't caught at compile time
and can only be discovered at run time. These invariants cannot all be
detected by tests, no matter how good your test coverage is. This is a
feature/limitation of dynamic languages. Our users will likely be finding a
long tail of miscellaneous bugs on Python 3 for years."

~~~
indygreg2
C/C++ is a language with limited facilities to ensure correctness at compile
time. The languages are riddled with undefined behavior in common features
that programmers with multiple decades of experience still get tripped up by.
NULL access - the so called "billion dollar mistake" \- out of bounds reads
and writes, and use after free create a litany of security issues and create
massive liability for companies who choose to author software in these
languages.

Not what you want to hear about an operating system :p

~~~
otabdeveloper4
"C/C++" is not a language.

Good lord, how much ignorance can hackernews handle??

------
weberc2
> One of the biggest early hurdles in our porting effort was how to overcome
> the string literals type mismatch between Python 2 and 3. In Python 2, a ''
> string literal is a sequence of bytes. In Python 3, a '' string literal is a
> sequence of Unicode code points. These are fundamentally different types.
> And in Mercurial's code base, most of our string types are binary by design:
> use of a Unicode based str for representing data is flat out wrong for our
> use case. We knew that Mercurial would need to eventually switch many string
> literals from '' to b'' to preserve type compatibility. But doing so would
> be problematic.

> In the early days of Mercurial's Python 3 port in 2015, Mercurial's project
> maintainer (Matt Mackall) set a ground rule that the Python 3 port shouldn't
> overly disrupt others: he wanted the Python 3 port to more or less happen in
> the background and not require every developer to be aware of Python 3's
> low-level behavior in order to get work done on the existing Python 2 code
> base. This may seem like a questionable decision (and I probably disagreed
> with him to some extent at the time because I was doing Python 3 porting
> work and the decision constrained this work). But it was the correct
> decision. Matt knew that it would be years before the Python 3 port was
> either necessary or resulted in a meaningful return on investment (the value
> proposition of Python 3 has always been weak to Mercurial because Python 3
> doesn't demonstrate a compelling advantage over Python 2 for our use case).

As a general rule, this seems like good practice, but surely b-strings,
print_function, etc are a trivial upfront cost, and one that would have to be
paid sooner or later anyway?

~~~
kingemer
It sounds like the cost was non trivial for them, partially because they
weren’t allowed to break things for python2, or even disrupt the efforts of
those using it.

The language wasn’t ready for the transition, but it feels like it may have
been even harder on them because of the requirements imposed on their project.

~~~
kingemer
Considering how much opposition there is in moving to python3, has there been
any significant effort in the community keeping python2 alive?

~~~
acdha
Most of what opposition there is comes from people with projects which are
some combination of under-staffed, poorly tested, or with a marginal approach
to scheduling necessary maintenance work. That is not a great place to expect
to find reliable maintenance contributions.

Where you are more likely to find this is from the major Linux vendors: e.g.
Red Hat is committed to support it through 2024 and I would expect that they
won't be alone in offering paid support for remaining users.

~~~
weberc2
I'm actually (pleasantly) surprised that someone like Google (or a league of
someones like Google) who have deep pockets and so much Python 2 that it's
cheaper to maintain Python 2 than it is to port to Python 3. Perhaps they
(correctly) were concerned about the ecosystem moving on toward Python 3,
leaving them behind?

------
j88439h84
A lot of Mercurial's issues would have been resolved much easier if they'd
used the common tools for maintaining polyglot 2/3 code instead of trying to
invent everything themselves.

Futurize and Pasturize in particular provide essentially all of the features
that this post laments missing.

[https://python-future.org/](https://python-future.org/)

~~~
lacker
The author does touch on that.

 _When Mercurial accepts a 3rd party package, downstream packagers like Debian
get all hot and bothered and end up making questionable patches to our source
code._

Some environments just can't use dependencies like this. IMO Python 3 was too
much of a breaking change, and in particular, the ability to transition from 2
to 3 should have been better in Python itself.

~~~
j88439h84
Six, for example, is designed to be a single file -- specifically to ease
copying it directly into the code base. But the idea that Mercurial couldn't
use dependencies because of fear for what Debian might do...I find it so hard
to believe that's the best choice. Vendor if you must, but _do not_ reinvent
the wheel.

~~~
masklinn
Having done so, six really is not necessary for a transition. We went with
forking werkzeug's minimalist compat file because six was waaay overkill, and
that made it easier to progressively rip it out later.

python-future is invaluable for the fixers (which are way better than 2to3's),
the runtime stuff you can easily do without.

------
mbar84
I guess this is as good a time as any to pimp my work in this area:
[https://pypi.org/project/lib3to6/](https://pypi.org/project/lib3to6/)

lib3to6 is a Python compatibility library similar to to BableJS. It translates
(most) valid Python 3.7 syntax to valid Python 2.7 and Python 3 syntax (aka.
universal python). If you would like to develop with a modern python version
and yet still maintain backward compatibility or if you want to bring a legacy
codebase forward step by step (my use case), then please have a look.

------
TeeWEE
We just migrated a big codebase to python 3 in about a year. It was not easy,
but also not super hard with tools like futurize, and mypy.

Gladly we already had mypy hints, this helped us find a lot of mistakes when
(not) using bytes.

Now we're on python 3 we're auto-migrate the type hints to be inlined with
tools, like com2ann
[https://github.com/ilevkivskyi/com2ann](https://github.com/ilevkivskyi/com2ann)
And we're auto rewriting code to be more python 3 like with libcst and custom
codemods...

------
alangpierce
It's interesting that they wanted to add b' prefixes to all strings, and I
wonder if they would have had a better experience by embracing regular strings
instead. At least in Python 3, if your string only contains ASCII, then the
underlying representation will use one byte per character, so ASCII-only
strings are stored just as efficiently as ASCII-only bytes instances.

I think there are two mental models for how to approach the str/bytes split:

1.) A `str` is for unicode use cases, and a `bytes` is better for cases that
don't support unicode.

2.) A `bytes` is an array of numbers between 0 and 255. A `str` should almost
always be used when your value is conceptually a sequence of characters. `str`
doesn't imply that arbitrary unicode is allowed, and it's fine to have a
convention that a particular `str` is ASCII-only, just like other conventions
you might have on variable values.

My impression is that #1 is the Python 2 mental model and is tempting for
Python 3, but that #2 often works better when writing Python 3 code. Under
mental model #2, asking for "%s" formatting is really asking for a replacement
strategy that detects the number 37 followed by the number 155 in an array of
numbers and fills in a sub-array, which seems more strange and likely to get
false positives if you're really working with binary data like the bytes of a
.jpg file.

That said, I'm sure the devil is in the details, and maybe a project like
mercurial has to stay backcompat with bytes data that is neither ASCII nor
valid UTF-8, or some other compelling reason to stick with bytes everywhere.

~~~
CJefferson
The problem is many strings might contain things like commit messages, or
filenames, neither of which has to be valid unicode.

I've had the same problem with a few Python 2 -> 3 conversions -- everything
is fine until you have to operate on text or filenames which aren't valid
utf8/unicode.

~~~
alangpierce
Got it. So I understand, maybe someone saved a filename as the latin-1
encoding of some non-ASCII text, and Mercurial would need to support such
files (but also would have no contextual information that it's latin-1)?

I'm tempted to say "nobody should have filenames like that", but I guess a
project like Mercurial needs to be as compatible as possible. Are there modern
use cases for filenames like that, or is it fair to say it's all legacy data?

~~~
xorcist
It's going to be the case every time you mount a Windows file system, for
example.

A big part of the problem is that a project like Mercurial doesn't have
control over what files people use it on. They have to design for the pessimal
scenario, because when the tool breaks, users complain.

------
gfxgirl
I wish more devs cared about backward compatibility not just in python but in
general. I know a particular, very popular library, 60k stars on github, who's
maintainers break stuff every month. They don't care how many developers time
it wastes they just decide FooBar should really be named FooB and rename it.
No Effs given how many people it disrupts. You'd think people would complain
but cult of personality and/or popularity of library turns people in to fan
boys where they seem to think "If these geniuses are doing it this way then it
must be good". .... sigh

------
rurban
For me this is the most exciting outcome:

> The only Python 3 feature that Mercurial developers seem to almost
> universally get excited about is type annotations. We already have some
> people playing around with pytype using comment-based annotations and pytype
> has already caught a few bugs. We're eager to go all in on type annotations
> and uncover lots of dynamic typing bugs and poorly implemented APIs.

Over in perl land people still spill their hate on types, which caused hard
forks.

------
raymondh
This criticism of the dev team seems naïve, "It should not have taken 11 years
to get to where we are today." Core developers can make tooling available, but
they can't control adoption. That is a user decision. Users switch-over on a
timetable governed by their own individual cost-benefit analysis.

~~~
yjftsjthsd-h
> Core developers can make tooling available, but they can't control adoption

Core developers made the design decisions that made nobody want to adopt it.

> the ecosystem of users and projects are collectively much better-off than if
> the transition had not occurred at all.

The question seems more like, "could the same benefits have been had with less
pain", and a reasonable reading is that the answer is yes. (ex. 4 years of not
being _able_ to work with bytes reasonably even if you _did_ need them)

------
bschwindHN
I wonder how many human lives' worth of work has been wasted from the decision
to use Python and having to deal with 2/3 transitions, and if it was worth it
for the speedup of using an interpreted language.

------
2T1Qka0rEiPr
Not knowing really anything about Mercurial, the `skip-blame` feature seems
interesting, but seems Git doesn't have something similar (has to be
constrained _when_ calling `blame`)

------
loxs
It would probably be less painful and much better (for other reasons) to
migrate to some other language. Some projects did that successfully, or are in
the process of doing so. Most notably reposurgeon:
[https://gitlab.com/esr/reposurgeon](https://gitlab.com/esr/reposurgeon)

~~~
sfink
Greg says as much in the article: in hindsight, porting to Rust would have
worked out better. Which is a pretty bold statement, but very interesting to
hear from someone with intimate experience to back the opinion up.

~~~
cookiecaper
Mercurial's dependence on Python has always held it back, IMO. Self-contained
Rust or Go-style static binaries work much better for "install everywhere"
system utilities. I'd love to see Hg port to a more concise ecosystem and
potentially claw some of the market away from Git.

~~~
mixmastamyk
The author wrote pyoxidizer for that purpose.

------
Ohn0
For such a thorough article, I wish there were mention of Python 4

------
luord
1\. Introduce a new version with the plan of discontinuing the previous
version _11 years_ later (that's almost half of the time that, by then, python
had been a thing), that itself was released only three years after the very
tool you're talking about was released.

2\. Don't even pretend to be interested in trying to do a migration until
seven years later.

3\. Make sure that your migration plan includes a development cycle that's
deliberately hostile to the migration process.

4\. ?

5\. How could the _python maintainers_ do this to _us_.

The description of the migration process was a good read. The fud
afterwards... wasn't.

And there were a few inaccuracies (I'm being charitable, some of them were
straight up lies).

> Python 3.0 was released on December 3, 2008. And it took the better part of
> a decade for the community to embrace it.

False, I've been using python 3, python 3 _exclusively_ , since 2014, for all
my projects.

> Yes, Python is still healthy today and Python 3 is (finally) being adopted
> at scale

False, same as above.

> I am ecstatic the community is finally rallying around Python 3

Again, false. Not only did "the community" rallied around python 3 years ago,
he isn't really happy about it, but I'll get to that later.

> For nearly 4 years, Python 3 took away the consistent syntax for denoting
> bytes/Unicode string literals.

Or, to put it another way, python 3 was compatible with python 2's string
types almost eight years before python 2 reached end of life.

> An ecosystem that falters for that long is generally not healthy

This entire paragraph was a hypothetical. It seems he really wanted to
criticize something that did _not_ happen.

> The only language I've seen properly implement higher-order abstractions on
> top of operating system facilities is Rust

And here's where his true point becomes evident: this is a hype piece for a
language he found that he likes better. He's just attacking something in his
previous language that he thinks is valid just as an attempt to highlight why
the new toy is truly better. In short: He felt like complaining about the
migration would be a good way to proselytize.

Just in case: no, it isn't better, and I say this as someone who currently
isn't using python nor rust. I'm using a language that I'm quickly growing to
hate more than I do either of them at their worst (no, it's not JavaScript).

> if Rust were at its current state 5 years ago, Mercurial would have likely
> ported from Python 2 to Rust instead of Python 3. As crazy as it initially
> sounded, I think I agree with that assessment.

So... The best he can say about rust is that it might be better than python 3
five years ago that, by his own opinion on everything he wrote before this,
was terrible? Well, that's a recommendation __not __to use rust if I ever saw
any.

When a hype piece defeats its own point.

> And speaking as a maintainer, I have mad respect for the people leading such
> a large community.

No, he doesn't; he used several appeals to emotion beforehand to try to paint
them as terrible people.

> It should not have taken 11 years to get to where we are today.

This statement by itself is a truism that doesn't really mean anything, but
the implication is that python 3 is only worthwhile 11 years later and it took
that long for it to be so I'll reply to _that_.

No, it didn't. It didn't even take that long for mercurial, they started the
migration four and half years ago, not eleven.

> am confident it will grow stronger by taking the time to do so

What is it to him? He should just move on to rust and be happy with it (sure,
there are many people unhappy with it, but he wouldn't take the effort to
proselytize if he wasn't).

In conclusion, I just don't understand the need to tear something else down to
prop up a new thing. I'm sure I would have liked a post about things he could
do with rust, but now...

------
ascotan
Python 3 is the new Windows Vista.

~~~
marcosdumay
You mean they just have to fix the issues behind the scenes, then rename the
last version (like to "Python 4"), and it will become the greatest version
ever?

~~~
yjftsjthsd-h
Yes, actually. Now that we've gotten through the pain of the first years of
Python 3, if we could have a clean start and call Python 3.8 Python4, it would
probably be well-received.

------
sprash
The transition from Python 2 to 3 was one worst things that could happened to
the whole community. The costs of the transition never justified the benefits.
The new features were negligible at best or a regression at worst and in some
cases performance got even worse. One could even assume flat out sabotage.

Let's just hope there will never be a Python 4 and the developers now finally
start focusing on the greatest flaw of Python: performance.

~~~
Tanooki_Mario
What's crazy is if they had removed the GIL python 3 adoption would have been
huge. All python 3 offered were marginal benefits for adopting functionality
that broke huge code bases.

------
raverbashing
Ok, yeah, maybe mercurial's case was special, but still, this seems like they
made it harder on themselves needlessly.

> One was that the added b characters would cause a lot of lines to grow
> beyond our length limits and we'd have to reformat code

ORLY?! Well, guess what: hard line size limits are stupid. Now you know why.

That's why "foolish consistency is the hobgoblin of little minds" is one of
the 1st phrases of PEP-8.

But I'm tired of people saying "oooh let's cut all lines to be under
80-characters" like it's some kind of Biblical Mandate. No, it isn't. And the
80 chars limit is BS. Probably the part I hate the most about PEP-8 (and
especially how people interpret the PEP-8)

> is its insistence that the world is Unicode

Oh please. Yes, the world is Unicode. Get over it. Maybe not bytes on
disk/network. But apart from that? Yes. If libraries take bytes or unicodes I
can agree it's a thorny issue, but let's move on because a happy day is a day
where I don't get an UnicodeDecodeError because Python2, to add insult to the
injury thinks the world is not only not Unicode, but it's all ASCII.

Windows made the right call a long time ago when it decided to make all
strings Unicode. Ok, maybe UTF-8 would be better than 16, but it still does
the job.

But I have to agree with them that any version < Py3.4 or 3.5 was really not
worth it.

~~~
yjftsjthsd-h
> the world is Unicode. Get over it. Maybe not bytes on disk/network. But
> apart from that? Yes

So, just ignoring the 2 things that a version control system exists to work
with directly?

~~~
raverbashing
You're ignoring the parts where they have several hardcoded byte/unicode
strings.

Otherwise just convert to and from when saving and sending it to network

