
The Linux 2.5, Ruby 1.9 and Python 3 release management anti-pattern - BuuQu9hu
http://www.lucas-nussbaum.net/blog/?p=916
======
fpgaminer
Arguably one of the most successful companies in the world is Microsoft. The
reasons for their success are many, but at least one is the focus on backwards
compatibility. Hook users, and never give them a reason to leave.

As developers we know it is painful to maintain backwards compatibility, but
it's incredibly valuable. It builds user confidence in your products, and most
importantly it builds user trust that upgrading is safe and thus encourages
adoption of newer versions of your software/libraries. The only mistake
Microsoft made was not doing rolling releases. Major updates are slow and
scary, so even though upgrading to a new version of Windows didn't mean loss
of compatibility, it was never "easy" and always jarring.

Backwards compatibility, automated testing, rolling releases. That's the magic
bullet to keeping everyone happy and on the latest updates. If your users have
to think about version numbers, you're doing it wrong.

* Yes, there are instances of Microsoft breaking compatibility, more-so post Windows 7. The point is that they tend to focus on it, and they get it right 97% of the time which is a technological feat.

 __See the Rust programming language for a great example of this recipe.

~~~
ebbv
This isn't really true. If Microsoft of 2016 has a focus on backwards
compatibility it is only because of their myriad mistakes in the past. I know
people have this false impression because of fluff pieces that have been
written about Microsoft having a focus on backward compatibility but anyone
who has lived through the last 30 years of computing history knows this isn't
really true. They have broken backward compatibility numerous times.

Many, many DOS programs would not run under Windows 3.1, necessitating users
to boot into or exit out into DOS to run those programs, and of course Windows
programs could not run under DOS.

Windows 95 generally did better at running DOS programs but it was far from
flawless. Many programs required all kinds of workarounds and even patches.

Windows NT was originally a complete departure from DOS and Windows 3.1, 95,
etc. only later on did Microsoft realize that they needed a compatibility
layer for NT to ever go mainstream (along with a rebranding.)

Windows Vista broke tons of stuff too. Including everybody's printer drivers.
When Vista was released you could buy a Vista computer along with a printer
which had no drivers for Vista and which HP would tell you there's no Vista
drivers ever coming.

Windows 8 may not have broken execution of Windows 7 apps but the entire UI
was a complete shift from what users were accustomed to. Whether you liked it
or not, that kind of major change is a big mistake when so many of your
customers are tech novices who are not going to have an easy or enjoyable time
learning how to use their computer from scratch.

This is just scratching the surface. My point is let's tell the truth here,
Microsoft is not some hero of backward compatibility who has never made a
mistake and always kept their users at heart.

~~~
jzwinck
While it would be hard to call any kiloperson organization a hero, it is
remarkable how people on the Windows team keep old software working even if it
does not deserve to.

The most accessible documentation of this is Raymond Chen's blog. For example
here's a piece written in 2007 about Windows 95 and DOS:
[https://blogs.msdn.microsoft.com/oldnewthing/20071224-00/?p=...](https://blogs.msdn.microsoft.com/oldnewthing/20071224-00/?p=24063)

While the herculean task of billion-device compatibility has not always gone
off without a hitch, the idea that the past was all gaffes and trial and error
at Microsoft does not ring true if you read the above blog from one of its
more prolific public writers.

~~~
ebbv
Maintaining hardware compatibility with any PC hardware ever created is indeed
nigh-impossible, so you'll note I didn't even mention that. I only talked
about maintaining compatibility with applications written for their own
operating system. Something that Microsoft _does_ have pretty much universal
control over.

The only mention I made about devices was regarding the printer driver issue
with Vista, which had nothing to do with the devices and everything to do with
Microsoft totally redoing their printer driver implementation on the OS level.
It necessitated all drivers be redone and some manufacturers (like HP) took
the not very admirable path of telling customers they were up a creek.

------
captaincrowbar
The "Third Way" is to avoid both problems (diverging stable and unstable forks
vs. a single line with frequent minor breakage) by making heroic efforts to
maintain backwards compatibility even in the face of major changes. Someone
already mentioned Windows as an example of this. I'd point to C++ as another
example: the C++ committee's longstanding commitment to maintaining backward
compatibility at all costs has always been one of C++'s greatest strengths,
and is certainly one of the main reasons why it's still in widespread use
despite its age and many flaws.

~~~
Yokohiii
Well no one wants to hear it, but PHP devs do a good job keeping BC breaks
minimal.

~~~
wvenable
PHP4 to PHP5 was almost as big of a deal as Python2 to Python3. In fact, while
I have PHP5 code that can easily ported to PHP7 -- I still have PHP4 code that
will be forever on PHP4.

Because of this, PHP developers strive to avoid this kind of large breakage.
Instead it's a smooth stream of deprecation and removals/changes and really
large changes (like native unicode strings) have been non-starters.

~~~
gkop
I might be misguided, but the last time I used PHP was over ten years ago,
using PHP3, and we had to name the files my_file.php3 in order to use PHP3.
That's the opposite of minimal breakage. On the other hand, my simple PHP3
scripts continue to work to this day, php3 extension and all!

~~~
wvenable
That's just how your server was/is configured.

------
Sanddancer
I think as a counterexample, one could look to projects like FreeBSD, which
have had a long-running development branch and branches for release for a very
long time. Breaking changes happen in -CURRENT, but within a stable branch --
10.x, 11.x, etc, you even get things like ABI compatibility, so a module
written for kernel 10.1 will work for kernel 10.2.

The example projects show more a lack of development discipline than a problem
with the big break model. A version number change from 1.8.7 to 1.9.0 doesn't
suggest big changes that one should be cautious of, causing problems. Linux
Kernel 2.5 had the problem of a lack of developers willing to write more
inclusive tests and a lack of a pre-determined roadmap. Python had the problem
of a lack of a transition version where deprecation warnings were on by
default. Successful breaks can be made when they are declared in advance so
people can have reasonable feedback. The projects in question show
communication problems, not problems with the idea.

~~~
IanCal
> Python had the problem of a lack of a transition version where deprecation
> warnings were on by default.

I'm really not sure this is necessary, though perhaps I'm wrong. I expect that
fewer people would have upgraded to 2.6 or 2.7 if they were full of warning
printouts.

> Successful breaks can be made when they are declared in advance so people
> can have reasonable feedback

Python has an enormous amount of advance warning. Python3 was released at the
same time as 2.6, which came with a warnings mode for breaking changes in
python 3, then 2.7 allowed people to backport many of the newer features.
There is an automated tool to do many of the conversions and you've been able
to test your changes for the last seven years against a fully supported
alternative, and 2.7 will be supported until 2020. That's a total of about 11
years from the launch of a working upgrade, it was being discussed a lot for
at least a couple of years before the 3.0 launch and Guido says he first
brought up the idea of py3k in 2000. I'm also sure you can write code that
works in both 2 and 3, so there's even a fairly clear migration path.

If developers aren't going to change things given such a runway, would adding
default deprecation warnings really make such a difference?

I'm sure I recall at the time that the transition was intended to take 10
years, and so is the problem simply that people are using a different measure
of success than python did themselves?

~~~
tormeh
>If developers aren't going to change things given such a runway, would adding
default deprecation warnings really make such a difference?

I think many developers don't pay attention to these things. If it compiles -
they've turned off the warnings - then it's fine. What's needed is the use of
a "\--use-deprecated-features" flag in order for the code to run/compile
successfully. People pay attention when they need to change the build script
in order to keep doing it wrong. Sure, they'll keep doing it wrong, but now
they'll know it's serious.

~~~
IanCal
But who is that targeting? Python programmers that haven't heard of python 3.X
in the last 7 years?

------
filereaper
I hazard that there's no easy answer to this.

If you 'fork' and start off on a new development trail, its hard to retain
compatibility.

If you keep with the legacy branch, you can end up with a frankenstein
monstrosity.

Most of the examples used are of very highly coupled pieces of monolithic code
ie. kernels, language runtimes, and probably also extends to databases. Its
hard to take these apart without losing performance in some aspect or
another...

I suspect most 'user-land' developers have moved to the microservices/SOA type
development methodology where API's dominate so this is less of a concern.

------
zzzcpan
I think it's a lot more subtle than that. Users are always ready to jump ship,
even if you don't break compatibility.

Merely introducing new functionality that competes with the old ways and
shifting dev focus to it forces users to put an effort into the new ways,
which irritates them, so some might not do that and might start looking around
instead, depending on how deeply invested they are and how fed up they are.
Others have not tried even the old ways yet and are ok with the focus on the
new ways. And only few are those benefiting from the new ways. Simple strategy
just can't work for everyone. There has to be a separate strategy for separate
kinds of users and things that introduce competition and fragmentation must be
carefully weighted.

Given all that, breaking compatibility early and often as a proposed solution
seems like even worse strategy as it not just demands effort, but demands it
immediately and irritates users even more, which means more users are probably
going to get fed up a lot sooner.

------
mjevans
With respect to Python 3, I think it /is/ in fact a similar but different
language.

Primarily my focus is not on small annoying easily covered things (a print()
function is better than a print keyword), but on more systemic changes like
it's decision to make a unicode str the default type of string for the
language.

I would like to see a new version of Python (maybe 4?) that centers around
basic, commonly atomic, types and arrays of those types. This would prevent
the issue of Python 3 either mangling input strings or choking as it tries to
digest legacy input that isn't in the anticipated encoding (at the cost of not
making the mess larger). It would not-modify data that is being passed through
the language UNLESS requested as an operation or as an output filter (implicit
operation). Traditional UNIX like 8 bit clean handling is why UTF-8 is a super
set of ASCII (code-points above 0x7f undefined/locally specific); UTF-8 was
able to define an in-situ character sync method and was still compatible with
existing files, and programs that did not enforce a specific encoding on the
data stream.

~~~
est
Py3k's problem is lack of incentives. If Py3k is 30% faster in execution speed
ppl will migrate in no time. But currently everything is do-able in py2.7 and
maybe even faster.

A successful example is PHP7.

~~~
mjevans
I suspect a reason Python 3 might 'be slower' would be precisely how expensive
everything being a /validated/ (is it UTF-16 internal? Some other crazy
Windows Native format IIRC) string, including internal object elements like
names and dictionary (map) keys.

~~~
guitarbill
UTF-8/UTF-16 are encodings, used to encode strings for transport/exchange.
Internally, strings are stored in UCS-2 or UCS-4 representation.

AFAIK, this hasn't changed between Python2 and Python3, after all, Python2
also supports unicode. It has more to do with what a "string literal" in
source code means. I don't want to go into it too much, but in Py3, `"a"` is a
unicode string literal equivalent to `u"a"` in Py2. Py3 `b"a"` is roughly the
same as Py2 `"a"`. But note the ambiguity in Py2 - could be a sequence of
bytes, or an ASCII encoded string. This is, of course, an oversimplification
btw. But not too far of if you think how e.g. `from __future__ import
unicode_literals` works in Py2.

Is parsing UTF-8 literals in a file slower? Maybe negligibly. Does it affect
runtime? I have no idea. Probably not, you can still do string comparisons
byte for byte once you've converted to UCS-2/4\. It might use more memory
though.

~~~
gsnedders
> UTF-8/UTF-16 are encodings, used to encode strings for transport/exchange.
> Internally, strings are stored in UCS-2 or UCS-4 representation.

That makes no sense; UCS-2 and UCS-4 are encodings too.

~~~
guitarbill
It isn't completely false. It's a simplification (as UCS-* is often used to
denote internal encodings), because oh god, Unicode. This does a great job at
explaining some intricacies: [http://lucumr.pocoo.org/2014/1/9/ucs-vs-
utf8/](http://lucumr.pocoo.org/2014/1/9/ucs-vs-utf8/)

Also, in Python 3, strings can be UCS-1 (latin1?), UCS-2 (UTF-16?), UCS-4
(UTF-32?) or other:
[https://www.python.org/dev/peps/pep-0393/](https://www.python.org/dev/peps/pep-0393/)

It's complicated. Really complicated. Sorry.

~~~
gsnedders
I'd lean against referring to them as any Unicode-based terminology, as all of
that is ultimately designed to represent all scalar values (e.g., even UTF-16
code units compose in such a way as to give some way to represent scalar
values outside of the BMP, though they obviously cannot directly). The other
notable thing about Python 3.3+'s representation is their ability to represent
surrogates, which UTFs cannot.

Realistically, simply referring to it as "flexibly sized codepoint sequences"
is probably about as good as we can get, IMO, because that's what they
fundamentally are. (And there's very little terminology for a sequence of
codepoints!)

------
maxlybbert
The article seems to believe that Debian packaging practices can be used to
illustrate how the users responded to a project's actions. I find that funny,
but it makes it hard to accept the article's conclusions.

(I'm not saying anything about Debian, really; I would have the same complaint
if the article pointed to Fedora, Slackware, Ubuntu, etc. as proof about
whether a change confused users).

~~~
lnussbaum
(I'm the author)

Well my main free software affiliation is with Debian -- that's why I used it
to illustrate.

Still, I think it's relevant. One of the difficult choices that distributions
have to make, is to decide which branch of projects should be shipped to users
(of if multiple branches need to be shipped, which one should be the default).

Of course Debian is often on the conservative side of things, and also made
those decisions based on its own ability to introduce patches to fix
breakages.

It would be more interesting to have data about which python version is
supported by how many modules, over time (and which ruby version is supported
by how many ruby gems, over time). But I don't think that this data is
available?

------
upofadown
Python 3 was to some extent a reaction to the constant breakage that people
were complaining about at the time. It was decreed that backwards incompatible
stuff would go into 3 and 2 would remain relatively static. Eventually it was
announced that 2 would remain entirely static.

That created an odd situation where the version that people were expected to
transition from became a much better place to do new development in that it
was a stable target. Heck, in a sense it is _still_ better to target Python
2.7 for new development as the people running the reference implementation
(cpython) have promised to stop changing things as a matter of policy. Your
2.7 based app could run basically forever with no changes.

So you are dammed if you do and dammed if you don't. There is no definite
answer here.

------
hyperpape
Are there examples of widely used, mature projects that have gone the opposite
way?

~~~
tedunangst
OpenBSD, llvm, and go all have six month cadences.

~~~
hyperpape
And do the same sorts of breaking changes get made, just distributed in
smaller chunks through those six month releases?

~~~
mixedCase
I don't know about OpenBSD and LLVM, but Go has made good on the Go 1
compatibility promise, with only a few minor changes being disputed as wether
they break it or not.

Go 2.0 will probably be a different story, assuming it ever comes out, but I
have a hard time believing the devs will release anything that won't be
fixable using the "go fix" tool.

------
epx
From this point of view, Swift language is doing right. It is irritating to
have to fix a project every few months because Swift language is changing
under our feet, but it is still better than having multiple active flavors of
Swift. (I still don't love Swift, though.)

~~~
EuAndreh
Well, Apple can do whatever they want, with any language they want, better or
worse, and they can still force developers to change to the latest thing.

If they say that you can develop for iOS 11 only with Swift 4, not Swift 3< or
Objective-C, everybody will have to switch to Swift 4, or else they won't be
able to develop to iOS 11.

Even though I also think that Swift is better than Objective-C, and getting
better in each version, is not really comparable with Ruby/Python/etc.

------
echelon
Ugh, looking at the comments it seems that the Rust "nightly vs stable"
article a few days ago [1] has sparked a lot of confusion. Either that, or
people are trying to spread FUD about Rust.

The original author ought to write a retraction.

[1]
[https://news.ycombinator.com/item?id=13251729](https://news.ycombinator.com/item?id=13251729)

------
Annatar
_In retrospect, and looking at what those projects have been doing in recent
years, it is probably a better idea to break early, break often, and fix a
constant stream of breakages, on a regular basis, even if that means
temporarily exposing breakage to users, and spending more time seeking
strategies to limit the damage caused by introducing breakage._

It amazes me how the author managed to conclude completely the opposite from
what he described. Make the users experience the breakage. On purpose. To me
as an engineer, that's utterly unacceptable.

Empathy is still a core engineering value

[https://www.listbox.com/member/archive/182179/2013/05/sort/t...](https://www.listbox.com/member/archive/182179/2013/05/sort/time_rev/page/1/entry/1:75/20130508172342:8CF96552-B825-11E2-98EE-8CBB5940B2DC/)

the solution isn't to expose breakage to users in order to make the
developers' lives easier; we are developers precisely because we are to tackle
the difficult problems, so that users could get work done.

As engineers, we convert the science and the theory into working products and
tackle difficult problems so that useful work could get done. If we are
disrupting the users so that our work and lives would be easier, we don't
respect our users. We should never make our work easier for us at the expense
of our users.

The solution is to design for backwards compatibility from the onset; version
interfaces if you can't predict for the future, but always remain backward
compatible. This way, new consumers of the interfaces benefit from the new
features transparently, and existing consumers continue to function without
disruption. Plan ahead; don't just code: planning is 50% (or more) of the
work.

~~~
lnussbaum
(I'm the author)

I did not mean that developers should break things on purpose. Of course,
seeking strategies to limit the damage caused by introducing breaking changes,
by providing some amount of backward compatibility where possible, is always a
good idea.

But I think that what those examples show is that strategies where disruptive
changes are introduced on a regular basis (what I call "break early, break
often"), requiring some fixes from time to time, work better than strategies
where a different branch is used to break the world, and then require a huge
transition for users.

Maybe what I failed to articulate in the article is that most compilers,
interpreters, libraries introduce minor breaking changes on a regular basis,
with most major or semi-major releases. When that amount stays manageable,
it's usually not a big problem for users (= developers that use those
compilers, interpreters, libraries), that just deal with that.

Also note that real end users are usually protected from this kind of breakage
because they use software coming from distributions (that do the work of
trying to make sure that everything they ship can work together), or package
managers that allow for strict versioned dependencies.

~~~
Annatar
_Maybe what I failed to articulate in the article is that most compilers,
interpreters, libraries introduce minor breaking changes on a regular basis,_

You appear to assume, or even imply, that this is acceptable because it is the
norm today.

There are those of us, just like Keith Wesolowski whose reply I cited
previously, who vehemently disagree with that implication and that assumption:
we represent the engineering ethos of sitting down, thinking things through
("what is it that we're trying to solve?"), and _planning_ ("how can we
implement this so that it can be made backward compatible, so we do not break
things for users?")

And that's my problem with your essay: that you accept that, because there is
so much haphazard hacking without planning going on, it is okay to sometimes
(in reality often) break things. I personally do not, can not, accept that
things will break because I had been unwilling or even lazy to think things
through and plan in advance, or because I had assumed that everyone has the
same amount of spare time as I do or did. It's an engineering ethos.

That is not to claim that things will never break, but I can assure you that
exceptional effort will be taken on my part to remain backward (and forward,
wherever possible) compatible. That's my engineering promise, to all my users.

The core idea of the essay is based on the premise that haphazard
implementations are the reality we should simply accept without second
thought, and the proposed solution is reactive, not proactive. Tolerating
organic growth instead of planning is the root cause.

 _Also note that real end users are usually protected from this kind of
breakage because they use software coming from distributions (that do the work
of trying to make sure that everything they ship can work together), or
package managers that allow for strict versioned dependencies._

I don't discriminate: users are users to me. Their time is extremely valuable,
and I appreciate that all of them use the computer because they want to get
something done. They all have certain rights, and deserve to be treated with
respect.

Sadly, the computer industry lost a great spirit when Keith Wesoloski decided
he had had enough and retired[1], but the engineering ethos is no less valid
today than it was when he wrote these words, and lives on in illumos and
SmartOS:

 _In other words, every entry in the test matrix is either identical to the
behaviour seen today or superior. The state of the system has advanced; its
capabilities have improved in newly-built software, nothing works less well
than before, and customers have the opportunity to apply new capabilities to
old binaries if they have the kind of supernatual awareness of those binaries
that the GNU /Linux model assumes of everyone. Most importantly, the approach
is _safe _: we do not attempt to change the system 's interfaces as presented
to existing software. Those contracts are honoured unless specifically
requested by the customer, with (one hopes) full knowledge of the
consequences._

[1]
[http://dtrace.org/blogs/wesolows/2014/12/29/fin/](http://dtrace.org/blogs/wesolows/2014/12/29/fin/)

------
tehwalrus
This article talks about problem porting patches back to Linux 2.4, and then
makes the comparison with python 2.

Conceptually, the "breaking backwards compatibility" argument works,
especially for the dependent ecosystem.

But, it is worth noting that python 2 and 3 build out of the same branch of
the same tree. Indeed, you can build the 2.5, 2.6, 2.7, 3.X grammars all off
the same code. Sure, there will be some python 3 specific stuff that won't be
included when you build 2.X, but any patches are _by definition_ already
backported.

------
je42
merge early. push automated tests to the limit. use feature flags to disable
unstable features.

------
stefantalpalaru
The author is mistaken in adding Linux to this list. There was no "big,
disruptive change" in the Linux userspace-facing API (stable syscalls,
basically). The Linux developers are quite serious about not breaking that and
what Linus said back in 2005 still stands -
[http://yarchive.net/comp/linux/gcc_vs_kernel_stability.html](http://yarchive.net/comp/linux/gcc_vs_kernel_stability.html)
:

> We care about user-space interfaces to an insane degree. We go to extreme
> lengths to maintain even badly designed or unintentional interfaces.
> Breaking user programs simply isn't acceptable. We're _not_ like the gcc
> developers. We know that people use old binaries for years and years, and
> that making a new release doesn't mean that you can just throw that out. You
> can trust us.

