There is fundamental a difference between Unicode and UTF-8/16/32. Unicode says nothing at all about how strings should be stored in memory. It is just a set of symbols and code-points. On the other hand, UTF-8/16/32 are implementations of Unicode. See http://en.wikipedia.org/wiki/Unicode#Mapping_and_encodings for others.
Statements like "first 256 code points in Unicode map to Latin-1" make little sense. They are true if you s/Unicode/UTF-8/g. However, they are not true for other encodings such as ones that use 2 or 4 bytes per character. There 'abcd' is not 4 bytes, but 8 or 16.
> Statements like "first 256 code points in Unicode map to Latin-1" make little sense.
That's not true. Latin-1 is both a character set and an encoding, so "the first 256 Unicode code points map to the corresponding Latin-1 characters" is a reasonable statement.
You can also say "the first 128 Unicode code points, when encoded in UTF-8, are equal to the corresponding Latin-1 encoding".
My point is that in Python land it is silly to call u"" strings, "Unicode" strings. Unicode strings are strings in UTF-8/16/32 and a bunch of lesser-used encodings. For that matter "" could be used as a Unicode string as long as it's only ASCII. What the docs should be talking about is ASCII vs UTF-16, not ASCII/Latin-1 vs Unicode. This starts making a difference when questions like "How much memory is consumed by this string?" or "What characters can I not store in Python?" are asked. In this light, Python 3 makes a big improvement: it has immutable byte arrays and it has encoded strings.
"Python 3 is unarguably a better language than Python 2."
I would say that's unarguably marginally better. What provides Python 3 that I miss in Python 2? For me nothing at all. Ok, dealing with Unicode it's easier. So what? Optional type annotations, that's something I would miss.
Unarguably marginally better is still unarguably better.
Set and dict comprehensions are pretty rad. Ordered dicts are useful. The new unittest module is great.
Of course, all of these have been back-ported to 2.7, but even if they hadn't, Python 2 is still a very capable language. Don't have dict comprehensions? Well, you can build an equivalent loop without too much difficulty, and you're used to doing that.
So there's not a great value proposition when it comes to moving from Python 2 to Python 3. You're used to working around the warts, and the workarounds aren't that painful. But what about going from 3 to 2? Once you're used to range() being efficient, to dict comprehensions, to a cleaned up stdlib, and to sane Unicode handling? You won't be crippled, but I'm fairly certain you won't find it quite as pleasant.
I believe that's the measure that matters, in the long run. Would the old thing feel worse than the new thing, from the perspective of the new thing? If the answer is, to any degree, unambiguously "yes," then it only makes sense to plan on making that move.
Python3 core has a better design: strict separation between bytes and characters, no more short integer type, no more
old style class, etc. The stdlib is there. The rocky bits are being fixed all the time. The other important modules all have strong development communities.
Upstream numpy works with Python 3 already. PyGObject is also working on Py3 support.
The point is python2.7 is the last 2.x version. It will be supported, bugs will be fixed but no new feature will be added. Core devs will concentrate on python 3.x. You can still use 2.x, no one is stopping you. But assume that it is stagnant now. After some time only security related bugs will be fixed. Sooner or later you will have to switch.
What you will miss is 3rd party libraries. Some like numpy are ported, but many like mechanize needs to be ported. Google App Engine doesn't yet work on Python3. Not many hosting providers have Python3 as option.
However, i must agree that many of the points people bring up for not switching to Python3 are real. People ARE stuck in limbo right now and there is no good direction to go... Stick with 2.x and have all the 3rd party support but soon enough you will need to do a major bug fixing, or move to 3.x -- oh wait i can't move to 3.x because module xxyy is not 3.x compatible!
Is there really a limbo though or is that just a perception? I am starting to learn Python. I am learning Python 3. I believe that close to 95% (or maybe more) of the skills I am learning still apply to Python 2. The move to Python 3 will come. It was a 5 year plan of which we are half way there. Chillax. Really.
A lot of the more visible features from 3 have been added into new versions of 2.x. Off the top of my head, dictionary comprehensions and new set literals are two nice ones.
Statements like "first 256 code points in Unicode map to Latin-1" make little sense. They are true if you s/Unicode/UTF-8/g. However, they are not true for other encodings such as ones that use 2 or 4 bytes per character. There 'abcd' is not 4 bytes, but 8 or 16.