Hacker News new | past | comments | ask | show | jobs | submit login
Release of IPython 5.0 (jupyter.org)
334 points by trymas on July 8, 2016 | hide | past | web | favorite | 103 comments

Glad to hear improvements to the shell ipython interface, especially up/down arrows on pasted code.

The most interesting part of this for me is that IPython 6 will not support Python 2.

>Projects such as Matplotlib and SymPy plan to drop support in the next few years, while a few projects like Scikit-Bio are already ahead of us, and should be Python 3 only soon.

This was also very surprising for the standard reasons, especially for a library like matplotlib. Glad to find Python moving forward. But what will companies stuck on Python2 do? Will libraries like numpy, matplotlib, and scipy all maintain a Python2 LTS?

Companies stuck on python 2, have known for several years that life-time support was disappearing and will do 1 of 3 things:

1) Get caught by surprise because they aren't planning. 2) Plan and pay the price of upgrading their legacy code. 3) Join an active developed (currently hypothetical) fork of python 2

I think in 2020 we'll see all 3 routes done by several companies. Better get the popcorn ready!

> 3) Join an active developed (currently hypothetical) fork of python 2

RedHat is probably going to keep Python2 on some degree of life-support at least until 2030 or so, just based on a guess that RHEL8 most likely will still have python2, and its support will continue for 10+ years after its release.

https://github.com/dropbox/pyston seems to be option 3

Or PyPy. It's development is still heavily focused on Python 2 because that's where the corporate sponsorship is.


4) migrate to other languages, such as Go

... which is not uncommon, and in many cases a pretty good idea.

migrate to other languages, such as Go

This depends heavily on the type of work someone is doing. People doing numeric/scientific computing work with Python can't really do this, for example, because there's a distinct lack of other languages with equivalents to the libraries and tooling Python gives them for their work. And that's a chicken/egg feedback loop problem: Go needs the libraries and tools, but in order get them needs people to move, but in order to get people to move needs the libraries/tools, and so on.

And if you don't have time to switch from Python 2 to 3, you definitely don't have time to switch to Go.

Julia will be a suitable (superior) alternative soon.

Remember that the other thing about Python is that it's got good support for other types of programming -- for example, I work at a company which settled on Python (in part) because it allows a single language end-to-end. Our data ingestion, analytics/processing/number crunching, and end-user applications based on the data can all be built in Python, and even if you're not a domain expert in one of those areas you can at least read code to see what's going on and not be totally lost.

Julia can do all that. its also a general programming language

It's like you're recommending a restaurant by saying that its food is "edible".

All in all, this is a bad recommendation. Julia is version 0.4. Its syntax changes with every version. It is not going to be stable soon. It is so far from stable that it doesn't have a plan for what stability will look like.

It's fine if you want to get in on the ground floor of a programming language, or to learn a new language for the fun of it, but it's completely unreasonable to suggest that you could replace a programming language that people use for their jobs that way.

Well sure, in the same way MATLAB is a general programming language. It can be done, but it'll make you feel icky.

I've seen people saying this for years, yet I don't know anyone that has actually switched over. The people I do who have tried came right back to Python, often within a few days or weeks.

careful continuing that linear extrapolation...I think it's about to hit a tipping point due to packages and tooling being much better.

Python 2 isn't going anywhere anytime soon. Tools and libraries which drop support for Python 2 will suffer and will either revert their decision or be replaced with something else.

Found the Python 2 developer folks!

Joking aside, Python 3 is the only Python. The only people who should use Python 2 are those supporting legacy codebases, and even they should start migrating.

This is fine as far as wishful thinking goes, but simply doesn't correspond to reality. Over the past 15 years or so I have been working with and consulting large corporations using Python, all over the world. And I can tell you for a fact: Python 3 is simply not an option in most cases. I HAVE worked on transitioning several large codebases from 2 to 3 and in each case this has been months of work, incurring completely unjustified costs. The benefits have been practically negligible. I would never advise a client with an existing codebase to move on to Python 3, unless there were damn good reasons for it.

For new, smaller projects, that don't have too many external dependencies - Python 3 is fine. Again, even in these cases, the benefits are largely superficial.

Making new release of Open-Source & free library python 3 only does not prevent your clients to still use the old versions. Those will never become incompatible. If the want longer support they then can consider giving some money to the Open-Source project they use. The switching at this point for many of the libraries/project that make these decision is almost 0 cost, and number of benefits. Especially when these libraries are maintained by Night and Week-end contributors. I think many contributors to these library would love to be hired to work on it, and many of these library would love8 to get funding to hire their contributor. I'm actually seating to 2 of these people that were hired by such kind of funds, and are now looking for more funding to hire more dev.

Saying new versions will not be Python 2 compatible is not a request for Companies to upgrades. It's just a statement that they will be stuck on old version of libraries, unless they buy* support for a vendor.

We (the IPython team) have no objection if Continuum or Enthought provide a Paying version of IPython 6.x that is Python 2 compatible, but that's their problem. It does not prevent either the core developer of the Python-3 only project to consult on potential Python 2 internal fork.

I honestly think that this would be a much sane model that would make both sides of the Py2vsPy3 battle happy.

> I HAVE worked on transitioning several large codebases from 2 to 3 and in each case this has been months of work, incurring completely unjustified costs

Out of interest whats the biggest time sink while doing this? I've had a lot of luck using 2to3 and with decent test coverage and a UAT environment that people can access, its not exactly months and months of work changing "xrange" to "range", and "print" to "print()"

> decent test coverage

You were indeed very lucky if all projects you've worked on had decent test coverage.

The unicode/bytes change it the most time consuming in my experience. Second comes the fact that you can't sort arbitrary types anymore and that None is no longer smaller than any other value.

In order to ease porting, I've created a modified Python 3.6 interpreter that generates warnings in a lot of the cases that real Python 3 would generate an exception. See https://github.com/nascheme/ppython .

The goal is to have a 2to3 script that generates something that will run, with warnings, under me modified Python. Once you fix all the warnings, your code should run correctly in Python 3.

> Second comes the fact that you can't sort arbitrary types anymore

Surely that's a bug in the system that you've just uncovered and fixed? Py2 used to use the memory address of the object when comparing by default, which is just... crazy, especially for a language with so few WTFs:

   >>> object() > object()
   >>> object() > object()
   >>> object() > object()
   >>> object() > object()
> In order to ease porting, I've created a modified Python 3.6 interpreter

That looks amazing! We're going to end up porting a fairly large + critical Django app to py3 and I definitely think this could ease some of the pain. I'm going to give it a go when I get the chance.

> Py2 used to use the memory address of the object when comparing by default, which is just... crazy,

Its true that that's crazy, but, OTOH, everything-can-be-sorted is a useful feature (which Erlang has, for instance). Py3 could conceptually have retained it with a different implementation (this would probably still have been a breaking change from Py2, but not a feature loss.)

For me it's been libraries which we rely upon which don't support 3.

True, I didn't consider the cost of updating packages (and any associated API changes). But in my experience there are very few packages that simply don't support Python 3, and the ones that don't are just dead.

http://py3readiness.org looks pretty good, most of the blockers for me (like twisted and a few others) are gone by now.

care to give some examples ?

unicode. That in itself is a huge win for the kind of the stuff I work on (100000LOC sized project). It makes things so much easier.

Absolutely. If you live in a world of ascii, then maybe python 2 is adequate. But I always deal with unicode, and python 3 is a big benefit just for that.

>I would never advise a client with an existing codebase to move on to Python 3, unless there were damn good reasons for it.

The end of life of python 2, and the end of python 2 support in popular libraries isn't "a damn good reason"?

Python 2.x support ends in 4 years. It seems irresponsible to not be planning for a move now. Large firms do take longer but that's why now is a good time to begin.

The cost is justified if you want to use a supported language.

I can't think of a way that this makes any difference, or see any reason why anyone should be bothered by python 2 support "ending". I don't even understand what this means, I mean it sounds like a scare tactic without substance. At the end of the day, if the PSF won't support it, I will (and I'll happily take the money for it).

Big companies aren't going to gamble their future without support. And they probably don't want to deal with a single guy they find on the Internet for security/bug fixes in a forked version of Python.

Your guesses are wrong.

Obviously when I say "I will support it", I don't mean myself, single-handedly. I mean my company. Or someone else's company. The point is, commercially speaking, nobody cares about official PSF support. The world will go on unaffected, with or without it.

And you're welcome to support IPython and matplotlib on Python 2 at the same time. We're just saying we're not going to do that, in the same way that the PSF will stop supporting Python 2. At some point, the cost of third party support outweighs the cost of moving to Python 3.

Can you clarify? Will these companies open source their support? Making a case for sticking with 2.x will be easier if there's one "official" repo. Otherwise, the number of patches and forks could quickly get out of hand.

I'm not saying you're wrong, but I find that very surprising. It's a far cry from the Enterprise Java consultant world I live in.

Sure, but PSF is not the equivalent of Oracle. If someone wanted a big enterprise to provide support they would use a language from a big enterprise. The difference between the PSF and another small company stepping up to provide support is negligible.

A few weeks back I surveyed my 3,500 members at PyDataLondon (London-based Python data scientists) about their Python version usage. At work 60% use Python 2.7, 33% use Python 3.4+. At home 44% use Python 2.7, 49% use Python 3.4+. I'm predicting 50% at-work usage of Python 3.4+ this time next year for my London community: http://ianozsvald.com/2016/06/20/results-for-which-version-o...

Note - I'm the co-chair for PyDataLondon (and the local conference series). I regularly poll my monthly meetup audience to see how many folk upgrade each month. Normally 1-10 per month stick their hand up, nobody sticks their hand up if I ask "how many downgraded from Python 3 this month?".

Many people use Python 2 for the sole reason it's the default system Python. I assume a lot of them will instantaneously move to Python 3 the moment typing "python" on the command line will invoke the Python 3 REPL.

All my python 2 only code explicitly invokes python 2 on the shebang line.

Though, more likely they'll find that typing "python" returns "command not found", and in that case they might find it easier to just type apt-get/dnf install python.

Tricky one. I've just had to sell my clients on an update from Django < 1.8 as that fell out of it's support period.

"Hey folks. I've got to do lots of work that will result in no new visible functionality and I'd like to be paid for it"

I did consider slipping a Python 3 upgrade in there at the same time but there are limits to my powers of persuasion.

So - Python 3 will have to wait.

I don't do this kind of work (agency) but I find it strange that ongoing maintenance isn't discussed and priced into the statement of work right from the beginning.

Something like 2 days maintenance every 9 months to stay current, or 4 days every 2-3 years to go from LTS to LTS. If a customer doesn't understand why they need to be paying maintenance every x months or years, isn't that your fault for not making that clear in the beginning? Sorry if it seems like I'm picking on you, but I've seen this kind of comment quite a lot, and I've never asked the question before.

I actually agree and I should factor that in more.

However for many reasons it's often tricky:

1. This is something you have to address in the initial stages of a client relationship and with price-sensitive clients unused to software development if you're not careful it can sound like you're just rent-seeking.

2. 'Big but infrequent' changes are difficult to cost. Django LTS upgrades are every 5 years. Breaking changes like Python 2 to 3 are even rarer. What figure would I have proposed during the sales process that would have correctly covered me for these? I'm almost certain I would have significantly underestimated.

3. It's not always clear if you're entering into a long-term relationship with the client. They might expect the website to be completely replaced in a year or so, they might be intending to bring development in-house or various other things that make 'saving for a rainy day' a tricky proposition.

> 3. It's not always clear if you're entering into a long-term relationship with the client.

I've also found that clients will often be acquired, go through management changes, etc that make having these things built in become unstable. One of my clients was acquired by Merck, then Merck sent a letter saying they were terminating all contracts. 18 months later, I get a frantic e-mail from some low-level people asking why I was no longer updating things for them.

The median life of many companies makes building in release->release upgrade support difficult in practice.

I personally charge my clients with a flat monthly maintenance fee which includes upgrades, but also minor bug fixes. In some ways it's a guarantee that the website will keep running, though without an actual SLA. It means I don't need to spend time selling the client on the upgrade.

1. Yes. I always put the on-going fee in the initial project estimate, so there's no surprises.

2. Yes, it requires some guess-work. I've also found that it saves time to take one feature migration (like removing django's url patterns() function) and making that change to all projects at the same time (rather than making all the upgrades to one project at once). I also have found it helpful to reduce the number of 3rd party dependencies. (Fewer things to upgrade, and fewer things that may potentially stop getting supported.)

(Also "Django LTS upgrades are every 5 years" - I was going to correct that to 2 years, but I suppose you can completely skip an LTS line if you time it just right (like, jump from 1.8 directly to 2.0). But you still need to upgrade every _4_ years. Right?)

3. Yes, it's something to figure out initially.

Thanks for the response. I forget sometimes that not all customers are reasonable.

I'm coming from the Java world, but as a consultant for enterprise customers, I'll say that they only understand two kinds of ongoing maintenance: 1. Dollars over time (e.g. licenses) 2. Purpose-specific hire (i.e. hire a guy to do nothing but support this new thing between now and forever)

In other words, they understand continuous maintenance, but not continual maintenance. Anything else is either too awkward to write on a budget. If you suggest that it needs periodic review, they'll think you just need to "do it right the first time, so this isn't an issue going forward", whether or not that makes any sense.

How about car oil changes for an analogy? You could probably drive for several oil change periods without apparent issues, until eventually there would be big issues. Django upgrades are similar, not "that" difficult, and it's a good idea not to fall too far back in versions, running into more and more incompatibilities. Py 2 -> 3 is a bigger one, so maybe like a timing belt change? ;-)

> Python 2 isn't going anywhere anytime soon.

This can be read two very different ways. ;)

Good! Both apply.

If you don't need Python 3, then you won't need IPython 6.

Companies using Numpy and Scipy who need python 2.x support will probably buy a support plan from someone like Continuum Analytics.

Probably until 2020 when Python 2 finally will be phased out. http://legacy.python.org/dev/peps/pep-0373/

Except Python 2 isn't going to be "phased out". Just support from Guido and company will end. Alternate support will continue on with fully compatible py2 implementations such as PyPy and pyston. "python" will still point to python2 on Linux distributions and Python distributions such as anaconda will still have the established, mature, fully functional python2 suite available.

No actually phasing out of Python 2 has already begun.

Several libs today are Python 3 only. Many, many more are not going to pay the cost of supporting Python2 and drop it in next few years.

Unless you are retiring next couple years or live with head in sand, any new code you write should either by python2-3 compatible (it's not hard at all, do it all day every day) or Python 3 only.

> "python" will still point to python2 on Linux distributions

That's already false in many cases for at least a year. Arch and Gentoo come to mind.

Sure, but it's true for both debian/ubuntu and fedora/rhel/centos.

"support from Guido and company will end" - I find it very ironic that Guido's employer will keep supporting py2 (pyston).

> It is important to note that users will always be able to use a Python 2 kernel with the Jupyter Notebook, even when all of our projects have transitioned to Python 3

The way I understand this is that you will need python 3 to open ipython 6, but once running, you can interact with python2 and run and inspect python 2 code. I think that is fine. I can't imagine a modern scientific setup that can't readily create python 2 and 3 virtual environments.

No, not really, When you start a notebook, you could have an IPython 6 - Python 3 kernel, OR an IPython 5 - Python 2 kernel. But Notebook & Jupyter != IPython

I thought that just like the notebook, IPython was a "view" into a python kernel. Is that wrongly understood?

It's a bit confusing because parts of IPython became Jupyter. But since they split, IPython is two things: the Python kernel for Jupyter, and a terminal interface that uses the same core execution machinery within the process.

So it will only be possible to run Python 3 code in IPython 6, but it will still be possible to use IPython 5 as a kernel for future versions of Jupyter.

This is going to take some getting used-to. The visuals (specifically the syntax highlighting and code completion) are very reminiscent of bpython.

On the other hand, I'm really happy to say farewell to readline. I've been stuck with readline v6.2.4.1 for ages just so I can have proper linewrap [0]. Of course this breaks virtualenv creation so you end up having to override the system readline [1]. Needless to say, this is well overdue.

[0]: https://github.com/ipython/ipython/issues/3329/

[1]: https://github.com/pypa/virtualenv/issues/4#issuecomment-966...

As someone who's never used ipython before, but used the standard python interactive terminal a lot, I'm very impressed!

The best feature I've discovered so far, is that when I want to change a function, I can simply 'up arrow', and I get the whole thing! Not a single line of the function, as in the normal python terminal. And if I write a syntax error while typing the function, it tells me immediately!

Does anybody have other examples of great features in ipython over the standard python terminal?

For one, you can just paste in a bunch of code. Try doing that in the standard terminal!

Another feature I end up using constantly is the documentation support. Say you want to read the docstring for an object, just prefix or append a '?' to the object, e.g. 'subprocess?'.

If you want to examine the source code for the object, just prefix or append '??', e.g. 'subprocess.call??'.

> For one, you can just paste in a bunch of code. Try doing that in the standard terminal!

Usually I can paste code just fine, I think? As long as the indentation is correct and I have double blank lines between functions, etc.

> Say you want to read the docstring for an object, just prefix or append a '?' to the object, e.g. 'subprocess?'

Isn't that the same as typing `help(subprocess)` in the standard terminal?

> If you want to examine the source code for the object, just prefix or append '??', e.g. 'subprocess.call??'.

That's pretty cool :-)

Just try pasting the following into the default interpreter, exactly as-is:

    class Sample:

        def __init__(self):

    s = Sample()
IPython wouldn't flinch.

Note that if you're doing this in a terminal, you MIGHT need to first do `%cpaste`:


It seemed to work fine when I didn't use `%cpaste`, though, so that might be a vestigial thing? (I remember having to do it on my linux workstation, so maybe it's just my mac terminal behaving differently.)

The cool thing about directly pasting is, if you fumble the indentation like I did when I initially pasted, it auto-indents for you. ;)

In previous releases, you had to use %cpaste. With IPython 5, so long as your terminal supports 'bracketed paste', it should just work. We'll probably deprecate %cpaste eventually.

We got this feature with no effort on our part - it's built into prompt_toolkit.

The ability to customize the IPython launch sequence and configure things is something I make heavy use of. For example, here is my IPython launch script [0].

I started out adding to the launch script so that I would have data and helper functions ever-present in my terminal when I was working to rapidly answer Stack Overflow questions, especially for often repeated tropes like group-by in pandas or some broadcasting issues in numpy.

But then I started accumulating other things, like the boilerplate for reflecting some local toy databases using SQLAlchemy, and some toy stuff with metaclasses and decorators, and kept going from there.

You should read the docs on IPython configuration -- you can even create different profiles and then launch IPython with a specific profile if you want to get different defaults in your launch environment for different use cases [1].

[0] < https://github.com/spearsem/configs/blob/master/ipython_conf... >

[1] < http://ipython.readthedocs.io/en/stable/config/intro.html >

Wow. That's pretty cool! I had wrapped ipython in a helper that would always have the Right PYTHON_PATH, but it's neat to see just how much of ipython I have not been using. Thank you for sharing that.

Whenever I'm helping someone new to Python, I show them the standard python shell (which itself is often a revelation). Then I show them ipython, and suggest they never go back to the standard shell.

Ctrl-R for command history search. Similarly if you type a few characters and do up-arrow, it'll look up commands that start with the same characters. ipython saves every result, you can get it by typing Out[1] (... and it'll take up memory, so beware). Call it with --autocall 2 and then you can omit parentheses in function calls.

%debug after an exception :)

Tab to autocomplete is pretty sweet.

You can still get tab to complete with the standard Python interpreter.

    python3 -ic "import readline; import rlcompleter; readline.parse_and_bind('tab: complete')"
Admittedly, IPython's is better.

file_list = !ls

So much nicer than messing around with subprocess and pipes.

Wait, what is the syntax for this? I can't do `len(!ls)`?

No, `!command` only work on assignment. Please read http://ipython.readthedocs.io/en/stable/interactive/tutorial...

See tutos like https://www.youtube.com/watch?v=xe_ATRmw0KM , most of what works in the notebook work in IPython.

But yes you can do things like:

    files = ! ls   *.txt

    for name in files:
        with open(name) as f:
            if 'Content-Type: text/x-rst' in f.read():
                !git mv {name} {name[:-3]}rst

adding a `?` at the end of a function name or other objects brings up the documentation.

Extremely handy.

There is a particular behaviour in RStudio that I would really love to be able to do in python, but haven't found the combination of IDE/or notebook that will do it yet:

I want to execute through lines with cmd-return, or by highlighting and pressing cmd-return, and then see the change in the variables in a separate pane, like RStudio's environment pane. Bonus points if I can click on table variables in the environment pane and examine them in a separate tab with sorting and searching. Spyder comes closest but the execution part doesn't work as fluidly.

Sounds like you're looking for something like: https://github.com/yhat/rodeo

Ah! I saw a very early version of that but never tried it, looks great! Thanks.

why do you want the results in a separate pane instead of in the notebook right below the code? can you explain this in more detail?

this might be an upcoming feature for beaker notebook (it definitely has the tables with sorting and searching).

When you clear the output of a cell, the variables lives on. Also a cell might change several variables but not print it.

It is extremely handy to have a spreadsheet like "variable explorer" handy to get an overview of what you have in memory and what your numbers look like and a feature that I miss from rStudio.

thanks, i think i get it, you like having a list of all the variables defined and their values (like in a debugger), in a panel that you can monitor while you run code. yes that sounds useful :)

Thanks to ipython, radare2 (reverse engineering framework) now has a very useful python shell with autocompletion: https://asciinema.org/a/16ko4jd1e6kdrqkqjxeu248hm

Thanks to prompt_toolkit, IPython now supports:

    Syntax highlighting as you type
    Real multi-line editing (up and down arrow keys move between lines)
    Multi-line paste without breaking indentation or immediately executing code
    Better code completion interface (we plan to improve that more)
    Optional mouse support

"The awkward dependencies on pyreadline for Windows and gnureadline for Mac prompted Thomas Kluyver to replace the old machinery with a brand new pure-python readline replacement: prompt_toolkit."

I was waiting for something like this for years!

Multiline support - my prayers have been answered! Copying/pasting code and modifying multiline commands from your history was easily the most annoying thing about ipython.

Agreed, though %edit is still a better (and already existing) way to deal with multi line in my opinion ;)

For me it would be, if only the result of %edit would make it into the history...

Syntax highlighting and line navigation look awesome. The big news is this is the last release to support Python 2.x. Oh, and no more readline. Thank the gods.

Kudos to the iPython / Jupyter team for spearheading such an important dev-productivity tool. I use your software all the time. I also appreciate the effort to migrate everybody to Python3.

Is there a way to enable vi mode in iPython 5? Since it no longer depends on readline library, putting `set editing-mode vi` in ~/.inputrc file no longer seems to work.

Yes, please read the release notes.

for reference: http://ipython.readthedocs.io/en/stable/whatsnew/version5.ht...

tldr; add `c.TerminalInteractiveShell.editing_mode = 'vi'` to ipython_config.py

Twitching on the floor... blinky blinky cursor, is there no way to make it go away..? looks in the feature list. Went back to 4.

That's a terminal preference. E.G: iterm> Preference > Text > Blink Cursor -> uncheck it. Tada !

Damn, the storm already hit python 2.x. Have to move on.

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact