"The end result is an implementation several orders of magnitude faster than the current reference implementation in Java. ... [Python] makes the first version easy to implement and provides plenty of powerful tools for optimization later when you understand where and how you need it." [edited to be a statement instead of rhetorical question]
- Python has easy development (https://xkcd.com/353/)
- Great libraries (ie, free matlab)
- Cython for efficiency via C
- The algorithms themselves determine speediness (ie numerical methods)
We do scientific computing at my company. Numpy does 90% of the work, but there are some algorithms that just aren't easily expressed with arrays. That's where Cython comes in.
Numpy and scipy have been the core of a huge amount of my optimisations. The first question I try and ask is
"Could this be solved with matrix multiplications and summing?"
Often the answer is "yes" and allows you to group a huge amount of calculations all together, and use the heavily optimised code available numpy/scipy.
I recently swapped out something that was running at about 100 rows calculated/second to about half a million in about 0.2s.
for k in dict.keys():
This has been 'fixed' in python3 and .keys() now returns an iterable view of the keys, and if you actually want a list of the keys you have to explicit and write list(dict.keys())
for k in dict:
if k in dict:
The equivalent python2 behaviour can be obtained using list(somedict.keys())
I tried a very simple toy program the other day and while I had to write some things slightly un-pythonically (it can't deal with syntax like a[:] = b+c yet), it performed practically as good as hand-written C code.
Java is compiled to bytecode, but it isn't a "compiled language" since that bytecode has to be interpreted by the JVM.
All the good libraries the author mention are probably implemented in C or FORTRAN, and so a true implementation in C with the right compiler optimizations would for sure be faster than the python code.
True. But this assumes that time is not a constraint.
I think you need to think of it this way (as a thought experiment): you start two programmers off, one in C and one in Python, both with a vague understanding of how to solve the problem and approximately the same skill level. Then after X hours, you stop both and see how far they've gotten.
I think the argument then is that you might find that the C programmer hasn't yet solved the problem, whereas the Python programmer might have a solution + a deeper understanding of where the slow parts are, and started optimizing those.
So in the end the C programmer may win, but perhaps it's not about winning in this sense but more about how fast you can get to a point where you can move on to the next thing.
I think as developers we often underestimate the amount of work we need to put into the non coding part of development for larger projects, like testing, debugging, optimization, and design. When these components become important then despite the local productivity gains of using rapid development languages like python or Julia, it would overall be a wash if you're writing the system from scratch in a language like C.
Finally, you can release the GIL with Cython:
(And while I personally love Scala, there's nothing magic about it in this regard. There's no reason a language can't offer Python-like expressiveness and Java-like performance, and many modern languages do)
I consider it an accident of history that Numpy, Scipy, Pandas, Scikits got written for Python and not Lua. Thanks to Luajit I think Lua would have been a better choice. Now that cause has been taken up by Torch and Julia.
JVM by itself is terrible for reaching close to the FLOPS that the CPU is capable of. Try sparse matrix multiply with it and see it for yourself.
I could think some examples where lacking SSE/AVX support would hinder it, but I don't see the connection with sparse matrices.
A structural problem of JVM is that its runtime semantics is over-specified, there is very little room for the JIT to
do its stuff. For example function arguments are evaluated right to left, there goes an opportunity for parallelism.
But how are sparse matrices then generally laid out? A naive approach would be some hash map, perhaps with some locality, in which I don't see JIT problems.
Not so in my experience. Write a matrix multiply in pure Java, and pure C, C++, Fortran. The difference would easily be north of 5X, typically more.
Regarding JNI, please see the root of the comment tree. Rarely have I seen more stinky garbage. Its hopeless if you have to go back and forth across the bridge. If that be so I might as well be on the other side of the bridge.
JVM is fantastic, but if you are doing number crunching and performance matters, then JVM is a wrong choice.
It seems you misread the article as saying Python created a trade-off. Perhaps the author shouldn't have tried to create a surprise ending. Regardless, the conclusion was that the Python implementation was both easier to code and more performant than the Java implementation.
In contrast, if you work in Java, you are trying to use the same tool for both jobs, and you may well fall between 2 stools. And I say that as a typical Java-head.
My only question about the 2-tool combination is whether there are better combinations. Python has all the libraries and community support so any alternative would need similar. Maybe Node?
As for the number crunching, I think Rust would be a better choice here. Good memory management is its USP and that can have significant performance benefits.
Absolutely not. Python has a much more mature set of libraries which are much better designed, better tooling, and a much more reasonable type system, and the community has only recently begun to be polluted by Web 2.0 "move fast and break things" mentality. The Node ecosystem is built on that mentality, and while outlier developers at the front line of the JS community are able to be extraordinarily productive in JS/Node, developers building user-facing programs just get bogged down in breaking changes in dependencies, buggy and poorly-designed 0.x libraries, untraceable framework code that prioritizes configuration over convention, and intractable errors caused by a broken type system (this final issue is significantly improved in ES6).
Source: Worked in Python 2.x for a few years, worked in Node for a few years, currently work in Python 3.x and Node (general design is to limit Node to compiling and testing the browser-based part of our product). JS/Node is maybe 20% of our code, 20% of our features, 50% of our dev time, and 80% of our bugs.
Then I remembered Perl and realised that data pre-/post-processing is precisely why it was invented in the first place.
Perl + Rust would be an interesting combo IMHO.
Except, when you're on Windows, and it just doesn't. Just installing things and doing the 'hello world' for aforementioned libraries is laughably impossible.
So, use Python, but use it only on Linux.
(Okay, if you absolutely must do it in Windows: Use Anaconda).
Granted, it does not say "and by the way, when you do that you get tons of other great things like pandas, matplotlib, etc". But they are very clear about not trying to do this yourself.
Python works fine, in your own virtualenv. Outside of it... not so much.
What are your preferred ways to profile Python code? Coming recently from PHP, where we have XDebug/KCachegrind, the excellent Facebook-sponsored Xhprof, https://blackfire.io and https://tideways.io, it's felt a step backwards.
I've tried line_profiler, and used memory_profiler and cProfile with pyprof2calltree and KCachegrind. I've found the cProfile output confusing when it crosses the Python-C barrier for numpy, sklearn etc.
cython -a module.pyx && open module.html
the -a command produces an annotated html file, which lets you see how Cython is compiling your Python into C. There's a colour coding to give you an indication of which bits are fully typed and being translated directly, and which parts are hitting the CPython API at run-time (and hence will be slow).
What didn't you like about line_profiler?
Here's a good guide on how to write fast(ish) code in Python:
Generally, the best strategy for me has been to use NumPy wherever possible and to avoid creating many complex objects. Best to use built in dicts or tuples for things that store data. Thus the only time I run into issues is when implementing algorithms in which case I usually isolate the slow function and turn it into a Cython module.
Recently have been playing around with https://github.com/jboy/nim-pymod which seems like a much better solution.
I bet :) I have only scratched the surface, that's for sure.
> What didn't you like about line_profiler?
In this case I was trying to profile the overall simulation codebase to find the slow spots (rather than guess) - a simulation that takes 60 minutes to run. line_profiler wasn't great at giving digestible results from that - and I haven't worked out how to write all output (across multiple modules) to file, without specifying the file in each decorator.
I've started to break the codebase down into 'tests' to measure each algorithm separately, and will give line_profiler another go then.
Re memory_profiler, while the mprof command showed all peaks, the line by line output only showed the result after executing each line - so when 4GB of RAM disappeared in a skimage call, only to be released at the end - it wasn't reflected in the output. Which is tricy when trying to reduce overall memory usage.
python -m cProfile module.py | gprof2dot -f pstats | python -m xdot -
From what I've heard from a friend of using other people's code in one particular scientific field (stringly type some of the things, probably accidentally, don't document this), an at-least-passable type system would be a huge improvement.
The Haskell crowd seems to write monad tutorials that are either cute or unintellegible, and stratosphere-high level stuff where I wouldn't have the slightest clue what I can use them for (Arrows? Zippers?).
Lets say you want to do processing of some xml file.
Normally you'd walk the tree and do manipulations in place.
With zippers however, you can inspect every intermediary tree result, you can rethink you problem so that you walk the tree once to extract interesting information, then compute a changeset for the tree, maybe merge it with a differently computed changeset, and then apply the union of them.
I build a ocr system in clojure on zippers and it was a lot of fun. You could for example extract a list of all the words, with line wraps removed, then do the correction on that view/lense of your data, and reapply the changes without having to worry to reintroduce the pesky linewraps, because they were never removed from the original document.
There are "Real World" Haskell resources, in fact there is even a book named Real World Haskell, though I recommend Haskell Book these days since it takes an approach of teaching from first principles.
This means you have a much larger audience, and therefore it's more likely that someone will already have written a tool of your domain in Python.
This is subjective, but having tried Elm with its Haskellish type system, it seems that the type system makes me think hard about things that ultimately don't matter much. It might depend on your problem, but I doubt it. Even doing numerical computation with Theano I find myself better off just executing subexpressions and seeing results and the shape of the tensors interactively is more useful than thinking really hard how the operations transform the shape.
If it is true that the type system, perhaps static type systems in general, make you think hard about things that ultimately don't matter much... why not go for the most dynamic and interactive language possible? Clojure perhaps.
I've been toying with the idea that either fully dynamic and as interactive as possible or full dependent typing is the generally right answer.
By right, I mean what you should reach for with most programming tasks if you aren't going to put effort into selecting one tailored to your task.
Haskell tends not to be a go-to choice here because it... well... there are a lot of reasons. It's a tough language to learn compared to its competitors because it's basically a few really awesome modern features in a massive graveyard of failed academic initiatives that are now enshrined in the lore of the language because one or two useful libraries used them.
Err... can you qualify this? Based on my knowledge of Haskell this isn't a fair assessment at all.
And how many failed lens libraries preceeded the current (quite good) dominant library that still show up on google and even get included in? How many people still run versions of Yesod using now-unfavored abstractions?
The Haskell community really got its shit together over the last 3 years. Cabal got a lot better, a bunch of key libraries got good, our compiler is fixed some major bugs. But these developments do not magically erase people who have navigated the prior 5 years of Haskell where a lot of these technologies got established.
I love Haskell, I really do. I wish I could ship more code with Haskell, and like it as well. But the OP asked about why Haskell didn't take off. Its ecosystem was in what we might call a state of growth and flux at the time a lot of stream and batch computation systems were shipping.
Oh and uh, I think Haskell's community is full of some truly toxic people, an intersection of some of the ugliest and most arrogant personalities of the Scala world who's primary education strategy can only be called "negging." You will know these people when you find them. Compared to the Clojure community or the F# community or the Erlang community which are also forward thinking, interesting and skilled... it looks positively hostile. You may not care about that, but I refuse to endorse or use technologies who's community leadership is so toxic. Why, I've stopped using Linux in anything but legacy apps because I completely despair of that project from ever rising above is puerile and cliquish roots.
Alright, I'll give you that there are a ton of GHC extensions after getting a list of them, all 106 as of GHC 7.10.
This does make me wonder why having so many extensions was never an issue for me.
> The Haskell community really got its shit together over the last 3 years. Cabal got a lot better, a bunch of key libraries got good, our compiler is fixed some major bugs
> I love Haskell, I really do. I wish I could ship more code with Haskell, and like it as well. But the OP asked about why Haskell didn't take off. Its ecosystem was in what we might call a state of growth and flux at the time a lot of stream and batch computation systems were shipping.
> Oh and uh, I think Haskell's community is full of some truly toxic people, an intersection of some of the ugliest and most arrogant personalities of the Scala world who's primary education strategy can only be called "negging."
I've seen this personality type in other languages, but not Haskell yet. I too abhor the "negging" type, as well as the subculture that term arose from.
0: all ghc extensions
RecordPuns (Deprecated, use NamedFieldPuns instead)
Edit: I kid, but most of the languages I just mentioned have very fast native compilers, easy ways of invoking low-level interop, functional-style libraries if you want them, and (except for Python) C-like syntax making it easy to cut-and-paste.
If we limit the discussion to ML languages, they are generally very nice e.g. for datastructure transforms (including interpreters and compilers). Static typing with type inference makes implementations short, and, if written in good style, are correct when they compile.
There are various other reasons why one would not want to use them. For me it's platform stability and availability - while I enjoy functional languages in principle - I yet seem to choose C++ more often than not for my hobby projects.
I would say learning self-described fp-languages have made me a better programmer who values static typing more than before.
FP paradigm has indeed didactic value and the experience gained in a functional language may translate into a plus of productivity ...in other languages. When you have to get things done, the functional language's embedded arm-twisting (well, mind-twisting actually) doesn't help! That's when you'll either abandon the (idea of) "pure" functional language for a multi-paradigm language (that will allow you to keep a mainly functional style with limited deviations when the situation warrants it) or you'll keep pushing yourself and throw the blame on you for not being good enough with a language that wasn't designed from very beginning for much anything but the FP standpoint (for the sake of it).
I see FP languages being able to excel at the kind of projects with large, maintainable codebases that evolve over a decade and are shared between dozens of developers. Service oriented architectures serve the same goal -- isolating behavior and state locally, and being explicit about mutation and state changes over time.
"R, at its heart, is a functional programming (FP) language. This means that it provides many tools for the creation and manipulation of functions. In particular, R has what’s known as first class functions. You can do anything with functions that you can do with vectors: you can assign them to variables, store them in lists, pass them as arguments to other functions, create them inside functions, and even return them as the result of a function."
Weak example is Wikipedia list of Functional Programming Languages list R as one. https://en.wikipedia.org/wiki/Functional_programming#R
R describes itself as a Functional programming Language:
The language syntax has a superficial similarity with C, but the semantics are of the FPL (functional programming language) variety with stronger affinities with Lisp and APL. In particular, it allows “computing on the language”, which in turn makes it possible to write functions that take expressions as input, something that is often useful for statistical modeling and graphics.
Academic Evidence: "The Journal Trends in Functional Programming" also list R as a functional programming language.
R is an environment and functional programming language for statistical data analysis and visualization. Largely unknown to the functional programming community, it is popular and influential in many empirical sciences.
The issue might be it isn't Haskel so it is not 100% functional but there are more Functional Programming Languages then Haskel.
PS This is why so many programs in R are poorly written because people try to force OO into R when they could do things Functional. That is also why my eyes bleed when I see people looping through everything in their R scripts.
Yes, you can write R code in a functional style in places when convenient, but one floor down it's still running that loop you didn't want to see. It's a language based on mutable values and sequential execution and looping, not pure function calls and recursion. The syntax itself is very much sequential, unlike stereotypical FPLs. Even the lexical scoping design relies fundamentally on mutable environments. So yes, people do seem to enjoy proudly declaring R a FPL citing a few features, but as I said, those features make R as much a FPL as Python.
That was very condescending and I am sure don't know who Hadley Wickham is? I wouldn't call his understanding "Shallow."
Can you provide some Haskell examples that are cumbersome, then provide an example that isn't in another language of your choice?
Also, have you tried using lens for this?
to that in Julia
Despite the Haskell program using the elegant facilities provided by REPA, the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic action seems cumbersome to me.
> the fact that mutable part (the `writePolicy`) has to be written in a different style than the pure parts since it is a monadic action seems cumbersome to me.
It has to be written differently more because it's using the state monad. You can write monadic actions that resemble pure code, for instance:
> replicateM_ (readFile "/etc/issue")
["Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n","Ubuntu 15.10 \\n \\l\n\n"]
Because some of the scientists doing this are bellow average programmers. Look at the author singing praises to Python while benefiting from C, Fortran and Cython. He doesn't understand that what actually matters is what's under the hood, and you want him to learn functional programming and rewrite his algorithms?
That's the unfair part right there: lying to people about the proper uses of Python.
We did run into some resistance initially, because our stack is almost the opposite in every way of what our enterprise colleagues are used to. However, our development velocity, especially around analytical features and just in general, has made a most gratifying impact.
(I have to add that the 5-man dev team we have working on this is stellar. It's hard to determine scientifically what the interaction is between team quality and the choice of a Python-oriented software stack. See Paul Graham's essays for more discussion on that point.)
In terms of support: There are many highly professional often boutique software agencies that can support Django systems, if you're not around. To my mind this is even better than the normal commercial support you get from a different vendor for each different component in your commercial enterprise system.
My project would be to put different data sources together + to allow users to upload their own structured data via Excel (think financial estimates). The current system has about 450 users, the next might have much more depending if it gets extended to other divisions.
Continuum Analytics is founded by the creator of Numpy and employs many leading python developers, and they offer support contracts for basically the entire Python/Numpy data analysis stack.
My real question is, is it so much easier to do this excercise in Python than Java, assuming equal proficiency in either case?
That is just my guess.
It sounds like the Java version was also optimized - though it's hard to say to what degree. I think in general if you're doing a lot of numerical computing and matrix operations, the basic java builtins are not going to cut it anyway, and you're going to end up using something like ND4j or JBLAS to get comparable performance to something like numpy, in which case I have hard time imagining it could possibly be /easier/ than numpy.
Granted, the cython stuff is more "advanced", and you probably would struggle a bit (or just would not care for the task) if you were a researcher with a surface understanding of what's really going on, but it's no big deal for a programmer. The other thing is that the defaults usually work well enough that you don't really have to pull out the big guns most of the time. You can also decide to do your optimization progressively. "Let me rewrite this part in cython" is a much quicker win than "let me rewrite all of this to run on top of platform X" or something.
Also it definitely does depend on the kind of problem. For example, the JVM does have awesome tools for distributed computing and streams like http://akka.io/, vs python is more lacking.
In conclusion though, I've been always very pleasantly surprised by how far I can get away with by just dumping larger and larger datasets (current record is in the hundreds of GB for me) into the python / pandas / scipy / numpy stack, and how much of a pleasure it is to use compared to anything else. To toss that away, I'd want to have tried and failed at a problem with the python stack first.
- Could this be/Was this developed in Python 3.x
- what is this "notebook" he keeps on referring to?
The "Python" here isn't particularly material to the result, it's mostly a wrapper around C. Toss in Cython, and now you've really gone outside the bounds of "I'm just using 'Python' for HPC!".
I agree some of the tooling and niceties are beyond a doubt best in breed with Python, but it's disingenuous to equate this to "writing HPC code in Python". If you had written a RPython to Verilog translator that produced an FPGA of your algorithm would you call that "using Python"?
But they aren't, of course.
For science, the fact that so many great libraries are available and they all use Numpy arrays as base datatype (from image processing to GIS to machine learning etc etc) is a real strength of Python and I doubt another lanuage is going to achieve that any time soon.
Big parts of numpy (and to a lesser degree pandas) are not in python, but lots of libraries on top of it are mostly python (scikit learn, scikits image, etc.). It is no more wrapping than lisp is wrapping the C VM.
His point is that the article shouldn't make it sound like "Python is fast", because the speed actually comes from the libraries that have been implemented in C.
I believe what the article says is that "Python has tools that enable a savvy user to achieve better results with less effort". Python is extremely popular in HPC settings (including supercomputers) for this reason. I see nothing disingenuous.
But the article is titled: "Why I Still Use Python for High Performance Scientific Computing", and it gives the impression that Python - the language - is fast enough for HPSC.
In reality, the reason why he "still" uses Python is that the libraries are fast enough for HPSC. But that's not what people see when reading posts like this.
The message they see is that "Python is fast", not that some of its 3rd-party libraries are fast.
But I should probably stop repeating myself here.
An answer is Python.
If you use C for HPC (the way you mean it) it will be slow as a dog. Because HPC is done with BLAS, LAPACK, etc. If you do HPC in C you are calling into BLAS and LAPACK. They are Fortran libraries. Anything else is dog slow in comparison (a slight stretch, but not by much).
So, the "real" answer is Fortran. But not a lot of people are sitting at their desk writing Fortran for scientific computing these days. They are writing Python, C, or C++ for the most part. All of which use libraries that call into the Fortran libraries.
That's all this article is saying, and we all understand it. No one is wringing their hands about what language the libraries are written in unless you are trying to write one of those libraries.
Working professionals use Python for scientific software because it excels at data munging, making reports, and very high speed computation. Everyone gets that you need to add some libraries beyond the libraries provided from the base install.
Second, it is an inherent feature of the design of CPython that its C API allows tight integration with external libraries in C. Cython does not just glue C and Python together, it does this in a way which makes the integration easier and safer than doing it by hand (e.g., generating correct reference counting code). All of this is a direct benefit of Python and its scientific ecosystem.
Sure, but that's how people construe the post, which I think the author knows too.
The post could have been accurately titled "Python has certain libraries that are fast enough for HPSC", but that wouldn't have generated nearly as much interest as "Why I still use Python for HPSC".
People want to see others say good things about their favourite language, which in this case is Python.
I knew where the article was going (with the libraries), but still wanted to read it because even I wanted to see someone compliment Python itself, because it used to be my "primary language".
Python is a good language, but to the extent you're not using those highly optimized "C-libraries", it just doesn't perform well, and it's not suited for concurrency. That's a part of why I switched to Clojure.
You can make any title more accurate by making it longer and more specific, but picking a good title is a trade off against other things as well. A charitable reader would not necessarily insist on the narrow interpretation of Python the language you insist on.
And your title is not more accurate, because it's not just the existence of certain libraries, it's also the way they can be easily glued together and incrementally improved. Compare this to say Java, where you'd have to use JNI to use those same libraries, which is much more cumbersome to the point that most people prefer to write everything in Java.
> People want to see others say good things about their favourite language
To be honest I only think this only describes people that enjoy internet debates on programming languages. If what you are interested in is getting work done, you don't care about the merits of a particular tool in isolation but about end results. Hypothetically, you could find the "perfect" language but if it doesn't have users or libraries, it's of no use, and therefore the insistence of discussing only this or that language is pointless.
Language discussions always seem to go down the same dismal tube in the end: "Is C > Python, or is Python > C?"