"This is a bit awkward for me as I've paused the development of HLearn and emphatically do not recommend anyone use it. The main problem is that Haskell (which I otherwise love) has poor support for numerical computing."
Can anyone expand on this comment, and does that also apply to Grenade?
Of course, there is the point where such a library becomes a thin wrapper around an efficient numerical library written in C, but there are other advantages to Haskell that might make such a wrapper still worth the trade: an amazing type system, libraries that take advantage of this type system, and a compiler that generates fairly efficient code. If, in exchange for that, I have to marshal numerical data into and out of Haskell's FFI, I consider it a fair trade.
Even if all the computation were implmented in C (eg, DAG nodes in a TF style setup), being able to compose the network that forms the computation via Haskell would be a big win for me because it lets me use the type system and various powerful libraries to design my network. (And opens a straightforward path to GPU acceleration of Haskell designed code by using the C GPU libraries.)
It also seems like a terrible idea to use recursion to solve multidimensional array problems typically associated with loops and the indexing of those arrays.
This SO does a good job of explaining the trade offs. https://stackoverflow.com/questions/13042353/does-haskell-ha...
You'd use the FFI to call out to something like C to actually execute the tensor operation, much like the TF design, where you're constructing a DAG in Python of the dataflow through tensors, which have their implementations in another language.
The type system just gives you a powerful way to reason about the structure of your DAG, define higher level abstractions (eg, autoimplementing certain kinds of networks against a certain input type), or discuss the feedback of the system during training, since training is just a higher order function that takes the entire DAG function as input and returns a new DAG. (Okay, for performance reasons you generally update values, but you can model that easily enough in Haskell.)
So really, basically the same way Python for numerics -- calling C for number crunching and doing algorithm designing in the higher level language.
This locates the constraint on the authors ambitions relative to the desired abstraction level, rather than the performance level available
The issue comes from being lazy by default.
When you are writing lazy functional code, you essentially build up a system of thunks that collapse to a value when you need it. This works great for abstractions like infinite lists, or essentially only paying for the value you need. Unfortunately, they are kind of expensive (I believe they are usually implemented as a tagged union, where they are either a value that is returned immediately or a pointer to another function or thunk) when you don't need the laziness. In any sort of numerical computing, you are usually dealing with enough values that you lose a lot of performance when dealing with these thunks. Working around this is doable with a bunch of strict annotations (so that these thunks are immediately collapsed to values), but that is only half the battle: you still need to reduce the immutable function applications to a mutable version so you don't lose out by copying your values back and forth in memory, and you don't do a bunch of unnecessary calculations. All this requires "a sufficiently intelligent compiler" or a lot of compiler hints and peering into Core: the intermediate language of GHC. It isn't for the faint of heart.
There have been a bunch of work on avoiding some of this: hmatrix is a lapack and blas library for finally doing the calculations once you have reduced the graph, "subhask" is a standard library for Haskell that attempts to add some support for basic numerical work and "numerical" is a library that I believe is trying to make a well typed linear algebra library (I'm not sure if it is going to call into hmatrix/blas/lapack or not). The issue with a lot of this is that until there are good tools for this kind of thing, no-one does serious numerical programming in Haskell, and there are only a few people working on the tooling because there aren't that many people trying to do serious numerical programming in Haskell.
(note: I'm not an expert in this, so don't depend on any of this for love or money. I'm sure someone will come along and tell me how I'm wrong.)
 https://en.wikipedia.org/wiki/Thunk, https://wiki.haskell.org/Thunk
SubHask for example uses hmatrix and other LAPACK/BLAS libraries under the hood, which is good of course -- but almost all of the actual work in SubHask, then, is around designing a reusable hierarchy that works for all the 'weird' cases you find with this domain. Even 'non weird' things like rank/dimension polymorphic products, etc, require a lot of type-level infrastructure to accomplish today, and the features to support that are constantly evolving. (An example of APL-style rank-polymorphic stuff I typed up recently can give you an idea of how much work it is to do some 'seemingly trivial' things, though the examples at the end are rather nice) SubHask also has an extremely fine grained hierarchy, with lots of little classes to handle cases like "weird containers" with appropriate laws -- and figuring that out is a lot of work too.
Haskell's type system is getting more expressive all the time, so hopefully this will become less tiresome in the future, and iterating and honing the design will be far simpler. But getting the design right is much more important and the other stuff (better underlying numeric libraries, improved performance, etc) can come after.
I think you're spot on that the lack of attention here creatives a negative feedback loop, as well. Another example of this I think that was bad was that very little code took into account things like numerical stability of algorithms. The SubHask/HLearn author also tried tackling this in an interesting way, too.
I think part of this issue boils down to avoiding the laziness and immutability of the language. The "patterns we want for [...] numerical computing" not infrequently boil down to (a) build a matrix, (b) modify that matrix, (c) apply that matrix, or parts of that matrix, on vectors, (d) modify vectors, (e) partition matrices into submatrices to do work on, etc. Notice the "modify" part of so many of these.
These aren't really hard problems in imperative languages. The naive algorithm for Arnoldi iteration is inherently iterative and really isn't that hard to write with ok performance in an imperative language... but try writing it naively in a functional language! So, tools and abstractions need to be built to make these kinds of algorithms more natural to write functionally without costing too much in performance. In other words: the tools and abstractions are only necessary because of the limitations of a language that leans so hard on the compiler for performance.
I think this is one of the underlying things that differentiate Haskell and other languages: In Haskell, abstractions can be absurdly reusable, allowing for really elegant code to do things... if you know what abstraction to use. If you don't, then things can get difficult to do. In imperative languages, abstractions can't quite reach the level of reusability that they can in Haskell... but you don't have to rely on the abstractions as much to solve the problems that you want to solve.
Think about the Arnoldi iteration. The iterative version in a number of languages is 10 lines, but the functional version needs some way of drilling down into an array in a mutable way (a monad) in order to make it match the iterative in performance...
There exist mutable counterparts such as Data.Vector.Unboxed.Mutable and everything in the ST monad.
The question is if you are allowing yourself to utilize them.
Sure, but IMHO the monadic code necessary to implement something like Arnoldi is neither harder to write nor uglier than the same implementation in an imperative language.
If anything, this highlights a strength of the language: we can capture an imperative idea in a functional setting, and even have the ability to nicely isolate the mutability from the rest of our functional program (with `ST`).
Haskell could shine in numerical computing. I think the key is polymorphism: a vector is a matrix is a tensor is a number, and they can all be multiplied in the same way, once you define what multiply is.
Poor support is more about the lack of haskell coders doing ML. It's hard to compete without the numbers that other platforms enjoy.
I learned all this while trying my own hand at writing a sparse matrix library in Haskell and running into some of these issues. Unfortunately, I mostly gave up.
The main issue that keeps me from giving this a try is the lack of GPU support out-of-the-box, necessary for tackling problems of more interesting size :-(