
Zeta: Functional Neural Networks in Ocaml - frjalex
https://github.com/liaopeiyuan/zeta
======
Athas
A quite bright bachelor's student of mine did something similar for his
undergraduate thesis[0]. Conceptually, he represented neural networks as
functions from input to output, and connecting two networks required that the
output of the first matched the input of the second, and produced a new
network. Now, to enable gradient descent or other kinds of optimisation, the
networks were not _literally_ functions, but rather pairs of functions for
running the network forwards and backwards. While limited in some ways (e.g.
no recurrent networks), it was quite convenient, and close in performance to
TensorFlow on a single GPU.

[0]: [https://futhark-lang.org/student-projects/duc-bsc-
thesis.pdf](https://futhark-lang.org/student-projects/duc-bsc-thesis.pdf)

------
6177c40f
I had been messing around with a similar concept not too long ago in OCaml- as
it turns out, strong, static typing can be very nice to have in writing ML
models (and code in general). I find it reduces some of the cognitive overhead
that comes from having to keep track of how the functions interact in the
program (or perhaps, it reduces/clarifies the _potential_ ways the functions
can interact).

I'll be very interested to see where this project goes.

~~~
dmit
It's always been baffling to me that certain problem domains with extremely
slow feedback loops have standardized on using dynamically typed languages.
Machine learning and Python. Video game scripting and Lua. A lot of scientific
programming as well.

How did that happen? Did the trend start because the work was done by domain
specialists who were not necessarily expert programmers, and C was too
unergonomic for non-experts to use? Is that the reason a Skyrim modder can't
get immediate feedback from a compiler in 2020 - because 25 years ago game
devs thought that documenting a C API was harder than embedding a scripting
language in the game engine? Or perhaps because evaluating scripts was more
secure than loading third-party DLLs and exposing the game's innards to them?

Edit: also, if every CSV file came with a schema, that would be great. Even if
it says that every column is of type Option<Any> \- at least then I know what
to expect.

~~~
75dvtwin
I think it is more that 'expert programmers' where told to leave 'domain-
specific' bits for power-users to deal with.

Therefore,

\- game scene development in games, are in scripts. \- Equity derivatives and
swap contracts are in Excel \- scientific modeling is in python

I also agree with your other though that, dynamic loading enables 're-use' of
the core framework, so those frameworks/engines become marketable technology
assets so to speak.

The programmers, then in some instances, created DSLs to then auto-generate
performant framework-specific code from the DSL-written user specs.

But those, usually, are quite limited and also some power users migrated to be
descent (not the expert engine/core-level, perhaps) programmers

------
andrepd
ML in ML?

Jokes aside, see:

[https://github.com/LaurentMazare/tensorflow-
ocaml](https://github.com/LaurentMazare/tensorflow-ocaml)

Also

[http://ryanrhymes.blogspot.com/2017/03/build-neural-
network-...](http://ryanrhymes.blogspot.com/2017/03/build-neural-network-from-
scratch-in-60.html)

for how you can very quickly write something from scratch with "Ocaml's
numpy", owl.

------
allisterb
I'm also working on something like this with F# and TensorFlow:

[https://youtu.be/GUMTpTqzsJA](https://youtu.be/GUMTpTqzsJA)

[https://devpost.com/software/sylvester-
tf](https://devpost.com/software/sylvester-tf)

using typed natural number dimensions for arrays, vectors, matrices, tensors
et.al

~~~
brianberns
This looks awesome. I’m an F# developer and I agree with your intro paragraph
100%:

“One of the ironies of today's computer programming landscape is that
functional languages directly inspired by the declarative languages for
expressing abstractions and equations of logic and mathematics, have been
sidelined for mathematical and scientific computing in favor of imperative,
dynamically-typed languages like Python and Julia.”

Sadly, I’ve found that every road leads back to Python when it comes to deep
neural nets. I tried using TensorFlowSharp from F# but found it very
frustrating. Only Python seems to have full bindings to the TF API. Would love
to give your wrapper a try.

~~~
allisterb
I've created a Jupyter notebook here: [http://ec2-52-15-66-200.us-
east-2.compute.amazonaws.com:6677...](http://ec2-52-15-66-200.us-
east-2.compute.amazonaws.com:6677/?token=bb2874e7bf09422cb2920e767eb3a63224112289013b1c53)

that demos my ongoing progress with Sylvester and I'd love your feedback. It's
also available on NuGet:
[https://www.nuget.org/packages/Sylvester.tf/0.2.3.4](https://www.nuget.org/packages/Sylvester.tf/0.2.3.4)
with a native TF 2 package:
[https://www.nuget.org/packages/Sylvester.tf.Native.Win/](https://www.nuget.org/packages/Sylvester.tf.Native.Win/).

It's still WIP but coming along pretty well.

------
akotulski
It's great to hear that people are trying to use ocaml for ML :) To me it
seems like owl is the most mature ocaml project in this area. I recall they
even had some GPU support. Unfortunately, I didn't have a chance to try it out
so I can't comment first hand on how "ready" it is.

[https://github.com/owlbarn/owl](https://github.com/owlbarn/owl)

