
Yanni – An artificial neural network for Erlang - _nato_
http://blog.ikura.co/posts/yanni-sings.html
======
fenollp
It uses `array` (somewhat mutable Erlang structure) and NOTP (no idea how it
makes code "zippy" and the repo [1] does not explain anything... it seems to
be bypassing the normal way modules are loaded?).

I am unsure why anyone would use Erlang for number crunching. Training neural
nets is basically just multiplying big matrices. I was hoping this project
would come up with an interesting approach (how about using SIMD on the binary
comprehensions that can use it? now that would be cool) but performance /
memory usage does not seem to be looked at here.

It is naive / uneducated to think that "Erlang’s multi-core support" \+
distributedness will enable many things for you. How does the VM scale on 32,
64 threads? Have you tried making a cluster of 50+ VMs? Unfortunately Erlang
Solutions Ltd.'s marketing has hyped many.

I am not against projects like these, I am just looking for reasons behind the
choices made.

[1]:
[https://bitbucket.org/nato/notp/src](https://bitbucket.org/nato/notp/src)

~~~
rdtsc
> I am unsure why anyone would use Erlang for number crunching.

I've talked to a few people from the financial world who to trading with
Erlang. They use it because it makes to easy to take advantage of multiple
cores.

And like others pointed out, if there is a need to optimize things, they can
quickly build a C module and interface with it but they didn't need it so far.

~~~
nikofeyn
i am curious why traders don't look to labview for some of their work. multi-
core support is inherent to the language and fpga programming is just a step
away with the same language. crunching away on things in parallel is what it's
great at.

~~~
microcolonel
It's not the _parallel_ type of multiprocessing which Erlang is good for, it's
the _concurrent_ type. The platform is largely based around message passing.

~~~
throwaway91111
Unfortunately this comes with tradeoffs—relatively high throughput compared to
synchronous computation at higher latency.

Of course that's fine for many workloads.

~~~
microcolonel
I'm waiting for the time we finally realize the obvious best model for
pipelined SMP applications: rescheduling the next required process to the core
where the data are cache local.

~~~
chrisseaton
'Work-stealing' schedulers already do this - jobs are scheduled onto the core
which created them and presumably touched their data last, unless there is
load imbalance in which case other cores take jobs. I don't know about the
internals of Erlang but I'd be surprised if it was not already work stealing
as it's the usual technique.

~~~
microcolonel
As far as I'm aware, most work stealing schedulers still aren't cache-aware.
One really naiive (but possibly effective) way to do this could be to have a
per-core (or per L2, or per NUMA node) work LIFO which would be consulted
before looking to other cores for work. When your core/L2/NUMA node schedules
a task right before terminating, it is more likely that the next task will be
local. This, of course, doesn't work if you're more concerned about jitter or
latency under load.

I noticed a paper about a cache-aware work-stealing scheduler which I have not
yet read[0].

[0]:
[https://www.researchgate.net/publication/260358432_Adaptive_...](https://www.researchgate.net/publication/260358432_Adaptive_Cache_Aware_Bitier_Work-
Stealing_in_Multisocket_Multicore_Architectures)

------
slezakattack
This is a neat idea but it would be great if there was a bit more substance to
the post. Do we have any performance benchmarks? Why would I consider it a
strong contender? Stating "multi-core support" to me is not necessarily
scaling.

I'm in no way an expert, but I work in Erlang in my day job and just glancing
at the repo, this solution can't possibly be performant. A) Erlang is slow at
math. B) Arrays don't have O(1) access(ETS tables might be able to help with
this). C) You can't scale this solution with more Erlang nodes(without some
additional work).

I really like Erlang and want to evangelize it but I don't think this is a
good way of doing it. I only see this as a neat toy but not a selling point
for using Erlang..

As a side note: I noticed the repo has a feature note about adding NIF's for
performance bottlenecks (native C code for Erlang to talk to). If you end up
writing C code, then what are you gaining from Erlang?

------
Cieplak
My favorite ML book is still the _Handbook of Neuroevolution Through Erlang_.
A bit pricey but you can borrow my copy if you're in SoMa.

~~~
salimmadjd
Interesting. Can you provide more details please. Why do you recommend this
book? What sets it a part from other books? What's the best thing you like
about? What is your own background (skills, education)? Only asking to make a
buying decision given the price.

~~~
Cieplak
In many ways it's a companion to this codebase:

[https://github.com/CorticalComputer/DXNN2](https://github.com/CorticalComputer/DXNN2)

------
brian_herman
With the latest revision I keep on getting this error:

    
    
        src/yanni_trainer.erl:78: type array() undefined
    

It was introduced in revision 20.

    
    
        3 files updated, 0 files merged, 1 files removed, 0 files unresolved
    
        [bherman@archy yanni]$ hg update 19
    
        [bherman@archy yanni]$ make
    
        rm -f notp notp.boot
    
        rm -fr ebin
    
        mkdir ebin
    
        erlc -o ebin src/*.erl deps/*/src/*.erl
    
        src/yanni_lib.erl:77: Warning: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
    
        src/yanni_trainer.erl:112: Warning: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
    
        deps/*/src/*.erl: no such file or directory
    
        make: *** [Makefile:6: default] Error 1
    
        [bherman@archy yanni]$ hg update 20
    
        3 files updated, 0 files merged, 0 files removed, 0 files unresolved
    
        [bherman@archy yanni]$ make
    
        rm -f notp notp.boot
    
        rm -fr ebin
    
        mkdir ebin
    
            erlc -o ebin src/*.erl deps/*/src/*.erl
    
        src/yanni_lib.erl:77: Warning: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
    
        src/yanni_trainer.erl:78: type array() undefined
    
        src/yanni_trainer.erl:118: Warning: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
    
        make: *** [Makefile:6: default] Error 1
    
            [bherman@archy yanni]$ make

edit: formatting

~~~
ramchip
I wonder if the author is running an ancient Erlang? The code looks very old
school, with the lack of maps, no rebar, deprecated functions, etc.

From Erlang release notes:

The pre-defined types array/0, dict/0, digraph/0, gb_set/0, gb_tree/0,
queue/0, set/0, and tid/0 have been deprecated. They will be removed in
Erlang/OTP 18.0.

Instead the types array:array/0, dict:dict/0, digraph:graph/0, gb_set:set/0,
gb_tree:tree/0, queue:queue/0, sets:set/0, and ets:tid/0 can be used. (Note:
it has always been necessary to use ets:tid/0.)

------
JamesUtah07
Looking forward to a Elixir wrapper for this.

~~~
pselbert
There isn't any need for that. Hex is becoming the de facto package manager
for the Erlang ecosystem. You can easily install this as a dependency and call
it from your Elixir code.

~~~
JamesUtah07
Oh sweet, I wasn't aware of this.

------
lgleason
They should use this as the photo for the website ;)
[https://upload.wikimedia.org/wikipedia/en/b/b1/Nightbird_%28...](https://upload.wikimedia.org/wikipedia/en/b/b1/Nightbird_%28Yanni_album%29.jpg)

~~~
bjcy
And don't forget! Make sure to read the intro with this in the background:
[https://www.youtube.com/watch?v=JfSSQ3Vejao](https://www.youtube.com/watch?v=JfSSQ3Vejao)

~~~
lgleason
The first Yanni conference will be "Live at the Acropolis" :)

------
hartator
Isn't Erlang good for concurrency but bad at math performance?

~~~
tormeh
Yes, as many have pointed out. It's ideal for coordinating a cluster doing
machine learning, but ANN code itself should not be in Erlang.

~~~
digitalzombie
> It's ideal for coordinating a cluster doing machine learning, but ANN code
> itself should not be in Erlang.

It's funny that's what they did with disco
([http://discoproject.org/](http://discoproject.org/)).

Python + Erlang = Hadoop 1.0 like.

------
buttershakes
I really wish there was a way to do inline optimized code, i.e a gen_server
that transparently wraps another language without having to get into nif /
external servers. Basically an abstraction that hides all of that cruft and
builds optimized gen_servers for doing number crunching or heavy processing.
Maybe I'm just being lazy. :)

~~~
pmarreck
erlexec (or its elixir wrapper) might be worth a try:
[https://github.com/saleyn/erlexec](https://github.com/saleyn/erlexec)

~~~
buttershakes
I've used this, it's an awesome project, but it's basically just pipes all the
way down. I'm thinking of something closer to a NIF. I think Saleyn's c++ node
code might be the closest thing for a lower level language.

------
bitmapbrother
I'm not sure how I feel about a ML project using the name of one of the
greatest instrumentalists in history.

~~~
mulmen
Yeah, that's more of a web framework thing.

------
nukifw
Nice work ! At Dernier Cri, we begon a similar work :
[https://github.com/derniercri/multilayer-
perceptron](https://github.com/derniercri/multilayer-perceptron) But we were
far less advanced than you!

------
Ono-Sendai
If you use one Erlang node or whatever per neuron, it's gonna be slow as f*ck.

~~~
pselbert
In Erlang you usually run one "node" per machine, though you can call between
them with transparent RPC.

This library uses one "process" per neuron for concurrency. Processes are
extremely lightweight and entirely unrelated to system processes or threads.

~~~
Ono-Sendai
Running one process per neuron is going to be extremely slow.

~~~
btbuildem
Erlang's inter-process messaging is ridiculously optimized. Processes are
extremely low-weight, it costs approximately nothing to start and stop them.
This is one of the core strength of Erlang.

Running one process per neuron would actually be a very efficient way to do
it.

~~~
ramchip
I'm quite sure that would be a grossly inefficient approach. Sending a message
is expensive in Erlang, less so than in other languages, but it's still very
large compared to a few math operations. It's a common mistake to use
processes to represent objects [1].

The recent article from Discord [2] also mentioned "Sending messages between
Erlang processes was not as cheap as we expected, and the reduction cost —
Erlang unit of work used for process scheduling — was also quite high. We
found that the wall clock time of a single send/2 call could range from 30μs
to 70us due to Erlang de-scheduling the calling process."

[1]
[http://theerlangelist.com/article/spawn_or_not](http://theerlangelist.com/article/spawn_or_not)

[2] [https://blog.discordapp.com/scaling-
elixir-f9b8e1e7c29b](https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b)

------
toisanji
I wonder if you could make computation run faster by configuring the network
to cut up the work into gpu vs non gpu work and have each node efficiently
process the work and then have the results reassembled.

------
mehh
Each neutron as an individual process is indeed interesting but no so much if
your using backprop for the training algorithm as it doesn't really fit the
paradigm.

------
brandonhsiao
What is the 1-to-1 mapping between the Erlang concurrency model and neural
networks?

~~~
btbuildem
Each node in a NN can be represented as an Erlang actor / process. They
communicate via messages.

------
artur_makly
and it also plays a sexy clarinet??

