

Neural networks in JavaScript - Jack9132
https://github.com/harthur/brain

======
zackmorris
This is fantastic. I wonder if we've passed a tipping point where some of
these fundamental machine learning algorithms will be taught the same way we
learned the C++ STL back in the day. Really something like a spam filter
should look like a shell script, where you pipe things around and use off the
shelf tools, rather than spending inordinate amounts of time trying to
implement building blocks (imaging having to write grep before you could even
start).

Just out of curiosity, does anyone have a link on NN training parallelization?
I’m just hitting paywalls. I’m curious if it’s simple or if there are
dependencies between neurons that make it hard. As I see it, the next big
hurdles after ubiquitization are distributed training (possibly with GPUs) and
empirical data on the best tuning parameters for various problem spaces (which
will come largely for free when training happens 5 or 10 orders of magnitude
faster than today). Eventually we’ll have an answer as to whether AGI can be
composed of basic building blocks or if we’re still missing some fundamental
insight.

For example, genetic algorithms are fairly easy to parallelize because by
their nature they use large and independent populations. About the time I was
graduating college in the late 90s, there was great interest in using them to
find the initial weights in NNs to avoid local minima, but then I didn’t hear
much about it since. Have there been any breakthroughs?

~~~
joeblau

      > the same way we learned the C++ STL back in the day
    

My school forbid you from using STL. You had to build all of your data
structures in freshman/sophomore years and then use those same data structures
you built in your sophomore/junior/senior projects. It was really crazy
because if you never got your lists, trees, graphs, and hashes working you
basically wouldn't make it through the CS program. It sucked at the time, but
I'm grateful now.

~~~
squeaky-clean
Wow, that sounds very bad to me. It's important to learn how data structures
work, but I believe it's more important to learn how to find and re-use
existing code/libraries in your work. This is something schools seem to be
avoiding and even discouraging (mine didn't take it as far as your school, but
they never once covered the standard libraries).

I've tutored students in Java classes who would write a bubble sort every time
they needed to sort a List, instead of using Collections.sort, simply because
they didn't know Collections.sort existed! Or their own ugly code of looping,
indexOf and substring to get values from a CSV instead of using
string.split(",") or a premade csv parser (which I know that professor would
have been fine with, he would have even encouraged it).

Unless point of the project is to learn how to make one from scratch, you
shouldn't be wasting your time writing an alphabetical sorting function when
you can be solving novel problems.

~~~
joeblau

      > Wow, that sounds very bad to me.
    

It might, but today, I run into tons of developers that don't understand the
fundamentals. I worked with a guy who I affectionally named "Stack Man." He
basically used a stack for any list implementation. He didn't understand the
difference between a list, a queue, and a stack.

You're teaching your students Java and telling them "Just use
Collections.sort" and that's fine, but do you understand when Collections.sort
is bad? If you don't understand the underlying algorithm of how a merge sort
works, you might be telling them to use something that is worse than could
actually be worse than a bubble sort in certain cases.

Also Java wasn't taught to CS students at our school, that was only for BIT
(Business Information Technology). Teachers wanted you to understand memory
and pointers.

~~~
squeaky-clean
>You're teaching your students Java and telling them "Just use
Collections.sort" and that's fine, but do you understand when Collections.sort
is bad? If you don't understand the underlying algorithm of how a merge sort
works, you might be telling them to use something that is worse than could
actually be worse than a bubble sort in certain cases.

When your assignment has to be uploaded in 3 hours, and your dataset is 120
items, Collections.Sort is always better than writing your own sort.

I never said I taught them only to use Collections.sort, as that would be just
as bad as never teaching them the standard libraries and only teaching data-
structures. You need a healthy balance of both fields of knowledge. Otherwise
you end up spitting out either programmers who can use library functions
without understanding what's actually going on, or programmers who know all
about the underlying methods and logic, but can't write code efficiently and
quickly enough to actually help a team.

That's not to say you cannot write code efficiently or quickly. You likely did
as most good programmers do and learned on your own time as well. But for
students who don't do this, they will be lacking huge amounts of practical
knowledge.

------
radq
Here's a similar project which has more functionality:

[https://github.com/karpathy/convnetjs](https://github.com/karpathy/convnetjs)

~~~
cyorir
I like this one because it is easier to get an idea of what the layout of the
neural network itself is, and gives more control.

Off the topic of neural networks in javascript but on the topic of neural
networks in general, the readme links a cool paper on applying convolutional
neural networks to old games.

[http://arxiv.org/abs/1312.5602](http://arxiv.org/abs/1312.5602)

------
domlebo70
Let me warrant this by saying I don't know the first thing about how neural
networks actually work, but can someone help me out and explain why the
following isn't working?

I added an extra input to the training data for the XOR example:

    
    
      net.train([{input: [0, 0], output: [0]},
                 {input: [0, 1], output: [1]},
                 {input: [1, 0], output: [1]},
                 {input: [1, 1], output: [0]},
                 {input: [3, 4], output: [7]}])
    

But the result of running:

    
    
      var output = net.run([3, 4]);  // [0.987]
    

Is

    
    
      [0.9999823694382457]
    

which strikes me as odd. Shouldn't it be closer to 7?

~~~
dmitrig01
From the documentation:

    
    
        Each training pattern should have an input and an output, both of which can be either an array of numbers from 0 to 1 or a hash of numbers from 0 to 1.

------
stared
Just curious, are there any visualizations of how weights of these networks
change in time?

~~~
chestervonwinch
Do you mean for this particular implementation or in general? If the latter,
here's an animation of input-to-hidden weights during the training of an
autoencoder on MNIST data:

[http://i61.tinypic.com/micsrc.gif](http://i61.tinypic.com/micsrc.gif)

~~~
warfangle
warning: tinypic often shows NSFW images on the same page as the image linked
to.

In my case this was a big ol' row of titties, right in the middle of my
office..

~~~
chestervonwinch
Yikes. Sorry about that. I don't remember the last time I used a photo upload
service, so I just used the first one that came up. I didn't notice the NSFW
stuff due to adblock probably. Anyway, here's a safer link for anyone else:

[http://i.imgur.com/CPe5Frh.gif](http://i.imgur.com/CPe5Frh.gif)

------
iamsalman
Nice! Although training 100k+ samples may crash or be prohibitively slow.
Maybe enabling Decaf for training on the GPU in the backend would be a killer
feature!

------
novaleaf
what kind of neural network is used? I hear big things about RandomForest
Classifiers/Regressors, so hope it's that.

------
darkxanthos
Does it seem weird to anyone else that the first example is approximating XOR?

~~~
jackcarter
No, that's a pretty standard trivial example for NNs.

[http://en.wikipedia.org/wiki/Feedforward_neural_network#Mult...](http://en.wikipedia.org/wiki/Feedforward_neural_network#Multi-
layer_perceptron)

~~~
abrichr
The reason why it's used is because it's a very simple example of a non-
linearly separable function.

You can interpret boolean functions as classification problems: given some
input(s) x, classify the output as either 1 (class A) or 0 (class B).

The truth table of an XOR function looks like this:

    
    
      x1 | x2 | XOR | Class
      ---------------------
      0  | 0  | 0   | B
      0  | 1  | 1   | A
      1  | 0  | 1   | A
      1  | 1  | 0   | B
    

If you were to plot the XOR function in R2, it would look something like this:

    
    
      x2 ^
         |
         |
         A     B
         |
         |
         B-----A----->
                      x1
    

To be non-linearly separable means that there is no straight line that you can
draw in the above plot that will split the classes. Thus, we need a non-linear
classifier, e.g. a neural net.

~~~
klapinat0r
Would you mind elaborating the linear separable part?

As in, the following _would_ be linear separable (all labels on the same side
of the line)?

    
    
       x2 ^
          |
          |
          A     A
          |
          |
          B-----B----->
                       x1

~~~
cyorir
It's a bit hard to elaborate. You have to rember that a classification problem
may be generalized to more than just two dimensions, in fact it can be
generalized to however many dimensions are necessary to describe the data. In
the general multidimensional cases linearly separable relates to the number
and type of hyperplanes (subspaces) necessary to correctly classify the data.

In this simple 2-d case the subspace is a line, so this example is linearly
separable. In the 3rd dimensional case linearly separable refers to
classification using a single 2-dimensional plane. For 4 dimensions with the
data, you need a single 3-dimensional hyperplane for it to be linearly
separable, and so on.

------
relfor
server in javascript, 3d engine in javascript, brain in javascript, next
kidneys in javascript

