
A Neural Network in 11 Lines of Python (2015) - williamtrask
http://iamtrask.github.io/2015/07/12/basic-python-network/
======
ocharles
3blue1brown is currently doing a terrific series on how neural networks work,
which nicely compliments this blog post.
[https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQ...](https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)

------
amelius
I'm wondering, given a random truth-table with N binary variables and 1 binary
output, what is (worst case) the smallest network that can learn it? (In terms
of number of parameters).

~~~
fooker
For a 100% success rate, it would have to be of the size of the minimal BDD.
If you can allow for some errors, this becomes an interesting problem.

~~~
HalcyonicStorm
Can you define BDD, please?

~~~
Someone
[https://en.wikipedia.org/wiki/Binary_decision_diagram](https://en.wikipedia.org/wiki/Binary_decision_diagram)

------
e19293001
The author is currently writing this book:

[https://www.manning.com/books/grokking-deep-
learning](https://www.manning.com/books/grokking-deep-learning)

------
dahart
Having gone through this tutorial (which is great!) and several others, I'm
curious what is a good second step for the casual neural network learner?

It sounds like there is a growing bag of tricks neural network researchers are
discovering to make training practical and stable for large data sets.

One example would be using relu activation - whenever I play with it in a
simple tutorial like this one, training seems to explode and fail much more
frequently, so I'm guessing either I'm missing another step people use, or
there are some extra constraints on initial conditions?

Using a Gaussian for activation in my tutorials has tended to be more stable
and converge much faster, but I assume there is a huge downside lurking
somewhere to having a non-monotonically increasing function?

What are the tricks of the trade that a weekend warrior should investigate?

~~~
mholt
I co-authored a blog post with my lab that has practical advice for debugging
DNNs - some of these tips might be helpful to you?
[https://pcc.cs.byu.edu/2017/10/02/practical-advice-for-
build...](https://pcc.cs.byu.edu/2017/10/02/practical-advice-for-building-
deep-neural-networks/)

------
ghosthamlet
this is my APL and J implemention:
[https://github.com/ghosthamlet/ann.apl](https://github.com/ghosthamlet/ann.apl)

------
zakoud
This is my Q implmentation:
[https://github.com/zakoud/mlq/blob/master/neuralnets/terser_...](https://github.com/zakoud/mlq/blob/master/neuralnets/terser_nn.q)

------
craigching
If this blog post interests you, you may be interested in his book that is
forthcoming: [https://www.manning.com/books/grokking-deep-
learning](https://www.manning.com/books/grokking-deep-learning)

------
yujinyuz
Nice discussion :)

------
carlosgg
This one has been posted before.

[https://news.ycombinator.com/item?id=9886555](https://news.ycombinator.com/item?id=9886555)

[https://news.ycombinator.com/item?id=11378022](https://news.ycombinator.com/item?id=11378022)

~~~
KennyCason
Quite a while ago, not sure if that's a problem. It's a pretty great write up
and worthy of posting again. For me, this was the first time seeing it.

~~~
SeanDav
You are both right. It is worthwhile knowing that something has been posted
before, in order to review old comments if interested and good content is
always worth repeating.

In any case, reposting is allowed, for several good reasons, which have been
discussed in the past.

~~~
KennyCason
Indeed, I had not considered that. I may have been used to people saying that
links shouldn't be posted twice. The extra context is in fact nice!

