
Gorgonia: a library like Theano or TensorFlow, mainly written in Go - boyter
http://blog.chewxy.com/2016/09/19/gorgonia/
======
thinxer
I don't see the point of writing a machine learning package in Go.

Go is designed for system programming, which is the opposite of scripting. In
scientific computing, interactivity is an important feature. Softwares like
MATLAB and IPython(now Jupyter) become popular because of this. Go doesn't
have a nice REPL like IPython, and it never will. Besides, Go doesn't allow
operator overload, which leads to verbose programs. Compare `c = a @ b` with
`c := Must(Mul(a, b))`, the former Python statement is much cleaner than the
Go one.

Usually, researchers use MATLAB, R, and Python for prototyping and training.
The final model will be put into production by using something like TensorFlow
Serving or by hand written C++ for performance. Even if the final product uses
Go for serving, the Go program mostly runs the model via cgo or RPC.

Write in C/C++, and wrap the library in Python and Go is a much saner option.
TensorFlow and MXNet are both good examples.

~~~
marmaduke
I've been

> scientific computing

for quite some time, and I've the value of a REPL steadily in decline: what's
far more useful is a tested code snippet or function & a debugger,
occasionally.

Also, if a scientific stack were available in Go, I'd jump on it, because
deploying Go is so much simpler than Python. Python's deployment story for
scientific software often starts with either )a) create the perfect storm of
dependency versions or (b) use Anaconda. Even MATLAB is way better in this
regard.

~~~
NegatioN
The "perfect storm of dependency versions" is a problem in Go as well though?

~~~
eikenberry
Not in practice. The vendoring solution that they are standardizing on makes
versioning issues very manageable.

~~~
NegatioN
Which solution is this? Do you have any links or more information about that
spesifically?

~~~
MoOmer
[https://github.com/golang/go/wiki/PackageManagementTools](https://github.com/golang/go/wiki/PackageManagementTools)

------
tristanz
The tensorflow team is working on a go wrapper as well:

[https://github.com/tensorflow/tensorflow/issues/10#issuecomm...](https://github.com/tensorflow/tensorflow/issues/10#issuecomment-245687757)

------
chewxy
Hi,

I wrote Gorgonia:
[https://github.com/chewxy/gorgonia](https://github.com/chewxy/gorgonia)

If there are any questions I'm available to answer them

~~~
IshKebab
How does it handle recurrent connections. I really like Microsoft's approach
with CNTK - they use a DSL called 'BrainScript' to define the network, and
it's basically a computational network (hence then name), but there are
special operations like

    
    
        x = PastValue(y)
    

and

    
    
        y = FutureValue(x)
    

for forward and backward recurrent connections. It even automatically only
unrolls the recurrent part of the network.

In contrast all of the other NN systems I've used have premade layers like
LSTM and GRU, and to be honest their methods are so hacky I haven't really
been able to work out how to do _custom_ RNNs in any other system.

How does Gorgonia handle it?

~~~
chewxy
The trick, IMO, is to manually unroll them.

So what I do is something like this:

    
    
        func (m *model) costFn() (cost *Node, n int, err error) {
            var prev *recurrentOutput
            for ...{
                // build your recurrent graph here
            }
        }
        
        func (m *model) runOne() (err error) {
            cost, n err := costFn()
            g := m.g.SubgraphRoots(cost)
            machine := NewLispMachine(g)
            if err = machine.RunAll(); err != nil{
                return 
            }
        }
    

The LispMachine type allows for rapid prototyping. Then once that's all done
and stuff, you would probably have figured out what the size of your graph is
and you can then create a TapeMachine that have the sizes you defined (though
I have removed all the bits of the TapeMachine that made it turing complete,
so it may be a bit hard to do that).

~~~
PeterisP
Manual unrolling fails as soon as the number of repetitions is not fixed (e.g.
NLP tasks where it matches number of tokens in a sentence); so you want the
looping to happen _after_ the graph has been made, preferably on the GPU.

~~~
chewxy
For Gorgonia it's the other way around. You build the graph by manually
unrolling it. It'll be a big goddamn graph, which is why you subgraph it and
run it.

source: actually have various LSTMs running. Some with attention, some with
ADHD

~~~
PeterisP
Is performance acceptable if you're required to re-build/adjust the graph for
every single minibatch, because the number of unrolled items is different
every time?

 _That_ is what I mean by "Manual unrolling fails as soon as the number of
repetitions is not fixed"; you can do manual unrolling in every framework but
generally repeated graph building makes it unusably slow if you're unable to
do the loop "in the system" with a runtime-chosen number of repetitions.

Padding is also not really a solution, since your average sequence length is
likely 5 or 10 times less than the maximum sequence length that you want to
support, so just padding to a fixed size will mean 5-10 times slower
processing.

------
dwhitena
This is awesome! Great to see so much momentum on the Go data science front
(see
[https://github.com/gopherds/resources/blob/master/tooling/RE...](https://github.com/gopherds/resources/blob/master/tooling/README.md)
for more). Go provides a great integrity and deployment story for data
scientists, and contributions like this are well worth the effort!

~~~
stanislavb
Yup, amazing work. I hope it gets traction. In fact, there's a lot happening
in the "Machine Learning" front of Go
[https://go.libhunt.com/categories/507-machine-
learning](https://go.libhunt.com/categories/507-machine-learning)

------
thr0waway1239
I think Microsoft should take some initiatives to help bring more of these
projects into the .NET ecosystem. C# is a fantastic language, and is generally
missing out on a lot of this ML stuff.

(Although maybe it conflicts with their desire to profit from it?)

~~~
dkarapetyan
There are no conflicts. The latest strategy across the board is to give away
everything for free and then run it on the cloud. In that sense microsoft has
a lot to gain actually by offering a solution in .net and then making it dead
simple to deploy to azure.

Software and hardware are now a commodity and all the big players know it. So
the only way to make money is to add value some other way like making it dead
simple to deploy tensor flow graphs in gce.

~~~
mark_l_watson
I agree with you on Microsofts/Googles/Amazon's cloud business strategy. In
spite of the apparent good profits Amazon is making, I question how profitable
this business will be long term. I expect large gross sales and small profits
because of intense competition.

------
perfmode
Number one on my wish list for Go 2 is the ability to go from A to B:

A =

if xw, err = Mul(x,w); err != nil { log.Fatal(err) }

if xwpb, err = Add(xw, b); err != nil { log.Fatal(err) }

if prob, err = Sigmoid(xwpb); err != nil { log.Fatal(err) }

B =

probs := Sigmoid(Add(Mul(x,w), b))

~~~
Zikes
Go 1 is already there:

probs := MustSigmoid(MustAdd(MustMul(x,w), b))

~~~
perfmode
That panics. What I want is monadic error chaining.

------
gtani

        Fixed up GPU work
    

I haven't thought about using golang to feed data to a GTX 1080 or similar,
but before i do that, i'd like some info on the above, and a quick google
shows that there may not be a complete wrapper for the Cuda Driver or Runtime
API's like pyCuda.

~~~
chewxy
Hi,

So I wrote my own CUDA wrapper. Turns out there was a bug. I'm not sure where
and how, but it worked for the earlier versions of Gorgonia but not this
release.

You may also note that a bunch of assembly for the float32 tensor are
missing... they seem to be affected by the same bug, so what I'm doing now is
I'm planning to rewrite from scratch

~~~
gtani
Did you reach out to Nvidia? It seems like they should be willing to help with
putting out tested/documented bindings for more languagues.

Also, where in the repo is your wrapper? (I don't have enough knowledge to
debug s.t. like that, just curious how they work)

~~~
chewxy
It's not released - simply because it failed to even build properly.

No I have not reached out to nvidia. Don't have any contacts there

~~~
imurray
You could possibly try
[https://twitter.com/AlisonBLowndes](https://twitter.com/AlisonBLowndes)

