
PyTorch 1.0 is out - kashifr
https://github.com/pytorch/pytorch/releases/tag/v1.0.0
======
htfy96
What surprised me most is the elegance of C++ API. Compared to its equivalence
in Python, the C++ version is almost the same if we discard the "auto" keyword
[0]. As mentioned in the doc, they put user-friendliness over micro-
optimizations, which also proves the expressiveness of modern C++ (at least
when they want to prioritize user-friendliness!) [0]:
[https://pytorch.org/cppdocs/frontend.html#end-to-end-
example](https://pytorch.org/cppdocs/frontend.html#end-to-end-example)

~~~
usgroup
i picked up c++ for some gpu stuff with the arrayfire api... felt the same.
firstly, modern c++ takes no time to learn if you come from java / c/ c# /
etc. secondly, things like operator overloading and type inference make for
pretty seamless apis. E.g. want to add matrices? auto C = A + B.

A lot of things suck (closures, generator functions, first order functions all
suck in c++), but oh my does it all run fast when you get it working; plus
wrapping it with Rcpp or Lua is easy as pie.

~~~
umvi
I'm not sure how I feel about operator overloading. It starts out simple
enough, but can lead to extremely confusing code.

Take this snippet of an API call[1] that parses ISO strings to
chrono::time_point using date[2]:

    
    
        std::istringstream in(iso_string);
        in >> date::parse("%FT%TZ", tp);
    

At first glace my brain cannot comprehend the second statement. Why is there
an input stream going into a _function_? Is that even valid C++? Why not just
pass `in` as a function parameter and return time_point as the return value??
When you actually dig into the source, it's a namespace overloaded operator
and it's extremely heavily templated to be generic. So now if I think of `>>`
as simply a second function call using the result of `date::parse` it makes
sense, but... I still don't understand all of the technical reasons behind
that decision. I assume they're valid reasons though since Howard Hinnant is
_the_ c++ datetime expert.

[1]
[https://stackoverflow.com/a/33438989/2516916](https://stackoverflow.com/a/33438989/2516916)

[2]
[https://github.com/HowardHinnant/date](https://github.com/HowardHinnant/date)

~~~
nwallin
> Why is there an input stream going into a _function_?

The function is constructing a temporary _object_ and reading into that
_object_. The object includes a reference to a named tp object, and the
temporary parser object is parsing from the input stream, which writes the
data into the tp object.

> but... I still don't understand all of the technical reasons behind that
> decision.

Because he wants to support input from an istream, and inputting from a
istream necessarily means overloading '>>'. That's just how things are done in
C++; if you want to support reading from or writing to an istream/ostream, you
overload the '>>' and '<<' operators.

It's a trade off; he could have designed it so that the constructor of tp
simply accepts the format string and the istream as parameters. But that has
the downside of not looking like idiomatic istream code.

I personally think that the way iostreams are implemented in C++ was a
mistake. (because I don't like the way idiomatic istream looks, and I think
friend functions are generally smelly.) Unfortunately, it's something like 35
years too late to correct it. (I'm aware that C++ is only 33 years old;
iostream.h and cout << "blah" << endl; and cin >> foo; preexist C++) But
here's the thing: it's badly designed iostreams that make the linked code
confusing, not operator overloading.

~~~
pjmlp
Maybe because my background coming into C++ was Turbo Pascal, and I already
knew OOP from 5.5 and 6.0, including a nice experience with Turbo Vision and
its stream framework, introduced with Turbo Pascal 6.0.

I never grasped the hate iostreams get, as they are type safe and composable
in ways that FILE will never be.

Sure they might be a couple of ms slower than stdio calls, but unless I would
be writing a HPC trading application communicating over stdio, I hardly see
the relevance.

And for the operators, oh well, as long as they are consistent.

Then again, it might just be my biased background as I got introduced to them.

------
nafizh
Really grateful to the FAIR team for Pytorch. I use deep learning for
computational biology. Pytorch lets me focus on the problem rather than
nitpicking with the framework (looking at you tensorflow) to make something
work.

------
wpietri
In case anybody else was wondering, since this isn't in the fine article:

"PyTorch is a Python package that provides two high-level features:

* Tensor computation (like NumPy) with strong GPU acceleration

* Deep neural networks built on a tape-based autograd system

You can reuse your favorite Python packages such as NumPy, SciPy and Cython to
extend PyTorch when needed."

~~~
gnulinux
ML noob hobbyist here. Would you use PyTorch for models not involving deep
neural networks too or is it just good for that. Say if I use linear models
(like least squares etc) or use custom algorithms (integer linear programming,
optimization, or something else...) but need very fast linear algebra support
is PyTorch a good lib? I'm a C kinda guy so I usually use blas, lapack etc or
numpy+pandas+sklearn in python. Would PyTorch give a "complete" enough feel or
would I just use it only for nn and use other libraries for other things?

~~~
p1esk
PyTorch uses CuBLAS [1] under the hood, among other libraries, so basic linear
algebra ops should be fast.

You might also look at CuPy [2], especially if you like NumPy.

[1] [https://developer.nvidia.com/cublas](https://developer.nvidia.com/cublas)
[2] [https://cupy.chainer.org/](https://cupy.chainer.org/)

~~~
amelius
Now I'm wondering if anybody wrote a DL library on top of CuPy, and if such
library could be competitive with PyTorch in terms of performance.

~~~
smhx
Yes, it is called Chainer and it's quite competitive.

------
pesenti
Link to blog post: [https://code.fb.com/ai-research/pytorch-developer-
ecosystem-...](https://code.fb.com/ai-research/pytorch-developer-ecosystem-
expands-1-0-stable-release/)

------
tbenst
The new JIT is very interesting. Anyone know if this is for inference only or
also for training?

~~~
smhx
training and inference.

~~~
dekimir
Can you actually train a torch::jit::script::Module? I couldn't figure out how
to feed its parameter tensors to a torch::optim::OptimizerBase constructor...

------
l3robot
__TL;DR __

\- New JIT feature that lets you run your model without python. It now seems
trivial to load a pytorch model in C++

\- New distributed computation package. Major redesign.

\- C++ frontend

\- New torch hub feature to load models from github easily

~~~
make3
I love PyTorch, but my experience with jits embedded in Python (eg. Numba) has
been everything but simple, nevermind trivial. I'll really have to try it to
believe it.

~~~
mlthoughts2018
I’ve had the opposite experience with numba in production. It works almost
flawlessly, very easy to reason about the generated code and inspect
annotations, easy to debug.

~~~
alexbw
Check out github.com/google/jax, it’s NumPy on the GPU with automatic
differentiation, JIT and autobatching.

~~~
mlthoughts2018
That’s very cool. Numba and Cython work extremely well with virtually no
overhead or extra effort on my part, so jax doesn’t seem like it would buy me
much for most of my work. But I can imagine a lot of projects where jax woukd
be useful, and I plan to keep current on best practices for it.

------
ritoune
I've been using PyTorch, and the PyTorch 1.0 pre-release for a while now. I
adore it but don't really want to write C++ backends in production.

Anyone want to start working on Golang bindings for C++ PyTorch?

~~~
metahost
Curious, can SWIG[1] help automate the process?

[1]: [http://www.swig.org/](http://www.swig.org/)

------
Rafuino
Is this the version combining Caffe2 and PyTorch into one framework? Is LMDB
continuing as the default data load/store method?

~~~
thelastidiot
According to this [0] blog, it it:

PyTorch 1.0 takes the modular, production-oriented capabilities from Caffe2
and ONNX and combines them with PyTorch's existing flexible, research-focused
design to provide a fast, seamless path from research prototyping to
production deployment for a broad range of AI projects.

[0]
[https://developers.facebook.com/blog/post/2018/05/02/announc...](https://developers.facebook.com/blog/post/2018/05/02/announcing-
pytorch-1.0-for-research-production/)

------
pilooch
If someone from fair is around, what does this mean for caffe2 ? Is the new
torch c++ API a form a replacement for caffe?

------
Solar19
Agreed. I've always thought, however, that D would be the perfect language for
a new Python implementation.

------
dspoka
Is there some sort of pytorch 1.0 migration guide or does anyone know if there
is any breaking from .41 to 1.0 ?

~~~
shafte
Breaking changes are highlighted here:
[https://github.com/pytorch/pytorch/releases#breaking-
changes](https://github.com/pytorch/pytorch/releases#breaking-changes)

------
openbasic
What would be a good book and project to get started with this? Object
recognition? Product recommendations?

~~~
dangom
I would suggest going through the Fast.AI course [1]. It's an excellent course
to learn more about DL in general, and some of torch API. The downside is that
the course material was produced when PyTorch was still at 0.3, so some of the
API has changed since then.

[1] [https://course.fast.ai/](https://course.fast.ai/)

~~~
jph00
A new version for PyTorch v1 will be released next month, FYI.

------
dekimir
Hm, the Mac version of LibTorch is suddenly unavailable!? [0] I swear it was
available for download until a few days ago...

[0] [https://pytorch.org/get-started/locally/](https://pytorch.org/get-
started/locally/)

~~~
smhx
we are fixing this shortly. it was an oversight.

Edit: fixed links should be going live in a few mins, via:
[https://github.com/pytorch/pytorch.github.io/pull/141](https://github.com/pytorch/pytorch.github.io/pull/141)

------
amelius
> The JIT is a set of compiler tools for bridging the gap between research in
> PyTorch and production. It allows for the creation of models that can run
> without a dependency on the Python interpreter and which can be optimized
> more aggressively. Using program annotations existing models can be
> transformed into Torch Script, a subset of Python that PyTorch can run
> directly.

Isn't python bytecode simple enough that it can be run anywhere?

~~~
skybrian
Alternative implementations of Python don't seem very easy to write, so I'm
guessing the answer is no.

~~~
yorwba
Python is not very difficult to implement [1], the hard part is making it
fast. That's because every property access involves a lot of magic behind the
scenes, like __getattribute__, __getattr__ and the method resolution order.
And because Python is dynamically typed, that dispatch logic can't be compiled
away but needs to execute every time. (PyPy's JIT can probably speed it up,
but still needs deoptimization checks in case types change.)

[1] see [https://github.com/nedbat/byterun](https://github.com/nedbat/byterun)
for a Python implementation

------
punnerud
Feels good to see torch-1.0.0 when I do: "python -m pip install torch
torchvision --upgrade"

