
Taking ML to production with Rust - LukeMathWalker
https://www.lpalmieri.com/posts/2019-12-01-taking-ml-to-production-with-rust-a-25x-speedup/
======
sandGorgon
Although I am a python guy, if I had to bet on an alternative language here -
its going to be Swift.

The amount of evangelism money that Google is spending on Tensorflow is
staggering. Its not just t-shirts, but an entire ecosystem - TensorBoard.dev,
Tensorflow Privacy, TF Enterprise, Explainable AI Beta, free TPU on Google
Colab etc.

And all of them will soon be backed using Tensorflow Swift
([https://www.tensorflow.org/swift](https://www.tensorflow.org/swift)). This
is MNIST for Swift Tensorflow - [https://github.com/tensorflow/swift-
models/blob/master/Datas...](https://github.com/tensorflow/swift-
models/blob/master/Datasets/MNIST/MNIST.swift) . Mind you this is runnable on
Google Colab using _TPU hardware acceleration_ in swift.

Swift vs Rust is a very political thing...but previous discussions on HN have
been tilted in favor of Swift [1]

And we havent even touched on the elephant in the Room - Swift may soon hook
into OSX Metal (which is arguably the _only_ way to do GPU AI on macbooks now
that the nvidia-Apple CUDA divorce is official [2] ). You can already call
Metal from Swift...but not write shaders directly IMHO.

[1] [https://cloud.google.com/explainable-
ai/](https://cloud.google.com/explainable-ai/)

[2] [https://gizmodo.com/apple-and-nvidia-are-
over-1840015246](https://gizmodo.com/apple-and-nvidia-are-over-1840015246)

~~~
rob74
While Swift may be the "latest and greatest" language with official support,
if you look at
[https://www.tensorflow.org/api_docs](https://www.tensorflow.org/api_docs),
it's only one of several (the others being C++, Java, JS, Go and of course
Python). Plus, there are community-supported bindings for C#, Haskell, Julia,
Ruby, Rust and Scala). Just to make the picture complete...

~~~
throwlaplace
Swift for tensor flow aren't bindings - it's a ground up rearchitecting of
tensorflow

~~~
pjmlp
For Mac OS only apparently.

~~~
throwlaplace
not true

[https://github.com/tensorflow/swift/blob/master/Installation...](https://github.com/tensorflow/swift/blob/master/Installation.md)

~~~
pjmlp
Having a package available for a specific OS doesn't mean it is usable,
specially when it still requires conditional compilation to import either
Darwin or Glibc, instead of Foundation.

And I don't see any Windows there.

Then there is the whole set of libraries that only compile on Apple platforms,
no IDE, no Playgrounds and so on.

~~~
throwlaplace
>specially when it still requires conditional compilation to import either
Darwin or Glibc, instead of Foundation.

you don't need to compile from source - binaries are distributed (if you
actually read my link instead of assuming it was just github source you would
see that).

>And I don't see any Windows there.

yes they're aware of this and it's being worked on

>Then there is the whole set of libraries that only compile on Apple
platforms, no IDE, no Playgrounds and so on.

you're moving goal posts in order to be able to have an axe to grind.

your comment was

>For Mac OS only apparently.

the fact that there's a runtime that runs on linux shows that that was
incorrect.

>set of libraries that only compile on Apple platforms

i don't know what you're talking about here - s4tf libraries compile on all
platforms e.g. [https://github.com/eaplatanios/swift-
rl](https://github.com/eaplatanios/swift-rl)

>no IDE, no Playgrounds and so on.

i don't see how xcode being developed by apple (not the s4tf team) has
anything to do with whether s4tf is multiplatform.

~~~
pjmlp
Not moving goalposts at all.

Swift is only usable in Apple OSes.

Making a tiny subject of its ecosystem available outside of Apple platform
doesn't make it usable for something that pretends to replace Python ecosystem
on data science.

------
captain_price7
I was a little surprised to see that instead of directly benchmarking
prediction performance on a local machine, author chose to go through that
complicated web-server route. I saw author saying this is nothing more than a
remote RFC, still, a benchmark should try to remove unrelated factors as much
as possible. I would be interested to see how the results compare when this
web-service layer is removed.

~~~
cwyers
Serving model predictions via an API is a pretty common usecase for putting
data science models in production. If Python's web service layer is slower
than Rust's, that's a real concern even if it's not directly related to model
prediction.

------
k__
Good!

I think Rust will eat more and more of the stack.

It has similar performance as C/C++, but feels more higher level. It's a drop-
in replacement for C and is developed by a company that devs like.

~~~
madmax96
Rust is not a drop-in replacement for C, it’s a drop-in replacement for C++.

~~~
pnako
It's a drop-in replacement for neither since it's an entirely different
language. C++ went to some great pain to be a drop-in replacement for C; you
can "drop in" a C++ compiler in your toolchain and still compile your C code
(mostly).

~~~
littlestymaar
This was kind of true till C99, but not so much anymore. _restrict_ is the
most obvious offender but far from the only one[1].

[1]:
[https://en.m.wikipedia.org/wiki/Compatibility_of_C_and_C%2B%...](https://en.m.wikipedia.org/wiki/Compatibility_of_C_and_C%2B%2B)

------
wdroz
The 25x speedup is because the gRPC python implementation is very slow [0].

[0] -- [https://performance-dot-grpc-
testing.appspot.com/explore?das...](https://performance-dot-grpc-
testing.appspot.com/explore?dashboard=5636470266134528)

~~~
kbumsik
Yeah, by skimming the article, it sound like making gRPC microservices 25x
faster, not ML.

~~~
amelius
Of course the actual ML is not going to be faster in Rust, because the main
tensor operations as used from Python are already written close to the metal.
The cost of these operations is 99% of what it takes to run ML. Optimizing the
remaining 1% is just not going to have a noticeable effect.

~~~
jstrong
I've spent a lot of time writing python ML code, and I think 99% (of time
inside fast C/cuda code) is much higher than most programs achieve. Off hand
I'd say 80% is average and 90% is good (usually comes after a performance
refactor). Also, one aspect that people often overlook is that optimizing
compilers can often combine or vectorize operations in ways that are an order
of magnitude faster than the same operations performed consecutively in, say,
pure numpy. in that scenario, even though each numpy operation is very fast
for what it is, the python program that combines a bunch of them ends up
running much slower.

------
yubozhao
I think it is also worth examining what type of production load and its
importance within the business.

For mission critical production usage, I think to use high performance
systems/languages is pretty good starting point. With the assumptions that you
update your model conservatively(not often), there are enough engineering
resources to maintain and 'port' models from python, since most data
scientists are trained with python. I think in this type of workload and
context, it makes sense use Rust for deployment.

Another type of workload I think it is much more common. It is the
experimental projects that data scientists are trying to discover does those
have enough ROI to be part of the production system. Those projects and
deployments are requiring a quick turn around on iteration cycle. I am not
sure Rust or even Swift are good tools, when typical data scientists are not
well versed in those. Not to mention, usually in this setting, they don't have
a lot of engineering resources they can use. Python is still the go to option
for this type of work.

I think the article has the right intention, speed up ML production. For the
experimental work setting, I think we can have the cake and eat it too. Data
scientists, still use python and generate production ready deployment service
without help from engineers.

We create an open source python lib/platform called
BentoML(www.github.com/bentoml/bentoml). BentoML makes it easy to serving and
deploying ML models in the cloud, from ML model to production API endpoint
with few lines of code. You can try it out at this Google Colab
notebook([https://colab.research.google.com/github/bentoml/BentoML/blo...](https://colab.research.google.com/github/bentoml/BentoML/blob/master/guides/quick-
start/bentoml-quick-start-guide.ipynb))

BentoML works with multiply ML framworks(Tensorflow/fastai/pytorch/etc) and
could generate different distribution formats (docker/AWS Lambda/CLI/Spark
UDF) for your serving need. We also support custom runtime backend. Feel free
to ping me or ask questions in our slack channel. We are pretty active there.

------
lewq
Nice article!

------
psv1
Rust can never replace Python as a front-end because it's compiled and not all
that flexible or user-friendly.

Rust can't currently replace C or C++ at the back-end because it doesn't have
good enough support for GPU computing yet.

~~~
excerionsforte
"The Rust ecosystem is indeed rich in ML crates - just take a look at what a
quick search for machine learning on crates.io returns. No need to go and
rewrite everything from scratch: I picture linfa as a meta-package, a
collection of curated algorithm implementations from the Rust ecosystem. The
first stop for your ML needs, as scikit-learn for Python."

Rust doesn't have to replace anything. It can be used to create complementary
packages especially ones that use C/C++. The point of introducing Rust into
your ecosystem is for safety + performance which is hard to achieve without
discipline.

~~~
psv1
> Rust doesn't have to replace anything. It can be used to create
> complementary packages especially ones that use C/C++.

I'm not sure that I understand correctly - are you suggesting that people use
Rust as an additional layer between Python and C++?

~~~
mcherm
The original article is proposing that the existing C and C++ libraries be
rewritten in Rust for performance reasons (while continuing to use Python to
access the functionality).

~~~
blub
I guess you meant memory-safety reasons because Rust is attempting to be as
fast as C and C++, not the other way around.

~~~
cogman10
Depends on the domain.

For single threaded applications, as fast is the goal. There is potential for
it to be faster due to stronger restrictions around pointers.

Multithreaded code is slated to be faster mainly because rust provides safety
for patterns that would be difficult to do right in C++ (see servo css
handling).

That is to say, you could do the same thing in C++, but getting it right is
harder to do.

~~~
blub
Concurrency patterns that are difficult to execute in the decades old codebase
of Firefox, which also happens to be a browser aka nightmare of complexity.

The same environmental conditions don't apply to other software and it's
perfectly possible to write blazing fast multithreaded software in C++. In
fact that's SOP.

------
xiaodai
Rust will never work except in niche areas because it doesn't allow for easy
experimentation and does not provide an iterative workflow like Python, R, or
Julia.

I think the future is Julia, as it's as easy to write as Python, is
interactive, and can be compiled for fast speeds!

It will just takes time for Julia to have as many libraries, but at some point
the tipping point will be reached, and Julia will just eat Python's lunch.

Python was dying before the latest wave of data science and machine learning
craze saved it anyway. There are pretty good reasons to believe Python (and
Ruby) will slowly wither away. I am careful not to savy that they will die
out, because they won't as some computing problems do not require so much
speed and hence Python and the like can still pass mustard. But in a post-
Moore's-law-world, data is getting bigger, the demand for computing is larger,
but programs aren't getting faster, and there is a limit to how much you can
optimise Python. Eventually, a crop of speedy and easy to write programming
languages will rise up and eat Python's lunch. The early signs are there with
tools like Julia.

~~~
Ygg2
> Rust will never work except in niche because

It warms my heart to see people don't read TFA. Like at all.

From TFA:

> But it was important to clean the air of any possible misunderstanding
> before getting started. I don’t believe Rust will replace Python as the
> language of choice for ML - it just isn’t happening, it’s not a thing,
> neither today or tomorrow.

~~~
hu3
What's wrong in expanding on a topic that was barely mentioned in the article?

Your parent wrote detailed reasoning that's not present in the article.

Also this type of uncharitable comment is against Hacker News guidelines:
"Please don't comment on whether someone read an article".

[http://news.ycombinator.com/newsguidelines.html](http://news.ycombinator.com/newsguidelines.html)

~~~
fnord123
There can be no discussion with GGP since he obviously didn't read the
article. They think the article will claim Rust supplants Python. They think
there is a performance advantage in using Julia instead of Python (but the
Python code is just calling C, C++, and Rust as described in the article). GGP
thinks there will be a tipping point where Julia has more libraries than
Python but this is also addressed in the article.

There is no expanding on topics barely mentioned in the article. As far as GGP
is concerned there was just a headline and they Kanye'd with "Julia is the
best". As far as I care, mods can delete the comment and subtree.

