
Show HN: Neuropod – Uber ATG's open source deep learning inference engine - vpanyam
https://github.com/uber/neuropod
======
vpanyam
Hey Everyone! I lead the development of Neuropod. Happy to answer any
questions

There's also a blog post that has more detail:
[https://eng.uber.com/introducing-neuropod/](https://eng.uber.com/introducing-
neuropod/)

Super excited to open-source it!

~~~
jaredtn
What will the continued support for this project be, given that Uber has
shuttered their AI Labs?

~~~
mysterEFrank
I'm a former Uber AI Labs member - they're different orgs. Begs the question
though, who at Uber will use this now?

~~~
mysterEFrank
(EDIT there are many projects at Uber using neuropods)

~~~
mysterEFrank
Also it's great.

------
damvigilante
How does this compare to ONNX
[https://github.com/onnx/onnx](https://github.com/onnx/onnx) in terms of
feature completeness/performance and what made you develop your own runtime ?

~~~
vpanyam
This is a good question. I want to write a more detailed post about this in
the future, but here are a few points for now:

\- Neuropod is an abstraction layer so it can do useful things on top of just
running models locally. For example, we can transparently proxy model
execution to remote machines. This can be super useful for running large scale
jobs with compute intensive models. Including GPUs in all our cluster machines
doesn’t make sense from a resource efficiency perspective so instead, if we
proxy model execution to a smaller cluster of GPU-enabled servers, we can get
higher GPU utilization while using fewer GPUs. The "Model serving" section of
the blog post ([1]) goes into more detail on this. We can also do interesting
things with model isolation (see the "Out-of-process execution" section of the
post).

\- ONNX _converts_ models while Neuropod _wraps_ them. We use TensorFlow,
TorchScript, etc. under the hood to run a model. This is important because we
have several models that use custom ops, TensorRT, etc. We can use the same
custom ops that we use at training time during inference. One of the goals of
Neuropod is to make experimentation, deployment, and iteration easier so not
having to do additional "conversion" work is useful.

\- When we started building Neuropod, ONNX could only do trace-based
conversions of PyTorch models. We've generally had lots of trouble with
correctness of trace-based conversions for non-trivial models (even with
TorchScript). Removing intermediate conversion steps (and their corresponding
verification steps) can save a lot of time and make the experimentation
process more efficient.

\- Being able to define a "problem" interface was important to us (e.g. "this
is the interface of a model that does 2d object detection"). This lets us have
multiple implementations that we can easily swap out because we concretely
defined an interface. This capability is useful for comparing models across
frameworks without doing a lot of work. The blog post ([1]) talks about this
in more detail.

The blog post ([1]) goes into a lot more detail about our motivations and use
cases so it's worth a read.

[1] [https://eng.uber.com/introducing-
neuropod/](https://eng.uber.com/introducing-neuropod/)

------
manicksurya
How is the performance of inferencing compared to the native serving solutions
provided by frameworks like TFServing etc

------
aloknnikhil
Found it interesting that most of the commits are under 1 contributor (OP).
Are you the most active contributor or was this an artifact of open-sourcing
it? Just wondering if you get hit by a bus tomorrow, what would we do? :)

Thanks for this, btw!

------
leonfedden
This looks great thanks for open sourcing it.

Have you had a chance to try running your models on baremetal devices such as
ARM cortex M4?

Is there a list of OPs that are supported or crucially, unsupported?

------
xbsd98
Are there any examples of demand forecasting ? Thanks.

------
j88439h84
How does it differ from pyro?

~~~
mysterEFrank
Neuropods can wrap pyro models

------
voz_
Consider using ONNX instead.

~~~
dang
This is not a good thread for a disgruntled grudge post.

~~~
voz_
That is fair. It does not contribute to curiosity, discovery, or good
conversation. I will remove the negative bits.

