
Launch HN: Numericcal (YC.S18) – Lifecycle Management for ML Models on the Edge - radoye
Hey HN,<p>We’re Ranko and Shaoyi, co-founders of Numericcal (<a href="https:&#x2F;&#x2F;numericcal.com" rel="nofollow">https:&#x2F;&#x2F;numericcal.com</a>). We’re building developer tools to simplify building, optimizing and maintaining Machine Learning models for embedded, mobile and IoT applications.<p>This platform is the result of our own struggles while helping on two ML powered projects; one using Computer Vision and one Voice Recognition. In both cases running models in the cloud was too expensive and slow for a good user experience. Moreover, it was not clear whether the training data our Data Science team had, at the beginning of the project, was representative of the data to be encountered in production.<p>The solution for cost and speed was to run inference on some end-device. However, we did not know which device would be feasible, nor did we know what model the Data Science team would end up with.<p>We initially considered two options:<p>* Postpone working on ML related software tasks until the Data Science team figured out what they wanted to do. Unfortunately, this meant that we could not parallelize the development. Moreover, the Data Science team would not get any feedback on the model runtime performance until we integrated them towards the end.<p>* Make a bunch of assumptions about models that the Data Science team would use and implement something. This would allow us to work in parallel, but we would be running the risk of having integration issues if those assumptions turned out to be wrong.<p>We looked for tools online to solve this but we couldn’t find anything comprehensive. So we decided to build our own! We settled on a design that packages ML models into (very primitive) serializable objects, encapsulating all the information and supporting files to use the models. For example, an object detection model would package the DNN graph, shape of the input and output tensor, bounding box anchors, etc.<p>This gave us what we wanted. On the software side we could write against the “interface” of the package. During initial development, we simply packaged dummy DNN models that always returned the same value. Later, the Data Science team would simply drop in a new package and everything just worked. One more victory for abstraction!<p>Since then, we’ve added a number of features around the packaging system. We can now add models during compilation or remotely through a cloud service and a user-friendly GUI. Models are versioned and report basic performance metrics to the cloud so Data Science teams can get feedback and guide model exploration. Remote delivery also allows sending different models to devices with different compute capabilities. We also added the possibility of running models on different runtimes (TensorFlow Mobile, TensorFlow Lite, our own library, Caffe2 is coming soon). Finally, we wrapped the app side code into ReactiveX API to make working with multiple models asynchronously easier.<p>Today we’re releasing the Numericcal Cloud Tools and Android SDKs in Beta. You can read about the system in more detail at (overview.workspace.numericcal.com) and check out demos (<a href="https:&#x2F;&#x2F;github.com&#x2F;numericcal&#x2F;BetaDemoApp" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;numericcal&#x2F;BetaDemoApp</a>).<p>Machine Learning on mobile and IoT devices is yet to gain wide adoption. Making this transition will bring personalization of mobile apps and automation of business and industrial processes to a whole new level. We’re currently working on projects for automatic damage assessment on vehicles and preventative maintenance of mobile assets.<p>We hope these tools will add to the momentum by speeding up development iterations for other teams, as they did for us. In the long term, we plan to open source edge integration and packaging SDKs. We will also open up more of our cloud-hosted functionalities, such as automatic model benchmarking, deployment, model compression and runtime optimization as services.<p>We’re eager to hear comments, ideas and your experiences in building Edge applications with ML. Thanks for reading!
======
rkirkendall
Very cool and congrats on the launch. I see your demo is on android, are you
doing anything with iOS / CoreML? I put up a similar project for OTA updates
on iOS using CoreML to hot swap out device side models
[https://github.com/rkirkendall/MLEdgeDeploy](https://github.com/rkirkendall/MLEdgeDeploy)

~~~
radoye
Yeah, as the cloud/edge interface settles we plan to build SDKs for pulling
models from iOS, GNU/Linux through Swift, Python and C#.

Let me check your repo.

------
zitterbewegung
Your project sounds really interesting. Machine learning models that can be
updated and treated more like packages will really give people an edge over
people who can't perform that task.

I made a project called mms2concept. What it would do is given a picture taken
by a phone (or anything that could send an MMS really) and it would attempt to
identify the image based on the picture. I used Clarfai and Twilio to do this.

I think with very resource constrained edge devices either you would have a
much more powerful "basestation" style deployments where you have an arduino
or similar device collecting edge data and sending it back to that
"basestation".

~~~
radoye
Can you share some writeup or repo for your project?

We internally have an auto-migration tool (currently very primitive) that can
learn from cloud hosted model and cache it locally on the device (that's what
this infra & packaging was built for ;)).

~~~
zitterbewegung
A writeup is at [https://steemit.com/twilio/@zitterbewegung/mms2text-let-
your...](https://steemit.com/twilio/@zitterbewegung/mms2text-let-your-
computer-look-at-your-message-pics-so-you-don-t-have-to).

I have a repository of the software at
[https://github.com/zitterbewegung/mms2text](https://github.com/zitterbewegung/mms2text)
.

~~~
radoye
For some reason, your quick demo reminded me of this awesome MS project:
[https://www.microsoft.com/en-us/seeing-ai](https://www.microsoft.com/en-
us/seeing-ai)

Though we would be pushing it the boundary with 3+ DNNs on mobile, adding
speech2text and text2speech to your app would make it an interesting addition
to Seeing-AI, IMO.

------
jamesonthecrow
Congrats to the Numericcal team on the launch! It's great to see new runtimes
coming out to improve performance specifically on Android. It's been a real
pain for us to get things up to par with Apple devices running Core ML.

We’re building something similar but focused on existing Core ML/Tensorflow
runtimes if anyone is looking for similar management features for iOS, check
out [https://fritz.ai](https://fritz.ai). (Full disclosure, I’m one of the co-
founders).

~~~
radoye
Exactly, there is a lot of potential on the Edge, but dealing with
fragmentation is really painful.

------
mendeza
How easy is it to build custom architectures like segmentation and object
detection networks? I am working on a segmentation coreml tools and apple
coremltools only coverts a selection of layers.

~~~
radoye
That depends on your choice of training framework and runtime engine (TF
Lite/TF Mobile/Numericcal/Caffe2/etc.). We wrapped multiple runtime engines to
provide as much flexibility as possible, quickly. For example, some layers
available in TF Mobile are not available in TF Lite (and TF Lite is slower,
for now!).

That being said, if there is no converter between the training format and
runtime format, you're out of luck (for now). Our runtime (targeted primarily
at Qualcomm SnapDragon SoCs) supports the most common layers and some more
exotic ones that our users needed. For other engines we simply pull the
standard package that vendors provide.

------
sidcool
Congratulations for launching. The idea is very promising. Good luck!

------
tixocloud
Congrats on the launch. We’ve pretty much experienced the same pains as you
guys and are building a similar platform but targeted more towards web
deployment. Best of luck!

~~~
radoye
I'm quite interested in hearing how things turn out. Let us know if you
write/release something.

