
Cortex: Deploy machine learning models in production - ChefboyOG
https://github.com/cortexlabs/cortex/tree/v0.9.0
======
kuu
How does this work under the hood? Is the model loaded every time it receives
a request? Is it run in a docker or a lambda? How does it work after
"uploading it" to amazon?

~~~
ospillinger
Each model is loaded into a Docker container, along with any Python packages
and request handling code. The cluster runs on EKS on your AWS account. Cortex
takes the declarative configuration from 'cortex.yaml' and creates it every
time you run 'cortex deploy' so the containers don’t change unless you run
'cortex deploy' again with updated configuration. This post goes into more
detail about some of our design decisions:
[https://towardsdatascience.com/inference-at-
scale-49bc222b3a...](https://towardsdatascience.com/inference-at-
scale-49bc222b3ad1)

~~~
kuu
Thank you!

------
oli5679
If your model can be exported as PMML, this is really nice. Fast, minimalist,
battle-tested and with very clean API.

When I've tested, it's up to 10x faster than Flask + serialised model object
and uses far less CPU resources.

Plays nicely with lightgbm and Xgboost.

[https://github.com/openscoring/openscoring](https://github.com/openscoring/openscoring)

------
isubasinghe
This is basically my startup idea that I worked on for a while now
([https://aiscalr.isub.dev](https://aiscalr.isub.dev))

Looks like I am going to have to scrap that entire project now, seems
pointless to keep working on it given how similar this is.

~~~
sixhobbits
Similarity should be taken as validation, not a negative thing at all.

------
sails
How does this compare to MLflow [0]?

Considering MLflow has a few components, I suppose you are building something
closer to MLflow Models? How do they compare?

[0]
[https://mlflow.org/docs/latest/index.html](https://mlflow.org/docs/latest/index.html)

~~~
ospillinger
From the MLflow Models docs: "An MLflow Model is a standard format for
packaging machine learning models that can be used in a variety of downstream
tools—for example, real-time serving through a REST API or batch inference on
Apache Spark. The format defines a convention that lets you save a model in
different “flavors” that can be understood by different downstream tools."

Cortex is what they are referring to as a downstream tool for real-time
serving through a REST API. In other words, MLflow helps with model management
and packaging, whereas Cortex is a platform for running real-time inference at
scale. We are working on supporting more model packaging formats and I think
it's a good idea to support the MLflow format as well.

------
lettergram
Was discussed three months ago:
[https://news.ycombinator.com/item?id=20579166](https://news.ycombinator.com/item?id=20579166)

------
ecnahc515
Not to be confused with weaveworks/CNCF Cortex project for high scale
Prometheus monitoring
[https://github.com/cortexproject/cortex](https://github.com/cortexproject/cortex).

------
punnerud
Seems to work with: Pytorch, TensorFlow, Keras, XGBoost, sklearn

~~~
ospillinger
Yes, Cortex uses ONNX Runtime
([https://github.com/microsoft/onnxruntime](https://github.com/microsoft/onnxruntime))
under the hood so any model that can be exported to ONNX can be deployed.

~~~
solidasparagus
Is it only able to handle ONNX models? That's a pretty massive limitation
compared to a hosted SageMaker endpoint.

~~~
vishalbollu
Contributor here - Cortex supports Tensorflow saved models in addition to
ONNX. PyTorch support is on the roadmap. Do you have specific frameworks in
mind that you would like Cortex to support?

~~~
solidasparagus
Perfect. Nothing in particular other than TF.

------
kevinmershon
Unfortunately, a somewhat popular Clojure library for machine learning on
GitHub is also called Cortex, because this is going to make discussing machine
learning APIs in the context of Clojure that much more confusing.

~~~
notus
Last commit was like 2 years ago...

------
TaupeRanger
Stop naming things single word neuroscience terms. There are like 50 projects
called "Cortex".

------
mzanchi
Calling it an alternative to SageMaker might be a bit misleading, as SageMaker
is also a platform for training the models in automatically allocated EC2
resources, even on spot instances.

~~~
dang
We've changed the title from "Cortex: An open source alternative to SageMaker"
to the page's own title, as the HN guidelines request.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

------
manojlds
Sagemaker has notebooks, training and serving. This seems to be only about the
serving.

