
Benchmarking the Major Cloud Vision AutoML Tools - rocauc
https://blog.roboflow.ai/automl-vs-rekognition-vs-custom-vision/
======
RcouF1uZ4gsC
This is amazing content marketing, similar to Backblaze hard drive stats.

It was well-written and very informative. I enjoyed reading it and learned
something new and potentially useful. And it brought attention to a potential
pain point (trying to train and infer models on multiple platforms) and
suggested their product to help(roboflow). It got the marketing message across
without being irritating or obtrusive. A win-win situation for both reader and
company.

Again, great writeup. I wish more content marketing would be this engaging and
useful.

~~~
rocauc
Really appreciate it; we had fun writing it. More of this to come.

------
mlthoughts2018
The thing with evaluating these services is that business people or product
managers don’t understand that accuracy on benchmarks doesn’t map well to
accuracy on _your_ problem. If you have a wacky data distribution (hint: you
do), say like a collection of photos all taken inside the same building, you
can’t generalize to consider benchmark stats.

On top of this, these services charge you for usage, not for _accurate_ usage.
This is a major issue that so many people overlook.

What does it matter if it’s cheap per request? That only helps you if accuracy
per request is very high. Otherwise you’re paying for cheap garbage, or in
some use cases you must grow requests much larger to overcome errors per
request, like if you’re trying to get a bulk of labeled data via these
services.

Most use cases are still better off hiring ML engineers who understand how to
evaluate accuracy for the unique business use case, fine tune or train a
model, and do it in house (or at least give you deep assurances of the rare
cases when the big cloud services actually are cost effective to use).

For most use cases, you’re wasting your money trying for ML-as-a-service like
this.

~~~
yeldarb
There's a huge gap between paying a couple hundred dollars for a pretty good
model and investing hundreds of thousands (or millions) in hiring a team.

Certainly if you're building something mission-critical, like self-driving
cars, you want to get the best possible performance (regardless of price).

But there's a whole other set of use-cases where "pretty good" is good enough
(eg finding people sharing your company's logo in social media posts).

~~~
mlthoughts2018
> “finding people sharing your company's logo in social media posts”

that seems very different than your comment describes.

I agree _hobbyists_ may use these services. If a business is looking for
spending “a couple hundred dollars” on a significant ML problem, that business
is living in a fantasy and needs a reality check.

There is really not a market for low end cost but high specific task accuracy
_business_ problems.

Also your comparison with hiring a team is a very false dichotomy. Hiring a
team may cost you $1MM but it’s amortized over all the work and projects they
do. It’s not $1MM _just_ for the in-house equivalent of this one AWK
Rekognition task.

------
yeldarb
Biggest surprise for me was that they can’t train on COCO. Would have thought
their performance would be a major part of the marketing considering how
prominent that benchmark is in the research community.

~~~
junipertea
I guess research community would not use commercial products? The number of
classes and datapoints is quite extreme for a “common” use case, usually you
would want to train it by yourself.

------
kuzee
Just learned a lot about the different hosted ML options, thanks for the
write-up with backing data. How much did this cost you overall?

~~~
rocauc
The 'hard costs' were the training time for each model ($125) and conducting
inference on the valid set ($15) for each.

Now, we had a few failed experiments like trying to run the COCO dataset
through each tool. And none of this includes the implicit cost of the computer
vision engineering time.

------
jorams
I think there's a number missing under Google Cloud AutoML Vision Inference
Cost:

> Our tests yielded x predictions per second

