
Cost of serving billions of images per month - ghoshbishakh
https://medium.com/p/f499620a14d0
======
no1youknowz
I like Unsplash. They and Pixabay are what I use for apps I have developed in
the past.

But simply, this floors me. I looked at their costs and with some developer
muscle, you could find savings such as:

\- Move Fastly to Cloudflare. They don't expressly say what the cost is for
that. But moving to CF would eliminate it.

\- Move Heroku to Digital Ocean. It's not difficult to create a fully
redundant solution.

\- Move from Imgix to having Golang micro services which handle the resizing
of images and use something like Belugacdn for the CDN. Beluga is $5k a month
for a PB. (Or some other cheaper CDN if you don't like Beluga, but damn...
imgix pricing)

I'm pretty sure that a savvy CTO could save at least $50k a month with a well
designed project that does this over many months and achieves the same result
and keeps the redundancy concerns for the small team.

I do realise however, why the team have done this. In the same position (with
very few resources) I would probably have done the same. But damn, when a cost
of a service gets up to a years salary ($120k) for a good developer. Time to
seek alternatives.

~~~
marcinzm
$50k/month comes out to $600k/year which is the cost around 2-3 engineers
counting overhead.

To manage infrastructure at that scale themselves they'd need someone who
knows DevOps. To write services at that scale they'd need someone who knows
backend engineering and micro services. Then they'd need to be able to have
24/7 on call rotations for when things break. And of course, like any rewrite
it seems like a "couple month" project until you get into the details and it
becomes a two year project. Soon you've got a full 6+ person team and are
paying them a lot more than $600k/year.

~~~
Jommi
600k year for 2-3 is only real in the silicon valley bubble. You could get
just as talented of a team from a Nordic Country for example and only pay
120kish

~~~
spockz
Realistically the costs for the company are higher. At least in The
Netherlands the total costs for the employer is between 2-3x because of social
security, health insurance, pension, housing/office costs.

~~~
jacquesm
I've been an employer in NL for close to 30 years and the 2-3x fully loaded is
not supported by any data that I'm aware of. 1.5 or so is more like it.

------
manigandham
What's the business model here?

We would never run with such expensive options for this functionality,
especially if it was a free/donations powered service. This company raised
$7.5M though: [https://techcrunch.com/2018/02/15/unsplash-simple-token-
seri...](https://techcrunch.com/2018/02/15/unsplash-simple-token-series-a/)

~~~
subpixel
They raised money from the crypto-coin outfit run by the former fab.com CEO.
At Fab and at his previous company it seems the modus oprandi was to raise
millions and then pocket as much as possible before the shit hit the fan.

~~~
pram
I ordered so much stuff from Fab back in the day! I thank the investors for
their largess

------
amanzi
So they must have some form of business model, right? All I can see on their
website is that everything is for free, even their API which can be used for
free without any limits or restrictions. And given the already huge Imgix
costs, abusing or over-using their API to request and resize images would
directly increase their overall costs. According to Crunchbase
([https://www.crunchbase.com/organization/unsplash#section-
fun...](https://www.crunchbase.com/organization/unsplash#section-funding-
rounds)) they raised $10m last year, but they must be churning through $3m+
per year on staffing and hosting costs.

I use and love the service but I can't see how it can last for more than a few
more years.

~~~
vincent_s
"He said it’s too soon to know exactly what that [business] model will be, but
it will involve blockchain technology and cryptocurrency" [1]

[1] [https://techcrunch.com/2018/02/15/unsplash-simple-token-
seri...](https://techcrunch.com/2018/02/15/unsplash-simple-token-series-a/)

~~~
ALittleLight
Is there a way I can short this company?

~~~
manigandham
I think the way to do this for VC companies is to sell them expensive stuff
until they go out of business. This post and many others like it seem to
indicate it works well.

------
Ayesh
Wow, these are some serious expenses!

I run a service that does ~30 million requests a month, and I only spend $15
on it. It's an apples to orange comparison, but looking at their costs
breakdown, I'm sure a couple engineers with a couple weeks can dramatically
reduce their costs.

For me, the biggest cost savings were at browser caching. With `immutable` and
far future cache expiration dates, webp images, etc, I could pull of the 30m
and growing stats without the server load going more than 0.40.

~~~
jeandejean
That's very tempting to do so but not knowing the exact details of the
processing involved, you probably make a mistake on how easy it would be to
optimize their costs.

It could be true, but would mean engineering time not dedicated at features
and likely more risk and more maintenance. It's often hard to admit but costs
are not that easy to assess and trade of are tricky.

------
oliwarner
The numbers being thrown around on this post are staggering. Not the
bandwidth, storage or or the number of images —they're all pretty average for
a small image host— but the casual tone that they're willing to throw money at
services rather than engineering some quite simple problems.

Seriously, I _know_ I'm not in their shoes but I know what they do, I know
popular options in these areas and I know they are literally throwing cash
away by not trying to in-house any of this.

The worst thing is they seem _proud_ about their decisions; that their current
situation has justified $100K/month expenses. I'm not the only person in here
that thinks they could be delivering the same —even better— service on a
fraction of their current budget, even after hiring. I think they should be
ashamed.

~~~
sudhirj
Doing a cost benefit analysis and deciding that using a service instead of
hiring and paying humans is a perfectly rational decision. I’ve downvoted
because I see a lot of vitriol like this and it’s exhausting to argue each
time that these decisions are carefully considered and work well for the
people making them.

This is no different from saying people should be ashamed because they pay so
much for food, then it’s much cheaper (almost free!) to grow it yourself in a
pot.

~~~
oliwarner
Being able to come to the decision that in-housing isn't good value, is _of
course_ possible. But they do it for everything. And they've been exhaustive
enough in their breakdown here that it's trivial to any experience developer
that they're spaffing money away.

They're not in an all-or-nothing situation. There are so many individual
elements that they could improve efficiency on and they're just not.

And I think cooking your own food vs going to a restaurant is a better analogy
(because it focuses on a service provision, not resource gathering). They're
currently eating out, seven days a week.

~~~
sudhirj
Even if they are currently eating out, seven days a week, that was a decision
to focus entirely on the product without any distractions.

[https://medium.com/unsplash/scaling-unsplash-with-a-small-
te...](https://medium.com/unsplash/scaling-unsplash-with-a-small-team-
fbdd55571906)

It's absolutely nothing for them to be ashamed of. We can all disagree with
the decision and say that we wouldn't do the same, but we shouldn't call it a
bad of shameful decision.

~~~
oliwarner
I don't understand your stance in the context of HN. We're here to make money,
celebrate people who can make money from the things they build, the services
they provide... And most importantly to learn from it when it goes right and
learn from the mistakes when it doesn't.

Unsplash is eating out seven days a week. But they don't have anything to show
for it. They're burning through cash. The product is basic. There's no magic,
nothing clever. It just is.

They're saying everything is great! I think it's a shambles; simultaneously
mediocre at both making money and being an excellent example of engineering.
Why is anybody _here_ celebrating or defending this result?

I maintain what I said. Unsplash _should_ be ashamed. They shouldn't trot out
numbers like this label them a success. It's a record of successive failures.

------
reggieband
I'm curious what their price-per-GB is from Fastly/Imgix ... the monthly total
doesn't really tell me the full story.

At the last place I worked I was in charge of a team that ran our image
service so I have a very solid understanding of the infrastructure around it
and what resources we had in place to develop/monitor/support/etc. It wouldn't
be hard to do a calculation and see if they are using money inefficiently.

I will say that most people don't really have any idea of the cost of simply
pushing bits. The majority of what I did was video distribution which gets
into eye-watering costs pretty quickly. Without knowing a bit more detail on
the precise volumes they are storing/moving I don't think it is fair to
criticize.

------
bdibs
I’m stunned that a business of this size would still be using Heroku.

Heroku’s sweet spot (to me at least) is for anyone with less than 3-4 servers.
After that, it just becomes so expensive there’s no rationalization in the
world that makes sense.

~~~
quickthrower2
Moreover what if Heroku goes out of business? Then they’re f’d. Hosting on
k8s, using Terraform or whatever would give them a lot of automation and
provider independence.

~~~
quickthrower2
I’m guessing I’m being grayed because I’m suggesting Heroku might go out of
business? Ok let me explain. That is one scenario, it’s unlikely but there
might be other similar scenarios that pose risk being tied in to one providers
stack. DDOS, TOS issues, simply out growing them, Heroku themselves having TOS
issues with their cloud. There are almost weekly posts on HN of “this major
service is down” and “that major service is down”.

I understand a scrappy startup needing to focus on growth and using the most
convenient tools, but once you’ve been around a bit it’s time to think of
boring stuff like what if stuff goes wrong. It might be technical, security
related, business related or political. Eliminating SPOFs is wise.

Happy for you to downvote this but if you do please drop a sentence to say why
I’m wrong. I’m happy to be proved wrong it helps me learn.

~~~
stickfigure
Downvoter (of your previous comment) here.

Your comment is not only bad, but outright dangerous. This is a company that
is burning through hundreds of thousands of dollars on hosting and staff with
no business model yet. The primary risk to this company is not their host
going away; the primary risk to this company is that _they won 't figure out
how to make money_. Worry about optimizing your serving infrastructure after
you have a business that you know is worth saving.

~~~
quickthrower2
Cool thanks it’s nice to hear the story behind the downvote. I agree you’d
need to be flying their plane so to speak to know the reasoning behind their
business decision. I still forget there are businesses that burn through that
much cash looking for product market fit etc. it’s not the European way so I
often don’t correlate spending 100k/m on hosting with fighting to stay alive.
Anyway I was sort of making a general point about lock in. It may not apply to
this company.

------
idlewords
$99K/month in hosting costs and no visible source of revenue...

I think this baby is ready for an IPO!

~~~
danieltillett
The losses are not large enough for an IPO. They need to scale at least 1000x
to be IPO ready.

------
maz1b
I'm a little surprised as to how you guys afford to give such a quality
service away for free. Really cool of you guys to share this data though.
Wishing you guys continued success!

------
selterdurst
It’s interesting to see how so many people here focus on specific line items
on their costs. What it comes down to is management focus. It’s a lot easier
to switch a CDN than to lay off two engineers as needs change.

I Do wonder why they don’t charge for API at all as even currency APIs charge
and that is effectively public knowledge.

Very interesting though as this is some serious scale indeed...

------
jon-wood
Everyone here seems to be screaming about how high their Heroku bill is, but
I’m more interested in how they keep their New Relic bill so low. I find that
New Relic are fine early on, and then the costs explode as you scale up and
there’s no way to justify spending huge amounts of money on their platform.

------
systemBuilder
10(+) billion images served monthly for $100k = 1/1000th of a cent per image.
This is not bad but I think Netflix is 2c a gigabyte; you are 10c (I'm
assuming 100k per image).

~~~
sudhirj
Netflix sends their own cache machines to tens, if not hundreds, of ISPs and
interchanges all over the world, each of which have multiple terabytes of
storage and act as cache nodes. I don’t think Unsplash is going to do that.

------
blissofbeing
Now I'm interested to see what their business model is. It seems like they
have a lot of money to throw at services instead of developing it themselves,
so I'm interested in their ROI or opportunity cost decisions now.

------
lwansbrough
Holy cloud. Well I guess the silver lining is they're leaving lots of room for
cost improvements...

------
ogirginc
Does anyone know their bussiness model?

------
jeandejean
That's a great blog post. Very transparent. I'm especially glad he's willingly
giving up cheaper alternative for having more value from his services.

I just would have liked some more details on what it means technically... :-)

~~~
GordonS
Well, it's transparent about their costs, but I would have liked a little
transparency about how they actually make money

~~~
jeandejean
Very true but I suppose we can't have it all, plus maybe it's sensitive
information.

------
mharroun
I'm surprised that at that scale you are still not building some core
functions internally.

Though it makes sense, buy over build is almost always the best choice early
stage. It's hard to slow down product velocity to fix things that work fine
when costs start to add up. (Also it's easy to say "do you want to save 3k a
month or deliver features x,y, and z" and let costs slide)

------
marknadal
First time hearing about unsplash, thank you!!! This is awesome for us OSS
devs.

------
busymom0
Is there any particular reason they are still using Heroku instead of
something like DO or AWS+S3? They can probably save a lot of money that way.
Unless Heroku is offering them some good deal?

~~~
ron22
They tell you in the blog post.

"We continue to use Heroku as our main web platform. Despite its premium cost
over AWS, Azure, and Google Cloud, Heroku’s built-in deployment and
configuration tools allow our team to move faster, more confidently, and more
reliably.

As we’ve detailed previously, the alternatives would undoubtably be cheaper on
paper. But in reality, the increased simplicity and freedom offered by Heroku
for a small, product-focused team is a major cost savings advantage."

~~~
scarface74
I’m far from a Devops guy. But deploying to AWS using their native tools are
Azure using Microsoft’s tools is dead simple.

~~~
paulddraper
Yeah something doesn't add up.

I'll believe Heroku is easier, but can it be _that_ much easier? For people
spending piles of cash on their webscale?

~~~
photonios
I work for a company that runs multiple major websites pulling in tens of
millions of users a month. We run on Heroku. We made that decision when we
didn't have a single user. Right now we're a team with 25 engineers.

Every time we need to re-negotiate our contract with Heroku we ask ourselves
whether we should move away from Heroku or not. So far, the answer has always
been no. Complexity would go up and the time spent on DevOps would go up.
Right now, any idiot can do a deployment, or a rollback. Most of the things we
need are taken care off for us by Heroku and it doesn't cost us any effort. We
use Heroku Postgres for our database and we get point-in-time restore for
free, without doing anything. We can upgrade our database or create a new
follower in minutes. It just requires a couple of clicks. On top of that, all
the monitoring is taking care of by Heroku. It just gives us a nice dashboard.

As for costs, on an negotiated Enterprise contract, it's not that bad. If we'd
move to AWS or DO for example, we might save a little bit, but the extra
engineering resources would quickly make those savings useless. Even if it's a
bit more expensive to run on Heroku, it's still cheaper than cobbling together
all of those features on another platform.

We've made a couple of small engineering investments to get features that
Heroku doesn't offer. For example, we built a simple auto-scaler for dyno's
that consume from RabbitMQ. This helps a bit to keep our costs down and
respond to massive spikes quickly.

We run our web servers, database, caching, data pipeline etc on Heroku.
However, we are not _that_ dependent on Heroku. We use Docker for deployments,
so we can easily reproduce what's happening and it reduces our vendor lock-in.

I hope this explains a little why some people might choose to stick to Heroku.
It works for us, it might not work for you or anyone else.

~~~
scarface74
_We 've made a couple of small engineering investments to get features that
Heroku doesn't offer. For example, we built a simple auto-scaler for dyno's
that consume from RabbitMQ. This helps a bit to keep our costs down and
respond to massive spikes quickly. We run our web servers, database, caching,
data pipeline etc on Heroku. However, we are not _that_ dependent on Heroku.
We use Docker for deployments, so we can easily reproduce what's happening and
it reduces our vendor lock-in._

I missed this part. If you are okay with using managed services, why are you
doing so much “undifferentiated heavy lifting”? Setting up autoscaling based
on queues, managed RabbitMQ based messages (Amazon MQ), and without knowing
what your data pipeline consists of, probably that too could all be done with
managed services.

~~~
photonios
Don't imagine anything fancy here. This is roughly 100 lines of Python code
that scales based on some tresholds. It took a day or two to develop, test and
deploy.

However, we recently found a managed service to do this for us so we'll be
switching to that as soon as we can.

If we'd have to spend more engineering resources on our infrastructure than
the occasional simple tool, then we'd consider switching away from Heroku. The
benefits still outweigh the potential engineering effort of not running on
Heroku.

------
jopsen
Curious, anyone who knows what the business model is?

The service seems pretty cool :)

------
umen
how do they monetize ? i can't find the monetization ...

~~~
amolo
Maybe API partners

------
tschellenbach
I work with these guys since they are customers of Stream. Super solid team,
really respect how they have such focus in their execution.

------
OrgNet
it would be nice if each category would line up in this graph: [https://cdn-
images-1.medium.com/max/2400/1*Bvw2zcdE146WaXB7i...](https://cdn-
images-1.medium.com/max/2400/1*Bvw2zcdE146WaXB7io3tZQ.png) ... kinda hard to
read as-is

~~~
Insanity
looking at that image fullscreen, the bars seem to be moving to me.
Interesting effect

~~~
OrgNet
visualization fail though

------
flywithdolp
I'm using unsplash for a while, They have the best quality free to use
pictures on the internet.

As someone who appreciate creative marketing i like how they generate millions
of backlinks by asking you to link back to their site and credit the author.

In few years they became among the most popular sites in their niche, a lot
thanks to their creative growth approach.

------
anonyxyz
How does unsplash make money?

------
sytelus
TLDR; $98K/mo. Year ago they were 10X smaller and used to cost only $17K.

~~~
tschellenbach
Sounds like they were focused on that 10x growth :D

------
mychael
It's cool that they're sharing this information, but using Heroku at this
scale is wasteful and fiscally irresponsible.

------
dna_polymerase
> We continue to use Heroku as our main web platform. Despite its premium cost
> over AWS, Azure, and Google Cloud, Heroku’s built-in deployment and
> configuration tools allow our team to move faster, more confidently, and
> more reliably.

This is just bullshit, really. I'd buy this excuse from a service like
Snapchat/Facebook in their early years, but unsplash is nothing that changes
dramatically all the time.

Every single point in their analysis just shows how they like to waste money.
Imgix as the main image CDN? Start serving images using thumbor and a cheap
CDN like BunnyCDN or KeyCDN.

They could easily save 60% on their monthly bill. That would enable them to
hire 1 or 2 engineers capable of maintaining a few dozen servers on AWS, DO or
GCP.

~~~
forrestthewoods
> They could easily save 60% on their monthly bill. That would enable them to
> hire 1 or 2 engineers capable of maintaining a few dozen servers on AWS, DO
> or GCP.

If they reduce their bill and then hire engineers to do the same job they've
gained nothing other than an increased headcount for no discernible benefit.

~~~
manigandham
If they're not really adding features then its the same staff. Not every
change needs a new developer.

