
How much is Spotify paying Google Cloud? - dmytton
https://medium.com/@davidmytton/how-much-is-spotify-paying-google-cloud-ebb3bf180f15
======
i0exception
You're making a big assumption here - that they don't batch events before
publishing to Google's Pub/Sub system.

The blog post says "We also added queuing and batching in the client. (This
wasn’t first time that we needed to roll our sleeves up and reimplement a
Google Cloud API client: in another project we implemented a high performance
client for the Datastore API.)"

~~~
dmytton
Batching doesn't make any difference to the pricing:

> Batched calls are accounted for as multiple operations.

[https://cloud.google.com/pubsub/pricing](https://cloud.google.com/pubsub/pricing)

~~~
patrickmclaren
Note by "queuing and batching" Spotify may be aggregating events client-side.
Given that messages are priced in 64K increments, it would be most cost
efficient to to get each message as close to 64K as possible.

i.e.

    
    
      # serialized: 1 message, 1 op
      publish("['foo', 'bar']")
      
      # instead of:
      
      # a) 2 messages, 2 ops
      publish("foo")
      publish("bar")
      
      # b) 2 messages, 1 call, 2 ops
      publish("foo", "bar")

~~~
dmytton
Yes then it would be down to their consumers to split them out. However, the
graphs Spotify show from Google's monitoring do not support this and they show
the actual volume of messages as quoted in the article.

~~~
patrickmclaren
I agree that the current graphs from Spotify don't support this. They do seem
to show a one-to-one mapping between events and messages emitted. Given the
comparison in my GP, I think that client-side message collation/aggregation
would be a natural next step.

------
gberger
$290k per month is $3.5M per year.

A software engineer at Google earns, on average, $127k per year [1]. We
multiply that by 1.4 [2] to obtain a cost per employee of $177k.

This means that according to this estimation, Spotify pays for 20 Google
software engineers.

[1] [https://www.glassdoor.com/Salary/Google-
Salaries-E9079.htm](https://www.glassdoor.com/Salary/Google-
Salaries-E9079.htm) [2] [http://web.mit.edu/e-club/hadzima/how-much-does-an-
employee-...](http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-
cost.html)

~~~
eloff
So I don't get this. 1gbps and 2 million events/s could be comfortably handled
on a single beefy 10gbps server. Add a second for failover. That's maximum
$20K/year which is far cry from $3.5 million. How does this even begin to make
economic sense? The economy of the cloud has never made sense to me, since
it's cheaper to overbuy your capacity by 5x in dedicated machines than use AWS
or GCE. After just one year you've paid for all your capital expenditures and
then you're minting it. For a startup the appeal of the cloud is obvious, but
for a big established company with known technical requirements I don't get it
at all.

~~~
dogma1138
ISP cost, backups, patching, hardware upgrades, co-location, DR, insurance,
paying for physical and cyber-security, personnel etc.

outsourcing allot of those things saves money it's also a different expense of
you buy infrastructure it has value which means depreciation overtime which
affects how your accounting is done.

Cloud makes sense for allot of companies not just startups the biggest reason
why companies with eatablished infrastructure don't migrate is the initial
cost and regulation.

~~~
Matt3o12_
Why can't you just rent the server from some company? Such a company is only
responsible for making sure the hardware works (there are no power outages,
the disks work, the network is up and reachable) and you have to make sure
your application works and have some failover (because the hardware will
eventually fail and they will only replace it, not make up for the loss).

You'd still save a lot of money compared to cloud servers, need little admin
work (the same compared to compute instances – amazon's EC2 servers), and have
a reliable server. Providing the servers doesn't take forever, either. Some
bare metal servers are up and running within 120sec, though, you can't just
shut them down and stop paying like cloud servers.

Companies that offer these services are OVH.com and softlayer, though
softlayer always seemed to be a bit expensive. I'm sure you'll be able to find
similar companies though.

I can see that this makes little sense to startups because the admin work is
way more expensive but for bigger companies, this should not be the case (I
don't consider Spotify and Netflix a startup anymore).

~~~
Johnny555
_Why can 't you just rent the server from some company? Such a company is only
responsible for making sure the hardware works (there are no power outages,
the disks work, the network is up and reachable)_

Isn't that what Amazon does? (even though they are virtual servers)?

 _You 'd still save a lot of money compared to cloud servers,_

Can you post some numbers? A 24 core 48GB Dell server with 3x120GB SSD's costs
around $11K, or around $300/month for a 36 month lease.

An 8 core 15GB 160GB AWS c3.2xlarge costs around $200 each per month with a 3
year reserved instance. 3 of them will fit in one of the Dells, so it would
cost around $600/month, or about $300 more than the raw purchase price for the
hardware alone.

But also with that price you get network, power, cooling, rack space,
maintenance, etc, so it's not clear that buying and running that server is
significantly cheaper.

 _Some bare metal servers are up and running within 120sec_

How long does it take if you want 500 of them? And 2 hours later you want to
turn them off. Then 2 hours after that you want to bring them all back online
again.

 _you can 't just shut them down and stop paying like cloud servers._

But that's the big value-add in Cloud servers, it's capacity on demand so you
only pay for what you us -- need to spin up 500 servers to run month-end
processing? You only need to pay for them for the few hours or days a month
when you need them.

Further, you can add multi-datacenter or multi-region redundancy at very
little additional cost - you don't need to keep those redundant servers
running when you're not using them.

~~~
lbradstreet
Just a side note. After some benchmarking, I recently discovered that
c3.2xlarge does not actually have 8 cores, rather it had 4 real cores = 8
hyper threaded cores. I was very surprised by this and suspect most other AWS
users would also be surprised. This is not to say that AWS is a bad deal, but
I think Amazon could be a little more transparent about it.

See [http://www.pythian.com/blog/virtual-cpus-with-amazon-web-
ser...](http://www.pythian.com/blog/virtual-cpus-with-amazon-web-services/)
for more details.

------
stephenitis
As far as I've heard from Netflix's end the bottom line cost was not as big a
factor in their infrastructure decisions as their content contracts were a
much larger % of their budget. I suspect Google wants to highlight more the
fact that it works and works well rather than highlight cost competitiveness
at the moment.

thanks for the post @dmytton.

not sure if it's accurate It'd be lovely for Spotify or Google to chime in.

~~~
simonebrunozzi
My guess is they won't. I suspect that Google heavily subsidized the deal. One
way to get market traction is to get market awareness, and Spotify on GCP
achieves that. Spotify would have been dumb not to get a huge discount in
exchange for its logo.

------
jamesblonde
This article misses the biggest cost Spotify have at GCE - analytics. Spotify
has migrated a 90 PB Hadoop cluster running on 2000 servers to GCE. That
cluster was running Hadoop jobs (mostly Hive) 24x7. The YARN job queue was
always full. They now have to pay for the storage and processing costs for
that. How much will they pay? I don't know, but I know from their history they
are very good at getting discounts on infrastructural software (e.g., their
Hadoop software was licensed from Hortonworks and they got an incredible
price, because Hortonworks needed a reference customer - sound familiar?)

~~~
dmytton
I could only write about pub/sub in this article because that's the only
service we have a full enough set of numbers for. I hope Spotify will be
releasing more stats in their future blog posts about the other GCP services
they'll be using.

~~~
jamesblonde
I have the stats from late January (at Stockholm Hadoop User Group). It's 90PB
of data, running jobs with 95 TB of RAM, 40K jobs/day (approximately 12,000
CPUs with probably 60-90% utilization).

------
bkovacev
The author also failed to notice "peaks" from the original spotify blog post
about events per second. So 700k requests is not average, it's peak. They
don't have that every second.

~~~
dmytton
This is a good point, but it could be somewhat mitigated by their expected
growth of up to 2M per second. To quote the full line from their blog post:

> Currently our production load peaks at around 700K events per second. To
> account for the future growth and possible disaster recovery scenarios, we
> settled on a test load of 2M events per second.

In the absence of any other numbers, using the 700k number seems reasonable
otherwise you would start to get into modelling without any real data to back
up your assumptions. This at least gives you a maximum they would be paying
under current production conditions.

~~~
mkagenius
How about a gaussian distribution which peaks at 700k?

Assuming max as the norm is not too right. (just to make the number look high)

~~~
EricBurnett
You could guess that, but with no real expectation that it's right either.
E.g. for a global service I'd guess it's better viewed as a sum of three
offset distributions, one per continent, but even then the individual
distributions are unlikely to be gaussian.

Having run global services myself I can tell you that the shape is usually no
standard distribution. Peak/trough depends on the exact usage demographics,
but the one I have on hand has a trough of ~1/2 peak, and an average of ~3/4.

~~~
smarinov
Just curious: how does the real distribution look like, from your experience?
Do you have any graphs by any chance?

------
mangeletti
As a side note, $300k/mo is a pretty small amount of money, compared to the
+$100MM/mo revenue Spotify brings in.

------
yeukhon
> Of course, for such a high profile customer and for the priviledge of a case
> study, we can assume Spotify has some heavy discounts.

I am pretty sure it is also not unusual for average customers on AWS to get
some discount, with some reserved instance committment). One such example is
probably Redshift which is extremely expensive.

Would be nice to get an estimate quote for AWS but just for SQS I don't expect
to be too expensive.

------
prdonahue
I'd be willing to bet they're paying almost $0 in the first year. Google
REALLY wants a big name customer, so you have to think they gave this deal
away for negative margin.

------
Pyxl101
> Thankfully, Google’s pricing is very transparent and simple to work with, so
> we can use Spotify’s blog posts to calculate some list pricing for what
> Spotify might be paying Google.

The biggest way in which all such assumptions are likely to be wrong is that
they overlook the likelihood that Spotify has been offered a potentially
substantial discount in exchange for attempting to win their business, and to
be an early adopter. Not that there's anything wrong with that, but it's
something to take into account while evaluating what you think Spotify might
be paying.

Many cloud service providers offer a discount to, for example, qualified
startups participating in certain accelerators. In the same fashion, it would
not be surprising for a provider to entice a major potential user and hopeful
net promoter with a significant price reduction as well as other perks, such
as more access to senior people for operations support, as well as influence
on the roadmap.

So while I think this blog post could plausibly estimate what a typical
customer with Spotify's usage pattern might pay, assuming the estimates are
correct (I haven't checked the details), one cannot draw conclusions about
what they're likely to _actually_ be paying.

[Edit: I see now that the article recognizes this at the bottom. At the time I
made my comment, it seemed to have been overlooked in the discussion here,
where only one comment acknowledged it and the rest seemed to discuss the
numbers as factual and face-value:
[https://news.ycombinator.com/item?id=11242166](https://news.ycombinator.com/item?id=11242166)
]

~~~
joshAg
The author covered this:

>>> Is Spotify really paying $290,518/month?

>>> Of course, for such a high profile customer and for the priviledge of a
case study, we can assume Spotify has some heavy discounts. Cost is important
but using public cloud and especially managed services like pub/sub, the
biggest benefit is in ease of deployment and not having to manage/scale
everything.

------
neom
I recall David Byttow mentioning that secret cost about this much to run on
GCP so I'm not surprised at all.

~~~
joeblau
Yeah, Secret was spending a ton of money on GAE.

> For example, in one month, our server costs were well over $4M on Google App
> Engine after spiking 5M users in the course of two weeks. We scrambled and
> optimized, but the cost was real[1].

[1] - [https://www.producthunt.com/live/david-
byttow](https://www.producthunt.com/live/david-byttow)

------
stephenitis
Is there a post talking about how much Netflix pays Amazon?

------
h1srf
The post is a cool example but I'd be shocked if Spotify doesn't have a
contract for up to X events/mo and overages at some reduced rate. No way
they're paying per request. The only place I've never seen negotiate rates is
Bloomberg.

------
Hengjie
The bigger question is if Google Cloud Platform is cheaper than AWS?

------
capkutay
There must have been a steep discount given that Spotify has been an enormous
reference customer and blogging about how much they love it.

------
kordless
A good guess using non-technical means would be $15-18M a year. I based this
on the assumption that a customer doing less than $1M a month really isn't
worth mentioning in the news or noteworthy. Also, Spotify did $1.2B last year.

P.S. Howdy David!

------
lateguy
Does Google cloud work in china?

