
Spotify moves its back end to Google Cloud - dmichel
https://news.spotify.com/us/2016/02/23/announcing-spotify-infrastructures-googley-future/
======
latchkey
What Google Cloud needs is a better sales story. AWS consistently beats out
Google in this respect. This story is welcomed and there should be more
success stories published like this.

My business is a heavy GC user (AppEngine, Datastore, CloudStorage, BQ,
ComputeEngine, ManagedVM) and I couldn't imagine achieving what we have
achieved in such a short amount of time on any other platform.

We (myself and another guy) started at zero and shipped our product in 3
months on what is effectively an infinitely auto-scalable platform where we
don't have to do any devops or carry a pager. I'd much rather work on features
than chores. =) 4 months later and we've had profitable growth and zero
downtime (knock on wood) and we're hiring.

Thanks Google!

~~~
thibaut_barrere
I've been relying on multiple Cloud offerings (including PaaS, IaaS etc)
outside Google for many years, but for some unclear reason, I cannot manage to
consider Google a reliable partner for a type of hosting or another, at this
point.

I would love to be proven wrong really, because I often look at their GC
offering, and I /really/ want to dive in, yet I feel that I would risk having
to migrate quickly in 2 weeks at some point because they are changing things
significantly (either shutting down or other major change).

So to my question: for how long have these services been around in a fashion
that is stable (feature-wise, quality-wise etc)? Is there anyone here with a
longer history of using Google Cloud offering, able to comment on this?

EDIT: thanks to everyone who replied below. I think I'll give it a try :-)

~~~
boulos
For Google Cloud products that have made it to General Availability (GA), we
have a one-year deprecation policy
([https://cloud.google.com/terms/](https://cloud.google.com/terms/), Section
7.2). That is, if we're going to make some big change, we give you a 1 year
heads up.

Disclaimer: I work on Compute Engine, but I'm not a lawyer ;).

~~~
nkw
Promises to do something for a year which you can change or withdraw after 30
days are really promises for 30 days:

"Google may make changes to this Agreement, including pricing (and any linked
documents) from time to time. Unless otherwise noted by Google, material
changes to the Agreement will become effective 30 days after they are posted,
except if the changes apply to new functionality in which case they will be
effective immediately. If Customer does not agree to the revised Agreement,
please stop using the Services."
[https://cloud.google.com/terms/](https://cloud.google.com/terms/), Section
1.7b

Also, Sections 13.1-13.3 means even though Google promises nt to do something,
you really don't have any recourse if they do it anyway and cause you damage.

Although in the grand scheme of things, not the worst terms of service
document I have read. The mere fact the drafters addressed the issue is worth
... something.

Disclaimer: IAMALBIANYL.

~~~
kbenson
> Disclaimer: IAMALBIANYL.

I like that one. :)

I have a question though, is the prevalence of taglines in any way related to
legal requirements? Do you have to stipulate that you aren't a lawyer when
stating things because otherwise people to construe it as legal advice, or is
it just a fun little identifier of how valid we should consider information to
be (like it is in other acronyms of the same type)?

~~~
nkw
I think it may stem from laws and rules related to practicing law, but there
is certainly not any "requirement" to stipulate you are not a lawyer when
speaking about legal things. However there are a few things to be mindful of:
In the (all of the) United States (and many other countries) the profession of
lawyer or attorney is regulated and requires a license to practice as such. In
many places to practice law without a license is a criminal offense. Obviously
this doesn't contemplate criminal charges against a group of people spouting
off about something legal related on the Internet (otherwise we would have to
jail perhaps 80% of reddit), but providing legal advice and interpreting the
legal effect of contracts, statutes, or regulations is part of the practice of
law.

The reason we heavily regulate the practice of law is similar to the reason we
heavily regulate the practice of medicine. If you screw something up, the
consequences can be bad. If someone takes your advice thinking you know what
you are doing, and you are wrong, that person can end up a lot poorer or even
in prison. (Though some will argue we regulate it so lawyers can be the only
ones charging high fees for legal advice!) Just like one probably shouldn't go
around telling folks not to worry about their incredible chest pain and
shooting arm pain unless they are a trained medical doctor, the same could be
said for those that go around offering home spun legal advice.. ("Yeah man, if
you ask the undercover cop if he is law enforcement, he HAS to tell you!
Otherwise it is entrapment man...")

Also the reason you always see lawyers preface everything by saying "I'm not
your lawyer.. This isn't legal advice... Yadda yaddda yadda" is because once
there is an Attorney-Client relationship a lot of stuff happens. The lawyer
has a host of duties to the client, from confidentiality, to competence, and
many others. That obligation isn't taken lightly and many times if there is
any doubt whatsoever whether or not there is an attorney-client relationship,
the law will find in favor of one because --- well -- the lawyer should know
better. Therefore lawyers will make it annoying clear they aren't your lawyer
before spouting off some deep legal thoughts about whatever the topic of
discussion is.

~~~
jsmthrowaway
Great answer. Has the acronym "IANAL" ever been tested in court in such a way,
to your knowledge? Now you have me curious, and that'd honestly be quite
amusing.

~~~
nkw
Ha! Interesting question. I just ran an all state/federal Westlaw search for
"IANAL" and didn't hit any case law. Two or three briefs, a couple of
scholarly articles, but no court opinions. I also searched for "I am not a
lawyer", but that brought up about 130 caselaw hits which seemed to primarily
be references to various filings by pro se parties actually saying in a court
filing "I am not a lawyer".

------
simonebrunozzi
I want to commend some of the Google Compute Platform employees that commented
in this thread.

I was at AWS for 6 years (left 2 years ago... today!), and I've always been a
proponent of being more open and communicative with developers, but it rarely
happened - I guess that AWS' PR policy and such are a big showstopper for
these kind of discussions. Although, some individuals did their best (e.g.
Jeff Barr) to try to share as much as possible.

It seems that the Google guys know how to do it.

Keep doing it. It will help your business a lot.

~~~
abrookewood
AWS employees may not speak out very often, but they hardly need to. AWS is a
marketing machine, from Jeff's regular blog posts, to the volume of free
information they provide and to the conferences that they are running
everywhere. I think Microsoft is starting to catch on with Azure promotions &
conferences, but it's like Google didn't even know that the there was a race
.... and they left their running gear at home.

~~~
rifung
This might be true but I think the issue is that many AWS employees actually
want to speak out or respond to issues personally.

I used to work at AWS and the things we could say were very controlled. It
could just be my team/managers though.

~~~
simonebrunozzi
No, I agree with your last statement - at AWS you always feel the invisible
shadow of PR :)

------
waffle_ss
I tried Google Container Engine (GKE) and really liked it - it's the best
cloud solution for deploying Docker to production in my opinion, mainly due to
its use of Kubernetes. Unfortunately in my Web apps I make heavy use of
Postgres-specific features, and since Cloud SQL only supports MySQL, Google
Cloud is a total non-starter for me.

So for now I'm on AWS, using Postgres on RDS and deploying containers with
ECS. ECS is a lot simpler than Kubernetes, but since my apps are pretty simple
(a half dozen task definitions), it's not a big deal. I really hope Google
adds Postgres to Cloud SQL at some point.

~~~
crb
There are vendors running managed Postgres services on Google Cloud Platform,
like ElephantSQL [1] and Aiven [2]. And you can of course run your own on GCE,
even to the extent of 24/7 commercial support - you can run EnterpriseDB from
Cloud Launcher [3].

And, you can also run Kubernetes on AWS - we have a group focused on making
sure it's an excellent experience.

I work for Google Cloud Platform; ping me if you'd like more help with either
option.

[1] [https://www.elephantsql.com/blog/2014-11-17-google-
compute-e...](https://www.elephantsql.com/blog/2014-11-17-google-compute-
engine.html) [2] [https://aiven.io/](https://aiven.io/) [3]
[https://cloud.google.com/launcher/solution/public-edb-
ppas/e...](https://cloud.google.com/launcher/solution/public-edb-ppas/edb-
postgres-enterprise?q=enter)

~~~
waffle_ss
> There are vendors running managed Postgres services on Google Cloud
> Platform, like ElephantSQL

I did check out ElephantSQL but my pricing needs are somewhere between their
$100 and $20 plans and there seems to be a lack of configurability compared to
RDS's parameter groups (e.g. enabling extensions).

> you can also run Kubernetes on AWS

I've had success turning up Kubernetes clusters on AWS for demo purposes, but
I really don't want to manage a k8s cluster myself (anecdotes I've read about
etcd failures / partitions especially scare me). Also I use Terraform for
provisioning, and kube-up.sh is not something that fits into that paradigm.
I've also made the mistake running kube-up.sh with the wrong arguments after a
previous invocation that had created a cluster, which caused it to try and
create a new cluster, which wiped the local cache of the previous cluster I
had made, making kube-down.sh unable to automatically clean up the old cluster
(so I had to do it manually in the AWS console).

The other thing I tried was the kube-aws CoreOS tool, which is nice, but it
comes baked with a 90-day expiration due to TLS certificate expiration, so I'd
have to set up some sort of PKI process to make that production-ready. All in
all just too much work for a single person trying to deploy a small number of
containers for small to medium sized projects; if I was a medium-sized company
with hundreds of containers and some dedicated DevOps resources maybe it would
be worth it, but for myself I'd prefer a turn-key solution like ECS or GKE.

~~~
TheIronYuppie
We also just had a user check in a cloud formation template for Kubernetes on
AWS - would that meet your needs? (No kube-up.sh required)

Disclosure: I work on GKE and Kubernetes at Google

------
obulpathi
> Google has long been a thought-leader in this space, and this shows in the
> sophistication and quality of its data offerings. From traditional batch
> processing with Dataproc, to rock-solid event delivery with Pub/Sub to the
> nearly magical abilities of BigQuery, building on Google’s data
> infrastructure provides us with a significant advantage where it matters the
> most.

Big Data is the core strength of Google Cloud. Good to see this move by
Spotify!

> What really tipped the scales towards Google for us, however, has been our
> experience with Google’s data platform and tools. Good infrastructure isn’t
> just about keeping things up and running, it’s about making all of our teams
> more efficient and more effective, and Google’s data stack does that for us
> in spades.

What I really really liked about Google Cloud is the ease of use. Spin up a
VM, start Cloud shell, SSH into your instance, install a bunch of software and
you will know what I mean. It's "Quality" Cloud.

------
jhgg
At work we moved to GCE at the beginning of this year, from Linode after they
were having stability issues over the christmas break. No complaints from us.
So far have been very happy with it. We were considering moving to AWS, but to
realize the same pricing as GCE we'd have to purchase reserved instances - the
sustained usage discounts have been huge for us.

Additional things we liked: \- gcs is way more responsive than S3. also, was
fairly painless to migrate our S3 buckets to GCS via the web console. \-
peering with cloudflare lets us save on bandwidth costs! \- network load
balancer has shown itself to be very reliable and solid for holding open A LOT
sustained websocket connections. \- http load balancer has shown itself to be
very capable of ssl termination & routing (love that we can route /api to our
API servers, and the rest to our static servers). additionally, no pre-warming
was required. when we did the datacenter switch, it was just 5 mins of
downtime. didn't have to worry about pre-warming for our production load like
we would have on ELB.

Things we didn't like: \- salt-cloud's driver for GCE is still lacking. Can't
specify disk size for provisioned storage. Can't specify local SSD storage
either. Also parallel provisioning didn't work. Something about PyCrypto not
liking the way it forked. Not GCE's fault \- billing support non-existent
unless you purchase a support plan. \- documentation still needs some work.

~~~
jpatokal
Billing support is provided for free to all customers. If you had trouble with
a particular ticket, please feel free to ping me.

Disclaimer: I work in Cloud Support.

~~~
jhgg
Thanks for the reply! I don't remember the specific issue, but we had a
billing inquiry and were told we had to upgrade to a paid support plan to get
an answer.

~~~
jpatokal
Please feel free to forward the ticket to me at jani at google, and I'll look
into it.

------
jamesblonde
The big point of this move is the hit it will have on on-premises Hadoop
offerings: Cloudera, Hortonworks, MapR. It's a massive vote of no-confidence
in their offerings. Spotify are effectively ditching a 2000 server, 90 PB
Hadoop cluster to go GCE. As Spotify say themselves, that's big news....

~~~
nrh
Spotifier here. This is an important point. I have nothing bad to say about
Cloudera or HWX (disclaimer: we're an HWX customer - we've had a pretty good
experience), but I don't really see a compelling reason at this stage to
manage your own cluster(s) (HIPAA/regulatory constraints, maybe?)

Getting shared-storage and indepedently operated/scaled compute clusters on
top of that storage isn't easily achievable with the standard Hadoop stack,
and building that on top of HDFS is non-trivial.

~~~
jamesblonde
In fact, I don't think large orgs like you (Spotify) really want independently
operated clusters. That prevents easy sharing of data, causing data silos to
appear. You really want to have true multi-tenancy, which isn't in Hadoop yet.
Hadoop has worked more on Kerberos support at the cost of features like easy-
to-use access control - Apache Ranger or Sentry anybody!?!?

~~~
ec109685
Yahoo's Hadoop clusters operate in a multi tenant fashion for precisely this
reason: to ease sharing of data between groups.

------
dublinben
When did Spotify abandon their original peer-to-peer architecture?[0]

[0][http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.p...](http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.pdf)

~~~
philo23
Quite a while ago. Not sure on the exact time, but judging from this article,
I'd guess sometime in 2014 [https://torrentfreak.com/spotify-starts-shutting-
down-its-ma...](https://torrentfreak.com/spotify-starts-shutting-down-its-
massive-p2p-network-140416/)

~~~
api
I'd be curious if anyone knows the details about why: complexity, customer
complaints, reliability?

~~~
natarius4k
Too much complexity for a world that's moving to mobile clients that can't be
active peers of the network.

~~~
api
Seems to be a pattern: wimpy mobile endpoint devices drive everything to the
cloud.

If so this is perhaps a dated phenomenon. Moore's Law is still a thing and
today's generation of mobile devices are getting fatter and fatter.

~~~
LeoPanthera
Moore's Law has not been improving batteries, which is the real problem.
Mobile P2P nodes need to be awake most/all the time, which is a real battery
killer. Modern devices have improved battery life by being asleep almost all
the time.

~~~
api
Depends on what those p2p nodes are doing. We do p2p on mobile but it's to
allow the mobile device to act as a client to access an IoT device, not to
force the mobile device to act as a CDN node. Spotify was doing the latter,
which is more problematic.

Moore's Law eventually _will_ matter, since faster and more efficient chips
equal the ability to do more with lower power consumption. Batteries aren't
getting better fast enough, but power consumption per fixed unit of compute is
falling like a rock. As a result "effective battery life" for a given fixed
workload is subject to Moore's Law.

------
peterfschaadt
We host a lot of our applications on Google Compute Engine but have had many
issues with even simple features available from other cloud providers. For
example, Google's HTTP(S) Load Balancer offering does not support SNI [1],
HTTP to HTTPS forwarding [2], or sticky sessions [3], making it unusable for
us. The support we pay for has been pretty helpful, but all we hear is "we
have a feature request for this but I cannot provide an ETA on when and if
this will be implemented". Many of the issues in the Compute Engine public
issue tracker [4] haven't been updated in many months or over a year in some
cases. Additionally, it can be very confusing finding out which beta services
can only be accessed via the gcloud tool and not the web console. Without the
ability to even schedule a window for maintenance on our VMs, we often feel
like we do not have the control we need over our hosting environment. Please
increase your transparency around addressing Google Cloud issues, all the
Docker updates have been great, but many of these issues are over a year old
and still show-stoppers for some users. Thanks again for churning out new
features/services constantly, but please help close out some of these older
feature requests!

[1] [https://code.google.com/p/google-compute-
engine/issues/detai...](https://code.google.com/p/google-compute-
engine/issues/detail?id=289) [2] [https://code.google.com/p/google-compute-
engine/issues/detai...](https://code.google.com/p/google-compute-
engine/issues/detail?id=255) [3] [https://code.google.com/p/google-compute-
engine/issues/detai...](https://code.google.com/p/google-compute-
engine/issues/detail?id=240) [4] [https://code.google.com/p/google-compute-
engine/issues/list](https://code.google.com/p/google-compute-
engine/issues/list)

------
ceocoder
We (Sojern) moved to 100% GCP as well this month, admittedly not nearly at the
same scale as Spotify, but we did. Other than some growing pains - that GCP
team is already aware of - this has been a very smooth migration for us.

We use Google Cloud {Storage, BigTable, Container Engine, BigQuery,
Monitoring, DNS, Compute} and may be some services I'm missing. I'd be happy
to answer any questions.

------
boulos
This was mentioned somewhere deep in a thread, but Spotify (and others!) will
be speaking at our upcoming customer conference (GCP Next) in a few weeks:

[https://cloudplatformonline.com/NEXT2016-schedule.html](https://cloudplatformonline.com/NEXT2016-schedule.html)

You'll be able to livestream the sessions here:

[https://cloudplatformonline.com/NEXT2016-Live.html](https://cloudplatformonline.com/NEXT2016-Live.html)

Disclosure: I work on Compute Engine (and I'll be at Next).

------
matt_wulfeck
The GCE model is the future. Google knows this because they've already
discovered it internally by running their own services. They know containers
are he future and they built a lot of the LXC kernel updates that makes things
like docker work.

Most people don't need the future yet. Really people are just trying to get
out of the business of managing and scaling their hardware with as little work
as possible. Containerization is much farther down the line.

So I posit the future GCE customers are not people "moving to the cloud" like
AWS, but rather people "moving to the small, contained, service" \-- indeed,
it's current AWS customers.

------
dang
Also on this: [http://www.wired.co.uk/news/archive/2016-02/23/spotify-
move-...](http://www.wired.co.uk/news/archive/2016-02/23/spotify-move-to-
google-cloud) and [http://googlecloudplatform.blogspot.com/2016/02/Spotify-
choo...](http://googlecloudplatform.blogspot.com/2016/02/Spotify-chooses-
Google-Cloud-Platform-to-power-data-infrastructure.html)

------
pdknsk
I notice App Engine isn't mentioned in their stack. I'm a bit worried that
Google abandons it once Snapchat moves off it. It has already been in
maintenance mode for some years now (save for Managed VMs). The writing was on
the wall when Guido left the team. Apparently Google still uses it internally
(as the recent switch to Monorail made evident), so perhaps it's safe for a
few more years.

~~~
arouzrokh
Hi pdknsk,

I'm a Product Manager on App Engine and wanted to reach out and say that 100%
we are invested in app engine and are working to make it your favorite
platform. For example, we heard that GAE documentation needs enhancements and
we've kicked off a large project to do just this. It's an ongoing effort as
you can imagine, but please take a look at the new landing page and managed VM
content as well as the left-hand nav reorg we did. Here is the link. More to
come of course.

[https://cloud.google.com/appengine/docs](https://cloud.google.com/appengine/docs)

Please give the new set of docs a try and please give us feedback when you see
errors or generally when the content is not up to par with what you expect.
There is a feedback tool on the top right hand side of every page.

Thank you! Amir

~~~
izolate
Do you have any plans to launch a data center in India? AWS is launching one
this year, and I imagine this will be a huge boon to the startup industry
here.

As of 2016, AWS is the better bet.

------
sigmar
>"...feature testing and more intelligent user-facing features," the Google
staffer wrote.

I have to wonder if Spotify is doing this so that they can (eventually) use
Google's deep learning technology in improving their recommendation engine.
They purchased Echo Nest in 2014 to achieve this.

------
agentgt
I have some questions based on previous experience many moons ago experience
(3 or 4 yrs).

When I tried to port some of our platform to AppEngine I had some issues with
various Java libraries (writing tmp files, opening sockets, etc). Once I sort
of got past that and used some of Googles own APIs (which is sort of annoying
that I had to couple my stuff) I had timeout issues. See we have to integrate
with all these enterprise third parties (SOAP/REST) and these guys can be
really slow. We also couldn't use our own pub/sub (AMQP/RabbitMQ) so that was
going to have to be ported as well.

Things must have changed because its hard for me to imagine Spotify being able
to get all their needs met on a rather coupling platform.

* How does one write code for Google Cloud while being agnostic of Google Cloud?

* Maybe appengine supports AMQP?

* Or maybe the container engine fixes these problems? (ie need something more custom run it in a docker image).

Our current platform runs on Rackspace and Digital Ocean. These guys are IaaS
so your basically doing DevOps which is a pain... but Rackspace does have kick
ass customer service.

~~~
dragonwriter
AppEngine is Google Cloud Platform's PaaS offering. Its the oldest product on
Google Cloud Platform, but its not one of the ones mentioned that Spotify is
using (the Spotify post doesn't talk much about the products, but Google's [0]
does.)

The Google Cloud Platform services they are identified as using are: Cloud
Storage, Compute Engine, Cloud Datastore, Cloud Bigtable, Direct Peering,
Cloud VPN, Cloud Router, Cloud Pub/Sub, Cloud Dataflow, BigQuery, and Cloud
Dataproc.

So, its not surprising that they might be doing something that Google
AppEngine doesn't support particularly well.

[0] [http://googlecloudplatform.blogspot.com/2016/02/Spotify-
choo...](http://googlecloudplatform.blogspot.com/2016/02/Spotify-chooses-
Google-Cloud-Platform-to-power-data-infrastructure.html)

~~~
agentgt
I guess they are not entirely on Google Cloud but rather their data
warehousing / big data stuff is now?/going? to be on GCP.

I don't really call big data processing "backend" but rather the microservices
the backend. Its not clear to me if those are going to be ported over or if
they are already.

Does spotify plan on porting everything over or just the heavy computing? A
detailed techy case study I hope will follow soon as I am interested (the
DevOps pain is very real).

~~~
dragonwriter
> I guess they are not entirely on Google Cloud but rather their data
> warehousing / big data stuff is now?/going? to be on GCP.

I don't know why you would guess that: the Google announcement indicates that
the migration has both a "services" track and a "data" track. And Spotify's
announcement certainly seems to indicate that they are moving off of their own
datacenters, onto Google Cloud Platform, and not in a limited way.

~~~
agentgt
So... basically nothing is live yet? Are you from Google? Is anything live. I
admit I got confused by the OP title: "Spotify moves its back end to Google
Cloud".

------
atrudeau
This reads like they've been acquired by Google and are integrating. Now
Google just has to write the check and own music streaming.

~~~
simonswords82
Hmmm...interesting point but Google already has an awful lot of music
infrastructure in the form of Google Music and YouTube does it not? Would it
really make sense for them to purchase Spotify too?

~~~
tormeh
I've only really looked at Spotify and Tidal, but Spotify is really good.
There's more to streaming than content.

Makes me curious about how good Google's recommendation engine is.

~~~
vinkelhake
As a long time Spotify user who switched to Google a few months ago: I really
like the playlists and I've found a lot of new music that I like. The
recommendations have worked really well. I'm not looking back.

~~~
pgm8705
I've also found that Google's radio stations are more diverse and less
repetitive.

~~~
tormeh
Just checked out Google Music on Wikipedia. No native desktop clients. Not
even for Windows, and I use Linux. They can't be serious. Total dealbreaker.

~~~
kyrra
I tend to always have a browser open so a desktop client doesn't matter to me.
I keep Play Music open in a tab. Then I use an extension[0] to add the other
controls I normally use.

[0] [https://chrome.google.com/webstore/detail/prime-player-
for-g...](https://chrome.google.com/webstore/detail/prime-player-for-
google-p/npngaakpdgeaajbnidkkginekmnaejbi)

------
halayli
Congrats to the sales team at Google Cloud. I wonder how much Google's new AI
services played a role in convincing Spotify to use them.

~~~
vgt
Hey guys it's Tino. When will you contribute to our Big Data blog with your
excellent ideas? :)

[https://cloud.google.com/blog/big-data/](https://cloud.google.com/blog/big-
data/)

~~~
halayli
I am more than happy to contribute but I have a feeling you've mistaken me
with someone else? :)

~~~
vgt
yup sorry posted in wrong thread.. but please do :)

------
derEitel
I would be interested to get some more technical info. OK so you use GC, but
how? What products do you use, how do you build your scalable infrastructure
so anywhere in the world I can have all the music in the world? I love reading
those type of posts from Netflix and was hoping to find something similar here

~~~
boulos
As I mentioned in a top-level comment
([https://news.ycombinator.com/item?id=11159840](https://news.ycombinator.com/item?id=11159840)),
they will be speaking at GCP Next (which you can livestream) and should have
some more detail to share.

------
mratzloff
Today: Highly publicized move onto Google Cloud

Later: Silent, gradual move away from Google Cloud after experiencing a number
of issues and bugs, constant connectivity issues with services like Cloud
Storage, silent failures that do not return error messages on a random basis,
detailed billing info only available through arcane APIs and not the
administrative UI, broken functionality and rigid inflexibility of BigQuery,
and other non-obvious negative behaviors

------
icaromedeiros
I was wondering about Spotify's contributions to Hadoop stack such as
Snakebite and Luigi. What will happen to these projects if they're moving to
BigQuery?

~~~
crb
Spotify are not moving to BigQuery alone. They are also using Dataproc,
Google's managed Hadoop service (which co-incidentally went GA yesterday),
which lets them get value out of all the orchestration tools they already use,
like Luigi. (They are also adopting Cloud Dataflow, Cloud Pub/Sub and other
parts of our Big Data stack.)

Luigi has had BigQuery support for some time:
[https://github.com/spotify/luigi/pull/1002](https://github.com/spotify/luigi/pull/1002)

Spotify will be talking more about the specifics of their data pipelines at
GCP Next in SF in March, - I would expect to see a lot more on the new Google
Cloud Big Data blog at [https://cloud.google.com/blog/big-
data/](https://cloud.google.com/blog/big-data/) too.

[Disclosure, GCP guy, etc]

------
hanief
Om Malik sees it as a first step toward acquisition by Google (or rather,
Alphabet).

[https://twitter.com/om/status/702194246489546752](https://twitter.com/om/status/702194246489546752)

~~~
PhantomGremlin
This has been rumored before. E.g here's a headline from 2014:
[http://www.wsj.com/articles/google-considers-buying-
spotify-...](http://www.wsj.com/articles/google-considers-buying-spotify-but-
finds-the-price-too-high-1406061732)

------
jonesb6
Great now they can free up some engineers to bring their Linux client out of
beta, which has been in beta since dinosaurs.

~~~
kingosticks
Or work on a successor to libspotify. Ha! Who am I kidding?

------
NearAP
I've run a few small side projects on GAE (Google App Engine). I use it as a
very simple and fast way to get an MVP off. My major issue is that their
documentation tends to lag (sometimes way behind) when they change stuff,
especially things that are integrated to other services they provide. For
example after dropping support for OpenID 2.0, the documentation for
'Authentication Type' on Google App Engine was not updated to reflect this,
neither was the documentation for securing your urls. This cost me
considerable time and effort.

~~~
arouzrokh
Hi NearAP,

I'm a Product Manager on App Engine and I truly apologize for the trouble you
went through when you tried the platform. We have kicked off a large
initiative to update and enhance our docs, and we just recently launched our
new App Engine docs, an ongoing project of course. You can check them out
here:

[https://cloud.google.com/appengine/docs](https://cloud.google.com/appengine/docs)

Please give the new set of docs a try and please give us feedback when you see
errors or generally when the content is not up to par with what you expect.
There is a tool on the top right hand side of every page that explains this.

Thank you! Amir

------
flurpitude
Now how about Spotify fixing their front end so that Premium customers on
Android can just hit "play" to play the album they want to play, in the right
order? And how about an app in the Windows Store so that you can use it on a
touchscreen? And how about some improved metadata that work for classical
music? And a way to report corrupt or mistagged tracks and get them fixed?

/rant

~~~
voltagex_
I'm on mobile so can't directly link you but there's a support form for
getting songs fixed.

~~~
flurpitude
I have never found a form, but I have found forums where people complain about
persistent issues and nothing gets done. They seem to go out of their way to
make it difficult for users to report issues. In addition they seem to have
contempt for user experience, or they're incompetent at it, or they are under
weird nonsensical licensing pressures that require them to make things
difficult for their users. Their front ends need an overhaul and they could
easily introduce a "flag this track as having a problem" feature.

------
misiti3780
I had no clue spotify operated their own data centers.

Does anyone have a link to that compares google cloud vs aws on a per price +
per service basis ?

~~~
crb
Last year, we commissioned an independent 3rd-party vendor to prepare a
whitepaper comparing GCP pricing with AWS:
[https://cloud.google.com/files/esg-
whitepaper.pdf](https://cloud.google.com/files/esg-whitepaper.pdf)

We also have a handy guide for AWS professionals who are looking at Google's
cloud: [https://cloud.google.com/docs/google-cloud-platform-for-
aws-...](https://cloud.google.com/docs/google-cloud-platform-for-aws-
professionals)

[Disclosure, I work for GCP, but I love all TLAs equally]

~~~
mikeklaas
There are some half-truths in that whitepaper, like "reserved instances cannot
be resized" (They can, within the same "class" of instances)

------
balls187
I wonder if this has to do with Amazon's streaming music competitor being more
of a threat to Spotify than Google's?

~~~
ascagnel_
Amazon's streaming music competitor is awful. The apps are bad, the web UI is
bad, the selection isn't as big, and the audio quality is middle-of-the-pack.
In general, it's a "oh right, I have that Amazon service because I have
Prime"; a similar comparison would be Prime Video to Netflix.

Spotify is better off in most ways (their desktop app still sucks, but the web
UI is better).

~~~
balls187
I wouldn't judge a competitive threat solely on a V1 implementation, given
it's from a company with engineering chops like Amazon.

------
frakkingcylons
The short and sweet of my user story as a hobbyist fullstack developer and
wannabe data analyst is AWS has standalone services for your apps that either
don't exist on GCP or are better than what GCP offers.

I'm talking about pay-as-you-go compute functions with Lambda (I'm aware that
GC Functions are on the way, Lambda has been out for more than a year
already), managed Postgres with RDS, and managed task queues with SQS. But
GCP's console UX is killer compared to AWS, and using BigQuery feels too good
to be true compared to how difficult it is to accomplish the same thing with
Hadoop/Spark.

------
doxcf434
A little birdy told me that if you find a bug in GCE's live migration they'll
buy you a bottle of scotch. Apparently the team gets blamed a lot for problems
that turn out not to something else. :)

------
an4rchy
I'm curious regarding the economics of this but could anyone approximate the
cost associated with this move or expenses going forward. It sounds like this
would be a pretty big account for Google.

~~~
jamesblonde
You can bet your bottom dollar Spotify are getting heavy, heavy discounts.
Google are ramping up here in Stockholm, partly to get this account. This has
been in the works since before last summer, so they've taken time before
announcing it.

The precedent for Spotify was switching from Cloudera to Hortonworks a couple
of years ago. Rumours are that they got their licenses for close to nothing
from Hortonworks, they were so desparate to close them. Same rumours going
around vis-a-vi GCE...

~~~
jkestelyn
Just with respect to some details: Spotify changed from being a self-
supporting (free) user of Cloudera's open source platform to being a paid
support customer of Hortonworks. So, it was never actually a Cloudera customer
(despite what was widely "reported").

Disclaimer: Cloudera emp here.

~~~
jamesblonde
Nice, reinforces the point that Spotify likes and Spotify gets discounts on
infrastructure :)

------
tomschlick
Still no IPv6 on Google's cloud... why?

------
verbatim
Has anyone seen stats published on how big of an infrastructure Spotify runs?

Or how it compares to Netflix, which runs on AWS?

------
crimsonalucard
Why google over aws? Why isn't this man answering the real question?

------
KaiserPro
Sorry but spotify's infrastructure is not a shiny beacon of correctness.

They've compounded mistake upon "Oooshiny lets use that"

here is a post where someone dissects spotify's infrastructure bit by bit:
[http://www.secretbatcave.co.uk/software/docker-and-
you/](http://www.secretbatcave.co.uk/software/docker-and-you/)

This isn't an anti google compute, its got some really great features. But,
spotify's backend infrastructure ain't one

Yes, I appreciate that that post is from a while ago, and they might have
completely paid back that massive technical debt/innovation token debt.

~~~
nrh
Spotifier here. For the record, I hate puppet with the fiery intensity of a
thousand suns. It's also pretty hard to magically make go away. I could
probably do a whole talk on why puppet is difficult to kill, it's the kudzu of
config management. _I think it is the worst._

We've got our warts and a pile of tech debt, and I wouldn't want anyone to
think otherwise. Containers are a part of a long-term strategy to get away
from puppet and onto more idempotent units of deployment, and move a lot of
what is considered to be "configuration" back into the build process where it
belongs.

~~~
laxk
Is there any public available information where we can read about deployment
process in the company?

------
dmourati
Does Google Cloud offer an equivalent to AWS Direct Connect?

~~~
boulos
Yes. We have both "Carrier Interconnect"
([https://cloud.google.com/interconnect/](https://cloud.google.com/interconnect/))
and "Direct Peering" ([https://cloud.google.com/interconnect/direct-
peering](https://cloud.google.com/interconnect/direct-peering)).

------
lukasm
Isn’t it risky to move to competition's infrastructure?

~~~
Splines
Not really - many people rely on Google Cloud and if Google made it harder for
Spotify to compete (via, say, raising prices for Google Cloud for them),
they'd have to be pretty specific to Spotify to not cause other customers to
move to AWS or Azure.

Spotify has much bigger risks in their licensing deals.

~~~
lukasm
cost it not a problem and google wouldn't do it. I'm talking about stealing
recommendation algorithms and models. Have a peek new features before release
etc.

~~~
zenlikethat
Recommendation algorithms and models don't do you any good if you don't have
users.

~~~
lukasm
Google Music has a lot of users?

------
dkubb
(Serious question) Isn't it risky to rely on _any_ Google product as a
foundation for your business?

They've shut down more popular services that people depend on than Google
Cloud. They tend to enter a market, undercut all the competitors, and then
once they are dominant in the niche they terminate or change the service in a
way that makes it not fulfill the original promise.

~~~
mastazi
Well Amazon killed the Fire Phone though ;-)
[https://fortune.com/2015/09/09/amazon-killing-fire-
phone/](https://fortune.com/2015/09/09/amazon-killing-fire-phone/)

Jokes aside, probably you are thinking about free products like Google Reader,
because this:

> once they are dominant in the niche they terminate or change the service

would not make any sense if the product in question makes $$$

------
ergo14
I think I could try moving to GAE if they had Postgresql offering for database
storage.

------
tschellenbach
I wonder if they are still using Cassandra or moved that over as well.

------
ed_blackburn
I always used to think Facebook would make a move for Spotify..

~~~
ed_blackburn
-1 :(

Zuckerberg used to wax lyrically about Spotify and I always assumed Facebook
would one day like to move to the subscription streaming market. A barrier to
entry for Spotify to stream movies I always suspected would be cash.

Moving to google cloud won't stop a Spotify sale but it may discourage some?

------
shockzzz
Smells like a puff-piece

~~~
skj
A puff piece would be if some tech reporter wrote about the wonders of GCP.

This is the Spotify team, and they've clearly put their money where their
mouth is by actually making the move.

~~~
wcummings
Marketing blog spam is the new puff piece.

~~~
vgt
Would you prefer testaments like these from Spotify engineers?

[https://twitter.com/plamere/status/702168809445134336](https://twitter.com/plamere/status/702168809445134336)

[https://twitter.com/plamere/status/702169733454495745](https://twitter.com/plamere/status/702169733454495745)

[https://twitter.com/mpjme/status/702167231623516160](https://twitter.com/mpjme/status/702167231623516160)

~~~
shockzzz
Kind of. We all know that companies cut deals with providers in exchange for
blog posts, tweets, and other sorts of promotion. I'm always wary of posts
like these, especially when they lack detail or awareness of competing
products.

~~~
vgt
I think if Spotify dressed down Google Cloud Platform competition, no matter
the merit, there would be many more complaints (ex. see comments on this
technical post
[https://news.ycombinator.com/item?id=11029032](https://news.ycombinator.com/item?id=11029032)).

Spotify did say that they really enjoy specific aspects of Google Cloud
Platform, through their own engineering blog. Spotify engineers have also been
independently vocal on Twitter.

Netflix has been very similarly vocal of their support of AWS.

I'd love to hear your thoughts on a better mindshare model, perhaps we can all
learn to do things a bit better.

~~~
shockzzz
Right, I don't think they have to directly attack anyone. It's just a matter
of tradeoffs - here are the things the Google Cloud gets us, here's a specific
example that hones it in, and here's why that's critical to our business.

At the moment, the blog post just says, "Yay we moved to the cloud" and threw
out some marketing buzzwords. It should be beyond a shadow of doubt that they
got a lower price as a result. That's ok - now give us the details! On a
product level, what made Google Cloud so compelling that you were willing to
move your entire infrastructure to it? That question really wasn't answered,
and is really the biggest thing, as a customer, I'd want.

Right now, I'd have to call Spotify as a reference in order to do to get that
answer, which again allows them bring the price lower. Not necessarily
democratize the information.

~~~
nrh
Spotifier here. Frankly, price is not the biggest factor in a decision like
this. If we were going for the _lowest cost_ cloud option, it probably
wouldn't be either AWS or Google - there are other providers who are hungrier
for business that would be willing to do deep cuts at our scale.

The way we think about this is that there are basically two classes of cloud
services: commodities and differentiated services. Commodities are
storage/network/compute, and the big players are going to compete on price and
quality on these for the foreseeable future (as with most commodities).

The differentiated services stuff is a bit more interesting. Different players
have different strengths and weaknesses here - AWS has way, way better
capabilities when it comes to administration and access control and identity
management, for example (which is actually pretty important when trying to do
this in a large org). The places were Google is strong (data platform) are the
places that are most important for us as a business.

Compelling: dataproc+gcs, bigquery, pubsub, dataflow Made it safe: high-enough
quality, cheap enough.

What more would you like to know?

~~~
shockzzz
Hey nrh,

Nice! Did you have the data tooling built out before you went to Google Cloud?
If you did, I could imagine the migration was pretty hard as well.

Also, all of those seem relatively possible with AWS Redshift, Kinesis, and
Data Pipelines. I'm interested what Google Cloud had to offer, spec-wise.

~~~
nrh
I would categorize data tooling as a moving target - we have some, it's never
enough, it probably won't ever be enough. It's a moving target (p.s.
obligatory we're hiring!).

I think this Quora post does a good job of redshift vs. bigquery:
[https://www.quora.com/How-good-is-Googles-BigQuery-as-
compar...](https://www.quora.com/How-good-is-Googles-BigQuery-as-compared-to-
Amazons-Redshift)

More generally speaking, (excuse me for being a little hand-wavey here) many
of the AWS offerings feel like polished, managed versions of familiar tools.
Redshift, for example, feels a bit like "hey we figured out how to abstract
away a bunch of mysql instances to feel like a big processing cluster". That's
not a bad thing, necessarily. The google stuff feels much more intentional -
"we need to solve the problem of doing these sorts of queries at scale" vs.
"we need to solve the problem of scaling mysql to solve these types of
queries"

Maybe they're just better at abstraction, but whatever - that works for me!

~~~
shockzzz
So it sounds like BigQuery was the deciding factor in this case?

~~~
nrh
I'd say the data platform overall, bigquery is certainly great . So are some
other bits.

~~~
shockzzz
Awesome, good to know. Thanks!

------
snissn
In the same way that facebook shut down parse, I am afraid that google is
going to shut down google cloud. And reluctantly have become okay with relying
on AWS.

~~~
ariwilson
Google Cloud interacts with the core of Google's business (web services that
needs scalable data infrastructure) in a way Parse never did for Facebook.

~~~
snissn
Google's core business is ads. Parse makes a lot of sense strategically for
facebook, a big moat that they have built is around being your identity on the
internet. Parse's platform encourages app developers to use facebook login as
a core component of their apps. So even despite being ancillary to their
business, they shut it down. Building a technology company around an ecosystem
is potentially a lot riskier than people take it as. Especially one with a lot
of proprietary bells and whistles..

~~~
ariwilson
Google's core business is providing web services and putting ads on them. That
core business means Google focuses a lot on running those web services
effectively, which is why they are now focusing hard on the business angle of
running other people's web services.

Google spends $2+B per quarter on capital expenditures; where do you think
that money is going if not datacenters?

Using Facebook as a default login was never core to Parse (e.g. PFUser:
[https://parse.com/docs/osx/api/Classes/PFUser.html](https://parse.com/docs/osx/api/Classes/PFUser.html))
nor is Parse nearly on the same scale as Google's investment in its
datacenters / web software efficiency.

~~~
snissn
And at some point your company might decide that it's already its biggest
customer for its cloud products and close it to the public.. or it might not..
but it's up for me to decide where I host :)

