
Evernote’s transition to Google Cloud Platform - ShanaM
https://blog.evernote.com/tech/2017/02/08/part-1-evernote-service-options-migrate-google-cloud-platform-gcp/
======
cm2187
I can understand that a company wants to do away with the distraction of
having to worry about running a datacentre but is using one of the major cloud
services cheaper than running your own? Amazon and Microsoft seem to be making
healthy profits off their cloud services. That's part of your cost.

Now it may be that even if more expensive, it is a better use of your capital
(renting instead of owning so that you can invest your capital in higher
return investments like hiring better people). But is it cheaper, at an
equivalent level of service / redundancy?

~~~
vidarh
> but is using one of the major cloud services cheaper than running your own?

At list prices? Pretty much never. Nobody sane with installations like
Evernote are paying remotely close to the published prices, though.

The cloud providers rarely make sense in terms of cost for smaller businesses
unless you heavily rely on batch jobs (spinning servers up/down) or other of
their more advanced services. There are many valid reasons for smaller
companies to consider cloud deployments, but I've yet to see one where it was
the cheaper alternative.

~~~
edblarney
"> but is using one of the major cloud services cheaper than running your own?

At list prices? Pretty much never. "

???

It's almost always 'cheaper' for small companies to use the cloud.

It costs a lot of money to pay people to figure what hardware to buy, install
servers, networking gear, maintain it, pay for bandwidth, backup etc.

But the cost becomes 'non starter' when you start to access all the nice
things that cloud offers: Lambdas, S3, easy backups, etc. etc. and the fact
that it 'scales on demand'.

Then of course there are the financing aspects: you don't pay for expensive
servers up front, you 'lease' them effectively - which has serious working
capital benefits.

All those things have considerable business value.

So sure - if you want to compare only 'server instance costs' \- maybe 'buying
your own instance' \- specifically on a server by server basis ... maybe it's
cheaper to buy a server.

But for most small businesses - the advantage of the cloud far outweighs the
cost.

I'd argue that 'rolling your own infrastructure' would be risky and costly
initiative that boils down to a 'cost optimization' only, and can't really be
considered until a business reaches an operational degree of maturity and
visibility wherein someone could really sit down and plan quite far into the
future.

And even then I'm not so sure - this is why so many big and successful
companies still use the cloud.

~~~
Veratyr
> It costs a lot of money to pay people to figure what hardware to buy,
> install servers, networking gear, maintain it, pay for bandwidth, backup
> etc.

> Then of course there are the financing aspects: you don't pay for expensive
> servers up front, you 'lease' them effectively - which has serious working
> capital benefits.

There are solutions other than cloud and going to the extreme of owning
servers, switches, maintenance, bandwidth and space in a datacenter.

OVH (an example due to my familiarity) will sell you a dedicated server with 2
x E5 2650v3, 256GB of RAM, 4TB of disk and 500Mbit guaranteed bandwidth (1Gbps
burst) for $269/mo. This includes most of the annoying stuff, like replacing
failed components, dealing with network issues and such. The bandwidth,
conveniently, is also not limited aside from the speed you're given. You can
use 500Mbit/s for every second of a month if you want. That's 164TB.

The cost of a n1-highmem-32, which is the closest alternative I could find on
GCE is $1030/mo, plus disk ($348/mo for 2TB), plus bandwidth (164TB is $13k
from Google).

Also, the cost of owning and running your own hardware isn't necessarily
expensive, particularly if you're small. When you're running a fleet of
thousands, yes, you definitely want dedicated people for that. When you can
fit all your infrastructure in a single rack, it's not much trouble to call up
Dell and ask for a quote, install it in a local DC then forget about it. Most
DCs give you "remote" or "smart" hands that can do basic maintenance like
swapping a failed HDD for low or no cost.

Dedicated/colocation aren't for everyone but they're certainly not as scary as
some people around here seem to think. And in terms of cost, I daresay that if
you're doing anything bandwidth heavy, the cloud is near-certainly costing you
insane amounts of money. You can get decent gigabit transit (~300TB/mo) these
days for $400/mo while Google, Amazon etc. charge in the thousands for that
kind of traffic.

------
quanticle

        Evernote’s engagement with Google engineers was a pleasant surprise to 
        McCormack. The team was available 24/7 to handle Evernote’s concerns 
        remotely, and Google also sent a team of its engineers over to Evernote’s 
        facilities to help with the migration.
    
        Those Google employees were around to help troubleshoot any technical 
        challenges Evernote was having with the move. That sort of 
        engineer-to-engineer engagement is something Google says is a big part of 
        its approach to service.
    

It's interesting to read that, given that one of the perennial complaints
about GCE on Hacker News is the relative lack of support compared to AWS or
Azure. Is it just that when you're a customer that big, even Google is willing
to give you personalized service? Or is this a sign of a change in the GCE
support model?

~~~
duncanawoods
Option 3. This is PR and not a postmortem containing facts you can use to
inform your decisions.

I fear there is no journalism here. No attempt to uncover real motivations,
nothing remotely negative, no dissenting comment from current/past engineers,
nothing about who paid what to whom, the incestuous relationships of ex-
Googlers (Evernote CEO is ex-Google), how much Google is losing to take on
high profile customers, who is placing these stories or anything other than
pushing the narrative that "everybody must move to Google Cloud".

~~~
TillE
And it's not merely a positive piece; those quoted paragraphs literally read
like marketing copy.

------
unix-junkie
The Stanford HPC team reported last year having sent more than 2PB to Google
drive using Lustre/HSM. This was using the public API and the fact that gdrive
is free for *.edu sites.

See
[https://www.eofs.eu/_media/events/lad16/07_thiell_cheap_n_de...](https://www.eofs.eu/_media/events/lad16/07_thiell_cheap_n_deep.pdf)
(slides) and
[https://www.youtube.com/watch?v=WbE0nl5V8WE](https://www.youtube.com/watch?v=WbE0nl5V8WE)

~~~
khc
2PB costs 6 figures for regular GCS a year, I probably wouldn't rely on Google
to continue to provide that for free.

------
yeukhon
So a large part of choosing GCE is possibly because CEO was a Googler and he
was very familiar with how Google works and his contact at Google helped him
scored a better negotiation and possibly even better support. Although I won't
deny GCE team would snatch Evernote regardless because it's an important
client to show off in GCE's client portfolio.

> The decision to go with Google over another provider like AWS or Azure was
> driven by the technology team at Evernote, according to Greg Chiemingo, the
> company’s senior director of communications. He said in an email that CEO
> Chris O’Neill, who was at Google for roughly a decade before joining
> Evernote, came in to help with negotiations after the decision was made.

~~~
origami777
I'm sure that had nothing to do with it. Free donuts. That's how you influence
and win enterprise accounts.

~~~
yeukhon
Evernote didn't get everything for free. They still pay Google with some NDA
stuff in the contract (better support, better pricing model, some free stuff
here and there). I wouldn't have a second doubt the CEO's past employment at
Google influenced the decision. After all, he had to sign it.

~~~
origami777
You're right; they didn't get anything for free. They gave up access to their
logo and the story for discounts. That's how the game is played. And there's
no problem with that! Both sides win.

I just thought it was funny that they mentioned the donuts at all.
Sales/solution teams bring stuff like that all the time. It's totally
irrelevant, until you realize that the original article was a puff piece.

~~~
yeukhon
I see. I missed that reference, sorry I didn't catch it on time. I remember
those days when browsers would send each other cakes...

------
grx
Nice Google ad, well done!

> Evernote wanted to take advantage of the cloud

Evernote _is_ part of the cloud. Buzzzz

> Evernote houses a large amount of unstructured data, and the company is
> looking to do more with machine learning.

Meaning private data? Is this covered in the TOS? Can I influence the usage of
my data by Google before migration?

~~~
acqq
> Meaning private data? Is this covered in the TOS?

Your question is right on the mark. The last attempted privacy policy change,
less than two months ago, did cause some reactions:

"The latest update to the Privacy Policy allows some Evernote employees to
exercise oversight of machine learning technologies applied to account
content"

[https://techcrunch.com/2016/12/14/evernotes-new-privacy-
poli...](https://techcrunch.com/2016/12/14/evernotes-new-privacy-policy-
allows-employees-to-read-your-notes/)

"The policy changes have to do with machine learning, which Evernote says it
is using to “help you get the most out of your Evernote experience.” Evernote
wants to let its machine learning algorithms crunch your data, but it doesn’t
want to stop there — the company also wants to let some of its employees read
your notes so it can ensure that the machine learning is functioning
properly."

Also:

[https://techcrunch.com/2016/12/16/evernote-u-
turn/](https://techcrunch.com/2016/12/16/evernote-u-turn/)

------
abc03
Evernote wanted to take advantage of the cloud to help with features based on
machine learning that it has been developing

I noticed there are other companies citing machine learning as a reason to
move to one of the big three (Xero.com gave the same reason when moving to
AWS). Could someone give more Background Information to this and what are the
exact Advantages?

~~~
diamondo25
Think of provide better suggestions, analyzing comments for quality and reach
and other things you can figure out when you have loads of data.

~~~
acqq
According to this article, they intended to analyze the notes taken by their
users and also have their "employees read your notes so" they "can ensure that
the machine learning is functioning properly" and that to "help you get the
most out of your Evernote experience."

[https://techcrunch.com/2016/12/14/evernotes-new-privacy-
poli...](https://techcrunch.com/2016/12/14/evernotes-new-privacy-policy-
allows-employees-to-read-your-notes/)

I believe that they simply wanted to enable the targeted ads or sell the newly
generated data, and that they still want to do this. But maybe somebody here
has more insight.

------
StreamBright
Running a datacenter is still a hard problem to solve. Evernote just realised
that.

~~~
patrickg_zill
Running a datacenter is a solved problem. Period.

What I get from this is that they want to use Google's available horsepower in
machine learning and maybe, trying to build their own TensorFlow equivalent
would have cost them extra and have added risk of becoming rapidly out of
date, since Google is investing heavily in it.

------
nealmueller
>Right now, the company is still in the process of migrating the last of its
users’ attachments to GCP.

I wonder if Evernote users can tell if they are on GCP or the previous
infrastructure. I suspect not.

~~~
rifung
According to this article, you'll see some improvements.

[https://www.engadget.com/2017/02/09/evernote-google-cloud-
pl...](https://www.engadget.com/2017/02/09/evernote-google-cloud-platform/)

------
Kiro
I would pay good money to Evernote if they had a "backup all my data multiple
times, removing all possibilities of losing anything". I would gladly pay for
Premium if that was an option.

~~~
therealmarv
You should look at CloudHQ. Their PDF backups to e.g. Google Drive preserve
everything out of Evernote. Used it once to get away from Evernote while still
preserving search of old stuff.

~~~
Kiro
Thanks. I don't want to get away from Evernote though.

~~~
therealmarv
You can also do continuous backups there.

~~~
Kiro
True, that's an option! Thanks again.

------
20years
Next up -> Google acquires Evernote.

~~~
therealmarv
Nah. It's not a good simple nice solution. Google also hates Desktop Apps
(except their browser). And IMHO Google Keep makes a much better job for short
notes without wasting whitespace.

------
veritas213
rSync FTW?

------
neotek
That was a surprisingly frustrating article to read. It kept repeating the
exact same things just worded slightly differently over and over again, like
the four or five times it mentions machine learning being a major part of
Evernote's decision to go with Google, without ever explaining what, exactly,
that machine learning would be.

The whole thing reads like a press release targeted at an audience that
doesn't know anything about the actual mechanics of cloud computing.

Edit: Fortunately Evernote have published a great five-part series on their
blog[1] which goes into much greater detail about the transition to GCS,
although there's still no explanation of how machine learning fits into their
future plans.

[1] [https://blog.evernote.com/tech/2017/02/08/part-1-evernote-
se...](https://blog.evernote.com/tech/2017/02/08/part-1-evernote-service-
options-migrate-google-cloud-platform-gcp/)

~~~
dang
Thanks. We changed the URL to that from
[http://www.pcworld.com/article/3167594/data-center-
cloud/her...](http://www.pcworld.com/article/3167594/data-center-cloud/heres-
how-evernote-moved-3-petabytes-of-data-to-googles-cloud.html).

~~~
neotek
Thanks mate, I wish other online communities were moderated this well!

------
tpetry
Sadly the article never describes HOW they transfer the data. Simply sending
3PB on the wire will cost a fortune. Maybe Google does have some non-public
service like AWS Snowball?

~~~
aluminussoma
Nothing against the author, but it sounds like another story that Google PR
pitched and influenced. It is normal these days. I am sure AWS does the same.
That is why the technical details may be light. The submitter is likely a
Social Media Manager for Google (nothing wrong with that - kudos for not
obscuring it!)

I, too, have wondered about the technical details when I see stories like
this. If the data was originally stored in S3, they probably used the Cloud
Storage Transfer Service
([https://cloud.google.com/storage/transfer/](https://cloud.google.com/storage/transfer/)).
The secret sauce appears to be parallelization of the transfers.

When I copied hundreds of GB of data from one bucket of to another AWS bucket,
it took me days to do! Then I realized it was because I was running a single
threaded process. The slowest part was getting the list of objects to copy. If
I had a separate, faster index of the objects, then I could have done a copy
in parallel much more quickly and max out the bandwidth available to me.
(Maybe that is part of the rationale of Netflix's S3mper?).

I'm curious to hear other thoughts on accomplishing fast transfers of S3 data.

Edit: theoretically speaking, if you had 1 TB of data to transfer, and a
paltry 100 Mbps (12.5 MB/s) connection, you could do this in under 24 hours if
the entire network was fully utilized. Not a network engineer so go easy on me
if I am wrong!

As for costs, sending 1 TB out will cost $90,000 in S3 transfer costs, before
any other discounts.

~~~
belril
Hi! Author here. (Nice to meet you.) I can tell you that Evernote, not Google,
pitched this story to me. While the Google Cloud PR team was aware of its
existence, they did not contact or attempt to influence me when it came to its
content. (Literally, all that they did was confirm the on-site presence of
their engineers, and that an engagement of that sort wasn't a one-off favor to
Evernote.)

Unfortunately, a technical deep dive on the exact parameters of the transition
wasn't really in scope for this piece. I tried to include detail where I
could, and Evernote has a really wonderful 5-part blog series that goes into
many of the details of the transition. (It's now linked in this post.)

To provide some additional details on the transfer: Evernote was going from
data centers in California that it owned and operated to Google, not from S3.
Technically speaking, the migration isn't 100% complete, as Evernote is still
finishing the transfer of its attachment files to Google.

Evernote did not disclose the exact details of how that transfer was executed
on the record, and thus it did not end up in my piece. Thanks for reading!

~~~
aluminussoma
Thank you for your response and adding clarity around how the story came to
be. You must forgive us if we are inherently suspicious.

~~~
belril
Happy to add clarity where I can -- it is my job, after all. Suspicion and
skepticism are healthy, and I totally hear the criticism that the piece feels
marketing copy-esque.

For what it's worth, the Evernote team seems genuinely happy with the way the
transition went, along with their interactions with Google.

------
djmobley
Guess Evernote are prepping for a sale/IPO?

Moving into the cloud at this stage doesn't make any sense otherwise.

~~~
StreamBright
Why would it not make any sense? They probably did an analysis on this and
decided that GCP overall is a better solution to their problem.

I regularly move companies to the cloud because they cannot afford to setup
their on services and just want to focus on the business side of tech and
write software only that delivers business value. These companies tend to use
queue services, CDNs, scalable file stores, clustered database servers,
disposable environments for research. If they had to operate these services,
it would be a horrendous cost and huge amount of effort to set this all up and
keep it running. Cloud vendors to these things for a reasonable price and it
is ready to be used today. I find it very hard to think of any scenario where
cloud does not win the game, this is why Google and Microsoft is betting on
it.

~~~
vidarh
I regularly move people off cloud providers once they realise how expensive
they are.

We usually cut peoples fully loaded costs in two or more by moving them to
dedicated hosting. Far more if they're bandwidth intensive (Google and AWS
charge in the region of 5x to 50x as much for bandwidth as most other hosting
alternatives)

There is certainly segments that use a lot of disposable environments and
complex services where it makes sense to go to cloud providers, but I've yet
to see a client where it came out as the cheapr option.

If they're big enough to negotiate steep discounts, then that certainly
changes - I know one company that has 75% discounts over the public prices
from one of the major cloud providers, and they're not that large (yearly
spend in the high six digits).

~~~
StreamBright
How would you move an S3 user off cloud? Do you have networking, load-
balancing and performance experts on call if something goes sideways?

I guess we are aiming for different companies, my clients want scalability and
do not want to deal with BGP issues, power outages, vendor RMAs and the other
things people forget to add to the basket when comparing cloud vendors to
dedicated hosting. Amazon buys networking equipment and servers on a discount
you would not able to buy, they also get a great discount for the networking
bandwidth and energy as well. It would be very hard to beat them in the
datacenter game. I worked for companies where we were in charge of datacenters
(I have built few) and companies using cloud services. Based on my experience
you cannot meet with the reliability, security, availability and efficiency of
AWS/GCP when using a small dedicated hosting provider.

~~~
vidarh
> How would you move an S3 user off cloud? Do you have networking, load-
> balancing and performance experts on call if something goes sideways?

The vast majority of S3 users have loads that can be handled with a couple of
low end dedicated servers.

If you can afford to pay the S3 bandwidth charges, then by the time you can't
serve the load with a handful of servers, you're paying so much that you can
afford hefty retainers with consulting/devops companies larger than mine and
still make massive savings.

S3 _can_ be cost effective, but only if you rarely retrieve the data.

> I guess we are aiming for different companies, my clients want scalability
> and do not want to deal with BGP issues, power outages, vendor RMAs and the
> other things people forget to add to the basket when comparing cloud vendors
> to dedicated hosting.

Same with my customers, which is why they pay me to set up and operate systems
for them. Still ends up far cheaper than cloud setups.

> Amazon buys networking equipment and servers on a discount you would not
> able to buy, they also get a great discount for the networking bandwidth and
> energy as well.

I'm sure they do, the problem is they're not passing those savings on.
Especially on bandwidth.

As an example, S3 bandwidth out for up to 10TB/month is $90 per TB. I pay ~$2
per TB for most of my own projects. Depending client needs we use providers
that charge from ~$2/TB to $20/TB.

It sounds to me as if you've not actually looked at prices at alternative
providers.

------
some1else
Please excuse the loose relation to this specific aspect of their
relationship, but there's something bothering me about Google & Evernote.

I recently set up a consumer grade LG Android smartphone. A few days later I
got a push notification inviting me to install Evernote[1]. I wonder who gets
paid for that: the telco, LG, or Google? How does one unsubscribe from such
app adverts? By disabling Google Play Store notifications? I really didn't
know push ads for apps are a thing. Besides the annoyance, it seems like an
unfair advantage over all the other top apps.

[1] [https://cl.ly/2U0K2H3S071E](https://cl.ly/2U0K2H3S071E)

~~~
efraim
Tap and hold the notification and Android will show you what app it belongs
to. You can also view the most recent notifications in Android settings.

~~~
some1else
Thanks!! I was hoping someone might drop a tip like that, so I kept the
notification around. Indeed.. After a long press, it reveals that it's owned
by the Evernote[1] app. Apparently it comes preinstalled. The gear button
results in a crash, but I was able to block notifications under the info icon.
Probably best to take the time and purge all the bundled apps, before another
one pops up.

[1]
[https://d17oy1vhnax1f7.cloudfront.net/items/1Q1L2Y2f3R3v0m1n...](https://d17oy1vhnax1f7.cloudfront.net/items/1Q1L2Y2f3R3v0m1n3v20/16667748_10154110255416571_1615101833_o.png?v=32ac7a0d)

