
Why no company that values their data should ever "Go Google"  - e1ven
http://e1ven.com/2011/04/14/why-no-company-that-values-their-data-should-ever-go-google/
======
steevdave
Why don't people make backups anymore? I know that using Google Apps is much
easier than running your own stuff in-house. It requires much less time and
resources, but at the same time there are still best practices and disaster
recovery plans that should _always_ have something about backups in them. And
just having them isn't good enough. They also need to be followed. Having
worked at a place without these ( and attempting to get them implemented )
versus where I currently work that has them and follows them, I have to say,
there are a lot fewer times where I feel like screaming here. Do it, and stop
expecting 100% uptime and availability of all docs/important information from
any service, internal or external. Nothing is 100% but there are many ways to
make it less stressful.

~~~
sdizdar
100% agree. We are developing service which tries to solve some of these
problems, so I'll do a shameless plug here.

One way to address these kind of problems is to use cloudHQ to synchronize
your Google Docs with SugarSync (clickable: <http://cloudHQ.net/sugarsync>).
(We are releasing dropbox this weekend).

Here is our idea: there are many different cloud storage services and each of
them is very good for a particular use case but not so good for other use
cases. For example, Google Docs service is excellent for collaboration but not
so good for storage. On the other hand, Dropbox or Sugarsync service is
excellent for storage but not so for collaboration.

So if data is continuously synchronized between Google Docs and lets say
SugarSync storage, then you can always fall back to SugarSync when Google Docs
becomes unavailable and vice versa. In order words, you can use other service
as a standby site in case of service unavailability, malicious or accidental
deletion of files, etc.

~~~
edanm
That sounds amazing, actually. A service which synced Google Docs documents
directly to my Dropbox is definitely something I would be interested in.

------
msy
This is precisely why I think it's insane that people trust their email - the
nexus of their online presence, normally the key to every account and their
primary communication tool for everything from conversations to account
statements to a free service. It's not 1996 anymore, this stuff matters. I pay
for Rackspace email. When I have a problem I can call a human and get an
answer. As far as I'm concerned it's a bargain.

Google doesn't do human, it doesn't matter if it's a free gmail account, a
serious-money adwords account or your entire business on apps, you're just a
number if something goes wrong you're on your own, shouting at the unmoving
monolith.

~~~
calloc
I've got Google Apps for Business, and when all of our mailboxes were
unavailable I contacted Google at the number listed on their website, talked
to a human, got it resolved in about 30 minutes.

~~~
wewyor
I'm not sure that they followed the support instructions, the email excerpt
said to reply if that wasn't the right solution but it just says they sent
another support email.

It probably would have been better to call, I mean you are paying for 24/7
phone and email support.

I haven't had any problems with getting support through Google Apps for
Business, I might be an exception though.

~~~
e1ven
Not that it matters much, but I called them several times, and replied to the
original email. But I think you're focusing too much on the specifics.

The problem is that your mail is crucially important to you, but rather less
important to Google. In such a scenario, it makes sense to change your plan to
something under your control, even if it's painful.

------
ttyS0
Yes, because we all know that when we run these systems outselves, they never
go down.

~~~
chaosmachine
When Google breaks, you're basically stuck waiting for them. If your own
system breaks, at least you can attempt to fix it.

~~~
jrockway
When your hosting company goes down, you're basically stuck waiting for them.

The only way to ensure that you never have downtime is to not use the
Internet, and keep all your servers connected to a nuclear reactor, all inside
a mountain. That covers power and not being flooded or hit by a tornado,
anyway.

Even then you'd better be careful not to trip over any of the power cables or
type "rm -rf /" as root.

~~~
e1ven
For mission critical things, you want to reduce the points of failure, and
ensure that when things DO go wrong, you have a reasonable escalation path.

It's now been over 8 hours since it went down, and no fix from Google yet. 4+
days on the missing file.

If I was running in house, I could have entirely restored the mail server from
tape by now. I could have swapped over to a hot-spare in a few minutes. I
could have failed over to our backup internet service. I have a lot of
options.

With Google, my option is to wait.. And hope my business doesn't lose too much
money while Google gets around to fixing it.

~~~
beambot
As another example of handling cloud services going offline, consider this
case by SmugMug:

[http://don.blogs.smugmug.com/2007/01/30/amazon-s3-outages-
sl...](http://don.blogs.smugmug.com/2007/01/30/amazon-s3-outages-slowdowns-
and-problems/)

"""So what are we doing differently? Simple. Amazon serves as “cold storage”
where everyone’s valuable photos go to live in safety. Our own storage
clusters are now “hot storage” for photos that need to be served up fast and
furious to the millions of unique visitors we get every day. That’s a bit of
an oversimplification of our architecture, as you can imagine, but it’s mostly
accurate."""

You can always maintain a hot-backup, fail-over of your site on your own
servers -- perhaps with reduced functionality until the scalable cloud
services come back online. For a mission-critical site, this would seem to be
a reasonable tradeoff.

~~~
jlangenauer
Are you, by chance, a Python programmer?

~~~
jrockway
English doesn't have """triple double quotes"""?

------
yesbabyyes
Dropbox has an advantage here, in that all files are stored local (mirrored,
really) in user's computers, and they are backed up. It doesn't scale to
shared editing, though.

I'm turning more and more to plaintext/markdown on Dropbox.

~~~
roc
Dropbox-style mirroring is still susceptible to the hypothetical "service
problem killed/corrupted a file, the mistake was happily sync'd down to all
mirrors."

In short: if you're reliant on the cloud service for your _backups_ (as
distinct from your sync'ing), you're at risk.

~~~
cezary
Dropbox stores different versions of files so even if files are
corrupted/deleted and this change is sync'd up to them, you can still restore
your files.

~~~
edanm
But so does Google Docs. The problems start happening when all versions of a
document are corrupted.

------
eitally
In case anyone cares to hear a counterpoint, I run a nearly 20,000 seat Apps
domain and the service and transparency we receive from Google is leaps &
bounds better than from any vendor of similar size. Perhaps my happiness makes
sense given how much we're paying them compared to, say, e1ven's company.

------
gregable
I think a lot of this is not about security or service level. It's
psychological, it's about control/trust. When it's under your control and you
can fix things yourself, you psychologically feel more safe. If you rely on
someone else and they say "we're working on it" but don't give you an
estimate, you feel a lack of control. It may be that doing things yourself has
worse uptime/performance/etc than relying on someone else, but the
psychological effect of that loss of control when something bad happens is
huge.

------
Vvector
<http://www.google.com/apps/intl/en/business/features.html>

Support and reliability

24/7 phone and email support for critical issues

99.9% uptime guarantee SLA

~~~
e1ven
We have that ;) I called them, emailed them, and asked for updates in as plain
of terms as possible.

4+ days on the missing file, 8+ hours on broken email.

~~~
VengefulCynic
Quite simply, Google appears to be in material breach of your SLA. Where is
your lawyer?

~~~
pyre
99.99% over what time period?

------
minalecs
I might of missed it, but are you using Google Apps (free) or Google Apps for
business ? <http://www.google.com/apps/intl/en/business/features.html>

~~~
e1ven
Business. We pay them $50/user, for quite a few users. Thanks for the
clarification, I've edited the top paragraph to make that clearer.

------
schwit
I would be afraid that in a dispute Google could do a PayPal and lock you out
of your data. I would also be concerned that Google would provide law
enforcement access to your data without your knowledge, consent or a warrant.

------
slewis
Just Googled around and found <http://www.backupify.com/> or maybe something
like <http://www.ltech.com/google-apps/products/google-docs-backup>

Has anyone tried those products?

~~~
Erwin
Mixed reviews on the Google Apps marketplace for backupify:
[https://www.google.com/enterprise/marketplace/viewReviews?pr...](https://www.google.com/enterprise/marketplace/viewReviews?productListingId=5941+16825965296321823984)
\-- 27 users take 21 days to backup? Might well be the Google Apps
infrastructure that's not up to the task. Looking at my company's Google apps
domain, we probably have several million threads and are close to a terabyte
of data. I imagine backing that up via IMAP could take a while.

~~~
jgarmon
Those are fair criticisms of Backupify to date. We've recently totally
reworked our throttling mechanisms, to our initial backup times are decreasing
rather rapidly. But, to your point, we are at the mercy of how fast Google can
give us the data through their API. That said, given Google's general uptime,
the likelihood of Google losing a file before we can back it up is pretty
remote. Not zero chance, but really unlikely.

------
res0nat0r
What type of SLA's did you agree to when you signed up for Google Apps?

~~~
e1ven
That's sort of the point.

It's a big deal to us, not a big deal to them. We'd pay extra for a tighter
SLA if that option were available, but as it is, on outage hurts us a LOT, and
them not very much.

That would seem to be prescription for bringing it in-house, despite how
little I want to do that.

~~~
16s
Find a small local IT company to use. It's been my experience that they are
very responsive and do really care and many of them are very high-quality (old
Unix guys). Google are just so large and so far away. They can't be everything
to everybody (try as they might).

------
mark_l_watson
Personally, I use Google Docs every day, and I back up all of my docs on a
regular basis (easy to do). Same with my GMail: use POP3 to keep a backup copy
locally.

I would not consider running my own services and not back up my data. If you
use GMail and/or Docs for free or as a paid customer, still, why would you not
make local backups?

------
powertower
It gets much worse than this.

A lot of businesses are completely dependent on Google via organic search and
AdWords for 90%+ percent of their revenues (without even realizing it).

What happens when you log into your AdWords account and you’re greeted by this
cryptic message:

<http://www.devside.net/images/adwords-account-suspended.png>

Except that you've done nothing wrong, have never been warned, are not a
spammer, and are completely legitimate.

 _What happens is that_ after contacting Google you soon realize that a
monopoly with no customer support is the most dangerous one.

------
nolite
Please stop paying them.. companies seem to only learn through the wallet

------
joe_the_user
No person or organization should put itself in a position where it depends
only on an external source for either security or backups. Whatever else might
or might not be "mission critical", I think this inherent is.

Come back and tell me that relying on using Google docs while keeping local
backups is problematic or awkward. That would be a fair complaint.

Toshiba isn't going to help me if my drive goes down either. Whether they send
a nice or not nice email isn't the point.

~~~
gregable
This sounds great, but in practice is impossible. Are you personally as an
individual going to take care of 100% of your own backups? Are you going to
drive the tape over to the other facility that you own, make regular checks
that you can restore, etc? No.

Ultimately you are going to rely on someone else and something else. Be it
your IT guy, the company that hosts your backups, the power company, the
company that makes the machines you keep your backups on, etc. You can
distribute this so that you rely on lots of someone else's and any one or more
of them can fail, but often you'll still have one point of failure somewhere
(the guy who monitors the distributed backups for example).

~~~
joe_the_user
No,

I mean each organization needs to have in-organization backups and each
individual needs to back-up their machine.

Yes, _my_ IT guy. I need to have an institutional relation with my backup guy,
not just a paying relation. The IT guy could send the data to three different
online backup services but me having a personal and institutional relation
with him makes sure that whatever scheme he uses will _remain_ reliable as
other things change.

The point is that every organization has at least one _mission_ , one task
that it is worth doing within the organization rather than offloading to
another entity. And backup is always going to be close to that mission.

------
atacrawl
I worked for a company that switched to Google Business a few years ago. The
one thing I'll say about the experience is that administering the back end
totally sucks.

The UI in a lot of places makes no sense. For instance, there are settings for
when you set up an email list that are labeled so obtusely that you're not
really sure of what you're selecting.

Another thing that totally bit us once was when dealing with document
ownership in Google Docs -- a lady in marketing had created a bunch of
documents, then later left the company. After scouring the back end for about
an hour (literally), I concluded that there was NO WAY for the admin user to
reassign the owner of the document to a different person. I ended up having to
log in as that person (luckily we kept people's accounts active for 6-12
months after they left) and switching the owner that way. Totally absurd. (If
I'm wrong about that, please do tell.)

As far as customer service goes, it's definitely a minus. They don't make it
easy to tell you who to call or email when you do have a problem, and once you
get there, the time before your problem is resolved can vary wildly (from
hours to days). That really sucks when you have a salesman breathing down your
neck because he can't use his email.

------
slewis
So are you actually switching off Google?

~~~
e1ven
We've moved from Google Docs to Dropbox, at least as a placeholder; We're
still trying to find a better long-term solution. AeroFS is leading the pack,
once it is out of beta.

For Email, we plan to, but it's a bit more complicated.

I'll need to imapsync each account, so it's going to be messy, so I'll need a
week or two to plan it out, and build/test a new Zimbra server.

~~~
eduardo_f
Try <http://www.MigrationBox.com> if you want to save the messy.

------
mariusmg
In another news scientists discovered that water is wet...

------
simonhamp
This is precisely the reason why I don't use Google Docs for any mission-
critical documents. Store them in Dropbox or some other shared, automatically-
backed-up system where you can easily retrieve a copy from somewhere.

Always have redundancy too. We keep most of our files in Dropbox. But we all
use Time Machine too. We also use version control for a lot of our work.

This might seem like overkill, but each has a specific use and purpose with
the added bonus of providing us with backup redundancy.

------
lwhi
I think this is the problem with all cloud services. The liability costs
involved in dealing with worst case scenarios are potentially so huge, the
majority of providers have terms and conditions which relinquish all
responsibility when bad things happen.

The only real solution is backup, and the only backup that can truly be relied
upon is one you make yourself; in which case the point in making use of a
cloud service in the first place is pretty much reduced.

------
VladRussian
>We’ve been running flawlessly for 6 months, paying them $50/user to avoid
handling it ourselves.

if you open Excel and put some typical numbers into it (like commodity
hardware cost, networks, electricity, data center mortgage, basic maintenance
employees, 40% margin, etc...) you'd see that $50/user/month is just basic
usage - timeslice to run provided software on provided hardware. There is no
room in that number for a real person dealing with your specific issue. An
elephant can't squeeze into a needle eye, even if he promised it to you in
writing and accepted the money - just look at the elephant and at the needle
with your own eyes.

~~~
jsnell
You might already know this already and that's just a typo, but the actual
price is $50/user/year, not per month.

~~~
VladRussian
if it is per year then it is a real miracle of mass scale production at
Google.

