
How upgrading to Google Apps for Business killed my company's email for 6 hours - ValentineC
http://floriancornu.blogspot.com/2013/05/how-upgrading-to-google-apps-for.html
======
dangrossman
Rackspace Mail. $2/mo/mailbox. 100% uptime SLA. 24/7/365 phone, email and chat
support -- you can call and talk to someone with power to solve your problem
at 3AM on Christmas Eve. Big mailboxes, configurable backups, good spam
filters. Instant e-mail notifications in your choice of native desktop/mobile
clients with IMAP push, or use their hosted webmail.

If you aren't fully addicted to the Gmail interface, there are better options
for business mailbox hosting.

~~~
andybak
No reason not to use a 3rd party IMAP/POP and still use the Gmail front-end.

Best of both worlds.

~~~
mike-cardwell
You're just adding more points of failure.

~~~
pyre
I thought that a single point of failure was a bad thing. Sure you might want
to only have a single place where things can fail, but if that failure is
catastrophic...

~~~
nknighthb
"Single point of failure" means a point at which a failure effectively breaks
the entire system or workflow. A system may have many SPOFs, which is worse
than a single SPOF, which is, in turn, worse than no SPOFs (that is, the
desired case is no single failure kills forward progress).

~~~
pyre
This is true, but sometimes be introducing a new risk (point of failure) to
the system, you get the ability to route around another. In this instance,
separating POP and Gmail gives you the ability to route around Gmail when it's
down.

------
bravura
tldr; Author experienced an 6-hour outage in Google Apps service, after
upgrading to a paid Google Apps account. This behavior is not documented.

~~~
sneak
Is "outage" the right word? That suggests that it's an error, or down for
everyone.

They really screwed the pooch, here.

~~~
300bps
>Is "outage" the right word? That suggests that it's an error, or down for
everyone.

As someone who has been in IT for 20 years, I can tell you're in IT. This type
of semantics drives business users crazy.

Business User: Hey my system is down!

IT Person: No, it's not down.

Business User: I can't log in to it.

IT Person: We are doing scheduled maintenance so the system is unavailable,
but it's planned maintenance so it's not down.

~~~
sneak
Wrong direction. This is not failure-apologetics.

Outage suggests temporary and regrettable.

This is more like Google just straight up fucking people over via neglect. I
was suggesting that stronger negative terms be used.

------
biot
The solution to this is for Google to explicitly tell the end user that the
upgrade involves a migration to bigger and better infrastructure, during which
their accounts will be inaccessible for a number of hours. Then let the user
schedule the migration to minimize the inconvenience. During the migration,
show the account status as "Migrating..." instead of empty. Problem solved.

~~~
Silhouette
_The solution to this is for Google to explicitly tell the end user that the
upgrade involves a migration to bigger and better infrastructure, during which
their accounts will be inaccessible for a number of hours. [...] Problem
solved._

Sorry, but that isn't even close to a solution. The problem here is that a
business-critical facility was off-line for pretty much an entire business
day, at who-knows-what cost to the business.

Many small businesses that use facilities like this are trading
internationally or otherwise need 24-hour functionality, so the idea that you
could just schedule something to "minimize the inconvenience" doesn't really
help them at all.

~~~
JohnTHaller
It wasn't business-critical before, since they were using the free version.
Businesses don't use free services with no SLA for business-critical things.
Unless they don't know how to do business.

~~~
ozataman
Don't make assumptions on what's business critical and what's not. Not every
business has the same level of free capital, circumstances or domain knowledge
to handle everything theoretically correctly. There is implicit trust in
working with companies like Google; you expect everything to work properly.
They don't have to, of course, but when they don't, articles like this chip
away at the confidence we all have in Google.

------
codev
I had a similar situation where I switched a 50 person charity to Google Apps.
They had a 30 day trial period and during that time the CEO's credit card was
stolen and replaced. I changed the information but the Apps interface said 'No
valid payment information stored' so I called their support and they assured
me the new card was stored and everything would be fine.

Surprise, surprise on the day the account totally shut down. I called in a
panic and after a lot of fiddling about (and reassuring me that they had
recorded that I thought there might be a problem) they got it back up with
everyone's mail after almost a whole day of downtime.

To add insult to injury Google don't run a charity program in the UK despite
the non-profit laws being a lot stricter than the US and it being easy to
validate a charity - so they're paying full rate for Google Apps for Business.

~~~
shrikant
That last part is not entirely accurate. I did some IT volunteer work for a
charity very recently, and they definitely had a free Google Apps for Business
setup. This is in addition to a reasonably generous grant in the form of
AdWords credit.

~~~
ValentineC
Would you have any idea who we're able to contact about this? I'm involved in
a registered UK charity and we were looking to use Google Apps before, but
can't justify the monthly cost of $5 per account.

Thanks!

~~~
shrikant
Sorry for the misleading information, chaps.

I asked the organisation, and found out that it's not an official programme in
the UK. The only one they have here is Google Grants for AdWords -
[http://www.google.co.uk/grants/](http://www.google.co.uk/grants/).

Basically, an enterprising IT volunteer had set them up with vanilla Google
Apps back when it was still free for small business and had a 50 user limit,
and they just got grandfathered in to the new Apps for Business. They've never
had any problems with the user or space limit since they just use the setup
for email, analytics and AdWords campaigns and only have about 4-6 people
"working" there.

Apologies again.

~~~
ValentineC
Ah. The offering used to be called Google Apps for Domains back when users
could still sign up for free 100/50/10-user-limit accounts. Google Apps for
_Business_ , in some of our minds, is the post-December 2012 paid service.

Here's a link comparing the free edition with Google Apps for Business:
[http://support.google.com/a/bin/answer.py?hl=en&answer=17512...](http://support.google.com/a/bin/answer.py?hl=en&answer=175121)

Thanks for getting back to us though!

------
surferbayarea
We had another issue, where the catch-all address started bouncing. Apparently
they have changed the way they handle email routing. So if you upgrade to the
'new google apps', you better be very careful about your email routing rules.
Went through a very similar experience with google customer care. Again turned
out to be a bug in their system. But what sucks is for 3 days our emails were
bouncing! And we're a paid google apps customer...Would be happy to pay 5X,
just want some sort of customer service + acceptable SLAs on issues. And yes,
please have a test team go over important features like enterprise email! I
know it's Google policy to not have features go through a testing team, but
maybe they should consider it for Google Apps.

/end{rant}

------
nivla
If you are relying on a third-party cloud email service, please keep a regular
backup of your data. From random hacking attempts leading to a account lockout
to incorrectly being marked for spamming violations, anything could go wrong.
The easiest way is to install Thunderbird and let it automatically sync and
backup your emails to your computer.

~~~
VLM
From past experience at an email service provider, admittedly a small one, and
quite a few years ago, the most likely failure mode related to internet
outages knocking out the customer's internal email unlike a locally hosted
email server which customers could never seem to understand, and DNS issues
(failure to renew, got scammed / domain stolen because of noobs, etc).
Probably the single biggest support problem was people (on either side)
getting on spammer lists so genuine business communication can no longer flow.

Generally speaking the web interface being down, or the imap server being
bonkers, was very rarely if ever a problem.

------
wc-
For whatever it is worth, here is my experience from a few hours ago: my
google apps trial ended this morning. Google emailed me, I clicked the billing
link in the email and entered my info. After submitting, my account was
immediately reactivated and I received a few emails that had arrived while the
account was suspended. No downtime or anything... Maybe it has to do with how
much data your accounts contain when you switch over to paying for access?

~~~
fpgeek
For a trial, I'd expect you started on the paid infrastructure from day 1,
even though you hadn't paid yet. That would mean no migration and no downtime.

------
iuguy
So the first thing about Google Apps is that you still need to back it up.
Because it's (mostly) pretty easy to use it's easy to fall into the trap that
it does everything right first time and that your data is safe when you make a
change, unlike say running your own infrastructure. It isn't.

------
joemaller1
Different issue with Google Apps: We work with some big pharma companies.
During the free evaluation, Google flagged two of our partners and one of our
project managers as spammers and silently suspended their accounts. We hosted
our email somewhere else.

------
fareesh
I've known of a few cases where the Gmail account associated with Google Apps
becomes unreachable. Rather than upgrading to the paid plan, most of the
people involved just opened a support ticket which was resolved in a few
hours.

------
thracky
I migrated a company with 60 franchises from a broken, old, 25 MB per mailbox
mail system, to Google Apps last year.

The biggest suggestion I have for anyone wanting to use the paid Google Apps
is to go through an authorized reseller. You will get better pricing, and they
generally have faster access to higher level support than going directly
through Google. I had an absolute nightmare of a time with client support
(construction workers with little computer literacy) but any legitimate
problems during the transition were quickly dealt with by Google via the
reseller.

Also, good luck getting a response from the Google Apps sales team in the
first place.

------
mrt0mat0
I used to be IT manager of a company that wanted to save money and they had me
switch them to google business. it was a smooth transition, but the company
wanted it's employees (50ish non tech salespeople) to still use Outlook. It
was ok. google had an app that allowed this. All was good. I left the company
but my friend took my job from me when i left. He now has come across the
problem of google not supporting the newest outlook. he gets new computers in
and has to downgrade the outlook just to get email to work. headache

~~~
Karunamon
They recently got the Office 2013 connector out there, for what it's worth.

Agree that the transition took ENTIRELY too long.

~~~
mrt0mat0
thanks. i hope he noticed. i'll mention it to him.

------
brokentone
Killed it for 6 hours... not convenient, but probably shouldn't make big IT
changes during business hours and without warning staff. Our company's move
from GApps to Office365 + Outlook has been much more systemically affecting...

~~~
kngspook
How are you liking Office365?

~~~
brokentone
Hate it.

------
rexreed
I've been POPing my email since the 90s... am I doing it wrong? Yes, accessing
email archives on multiple machines is a kinda-headache (I replicate my
mailboxes), but never lost email.

------
vidyesh
tl;dr : Whenever you click on 'Upgrade Now' button just click it and go to
sleep. Once you wake you everything should be fine.

Author was awake and saw some horrible things.

------
qwerta
And where is your backup? Oh wait you dont have a backup...

------
rasterizer
So the upgrade took a few hours to complete and it didn't happen instantly in
defiance to the laws of physics. Next time play it safe and upgrade during the
weekend.

The post is dated May 27, is Google planning to announce a new feature for
Apps this week and this is some sort of a preemptive PR attack?

~~~
robryan
6 hours... We run a 4 person company and 6 hours during the weekend would
still be a problem for us.

Given a company larger than 10 might actually upgrade (small amount of shared
email accounts) I could see it causing big problems for some.

~~~
VLM
How do you get 24x7x365 coverage with only 4 people? The "standard" is it
takes about 5 to fill a position 24x7x365 over very long term on a larger
average with absolutely no failure (unstaffed) tolerated. As in, 50 people can
fill 10 positions absolutely positively all the time, but it doesn't downscale
well at all to just 5 people and one desk. You can "fill" ten positions with a
lot less than 50, but even at scale there's going to be a heck of a lot of
time when there's only 8 or 9 people there hence the quotes around "fill". For
instance you can "fill" 10 positions with 30 people but there's going to be a
heck of a lot of time where the supposed 10 positions only have 7 (or even
fewer) bodies actually on deck.

Depending on the business sector of course. If you're a stereotypical weekday
business then a 6 hour outage at 9am on a weekday would be a disaster, but a 6
hour outage from midnight to 6am on Sunday morning wouldn't even be blinked at
because no one cares.

~~~
robryan
I agree with you midnight to 6am Sunday would probably be okay. Just would be
a really bad time to get an email stating an issue with a payment gateway or
marketplace listings or an issue with the website.

