
Google Apps Disabled My Domain - vectorbunny
http://ctovision.com/2012/11/google-apps-assassinated-my-domain-beware/
======
robomartin
Just stop using these services.

Don't use any service provider that does not align well with what is important
for your business. If this includes guaranteed service availability and solid
customer service and the company in question just can't provide these
guarantees, well, don't use them.

It really is that simple.

I have not touched anything Google except for Search and Analytics for
probably three years. I learned early-on where my business needs did not align
with their offerings and we don't use those services. I couldn't be happier.
From my perspective Google is a great company.

There are other approaches to all of the other services. We host our own email
for a number of domains on different private servers. Easy. A license of MS
Office on every machine isn't a problem if you have a real business, just like
a license of the appropriate Adobe suite is almost a must. Tools like
GoToMyPC, for the technically challenged are a no-brainer and they are cheap.
There are a multitude of cloud storage solutions. And, frankly, for a lot of
stuff, there's nothing wrong with hosting your own access-controlled FTP site
to share files with your team (although a paid Dropbox account is oh-so-
simple).

So, yeah, Google is a good company. Just stick to the stuff that works the way
you need it to and you'll be fine.

~~~
kami8845
>We host our own email for a number of domains on different private servers.
Easy.

From what I've heard other people say, running your own mail servers can be
described with a lot of colorful vocabulary, though "easy" not being one of
them. Do you know how much it costs you guys to run your own mail servers
(both in actual "server has power" as well as technical ops cost) compared to
using Google Apps? I've heard numbers in the 10x - 100x range.

~~~
jiggy2011
You don't have to run it yourself if you don't want to. Plenty of people will
take that work of your hands for a reasonable price. Oh and they will give you
an actual phone number you can call.

~~~
thaumaturgy
Huh. For some reason, I'd never considered that people might be willing to pay
for a remote "on call sysadmin". We already admin a handful of mail servers,
that's something we could add to our services.

~~~
iamdave
When you have services like Intermedia.net doing a lot of the heavy lifting
for you, it's insanely easy to sell mail as a service. I will say though,
Intermedia is a LITTLE trigger happy with their routing security; one of our
PC's had a malware infection, already quarantined and was pending removal. Our
entire IP range was banned until it got cleaned up, and the machine itself was
a DMZ'd test machine that we purposely infect regularly for "war games" etc,
so it had no email clients, didn't even send any messages.

~~~
thaumaturgy
There's a reason for this. It occurred to me the other day, after dealing with
a stupid hover.com problem, that it has become almost impossible for network
admins at different networks to talk to each-other.

I have never, not once, in the five or six years I've been doing network admin
work, been able to contact another network admin to report a problem. Instead,
I have to go through incompetent and clueless frontline support first, and
spend hours or days navigating that until the problem is no longer relevant
anymore or I give up.

It became obvious that I wasn't the only one that had given up on contacting
network admins when I recently had to deal with a spam issue (same spammer,
multiple hosting providers, new technique) -- while I tried following the RBL
rules, it was clear that other service providers entirely skipped the "notify
the network admin, give them reasonable time to resolve the problem before
nominating their block for the RBL" step. the victim networks all became
listed on the RBLs within just a few hours of the first waves of spam.

It's just gotten to be too much trouble, nobody bothers anymore, and
unfortunately that will have to include me from now on too. If I see bad
behavior coming from another network, I won't any longer even try to contact
anybody at the other network; I'll just ban their IP and move on.

What a tremendously stupid situation.

~~~
iamdave
Your post is spot on and I've felt every frustration you've outlined. I maybe
should have clarified my post originally to suggest I'm not entirely
suggesting this is a fault of Intermedia, as they don't know our internal
systems architecture. But getting the issue resolved and being able to get the
message relayed that the machine posed virtually no threat to an admin who
could have done something about it was just as much of a chore as you've just
explained.

Granted, the hilarious irony in this is that it was Intermedia who pointed out
"there's an infected machine" on your network, so all of the wrangling around
and sending tracert outputs, just to get a reply weeks later "This IP address
is infected" and the resulting " _That's_ what it was? That machine is just
for testing, we know it's infected." was a bit of a grind.

tl;dr - I know that feel.

------
gojomo
Remember, to receive prompt support for failing Google services, you need to
contact Google via a story at a famous blog, a major social news site, or
(ideally) The New York Times.

~~~
capo
Or just store the account PIN outside the affected service.

~~~
Jamiecon
I have been a paying GAFYD user for ~ two years and wasn't even aware there
_was_ an 'account PIN'! I fear for most people this simple solution may come
with the benefit of hindsight.

~~~
chimeracoder
Well, this is not very reassuring:
[http://support.google.com/a/bin/answer.py?hl=en&answer=6...](http://support.google.com/a/bin/answer.py?hl=en&answer=60233)

Nope, no "PIN" for me under "Support". Worse,

> "Please note that for your security, your Customer PIN will be updated
> periodically.

So even if I find it now and write it down, there's a chance that I still
won't be able to access support at some point down the road if/when I need it.

~~~
irq
Only paid accounts and government/education accounts have a "customer PIN",
apparently.

------
meaty
Good. The more this happens, the more people will realise that contracting out
your critical services to a company with virtually no support and a false
reputation isn't a good idea. It also undermines "the cloud" which is a good
thing IMHO.

Sorry but I have little sympathy and I don't want to hear the "well it works
for me excuses" and the financial advantage crap.

~~~
kin3tic
This is an annoying stupid comment.

Yeah, let's ignore the success stories and the financial benefits because it
what is sort of a race scenario, Google's support failed. This seems like a
bug and resolvable as noted by others in the thread.

You have a really obvious chip on your shoulder.

~~~
meaty
No I'm just well versed in business risk assessment and sound engineering
decisions.

If we were to deploy google apps and that happen we'd be down approximately
£120,000 a day (we actually make money).

~~~
moe
If you're so well versed in business risk assessment then you surely have
heard of the advanced concept called _having a backup plan_.

~~~
meaty
Yes for data. Data is easily recovered, but under the circumstances that your
cloud apps suddenly become instantly unavailable, how do you recover your
applications?

You don't - you have to migrate and retrain everyone, which if you've used all
the features in your cloud platform requires major process and tooling
rewrites.

With an offline solution, you can shelve it and migrate slowly. When someone
else hosts it, you're at their mercy.

~~~
moe
_how do you recover your applications?_

You mean google apps? Well, you could keep a local copy of office around, it's
not exactly rocket science.

 _When someone else hosts it, you're at their mercy._

And when you host it, you're at your own mercy.

Matching the availability of google tends to cost a little more than a
hypothetical day of downtime would.

There are plenty valid concerns about hosting your office in the cloud.
However, availability and TCO are normally the main arguments _for_ google
apps, not the other way round. Those infrastructure costs really add up when
you're talking thousands of seats.

~~~
meaty
OK what about Google Apps Script, Forms etc? You're stuffed there. That's
where the rocket science starts to kick in. It's not as simple as it looks. Oh
and as part of the cost cutting, can you afford to splash out for Office
company-wide suddenly out of the blue?

If you have thousands of seats, you should have the budget lying around to do
it properly and a team of people to handle it. The TCO if you include risk is
an order of magnitude lower if you manage it yourself. I've done the figures
enough times for enough companies to know this.

Most companies operate "on the line". An event like this would take them
offline permanently.

~~~
moe
Well, no idea how you calculate that. Google would have to be down several
days every year only to offset the staffing costs for running your own
equivalent to google apps. And that's not even getting into hardware, licenses
and the expensive windows-lockin that everyone seeks to escape nowadays.

I guess we'll just have to agree on disagreeing here.

------
lambada
Are people still surprised by Google's lack of support?

I think one of the messages here is to take note of any support codes (account
#'s or PINs in this case) and store them outside of the service that requires
them.

~~~
chimeracoder
> I think one of the messages here is to take note of any support codes
> (account #'s or PINs in this case) and store them outside of the service
> that requires them.

These codes "will be updated periodically", so there's no guarantee you'll
have the correct one if/when you need it.

------
Spearchucker
I use Office 365 (Exchange Online) and made the mistake of using my own email
address as the primary Exchange Online address. I was unable to access my
email for close to two weeks. Other than that they've been better that
expected.

What I'm saying is that unless you do it yourself these things will happen. A
service level agreement can only be viewed as "this is what we promise as our
best endeavour", but when it goes south it goes south, and you're at the
provider's mercy.

I made the mistake of regarding"the cloud" as this "machine" that's impervious
to human error.

~~~
aneth4
These things will happen, by usually businesses care about their customers and
make humans available to fix the problem.

Google is notoriously awful at this.

------
markokocic
I haven't read the article. First thing that I saw when visiting this
blog/site whatever was a long pageover list with checkboxes and email input
fields asking me to subscribe to something. There was no close button
immediately visible, so I just closed the tab instead of proceeding.

Sorry, but if you really want your story visible, please stop treating your
visitors like that.

~~~
mappu
I couldn't find the close button either. The first thing i did? Right-click
the black background, choose Inspect element, untick "display:block".

If the web developer isn't going to respect the user, then i refuse to respect
their layout (and event handlers, and plugins, and...)

~~~
zizee
I didn't have this problem in Chrome, perhaps s/he has fixed it?

~~~
markokocic
Yes, he moved it to the right side of the blog. Glad that the complaint worked
;)

------
px1999
Google has a well documented history of it being next to impossible to talk to
a real person unless you have an $X0,000 hardware search appliance or similar.
As an individual, if you're not internet famous, you're taking your business
into your hands using Google for critical services like these.

That said, not everyone looks at the level of support provided before they
choose a service. So, if you get blindsided with this type of thing, you do a
quick search to see if it'll get resolved quickly, realise it won't, change
your MX records and learn your lesson / plan for a migration out of the
service if/when you get your account back. Just like with any problem of this
nature.

Usually it's just worth paying for something more expensive (and often less
featured) for which you know that there's someone you can call when the
service goes down in the middle of the night or you get locked out for some
random reason.

edit: I'm really only being overly harsh because the guy who wrote the article
is the CEO of a _cyber risk management_ and security company, and should know
better.

------
mevodig
Something similar happened at a company I work for; the account of the CEO
suddenly becomes "temporarily unavailable".

I take their word for it and 12 hours pass and the account is still
inaccessible. We're a paying customer, so I open up a ticket and it is
eventually passed on to the "technical team".

I give them a call but they cannot offer any additional details nor any
timeframe whatsoever when this will be resolved. Another 20 hours pass and my
requests for any update go unanswered. Finally, after a total of 48 hours the
account is available again. Still no update from Google.

Having been completely cut off for two working days, the CEO is now
(correctly) questioning their use of Google Apps so I update my original
ticket requesting any additional information they have that could assist me in
justifying staying with the service. They tell me this will be forthcoming.

A week passes, no update. I reiterate my request and 10 hours later I'm told
that the cause was a "server issue"...

~~~
kami8845
Yup. It seems Google really cannot be arsed to do Customer Service. For them
this was a "bug" and it is now marked as "fixed". Case closed.

------
ftwinnovations
TL;DR A bug in Google's internal systems took his Apps account offline for a
week. Slow customer service upset him greatly.

To be fair to Google, this was hardly an assassination...

~~~
kordless
> What could I do to restore email? Migrate the MX entries to another
> provider?

Yes.

~~~
spindritf
It gets better, even if you change the MX entries, there is no guarantee that
GMail will deliver mail to those new servers because the domain may still
exist in their system. So you may remian cut off from a significant number of
people.

~~~
stickfigure
Is that speculation or do you know it will happen?

I feel reasonably confident that ignoring MX records would violate some part
of the smtp process, and I would be be very surprised if Gmail did this.

~~~
spindritf
> Is that speculation or do you know it will happen?

I had troubles a few years back with e-mail routing from GMail to mail servers
for a domain that had previously Google Apps set up for it but was
subsequently moved to another provider. They disappeared after deleting Google
Apps.

Maybe it's not an issue any more, maybe it was just some weird caching
problem, maybe disabled domains are handled differently but it wouldn't also
surprise me if they had some system for handling what is essentially mail
internal to GMail.

------
aneth4
I've been looking for a functionally equivalent alternative to gmail largely
because I fear ending up in a similar situation. I'd like to host my email
with a company that has remotely competent and caring support.

What are the best services?

And no thanks to cloud haters - I will not be setting up my own imap servers.

~~~
tijs
I'm quite happy with Fastmail.fm, there are not that many options out there
i'm afraid.

~~~
aneth4
I used to use fastmail but switched to google as their interface fell behind
the times and I wanted features they didn't have. Perhaps I should look again.

------
mattwdelong
Just for my own personal curiosity, has any company made a reasonable attempt
at competing with Google apps core services? I know Microsoft offers similar
services, but it's not that well "put together" the last time I looked.

You're probably not going to be able to compete with Google Drive integration,
but if someone put together a well designed integration of inbox, calendar,
and contacts with an API, they would do well.

~~~
meaty
Yeah buy a file server and office. Seriously.

~~~
derekp7
Does anyone have a current best-of-breed list of web enabled applications
(preferably open source) that you could run on your own server? Ideally, I'd
like to put together a kit that is easy to install, either on your own server
or on any hosted provider. Bonus points if the kit includes replication tools
built in so that you can use multiple web hosts, and have your own redundant
cloud.

~~~
meaty
Rdp into your windows file server and work there (I work over rdp for about 7
hours a day).

The web is shitty for serious productivity apps from experience.

Regarding open source, use openoffice instead.

Alternatively use Windows' offline files feature when you are away from the
fileserver.

------
ditojim
im not suggesting every google apps customer should do this, but if you had
purchased google apps through a reseller, said reseller could have helped
resolve this issue within a couple of hours, most likely. without a PIN. also,
why wait so long to use another domain's PIN to contact support? why not do
that on day 1? lastly, if you change your MX records away from google apps,
your mail will route to wherever you point it. google looks up MX like
everyone else, for each domain. every time you send an email to or from a
google apps account, it goes outside of google and then if it is an apps/gmail
account and mx is pointed to google's mx, it goes back into apps. only then.
could support be better on google's end for normal paying customers? yes. so
could every company's IT support. just go through a premier reseller next
time. most charge the same price ($50/yser) and include support, especially
for issues like this.

------
gingerlime
I wasn't aware of this PIN either. Even if you do change your MX record, and
move your email elsewhere, you still get locked out of access to your old
email, which can be important.

I have a (free) google apps account for my domain, but I configured it so it
actually still goes through my own MX server. It took a bit of trickery,
because google wants you to use theirs, but I managed to get around it. It
works well for a couple of years now. My email always goes first through my
own MX, and then to google.

I would still be pretty upset if they blocked me though. But I guess I can't
expect much from a free service.

------
luser001
I am currently unable to login to one of my domains on Google Apps (i.e.,
www.google.com/a/example.com returns an error). When I try to register it, it
says the domain is already registered. Fun.

It isn't my primary domain, so I haven't bothered to pester one of my Googler
friends to see if they can do something about it.

~~~
pdwetz
Similar thing happened to me; it turns out someone else had somehow setup an
account a couple years ago with my domain (I've owned it since 2000 or so). I
was able to get control of the google apps settings for it (they give a couple
options, such as adding a DNS entry). However, it was useless because the
previous person had set it up in Russian and I couldn't find a way to change
it.

~~~
mikebracco
I had the same issue and contacted Google Apps support using PIN of another
account I already had. I told them the issue and they basically cleared out
all the internal Apps data of the previous domain owner. After 48 hours I was
able to set it up as new with Apps.

------
devost
Appreciate all the interest in this story.

As I mentioned in the follow-on comments, no service is perfect but I think
Google needs to try harder for instances like this. The account was disabled
and was also the Google Apps admin account. In that instance, I think Google
is at least obligated to kick-off an email to the Google Apps back-up email
address and alert to the issue with a time-expiring form that triggers a
support request with a slightly higher priority. This could all be automated I
would think.

I realize Google is huge and they provide critical services to many of us (I
continue to use Google Apps for several domains) but the "can't be bothered"
customer service attitude can have a significant impact on operations and I
wanted to generate some awareness about that. Looks like we succeeded on the
awareness front.

------
edgls
Yet another great example proving google's bad customer support. What is
google trying to prove by such pathetic customer service? I have heard
complain about almost every google service: admob, adsense, google app ....

~~~
jff
I haven't had any experience with Google customer support, but here's an
experiment. Go on Yelp, look up some highly-rated restaurants and shops in
your area. You'll still find that many of them have one or two reviews about
how terrible the service was, and how the product was awful, and the owner was
a dick about providing a refund. Sometimes things go really wrong, and even
the best will in the world can't fix it quickly. Only those stories make it to
HN, because we all love a dogpile, and because people seem to have realized
that bitching publicly is one of the best ways to get their problem solved
quickest... although sometimes it's like faking a heart attack so you can see
a doctor immediately, then asking him to take a look at a rash.

~~~
Dylan16807
There's a big difference between a company providing bad service on one
occasion, and a company actively blocking someone for days or weeks.

Don't imagine a restaurant providing the wrong food, burnt, and still charging
full price. Imagine a restaurant taking the customer's iphone and keeping it
in a safe for a month, employees ignoring them and telling them to leave. Are
you really going to see many stories like that on Yelp?

~~~
devost
Agreed. My biggest issue wasn't that an error occurred, but the fact that
there was no recourse available in a timely manner. I tried to be patient and
follow recommended support procedures (post to the forums, etc) but losing
email for over a week because Google has an internal issue and couldn't be
bothered to resolve it didn't sit well with me (thus the post to CTOVision).

------
ommunist
Not being a part of the Google ecosystem is always a good thing for business.
I grief with the author for the loss, and really encourage to use Rackspace,
which has more than enough human support options.

------
ams6110
Filed under _Don't put all your eggs in one basket._

~~~
Dylan16807
The main issue here is email. That is one egg. You're misfiling.

~~~
mylittlepony
What are you talking about?

 _"No App services – Calendar, Docs, Drive, and all Google App services are
completely blocked. These are not critical for this domain in particular, but
would be high impact if denied for some of my other domains."_

Email is the only thing that got broken AND mattered. And having all your eggs
in the same basket has the implication that the owner of the basket can take
your eggs as hostages, so you better be careful when you talk to them. Also,
remember to carefully read the ToS every day because they can change without
notice and f*ck you up.

~~~
Dylan16807
Right, only one egg mattered. It doesn't matter what happened to the rest of
the eggs. If Google only had the email egg, it would have been just as bad. It
was not a problem of centralizing the eggs. Google had to take just one
hostage to cause major problems.

Unless you have a suggestion of how to split the email egg into multiple
baskets?

------
donniezazen
Google has similar attitude on providing customer support for all its
services. A quick example, I have integrated my Sprint number to Google Voice,
all my messages and phone calls go through Google Voice, for past few months I
have stopped receiving text messages on my phone. I only get them through
Google Voice. There is no support, Sprint has no idea, and Google is not
reachable.

------
bobgourley
This whole discussion (post was written by my friend) and my personal
experience says to be very cautious about dependencies on any service
provider. A simple smart hack would be to auto forward 100% of emails to a
different account just for backup, then have the domain registered at a non-
google DNS so a fast change to an MX record could be made.

------
jiggy2011
I don't understand why people use google apps for something as important as
email in the first place.

Is making a marginal cost saving really that much more important than having
actual support?

~~~
maratd
The reason is fairly simple. As a small business, I need my email to work.
When I do it myself or outsource it to a small company, it works most of the
time ... but not always. I can't afford that.

~~~
Mahn
Also it simply makes no sense spending time setting up, configuring and
maintaining email servers as a business owner when there are more important
things to put your time in.

~~~
gruuby
Installing dovecot is not much harder then setting up a domain in Google Apps.
It works great for small set-ups and you own the data.

~~~
maratd
This is unbelievably not true. Installing dovecot or similar is super easy.
There are many options.

Unfortunately, setting things up to the point where your email isn't instantly
tagged as SPAM by Google's, Yahoo's, and Microsoft's servers which handle the
bulk of the email you want delivered ... is a monumental feat of engineering.

Frequently, they just mark it as SPAM when it doesn't come from one of the big
boy's servers. What's the point of sending an email if the recipient will
never have a chance to read it?

~~~
gruuby
Are you sure? I've been sending email from my workstations for years and never
had an issue. Maybe if I'd let spammers relay through them for a day or two
I'd get blocked. Even that would just be temporary. Let's see some proof that
you need a big boy's server to successfully send email. The only real problem
I can think of is that a lot of ISPs block 25 out.

------
pjmlp
Don't bet your productivity tools into the "Cloud" propaganda.

------
capo
To recap: In order to receive support from a human being for Google Apps the
account holder needs her account associated PIN, but since this guy was locked
out of his account he didn't have it available to take advantage of the phone
support. So after a week and by using some trickery he got hold of someone on
the phone and the issue was resolved.

I’d argues that Google Apps has reasonable support (bigger accounts, I assume,
have direct support lines) and these occurrences are relatively rare
considering the user base, but they get amplified rather loudly, especially
when there are plenty of cloud haters, and Google haters (and competitors)
ready to pounce.

There might be some risks or trade-offs involved but they’re much less
pronounced than some of the comments on any “Google support sux” thread lead
you to believe. I haven't had an issue with my personal Apps account but my
case is statistically insignificant, same as the case discussed here.

~~~
the_bear
I don't agree that their experience is equally as statistically insignificant
as yours. To me, a company should be providing an acceptable level of service
to _all_ of their customers, and they must have systems in place to make sure
that huge blunders don't happen, even in rare cases.

Your experience doesn't prove that Google always offers good support. The
occasional stories we hear about Google are enough to prove that they
sometimes offer terrible support.

------
mylittlepony
Good, now you can join the club of people who don't depend on google and its
bullshit. You are gonna like it here.

