

Google wreaks havoc on our company’s calendars: lost data and security breaches - pascal07
http://www.elezea.com/2012/12/google-calendar-lost-data-security-breach/

======
NameNickHN
This sounds like a severe technical problem that has to be analyzed and fixed
by real experts. I doubt that these kind of people do night and weekend
shifts. And unless the problem is widespread, I don't think they're calling
those experts in from their weekends for a single incident.

~~~
Osmium
If it can happen once, it can happen again. Calling in those experts on the
weekend is _exactly_ the thing you should do if you're Google. Pay them double
if you need to. But you need to catch this sort of thing before it escalates,
not least for the PR damage it can cause.

~~~
raverbashing
I'm sure Google employs several people on call

Hopefully they can help solve this problem or at least raise the (SNAFU) alarm

------
alexkus
Sounds like a database migration [by Google that is] or dump/reimport that
hasn't preserved the original primary/foreign keys and so items are now owned
by other people. Oh the joys of referential integrity.

The ownership of things seems to have been kept within your company but not so
for individual items in the calendar which could mean that random other people
are seeing your calendar entries appearing in their calendars.

"Oops" is the operative word from Google.

------
cpbotha
Are all of your employees using two-factor authentication? Are you 100% that
this could not have happened via a normal security breach via a number of your
employees' accounts?

GMail accounts get hacked regularly, often due to users using the same
password for different services, or just getting keylogged somewhere.

I'm very curious too see how this pans out. Good luck with the recovery!

~~~
driverdan
It's not possible to move calendars around in this way. Plus it doesn't make
sense to do something like this as a hack.

------
danielsamuels
We're having a different issue on our companies' shared calendar -- all of our
events are duplicated. Apparently Google know about it but there doesn't seem
to be any fixes happening.

------
imroot
I've used Google's services from both an end-user at a company who uses Google
Apps, and from a IT/Infrastructure Manager who moved his company to Google
Apps.

For 95% of businesses who don't do a lot of intensive activities (be it
scheduling, emailing, or anything else), google apps works fine.

If you're in that 5% who push the limits of your email/calendaring setup, then
Google Apps (and their lackluster support) aren't the choice to make.

We had a support issue where the CEO couldn't send email to the company-wide
mailing alias -- every time he did so, it was marked as 'bulk' email, and
rejected. Google's support was non-existent (read this support page -- did
that help? No? That's all I can do). Fast forward to a few weeks later, when
users would randomly get non-delivery notifications when sending email
internally. Google mentioned that they couldn't see the notifications come
from their servers.

Another option was when a trial ran out, they'll lock EVERYONE out of email,
calendars, and everything else until the trial is cancelled, or, until they
pay. If an admin unknowingly signs up for something that will cost him a few
extra bucks/user/month, and then forgets to update a credit card, every user
in the organization will be locked out, and they can't call support (because
you can't get a phone pin when you are locked out of a domain like that).

If you want minimal hassles, then Google Apps works great for small places (up
to 20 users). Then, it's probably wise, from both a business risk and a
infrastructure standpoint to just start running your own.

------
tallanvor
Obviously having something like this happen sucks. If nothing else, you should
be able to have a specified timetable for updates (every 4 hours, for
example), even if it's just "we're still analyzing this issue" or "we are
evaluating potential solutions".

That said, when you're dealing with a cloud-based application, fixing a
problem like this is not going to be easy. First they need a clear enough
understanding of how this happened to prevent it from happening again. Next
you have to start coming up with scripts to correct the issue. After that you
have to test the scripts to make sure you don't make things worse. Finally you
can update production.

Most likely the support team has engaged an operations team and probably some
developers as well. But even if they've been working through the weekend,
there's no guarantee that they'll have a solution yet.

Or maybe Google doesn't have the necessary people on call over the weekend so
nothing has actually been done. I can't say.

------
willyt
I sync everything with the cloud and have a local backup which allows me to
rollback any change to any file. So in the event that the cloud goes rogue I
can just disconnect and restore from backup. But also, in the event that my
office gets burnt down, I can still access all my data from the cloud. I
personally would never have a system where the only point of access to mission
critical data is through 'the cloud'.

------
leephillips
You failed to perform due diligence before deciding to rely on this service. I
wrote about this kind of thing over a year ago: <http://lee-
phillips.org/gcaldisaster/> If you just Google (hah!) "calendar privacy" today
my article, and a few other relevant warnings, are on the first page.

------
n0cturne
Remember, according to Derek Sivers, everything is your fault.

------
polskibus
Maybe they tested Google Spanner on Elezea's bit of data?

------
capo
So it's being dealt with then. Not sure when wording an attention seeking post
and submitting it to HN became standard operating procedure in these
situations. Patience remains a virtue when this stuff happens.

~~~
WayneDB
That's a really easy decision for _you_ to make when it's not your business
that's being held hostage.

Personally, If I can't run something on my LAN, I don't want it. So, I love
hearing about these issues; I'd like to someday see developers moving away
from SAAS for every little function.

~~~
JPKab
"That's a really easy decision for you to make when it's not your business
that's being held hostage. Personally, If I can't run something on my LAN, I
don't want it. So, I love hearing about these issues; I'd like to someday see
developers moving away from SAAS for every little function."

My former employer at a small business used to say the same thing, and
gleefully pointed out when gmail went down for 3 hours 2 years ago, screaming
about how dangerous it is to rely on an external entity for these services.
The outcome: we had our own servers for our email, and experienced DAYS of
unplanned outages a year, not counting the planned outages due to migrations.
On top of that, he had to pay the salaries of sys admins who could have been
devoting their skills to stuff that our business actually got paid for.

The justifications for insourcing IT vs. using a SaaS pretty much align with
the justifications for keeping cash in a safe in your house vs. using a bank.
Sure, they are there, but they all are much, much less likely than the
scenarios which will make you wish you weren't handling it yourself (local
server failures, local internet failure due to ice storm, hurricane, software
update, idiot sys admin) or in the case of money in a safe(house fire,
robbery, flood). My company had very competent sys admins, but I'll never
forget losing email for 3 business days due to an ice storm 3 states away
where the other office was hosting the servers. Then we decided to purchase
servers in our other office to prevent this, and then we hired more people to
run them. It was fucking stupid, and all due to a stupid mental paradigm of
"If i'm not in direct control then it's not being done right." We had less
uptime, less security, and a much higher operating cost. That's a high price
to pay for illogical, emotionally based paranoia.

~~~
WayneDB
I'm not saying that running your own servers is _always_ easier, nor is my
stance one of emotional based paranoia.

I'm saying that "private cloud" products that you host yourself _could_ be
just as easy to setup and use as cloud-based products if developers wanted to
build those products. Instead they want to nickel-and-dime everyone for every
business function while retaining absolute control over access to the product.
SAAS can absolutely be more expensive in the long run.

(EDIT: Coincidentally, gmail is down again today.)

------
pjmlp
Don't thrust the cloud...

