
Google Drive issue affecting a significant subset of users - pgrote
https://www.google.com/appsstatus#hl=en&v=issue&sid=4&iid=d9bab8e4a0fdfb97d4ac42a2c918af14
======
pgrote
At first, I thought my machine was hacked. I didn't know Google Drive was
changing their name and the error message I received looked fake:

[http://i.imgur.com/zAJVpAF.png](http://i.imgur.com/zAJVpAF.png)

Notice the grammar and punctuation issues not to mention the email address to
nowhere.

~~~
PuffinBlue
Hmm, that's interesting. Looks like an error message meant for internal Google
users.

uploader-eng@ would presumably be uploader-eng@google.com

The error message also looks like it is also geared towards users with
google.com accounts.

The whole thing smacks of someone pushing something internal from development
to production.

~~~
pgrote
Good point. I hope the details as to how this happened become public. I cannot
imagine a release to production can happen without multiple approvals.

~~~
PuffinBlue
It's definitely quite surprising, especially with Google's SRE strength and
multiple automated check systems that have been previously documented here.

They often do detailed postmortems so hopefully this one might get that
treatment - though given the accidental exposure of the 'not for public
release' UI and client they may well not go into much depth with this one.

------
johansch
From experience, I've learned to parse "some users" as "half a billion users"
when it comes to Google issues like these.

I really think they have some policy to say "some" unless it's literally
100.00000% of the user base. Cause PR..

I wish they would improve on their communication accuracy in this aspect. I
really do think it would make people trust them more in the long run. (Just
tell us a percentage estimate! Is it 1% or 70%? Don't have an exact count?
Just give us your best guess! Anything else than this really an insult. We're
all in this together.)

~~~
pgrote
The status board stil says, "Some users of the Google Drive Sync client for
Windows will receive the error message "Sorry, Backup and Sync needs to quit"
and will be unable to use the client. Other Sync clients are not affected.
Affected users may continue using the Google Drive web interface and mobile
applications."

What is odd ... I cannot find anyone who was upgraded to the new client that
isn't getting the error message. I wonder how they define "some."

It would be interesting to see what criteria was used for upgrading. I've seen
reports of Windows 10 and Windows 7 users being upgraded, but I have a Windows
7 machine that is still running the old Google Drive and hasn't been upgraded.

~~~
M_Grey
>What is odd ... I cannot find anyone who was upgraded to the new client that
isn't getting the error message. I wonder how they define "some."What is odd
... I cannot find anyone who was upgraded to the new client that isn't getting
the error message. I wonder how they define "some."

Well there's this one guy in Pacoima, and he's having no problems at all.
"Some".

------
jjkmk
I fixed this issue by fully removing google drive and reinstalling it.

It seems like google released an early beta / alpha version accidentally.

~~~
RachelF
It is also affecting online users, not just the desktop client - "The affected
users are able to access Google Drive, but are seeing error messages and/or
other unexpected behavior."

------
19eightyfour
It's unclear to me if Google Drive really has changed its name to backup and
sync. However it occurs to me that Google Drive is a name that might elicit
some kind of regret since it was a product named and created before self-
driving cars became a big thing and Google got into it with their in my
opinion kind of clumsily named waymo.

------
pgrote
Appears the issue is resolved by an updated client through a push or manual
download:

Google Drive service has already been restored for some users, and we expect a
resolution for all users within the next 12 hours. Please note this time frame
is an estimate and may change. We are deploying an updated client now. The
Windows client will automatically update with this fixed version within the
next 12 hours. Alternatively, affected users can download the updated version
of the client from
[https://www.google.com/drive/download/](https://www.google.com/drive/download/).
Affected Windows 10 users may be required to sign in to their Google account
within the client.

~~~
RachelF
Indeed. There is more at the Google Drive forum. Seems to be limited to the
Windows client from the reports. It could be serious as there are reports of
folders disappearing.

[https://productforums.google.com/forum/#!forum/drive](https://productforums.google.com/forum/#!forum/drive)

------
jhancock
Apologies for the long post...this is a bit of a cry for help...is there a
support group for programmers/admins that have been bitten by the Drive cool
aide?

My organisation did a migration to Google Drive (from internal Windows server)
in the last few months. I've had quite a few issues (I'm the administrator).

I'm posting this (abridged) laundry list of problems here...would love to have
some feedback or pointers to what's wrong or what I'm doing wrong. When I've
managed to get through to Google Drive tech support I don't get much in the
way of answers.

For context, I'll confine this to just two central share accounts:
fileshare@mydomain and records@mydomain. Each account has around 1.5 terabytes
of files. Our org has 85 G-suite accounts (users). This is a small to mid
sized business.

* I'll mostly skip the part of how difficult, slow and expensive a process it was to get around 3 terabytes into Google Drive. rclone was a good friend to us. It took months as we had to constantly re-sync as we migrate....we are running a business while we migrate. I'll also mostly skip the problems we had with the Drive sync client...its default behavior is to sync everything. I had to globally disable usage of this tool. Guess what happens to your company network when 50 users install the sync tool and click "add to drive" for the company fileshare. Good luck educating users with a wide range of PC skills on proper sync usage.

* Drive's file sharing operations are opaque and take an indeterminate amount of time to complete a share (could take days). If I need to restrict access to a large subset of files, I have no idea when the share is complete. I have found that sometimes it does not complete for all users and I'm left with some files not shared. I can't find a pattern. I have no tools to help me find what (under a large set of folders) didn't get shared. If a user or me as admin needs to unshare, this can take days. So there is no practical way to protect sensitive access files except put them in different root folders or different share account.

* No practical way to get a backup. I'm trying spanning backup at the moment. Its been running against my fileshare account all week. The spanning backup console tells me I have about 100 days to wait to complete the initial backup. Yeah, 100 days to backup 1.5 terabytes. What's the use of unlimited storage if I can't protect it? In this case, I'm suspecting Google Drive, not Spanning as I've become familiar with how performance variant the Google Drive API is.

* Google Drive allows an end user to easily do "destructive" things to the central file share. Last Friday I received a call from a staff. She knew something bad just happened and quickly reached out for support. She had accidentally used the move or delete operation in the Drive UI to cut out a huge subset of our fileshare. I say "move or delete" because she didn't know and as an admin I couldn't figure out what happened. I dug around both the fileshare account and her account and couldn't find the files. The Trash folder on both accounts was empty. This didn't make sense as the activity panel clearly showed her and other users deleting things. At first the Trash view returned a server error. I refreshed my browser and got it to show. Empty. I tracked through the activity list and found the operation of concern....it showed the folder had been removed. Fortunately, I was able to click on the folder in the activity panel and it took me to it. From what I could tell, the files were in the system, but not attached to anything...no parent folder, not marked as deleted or moved. I was able to move this unattached folder under a new temp folder at the root of the fileshare account. Then go to that folder and move it again back to its original place. Then we wait...for the removed share permissions to fall back into place. Good thing this happened late on Friday as we had the weekend to wait for the shares to come back.

* The Google Drive API is slow and unstable. I'm a programmer. We spent quite a bit of time writing a system to push files from our records management system to our Google Drive records account. Currently, we are adding less than 1000 files a day; mostly mid sized jpg's (1 to 3mb). So well below our API limit of 1 billion requests a day. Our server using the drive API tracks errors. Currently, around 1/3 of the uploads fail and need to be retried up to 10 times. Some never make it and we have to handle it manually. Good thing we wrote this in Elixir/Erlang as fault tolerance on the API client is our friend.

* The Google Drive API is "global". When you create a service account (Google's API method for server to server communication), you give access for the scope of the API but not the account. Our system needs read/write access to only the records account. No can do. You have to give your API keys full read/write access to every account. So yeah, your dev and test systems have full access to all production data. Better be a very careful programmer. If you mess something up in dev, you could mess up your production drive accounts.

* Combine these API problems with the inability to do a backup (and restore as well as restore to a test account) in any reasonable amount of time...and you can throw away your discipline of separating dev from production and being able to run repeatable tests before deployment.

I'm going to spend the rest of my day reviewing code with a programmer. The
code is to refactor our folder structure in our records account. It shouldn't
delete anything...just move lots of folders into a new structure. But if we
have just one bug in our migration scripts, well who knows. So I'll be
reviewing code today and on pins and needles Saturday while our migration
script runs on our production drive.

~~~
CPAhem
I feel your pain. We went through similar problems. Many others do to, if you
look at the online support groups. Some things that might help you:

* Set user permissions really tight. I've had so many issues with users dragging folders around resulting in everyone re-syncing.

* Use a better desktop client. The Google one is really basic. We use SyncDocs which is more robust and has the features you need.

* Your API issues with jpg files are probably due to you tripping Google's anti-piracy system.

* Set up a separate Google Domain for testing stuff. You'll have to pay extra for it, but don't test on production data.

~~~
jhancock
Thanks much. Good to know I'm not alone or imagining this stuff ;)

> * Your API issues with jpg files are probably due to you tripping Google's
> anti-piracy system.

Holy smokes! Didn't know this was baked in. Our school teaches art and
creativity. The jpg's being uploaded are pictures of student's original work:
painting, metal works, jewelry, glass blowing, hand written diaries and
sketches. Our images are pretty darn unique (and some amazing art as well).
Would love to know if Google's detectors are tripping up.

> * Set up a separate Google Domain for testing stuff. You'll have to pay
> extra for it, but don't test on production data.

Good idea. That solves part of the problem. But it doesn't enable me to have a
set of test data based on near-current production data. And god forbid I
someday make a mistake on production...no way to rollback.

~~~
CPAhem
>Good to know I'm not alone or imagining this stuff ;)

No, you just don't see it mentioned much, perhaps as it is just seen as "un-
cool" in some circles to paint anything Google does as being less than
perfect.

------
mustafabisic1
But I love the way they're dealing with it. :D it simply feels human from such
a big company. Kudos Google

