
Google Intrusion Detection Problems - K0nserv
http://www.fredtrotter.com/2016/08/22/google-intrusion-detection-problems/
======
colinbartlett
If Google is ever going to fix their cloud market share problem[1], they need
to reverse the deeply engrained perception that they don't support or maintain
their services for the long term.

The majority of Hacker News pundits will all have the same gut reaction to
these kinds of anecdotes: complete and total lack of surprise. And although
not all of us are in a position to control the vast budgets of the enterprises
that drive much of this marketshare, the opinions of the rank and file do have
a vast impact and I believe Google's unstable support and product commitment
is one of the biggest things holding them back.

1\. [http://www.nytimes.com/2016/07/25/technology/google-races-
to...](http://www.nytimes.com/2016/07/25/technology/google-races-to-catch-up-
in-cloud-computing.html)

~~~
foxylad
I just want to say that Google's supposed lack of support _is_ just a
perception, despite all the usual comments below.

Real example: we use Appengine, and the other day I spotted a bug in a request
handler. I filed an issue on the Appengine issue tracker, and have been having
a back and forth conversation with a a very real (and very skilled) engineer
about how to resolve it. This is outside our support contract, so anyone with
an software issue has exactly the same access.

This doesn't cover situations such as the OP experienced, of course - but if
they had a support contract, I think they'd have found the issue was resolved
quickly and efficiently. We pay $150 a month for a silver support contract,
and when we've needed help, we've got personal, prompt and effective support.

At the end of the day, you get what you pay for. If your organisation depends
on any service, you need effective support, and you need to pay for it. That
goes for Microsoft, Amazon, Salesforce or anyone else - good support costs a
lot, and they all have to cover costs. Each company uses a different model,
and some include more support costs in their base charges than others.

I happen to like Google's model where dipping your toe in the water costs very
little, and when your service proves itself you can ramp up support as you
need it. But it leaves Google open to criticism like this the OPs, who let
their small project grow big without contingency planning and then complains
when something goes wrong.

Finally, the only reason I'm commenting here is that I was so impressed with
the support from the engineer mentioned above that I said I would mention him
next time Google's perceived lack of support came up on HN. Promise kept!

------
abstractbeliefs
I don't understand why people are ever surprised about Google offering
substandard support for really anything.

Time and time again Google products prove to make the happy path happier, but
as soon as anything should go wrong, you're on your own with no explanation,
and no support.

Additionally, and as an aside to the core issue here, I've found the recent
Google Cloud UI updates to be a real pain. When I last used them, I did
acknowledge that it was beta, but I seemed to lose an awful lot of oversight
of an awful lot of things. It's put me seriously off it vs. AWS.

~~~
0xmohit
> It's put me seriously off it vs. AWS.

Can't really compare, but AWS isn't great either.

I've experienced instances when an instance (1) shut down, (2) rebooted on
it's own.

AWS support would point you to the SLA if you were to report the incident.

~~~
sargun
EC2 instances are meant to be ephemeral.

~~~
bpicolo
That doesn't sound factual at all. Got docs which support that? They send us
emails every time they plan on doing intentional maintenance on our instances.

~~~
0xmohit
I wish it wasn't true. On all the occasions, I was told that there was "an
issue with the underlying hardware".

~~~
CPLX
It is worth noting that "the cloud" is synonymous with "someone else's
computer"

~~~
dragonwriter
> It is worth noting that "the cloud" is synonymous with "someone else's
> computer"

Well, no, its not, otherwise "private cloud" would be incoherent.

------
tscanausa
My name is Terrance and I work on Google's Cloud Support Team. Our team
mission is to Reduce Customer Anxiety. We take that very seriously, and Fred's
experience obviously shows that we fell short this time. The process should
have been an easy flow to follow, and it was not. We are reviewing this
incident in detail to ensure that we make the process less error prone and
quicker in the future.

Best, Terrance

~~~
eadz
How does one reach someone at google in an emergency when you have no access*
to any google services?

* locked out of your account

~~~
nikanj
Write a good blog post and get it to the front page of a news aggregator

~~~
tscanausa
We became aware of Fred's problem before we knew it was here.

-Terrance

~~~
CountHackulus
That doesn't really answer the issue, or the question. How are we supposed to
get in touch when things go wrong? The blog post clearly showed that support
was an afterthought and not properly QA'd.

------
20years
This is a perfect example of why I refuse to use Google Cloud. Google in the
past has put me through similar automated loops with Adwords and Adsense. Even
disabling an Adwords account for 2 weeks for reasons I still do not completely
understand. After weeks of trying to get a hold of the right person and being
pushed through a ton of different channels, it was finally re-instated with a
generic apology and still no concrete reason as to why.

I have learned that it is way too risky relying on Google services for crucial
business operations.

------
nokya
The Nth story about that guy who runs a business depending on Google services
and suddenly starts telling everyone else they shouldn't run their business on
Google services because Google services flagged them and they realized it was
a bad idea to run a business based on Google services...

~~~
ocdtrekkie
Yup. This is note even a remotely surprising story. If you're building your
business on Google, you're building your business to be arbitrarily shut down.
You should not operate any business critical service or store any business
critical data on any Google product or service.

If anything, this is one of the more generous stories, because Google gave
them a day or two of notice first. Usually you just find your Google account
shut down one day.

~~~
Kadin
Well, I wouldn't go that far. If you run your business on Google Services, you
definitely need to have a contingency plan for what you're going to do the day
that Google decides to arbitrarily cut you off, though.

But in some instances it might be financially worthwhile to take advantage of
their services (particularly their free or very cheap ones) as long as you
have a plan for standing up something else when that happens.

For instance, I'm pretty comfortable using Gmail, even though I know they
might someday pull the plug on my account for no particular reason. I'm
comfortable doing that because all my mail is backed-up elsewhere, and in a
pinch I could restore everything or stand up a new mailserver in a day. It
would be obnoxious, sure, but it's not exactly breaking new technical ground
or anything. The years of free service I've gotten out of them make that risk
manageable.

I could easily see a company taking that bargain for various other services.
The problem becomes when you simply rely on a 3rd-party's infrastructure as if
it was your own.

------
redstripe
This is the google M.O. - optimize all processes through cheap machine
learning and heuristics and accept that some people will get screwed along the
way.

The number of people that can be served at very little cost without any human
involvement is apparently very lucrative. Google just considers customer
service as an archaic relic that predates the invention of behavioral
algorithms.

~~~
tlogan
And it works quite well from them. Yes, people on HN and on internet will
complain...

Also it could be that customer support is just overrated.... The problem is
that 95% of support calls / issues are anyway just people complaining about
things which are not at all related to the actual Google service.

But, on the other hand, hackers which try Google Cloud and are bitten by lack
of Google support do have influence on decision making process in big
corporations.

~~~
Jordrok
Yeah, how dare all those little people complain when they're inadvertently
crushed by the glorious Google behemoth. Don't they realize how insignificant
they are? Their entire lives are simply rounding errors at Internet Scale.

Sometimes the end result is less important than how you get there, and how you
treat the ones who don't fit into your plans. It happens all the time with big
companies, but Google even more than most. They KNOW that the potential for
problems exists, but so long as it only affects 5%, who cares? Fuck you, got
mine.

I think that the attitude conveyed by Google's lack of support is what
inspires a lot of people to post rants online, rather than the frequency of
incidents.

------
paradox95
Once had Google kill a Spark job after it was running for about 12 hours
because they thought we were doing something nefarious. It cost us thousands
of dollars in wasted time and Google spend because their automated system made
a mistake. They never attempted to fix it, refund us or even seem concerned.

~~~
bigiain
And you compare that to the many stories you hear about Amazon waiving
invoices when people inadvertently ran up multi thousand dollar AWS bills by
accident or when credentials were exposed. (Happened to an ex-colleague of
mine years back - AWS creds committed to a public git repo - minutes later
there's like 40 10xlarge or g2.8xlarge or whatever instances mining bitcoin.
First thing he knew about it was Amazon ringing him up saying "this $10,000
spike in your typical use, that's not really you, right?" and shutting it all
down for him and reversing the charge...)

Then you consider this when you decide whether to base the next big business
decision on an AWS or Google Cloud platform…

------
Animats
Intrusion detection false alarms can be a problem if you run anything like a
web crawler. I have one running at "sitetruth.com", which is a site rating
system hosted on a leased server. (Not a cloud service, a leased rackmount
server). One of the things it does is to find the home page of a site by
trying "example.com" and "www.example.com", with and without HTTPS. Some sites
will block access for 30-45 seconds if those requests are made too fast.

About once a year, there's a serious intrusion complaint, as the crawler,
which obeys robots.txt, examines about 20 pages on a site in a few seconds. No
more than three connections at once, but some sites are touchy. The server
leasing company sends me a warning letter, I reply and call tech support, and
there's no big problem.

I can recommend leasing servers from Codero as an alternative to dealing with
the Borg of Mountain View. I've been a customer for five years, and nothing
bad has happened. They now have "cloud services" too, but I haven't used them.

~~~
noja
> the crawler, which obeys robots.txt, examines about 20 pages on a site in a
> few seconds.

That seems like something that could be slowed down slightly with little
negative impact, and a big positive impact (no annoyed people, no interaction
with support).

~~~
Animats
Our user is waiting. We rate sites on demand, and the user is looking at a
rotating "busy" icon in their browser search results during the rating
process.

------
fhoffa
Posting this 4 days later, what happened next:

[http://www.businessinsider.com/google-cloud-won-skeptic-
afte...](http://www.businessinsider.com/google-cloud-won-skeptic-after-
shutting-site-down-2016-8)

 _Google then shocked him again, in a good way. Within four hours of tweeting,
someone from Google had contacted him and had restored access to his project._

 _Trotter says that, it turns out, he and his team bear some responsibility.
They had inadvertently set up a server wrong, exposing a hole, and a hacker
was using his company 's to conduct a "denial of service attack," which is
when hackers overload another website or online service with so much traffic,
it shuts down._

------
CaptSpify
Lol

Sorry to be so harsh, but Google has always been this way.

Why do people keep putting essential stuff in someone else's sandbox? Your
effectivly adding a SPoF, that you have no control over

~~~
numbsafari
Is Amazon or Azure or Heroku really any different?

I've had similar problems in the past with Amazon shutting down services and
providing little to no support to get it remediated. Even after paying for
technical support, it was impossible to recover (in our case, it was a billing
issue that was entirely Amazon's fault).

My general recommendations are:

\- be sure you have fully scripted your deployment process, \- be sure you are
making remote backups of critical data (often times this is as easy as setting
up simple replication of S3 to GCS or Azure's Cloud Storage) \- rely as little
as possible on their "value added" features \- if you do rely on their value
added features, be sure you still have someone on staff who can quickly
replace it with something minimal (e.g. know how to install your own mysql)

At the very least, this will put you in a position where you aren't 100%
locked out of your data, or have to completely rewrite your application in
order to move to another provider.

You don't have to go crazy with some kind of hot failover, or active-active
deployment. Just have daily snapshots of your data and mitigate the risk of
your PaaS provider.

No matter who your hosting provider is, you should probably have this sketched
out as part of a "disaster recovery" plan.

~~~
CaptSpify
> Is Amazon or Azure or Heroku really any different?

Nope, and thats why I don't use them. There are situations where they make
sense, but people should be much more skeptical about them. I don't have
stats, but IME I've yet to see a company that actually saves money by using
them. I've seen one company that did save money the first year, but got too
addicted to "just spinning another instance up" and stopped optimizing their
code. The next year they were bleeding cash.

So, speaking from my experience: It's more expensive, less configurable, and
unreliable. Why do people keep buying the cloud lie?

~~~
markstos
I worked on a project that costs about $5k/month to self-host. After moving it
to AWS, it was about $2k/month. So $36k/year savings in hosting costs. In our
case outgoing bandwidth at AWS was about 10x cheaper than what our ISP
offered.

Having spent years help running a physical hosting company, working with
virtualized servers saves LOTS of time. I still find it amazing and much, much
cheaper in staff-hours to reboot a cluster and have it come back up on 2x
larger hardware when a project grows.

~~~
CaptSpify
Right, I'm not saying hosted solutions are never the answer, they are just
less often the answer than people seem to think. Too many people buy into the
cloud-hype, and never look at the numbers until it's too late.

And that doesn't even touch the trust you have to have in your provider.

And I love virtual more than physical. But I set up my own virtual server and
run them myself.

~~~
oneloop
> And I love virtual more than physical. But I set up my own virtual server
> and run them myself.

Where do you keep the machines? In your bedroom?

~~~
CaptSpify
Depends if you mean for work or home.

For work, we have a DC. For home, yes, I keep them in my office.

~~~
oneloop
You're paying for a super ninja ultra special internet connection as well?
Yikes. Interesting concept though.

So your strategy for scaling up is building a data center then?

~~~
CaptSpify
> You're paying for a super ninja ultra special internet connection as well?

Not really. Just a normal home connection. If you design your services sanely,
and don't have much traffic, it's not really an issue.

> So your strategy for scaling up is building a data center then?

Well, it depends. You really should do a cost-benefit analysis, because each
product is different. But if it makes sense, then yes.

~~~
oneloop
But your whole argument was that for some serious application you shouldn't
trust on cloud services, wasn't it? The kind of things I was doing half a year
ago you couldn't run on your home connection. So this whole "run important
stuff on your home connection" is just a signal that you're not doing anything
great.

~~~
CaptSpify
> But your whole argument was that for some serious application you shouldn't
> trust on cloud services, wasn't it?

Well, with everything, there are pros and cons. My argument is that I think
it's silly to trust someone else with your infrastructure. There can be
benefits to doing so, however, your giving up a lot as well. And in terms of
cost (which is usually the main argument that I hear), I've personally never
seen it work.

Your trusting someone with something pretty critical with your business, so
you should think long and hard about which direction you want to go. If your
cloud-service gets arbitrarily shut down or has an outage, what responses can
you take to fix it? Usually not many. Thats essentially adding a SPoF, which
is generally considered a big no-no, although it does happen.

> The kind of things I was doing half a year ago you couldn't run on your home
> connection.

I'd be curious as to what your doing then.

> just a signal that you're not doing anything great.

I guess that depends on what you mean as great. I do a lot of _useful_ stuff:
email, web-apps, file-store, home-automation, etc.

------
aluminussoma
Reaching a human being is a common problem with all Internet businesses. I've
had similar frustrations with Google competitors. As an industry, we have to
do much better with customer service. Our users are real human beings.

~~~
ocdtrekkie
This is why when it comes to web hosting, I only do business with companies
that have 24/7 phone support. It limits my options a lot. GoDaddy and
HostGator both provide this, most others I've seen don't.

But before I got my first HostGator account, I called their tech support line
at 2 AM, and got a guy named Chris with a strong Texan accent. And I've had
escalated issues at odd hours where I've actually spoken directly to sysadmins
there. Ticket support only hosting simply can't compete with that service, no
matter the cost.

------
pjjw
Dollars to donuts these guys got hacked. Still pretty shitty the appeal
process was busted.

------
pellej_s
So, everyone's basically saying:

> Google sucks

> Don't rely on Google if you want your business to succeed

> This is old news! Happens all the time

> Google = SPoF

...you're all looking like trolls to me. Please, do tell how Spotify manages
to service 60+ markets with some core pieces of infrastructure on Google's
Cloud offerings.

~~~
eadz
Spotify will have a phone number of someone they can call.

I've had this issue with google too. They'll shut down your account, your
whole account - google apps email and all - if an automated system detects
something dodgy. In my case an automated system at ebay accused my site of
phishing which it wasn't.

There is a paid support option, but when your account is shut down you are
unable to access the required code.

I don't think you can appreciate how inaccessible Google is until they decide
you're a bad actor.

------
ne01
I hate that Google don't give a damn about its users. To them interacting with
humans is a waste of time and money... They prefer to automate everything and
manage servers, robots and programs instead.

The same thing happened to us in 2013 luckily had backup servers at Linode..
took them 2 weeks to solve the problem it was a bug in their billing system.

Google gave us $8k credit and I gave them another try and convinced myself
that GCE was very young in 2013.

Now I'm just worried! It's 2016 and our entire business depends on their
platform and their support sucks! Even the $400/month version!!

Google, why can't you have support like DigitalOcean and Linode?

~~~
hga
_Google, why can 't you have support like DigitalOcean and Linode?_

Because it's totally not in their DNA?

Because GCE and company are relatively low priority offerings? Would you agree
these are higher:

Ad Words etc., which make the bulk of their money.

Search etc. which among other things has you seeing those ads.

Android and all their other efforts to keep themselves from losing their
access to the users of the above.

For a while, Google+ was an anomalous offering they were pushing very hard
(with this sort of brain damage with linked accounts that hit hard with their
real names policy), but they might try such a stunt again.

Compared to the two service providers your mentioned, which only do one thing.
Even AWS is I gather rather like that, there's it + all the Amazon selling
stuff which is significantly separate. Some of the DNA is shared, but that's
not all bad, such as the focus on customers.

------
zippy786
Google, said to have very high hiring bar, yet I've seen more than a few times
now that their product lines are sub-par. I'm sure the explanation they give
is that we are very big and we can ignore the 1% corner cases and have the
mentality that "we are so big we don't really care about smaller parts of the
internet".

Also, the hypocrisy in using very logical questions to hire people but not
sticking to similar logic in the product line is laughable.

~~~
blakeyrat
It feels like they have an entire company of Mr. Spocks without a single
Doctor McCoy to round things out.

~~~
PhantomGremlin
Mr. Spock would have approached the issue of support more logically and would
have thought things through more carefully.

It's more like they have an entire company of Sheldon Coopers, without a
single Penny or Leonard Hofstadter to round things out.

~~~
zippy786
Sheldon would have never agreed to release a system with flawed design.

------
danpalmer
Of course it's easy to pattern match this to previous stories of advertisers
and publishers being banned from Google's ad products, people losing their
email access on Gmail, etc, but this is an entirely different area of the
company, and one where account managers are very much a thing. It strikes me
as odd that one couldn't just email or phone their account manager to ask for
clarity on a situation like this?

~~~
hackcasual
Google tries to get away with as little human supervision as possible. I doubt
they have an actual account manager

~~~
DashRattlesnake
They have an account manager, it's just a robot named Samantha West,
programmed to insist she's a real person:

[http://newsfeed.time.com/2013/12/10/meet-the-robot-
telemarke...](http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarketer-
who-denies-shes-a-robot/)

------
damm
I admit to having a gmail.com account

\----

I really don't understand how anyone can use Google for production. If you
don't pay for support to have a way to actually get a hold of Google...

... Someday you may be in a world of hurt and have absolutely no way to get
ahold of anyone. You can post a story and hope and pray that Google might read
it and have someone reach out and remedy the situation.

------
martinald
We got the 100k credit which is nice. Noticed the actual technical side is
excellent - vastly better than azure and AWS (AWS especially - I find the
whole UI and way it works so clunky).

However, despite using our credit to buy gold support when it came to using it
I couldn't work their tickets UI at all. Turns out you need to change products
from your Google apps support to cloud platform by some tiny link which is not
at all obvious.

I had to phone up and it was obvious people hit it all the time as the
operator instantly knew what the issue was.

Also related: azure by default didnt renew your free trial despite an active
cc being on file. It just shuts everything off with no obvious warning. What
other service does that?

Honestly think these companies need to do some proper usability testing on
their flows because there is so much clunky weirdness in cloud providers right
now.

------
jeffmould
Not sure if he missed it on the support page, or simply is not willing to pay
the added cost, but Google does offer a paid support option with both phone
and email support options. Obviously the more you pay the quicker the response
and more access you have to those options.

~~~
ocdtrekkie
I suggest reading the article.

"Google offers support solutions where you can talk to a person if you have a
problem. We view it as problematic that interrupting an “allergic reaction” as
a “support issue”. However, we would be willing to purchase top-tier support
in order to get this resolved quickly. But there does not appear to be an
option to purchase access to a human to get this resolved. Apparently, we
should have thought about that before our project was suspended."

~~~
jeffmould
Hey thanks, but I did read the article, and saw that part. That is why I said
that he may have missed that part of the page. And just because he says he
would be "willing to purchase top-tier support" doesn't mean that he doesn't
have a limit on how much he would pay for said support, thus my saying that
maybe it was too much for him.

~~~
eridius
My impression from the article (and the quoted bit) is that you can't pay for
support once your project is suspended.

~~~
DannyBee
As far as i can tell, this is false (I have a suspended test project, and the
support options still seem to be there)

I think he means "there is no 'pay to talk to a human'" option in the FAQ/etc
for _this issue_.

That is, he wants to pay to talk to a human about just this issue, and thinks
somehow that paying for the support option will not give him that (or is too
expensive for that).

------
yladiz
This makes me wonder, which do people dislike more: not having access to
support in almost any capacity except "help desk tickets" which might not get
answered for days, e.g. how Google handles non-free support tiers, or horribly
long wait times on phones? My gut says that the former is worse, especially in
this case, because you can just turn your phone on speaker and wait for the
queue to finish up (or with some systems, have them call you back when the
queue is to you).

I almost can't believe that Google doesn't have a support line for its account
holders for Google Cloud, when you have companies like Paypal and United
Airlines, which have many more users (paying or not) and have support lines.

------
76bd
The situation of having a project fully shut down without adequate information
or process by which to correct it is clearly unacceptable. That said, I was
involved in recently migrating to GCP and it has worked wonderfully for us
thus far. We run a fairly standard java/MySQL web app, with paid silver
support. Questions are answered promptly in our experience. We found compute
engine VMs to be faster (and cheaper) than EC2, the developer console easier
to use, and better quality documentation/APIs.

------
oconnore
Just wait until you can buy one of their self driving cars.

------
manigandham
I usually defend GCP as they have better technology and platform architecture
but I definitely agree that they have major communication problems. While
there are plenty of individuals available through social channels, it does
seem to be very informal most of the time which does not inspire confidence as
a business looking for a solid foundation.

------
yclept
This sounds like a nightmare. One that I can't imagine happening at Rackspace
Cloud...

------
hehheh
Bet you cash money the affected site is or was in a ddos botnet.

------
Safety1stClyde
Google = rain man.

