
Student Charged $14k on Stolen Google Cloud Credentials - nitins_jakta
Hi,<p>In 2017, I made a Google Cloud Account to use Google Maps API for a Computer Science student group project and put my debit card in. I naively put a $5 account notification in, thinking it was a cap. This project was defunct after 2017 and I should have just closed the Cloud account.<p>All was fine up until January 2019 when the Google Cloud Credentials were somehow stolen and over the course of two days on Google Maps API, racked up enough API calls to generate over $14k invoice. I disabled the Google Cloud Account a day after I noticed an email from Google Cloud. Google Cloud did try to use debit card to deduct from checking account, but I don&#x27;t leave thousands sitting around in it, so charge was declined.<p>I talked to Google Cloud Billing and they have not been helpful, telling me to contact my bank. Today, I got a scary email from a collections agency demanding I login to my Google Cloud account and pay the bill! Worst part is, this API used to be free, until Google started charging exorbitant amounts for it.<p>I know I did not make these API calls -- if you looked at the call volume history, there was nothing for well over a year, until those two days in 2019, it started going crazy (and the project is not running on any server or being used in any way). I suspect a group member might have accidentally leaked the credentials.<p>I know AWS has waived costs[1] like this in the past, but Google is not known for customer support. I should have been more proactive in setting up a cap.<p>Appreciate any advice or Google contacts to talk to an actual human. Should I see if Google is willing to actually verify this was unauthorized usage or just lower the bill? I&#x27;ll eat a few thousand just to make this go away.<p>To say GCP has left a sour taste in my mouth is an understatement!<p>Thanks for reading.<p>[1] https:&#x2F;&#x2F;dev.to&#x2F;juanmanuelramallo&#x2F;i-was-billed-for-14k-usd-on-amazon-web-services-17fn
======
boulos
Sorry this happened to you! Feel free to send me your case number (email in
profile), and I'll escalate it.

The Support personnel have hopefully been helping out, as all Billing Issues
are covered regardless of support tier. I obviously don't know the ins and
outs of payment instrument refunds / do debit cards mean that you actually do
have to contact your bank, but I'm sure people in Support do.

~~~
travisglines
Thanks for responding to this and offering to escalate boulos.

I certainly don't think a student just learning the ins and outs of a cloud
provider's services should be able to spend 10k+ without warnings/thresholds
that require configuration to exceed. It would be positive for platform
adoption to make that process better.

~~~
motherofzappa
This is hilarious. Student doesn’t understand security in depth model, gets
owned. Has a sour taste with said cloud provider. At what level do you accept
responsibility for shoddy security practices. If the project was truly defunct
then you should have closed the project or removed everyone’s access who isn’t
project owner. Hindsight is 20/20.

~~~
duiker101
Nice victim shaming you got there. The fault of all of this is 100% on whoever
stole the credentials and made those calls. OP could maybe have been more
careful but that doesn't mean it's all his fault or that we should be shaming
in oblivion. And Google can still try to help rather than just take advantage
of the situation. Life is easier when we are not dicks to each other, a little
empathy can go a long way.

~~~
relaunched
>>>Life is easier when we are not dicks to each other, a little empathy can go
a long way.

I found my new email signature.

~~~
highhedgehog
I am seriously wondering whether it's a good idea to put it in my business
email or not :D

------
scarface74
I’ve heard so many stories of something similar happening on AWS and after an
email to support, all of the charges were dropped.

This isn’t exactly helping Google to fight the narrative that it isn’t good
with customer support and they can’t be trusted as a platform for business.

So if you were a person deciding who to choose as your cloud provider, who are
you going to choose?

AWS - “No one ever got fired for choosing AWS”

Microsoft - well known for their enterprise support and there are plenty of MS
Shops out there.

Or

Google?

~~~
nitins_jakta
I agree with you 100%. This has certainly left a sour taste in my mouth.
Unfortunately Google Maps API is much better than the competition. AWS has
none here.

The google customer experience is just horrific.

------
segmondy
While you're figuring this out, backup all your data on Google. Google is
crazy and could possibly delete all your accounts and data.

~~~
nitins_jakta
Backing up now, thanks. Even worse I'm an Android user. I don't have a good
solution for the people that mail me every few months. Oh hey, Google might
lock my account because I didn't fork over 14k in fraudulent charges, can you
start emailing me from my Outlook email instead?

~~~
quickthrower2
The idea email is your own domain name. I’m yet to take my own advice though!

------
unknownkadath
Before disputing the charge, be sure to back up all data and contact info from
your Google accounts. Fighting charges has been known to trigger account
lockouts with no appeal.

~~~
nitins_jakta
This is even more terrible as an Android user. So much for "don't be evil".
Thankful for the HN forum.

~~~
unknownkadath
Hope it works out with a minimum of hassle!

------
applecrazy
Did you check your Github repos and associated commit history for accidental
push of secret files? There's an article on the HN front page describing
secret leakage in Github repos (the most common is Google API keys, go
figure)[1]. I imagine somebody out there has a bot to monitor pushes in
realtime to extract secrets. You or a team member might have leaked keys in a
similar manner.

[1]: [https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-
chara...](https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-
characterizing-secret-leakage-in-public-github-repositories/)

------
londons_explore
Google will typically waive charges in cases like this.

The only time they won't is if (by looking at the logs) they decide you were
probably scraping and storing all their data.

------
foobarbazetc
Make them prove you used it to generate the charges. Make them provide IPs
etc.

You need to say it was used fraudulently and you don’t agree to the charges.

~~~
nitins_jakta
Will do, thanks for the advice. The only thing they did was validate the API
count, but I was not disputing this, only that I made the calls.

The call volume itself looks suspicious. No calls for well over a year and
then suddenly an incredible amount? You would think Google would have some
algorithm to detect this?

~~~
foobarbazetc
Yeah exactly. But the key is not to say “lower my bill”, but rather “these
charges aren’t mine”.

If they send it to collections you can dispute it. A lot of laws and things
around that.

It sounds like it hasn’t been charged off yet though, since they want you to
pay Google, not them.

I wish you luck. :)

------
samfisher83
Next use a credit card. Basically thanks to credit card laws the bank will go
tell google to f off and give you your money back. Debit cards don't have the
same protection, but just call your bank or OCS
([https://www.occ.treas.gov/](https://www.occ.treas.gov/)). They have a little
more bite.

~~~
joecot
His debit card ended up declining it anyway, but that's not the point. The
point is Google thinks he owes them thousands of dollars, and a collection
agency is hounding him for the money.

Disputing a charge with your credit card company doesn't change that. The
credit card company might side in your favor and reverse the charges, but the
original company can still pursue payment with you directly. Their claim can
even show up on your credit report. I remember working with a payment
processor in Europe that even automated sending a bill to collections if it
was charged back by the consumer's credit card company.

~~~
motherofzappa
playing devils advocate, but taking peoples word prima facie is dangerous.
Especially when it deals with financial issues. Pretty sure you can any
contempt person could game the system if people found out “students”
automatically get refunds for bs charges. I think AWS is different as they
have their educate program and distinguish between students and free trial
customers. AFAIK gcp doesn’t have an education system for students, so it’s
harder to differentiate a legitimate request

~~~
quickthrower2
What will these system gamers do with computer power? Mining crypto will make
them pennies, and running a business on stolen accounts isn’t sustainable.

~~~
kkarakk
there are entire dark market businesses built out of finding secrets and
deploying a crypto miner as fast as possible and mining as much as possible.

that's just one use case, DDOSing an API/website through seemingly legit
requests from the G3's cloud infrastructure is another.

Lots of things you can do with massive amounts of computing power. It doesn't
have to be sustainable in the short term, you just keep repeating the process
and the value adds up fast

------
cjbprime
Keep trying to talk to them and explain -- so far every instance like this
I've heard about was refunded. Good luck.

------
kkarakk
I dunno if google lets you do this but amazon/azure will pretty reliably let
you create new free tier accounts with fake emails and access them from the
same IP. i just create a new debit/credit card every 6 months(it's pretty
hassle free in india).

i do pay for production instances, i just don't want to mess around on those
production instances

