
How a bug in VS2015 exposed my source code on GitHub and cost me $6,500 - MrMrtn
https://www.humankode.com/security/how-a-bug-in-visual-studio-2015-exposed-my-source-code-on-github-and-cost-me-6500-in-a-few-hours
======
Analemma_
This is becoming such a common story that I'm starting to think it's a little
irresponsible of both Amazon and GitHub to not react better to it. Clearly
Amazon is capable of detecting when this happens as evidenced by their rapid
response; when they see AWS keys published to GitHub, they should block the
account, or at least prevent it from spinning up any new instances. They can
then contact the owner over e-mail to follow up.

Same thing with GitHub. When they see an AWS key, they need to block that push
and send back a message saying "If you're really, really, REALLY sure you want
to do this, add .allowawskey at the root of your commit and push again".
There's no excuse for letting people shoot themselves in the foot like this
over and over.

(Edited for typo)

~~~
Alupis
Is it really on the web service companies to make sure you (the developer)
don't do anything stupid?

The article author could have avoided a lot of headache if he simply checked
the repo before walking away. Red flags should have gone up as soon as he
received the first weird Amazon messages, but he still didn't think to check
his own work.

The author calls this a "Data Breach" but in reality it was a "Data Leak".

~~~
mhuffman
Maybe not, but it think it is the web service company's responsibility to
alert and contact you on suspicious activity ... like a sudden $6,000 jump in
service usage of spot instances in regions you have never used before.

I mean a quick automated email would be enough for you to take action. The way
it is set up now, you pay for it and humbly beg Amazon to deduct those charges
if they see fit.

~~~
Alupis
I agree, and it seems this sort of thing is already in place on AWS, but the
author didn't heed the first two warnings. Short of just blocking things from
happening and risking inconveniencing other customers, I don't see much more
AWS could have done that would be reasonable.

------
famousactress
_" As a senior developer with over 12 years of experience in the financial
services industry, I didn't think it was possible that I could be the victim
of a data breach."_

We're only one sentence in and I'm already very, very nervous about the
author's hubris.

~~~
merlincorey
Thankfully, the author realizes later what it they _really_ did wrong here:

> What could be done to prevent this?

> Always test new version control GUIs before using them in the wild. There
> could be a bug that could expose your data.

> Encrypt access keys in config files instead of just leaving them exposed in
> the config file.

> Better yet, move access keys to a seperate config file, and exclude this
> from Git deploys with a .gitignore.

The fact this supposedly security experienced and conscious developer thought
it was okay to have any kind of access credentials stored in a git repository
is glaring.

You should consider all data in any repository public for security reasons.
Publishing to github publicly made it quite obvious - but he could have been
just as screwed over by people he gave access to the private repositories and
people who have broken into github and can directly access the repositories.

Sure the numbers on both those groups are lower than the numbers of people
apparently scanning public github for access keys - but the severity of the
risk of keeping access keys and other secrets in your revision control are the
same.

~~~
nailer
> Encrypt access keys in config files instead of just leaving them exposed in
> the config file.

How do decrypt them when you app re-launches? How do you decrypt them
automatically on every instance?

~~~
voltagex_
I've moved to using environment variables. I realise they have their own
security challenges but at least I'm not accidentally going to git push the
keys.

~~~
wereHamster
But you probably want to store the descriptions of the instances and what runs
on them in a git repository. It doesn't matter whether you use env variables
or something else. At some point you will have a git repository which stores
the CM for your whole infrastructure.

------
jongalloway2
I'm fairly certain this is a bug in the GitHub extension for Visual Studio
([https://visualstudio.github.com/](https://visualstudio.github.com/)), not in
Visual Studio itself.

Here's how the sync dialog appears in Visual Studio 2015 without the GitHub
extension: [http://i.imgur.com/EGjYziR.png](http://i.imgur.com/EGjYziR.png)

After installing the GitHub extension:
[http://i.imgur.com/shRBwXM.png](http://i.imgur.com/shRBwXM.png)

Try it yourself, then complete these fun, but completely optional activities:

1\. Re-read all comments on this thread using your best Andy Rooney voice.

2\. Wish that the author had comments on his blog.

~~~
jongalloway2
I logged an issue on the GitHub extension in case that's the problem:
[https://github.com/github/VisualStudio/issues/62](https://github.com/github/VisualStudio/issues/62)

------
d2xdy2
> As a senior developer with over 12 years of experience in the financial
> services industry, I didn't think it was possible that I could be the victim
> of a data breach.

Ok...

> Better yet, move access keys to a seperate config file, and exclude this
> from Git deploys with a .gitignore.

No shit?

~~~
curun1r
Even that's a terrible solution. AWS specifically changed their tooling to
look for access keys in ~/.aws and make it more difficult to load them from
inside a project directory. It's really not that hard to setup MFA and require
it when using STS to assume whatever role you need to make changes. For
automated changes, you can give instances the necessary an IAM role and only
run those from inside EC2.

He didn't get a $6.5k bill because of a bug in VisualStudio, he got one
because he didn't finish setting up his AWS account. Regardless of the
public/private mixup, checking in AWS credentials to any repository is just
bad practice and creates a huge risk of the exact situation that happened in
this story.

~~~
d2xdy2
Albeit not on AWS, I typically just set environment vars to reflect configs I
would have set in a .env (or .local.env, or .testing.env), or have some sort
of config manager daemon type thing running that keeps that sort of stuff
locked away.

> He didn't get a $6.5k bill because of a bug in VisualStudio, he got one
> because he didn't finish setting up his AWS account. Regardless of the
> public/private mixup, checking in AWS credentials to any repository is just
> bad practice and creates a huge risk of the exact situation that happened in
> this story.

I agree

------
morder
Visual studio does not integrate with Github. This was a 3rd party extension
created by Github that this user installed and used.

------
blablabla123
Most UIs for programmers still seem to lag behind the command line tools a
lot. I wonder if it will be possible again some day to create apps like in
Delphi without some black magic happening in the background.

P.S.: never ever put keys into repositories. Except those repos are deployment
config only.

------
AdamGibbins
Amazon are well known for refunding erroneous charges like this, they've done
so many times in the past - get in contact and ask for some leeway.

------
ocdtrekkie
I still find it amazing that AWS doesn't offer any way to hardcap your billing
and cut off service.

~~~
mbrutsch
There's really no financial incentive to do so.

~~~
tomschlick
There is when they keep refunding people for having their account hijacked and
on the other end are giving away CPU resources to the bitcoin miners.

~~~
ohashi
Here's a hypothetical, what is the real cost, he said they were using spot
instances too. A huge spike in demand might have an effect on price, pushing
the cost up for all legitimate users. As far as the reserved instances, they
were there, not being used presumably. They don't really lose anything except
possibly some electricity and bandwidth costs?

------
DenisM
If you're keeping your keys in source control, you're only one mistake away
from a disaster. It could be your own mistake, a bug in a VS, an error in
GitHub, a mistake by a teammate, anything, really.

It's similar to driving your car too close to the car in front - you will get
away with it most of the time and get lulled into a false sense of security,
but eventually it will catch up with you.

Don't be one mistake away from a disaster. Make room for at least two.

------
slg
Does Amazon actually end up charging you when this type of thing happens or do
they eventually wipe the bill after you talk to support? If Amazon tried to
charge me, I would definitely be in touch with the credit card company I have
attached to the account. This is rather straightforward fraud/identity theft
so you should be covered by the standard consumer protections most credit
card's provide.

~~~
e40
I accidentally posted AWS credentials to the AWS EC2 forum, and got > $10k
(US) of charges very quickly. They removed all the charges immediately,
without me pushing at all. They offered it.

~~~
slg
That is good to hear, but it also means the headline is a little on the click-
baity side. This is a bug that caused a lot of headaches and cost time to fix,
not one that cost $6,500.

------
apetresc
The most important part of this story seems to be glossed over – how did an
IAM key that the author insists only had access to the Alexa service get used
to spin up EC2 instances?

Either the author is just mistaken/lying about that point, or there's another
leak he doesn't know about, or there's a serious privilege escalation
vulnerability in AWS.

~~~
merlincorey
Someone else in a thread above supposes that they actually published amazon
master keys, rather than an AWS particular IAM role. The author of the post
states they didn't even use AWS prior, so they are most likely not aware of
such things.

------
merb
Why does somebody pushes AWS Keys into a Git Repository? Why should somebody
ever push some Authentications? Currently when I code an app I create a
database with name of the app and have new credentials with just name:name so
that the only thing that I check into my config if the app needs a
application.conf or resource.conf are default username:passwords for
development, so that bootstrapping is easier. However EVERYTHING else gets
read from the Environment Variables that even Works on Windows since Visual
Studio has a Feature for that, but be aware the sln file we keep the values.
So however never ever put anything into git / source code repo which shouldn't
be read.

~~~
zeveb
> Why does somebody pushes AWS Keys into a Git Repository?

If one wants to use version control to track credentials, then this makes
sense. Why would one want to do that? In order to track versions of one's
production system. This enables things like one-shot rebuilds.

Now, it's obvious that such a production repository should be separate from
one's development repo(s). And one could make a decent argument that it should
never be pushed someplace like GitHub anyway, even as a private repository.

------
medina
[http://ghcc.io](http://ghcc.io) \- Tool used to continuously monitor a Github
org for mistaken public commits (& lets you monitor repos you care about but
might not be able to enforce commit-hooks for).

------
lightlyused
pre commit hooks could have prevented this:
[https://gist.github.com/stevendanna/4db970716cabb8f5def4](https://gist.github.com/stevendanna/4db970716cabb8f5def4)

------
jerrac
Hopefully he posts updates about his unanswered questions. Since it sounds
like the compromised key should not have been able to create ec2 instances.

And I don't blame him for making that mistake. That kind of bug should have
been caught in QA.

On the other hand, I'm paranoid enough about things that I wouldn't push
unsecured access keys to any server I didn't control. Even if I was paying for
a private repo.

------
hoare
im impressed how fast a bot fetched the key from github, set up a trillion
instances and did what exactly on them?

edit: my guess, some kind of crypto currency mining?

~~~
JorgeGT
Mine bitcoin?

------
frozenport
Maybe secure keys are a lot secure than passwords?

------
pbreit
1) Seems like an inherent risk of closed source tools.

2) Someone other than the OP should pickup/waive the tab.

~~~
JamesBarney
If he is using the GitHub Extension then it's open source see
[https://visualstudio.github.com/](https://visualstudio.github.com/)

~~~
pbreit
I think you might be right:
[https://github.com/github/VisualStudio/commit/eb3f7f254de51a...](https://github.com/github/VisualStudio/commit/eb3f7f254de51af3a1f3d36fc164d6fd7eb2383a)

------
mahouse
Are we really going to run this story every month?

"I posted my private AWS keys by mistake on github, a spider got them, I got a
huge bill"

Yeah, you don't say

~~~
tonywebster
Don't be so quick to dismiss this one. This is notable for several reasons,
including (1) it wasn't his mistake, but rather that of a Visual Studio bug
not following his instructions to make a _private_ repo on GitHub; (2) the
speed in which this happened (minutes), and (3) it has useful analysis into
some sorely lacking functionality in AWS that lets this continue to happen.

~~~
nathancahill
I don't know. Even for private repos, it's bad practice to commit private keys
to source control.

~~~
zeveb
> Even for private repos, it's bad practice to commit private keys to source
> control.

Is it, though? Committing them to the same repository as one's code lives in,
sure, but committing them to a separate production-deploy repo seems okay to
me (although I'd much prefer that private repo never to hit a centralised
service like GitHub).

~~~
curun1r
> Is it, though?

Yes, it is. Period. Full Stop. Don't ever check access keys into any repo,
public, private or even self-hosted. AWS needs to do a better job of making
you realize that access keys are like 100-year-old sticks of dynamite and
should be handled with an equal amount of care considering they can cause a
similar amount of damage.

To their credit, they basically noted this when they changed the way access
keys are handled in the command line tool[1]. Quoting:

    
    
      An important point is that the default location for the credentials file is a user
      directory. It's no longer part of a project file structure, such as an app.config file
      (.NET) or .properties file (Java). This can enhance security by allowing you to keep the
      credentials in a location that's accessible only to you, and it makes it less likely that
      you'll inadvertently upload credentials if you upload a project to a developer sharing
      site like GitHub.
    

[1]
[http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-...](http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-
and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs)

~~~
zeveb
> Don't ever check access keys into any repo, public, private or even self-
> hosted.

So, how do you propose to handle version control of your exact production
configuration? Do you think that recording one's production configuration is a
valuable (or even useful) technique? Do you propose to have a single file with
production configuration relying only on point-in-time backups for history?
That gets us back to the bad old days…

A local or self-hosted repo _is_ 'a local location that's accessible only to
youation that's accessible only to you.'

~~~
curun1r
> So, how do you propose to handle version control of your exact production
> configuration?

Version control is perfectly acceptable for that, but the key point is that
access keys aren’t part of your production configuration. I personally don’t
allow my team to have long-lived AWS access keys that have access to
production. Production is only changed via an assumed role requiring MFA or an
IAM instance role.

If this kind of setup is too complex to create on your own, solutions like
Vault ([https://vaultproject.io](https://vaultproject.io)) can make it much
easier to get right.

~~~
zeveb
> Version control is perfectly acceptable for that, but the key point is that
> access keys aren’t part of your production configuration.

Why not? If your production configuration is stored as static-file in version
control, then either those credentials have to live in that version control,
or they have to live outside it, as a separately-backed up file of
configuration variables, which are then fed to a static-file-generating tool,
which means a) that one has to select a templating system and maintain those
separate templates and b) that there's another step involved in developing the
prototypes for those templates. This isn't a bad course of action, of course,
but it does have its costs (just like storing production
configuration—including credentials—in version control has its costs as well
as its benefits).

> I personally don’t allow my team to have long-lived AWS access keys that
> have access to production.

Of course! Developers shouldn't have access to production systems, period. But
operators can, and must, and I've no problem with them having access to their
own version-controlled repository of information that no other team can
access.

> Production is only changed via an assumed role requiring MFA or an IAM
> instance role.

Of course! I'm certainly not arguing for personal credentials to be stored in
an operations repo.

