Hacker News new | past | comments | ask | show | jobs | submit login
How a bug in VS2015 exposed my source code on GitHub and cost me $6,500 (humankode.com)
143 points by MrMrtn on Aug 31, 2015 | hide | past | favorite | 108 comments



This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to follow up.

Same thing with GitHub. When they see an AWS key, they need to block that push and send back a message saying "If you're really, really, REALLY sure you want to do this, add .allowawskey at the root of your commit and push again". There's no excuse for letting people shoot themselves in the foot like this over and over.

(Edited for typo)


Is it really on the web service companies to make sure you (the developer) don't do anything stupid?

The article author could have avoided a lot of headache if he simply checked the repo before walking away. Red flags should have gone up as soon as he received the first weird Amazon messages, but he still didn't think to check his own work.

The author calls this a "Data Breach" but in reality it was a "Data Leak".


Is it really on the web service companies to make sure you (the developer) don't do anything stupid?

Yes. The web service company should look at it as basic security - if people are using their service to discover private information and use it to defraud their users, even if the users have been stupid enough to upload that information publicly, the company should do what they can to mitigate the problem.

If I post a photo of my credit card on Twitter I would expect my bank to limit the damage as much as possible by declining obviously fraudulent transactions. Ultimately it's my own fault, and I would be liable for the costs, but that doesn't mean the bank should just let it happen if they can do something to stop it.


> mitigate the problem.

This seems like very minor engineering effort on behalf of GH - simple regexes, even.


To detect the key, sure. But then there's the interface design to inform the user, hooking in to an email to notify the user if they're committing via git, a system to disable the option if necessary, monitoring the AWS format in case it changes, dealing with other formats that are requested by AWS's competitors, and so on.

I don't think this would be a minor undertaking or simple in any way, but it is important enough that a provider (and target) the size of GitHub should still do it.


Amazon would save some money by doing this, since they usually reimburse "victims" of public AWS keys (at least on the first instance) and eat the cost.


They don't "reimburse" anything. They only cancel the due amount. The only thing they lose is their wasted computing power (and some support costs) and I think the RAW computing power is very cheap to them, especially considering their scale. The cost in terms of support for the victims is probably bigger but still not significant.


It's still a cost that could be mitigated with a couple hours of someone's time.


GitHub is a service. Its not their job to offer me issue tracking, or pull requests, or really much of anything. They CHOOSE to offer me things, and if those things are interesting, I CHOOSE to pay them. As users, we can certainly declare that we believe AWS key detection to be of foundational importance to a code sharing community, and signal them to thus take action or risk losing us as paying customers. I would certainly feel better if this was a feature, given its apparent ease of implementation and huge head aches it would solve.


> Is it really on th web service companies [...]

Yes. Or, it's not so much a responsibility as it is an opportunity. I'd choose the provider that stops me from shooting my foot.

I assume AWS has spending caps for each account. My credit card company blocks my card and calls me if I spend in a suspicious pattern. And so on.


Maybe not, but it think it is the web service company's responsibility to alert and contact you on suspicious activity ... like a sudden $6,000 jump in service usage of spot instances in regions you have never used before.

I mean a quick automated email would be enough for you to take action. The way it is set up now, you pay for it and humbly beg Amazon to deduct those charges if they see fit.


I agree, and it seems this sort of thing is already in place on AWS, but the author didn't heed the first two warnings. Short of just blocking things from happening and risking inconveniencing other customers, I don't see much more AWS could have done that would be reasonable.


You can easily set up billing alerts in cloudwatch to do this. I have done so. I agree by default it should prompt you to do so during the aws on boarding process.


> The article author could have avoided a lot of headache if he simply did not check in secrets into any source control system.

FTFY

It baffles me that people actually do this. Passwords and secrets DO NOT BELONG THERE. Even your 'if he simply checked the repo' statement is in my view completely crazy. No you should not have to worry about that, because that should not be there at all. Leaking your source-code should not result in a system-breach at any point.


It's not "on" them but it would behoove them to perform better than this tale suggests. And they are not doubt capable of such.

Or just waive the charges.


GitHub is a value added service.

Making sure I don't do something stupid and expensive is a value add.


There's a huge amount of value in a service that insulates you from mistakes like this. You really, really have to know what you're doing with EC2 because it is a loaded gun.

Personally, I don't like using things that are covered in spring loaded booby traps that require eternal hypervigilance in order to avoid nasty financial consequences. Certainly not when deploying something that would be right at home on something like Heroku.


> You really, really have to know what you're doing with EC2 because it is a loaded gun.

That's a fair point, but keep in mind the article author was given several warnings that something was wrong, but failed to correct the problem each time (which resulted in the huge bill he ultimately received).

At what point is the responsibility on the author here?


The author deleted the old key and created a new one and changed his password as soon as he was notified. How did he fail to correct the problem? What else should he have done?


> or at least prevent it from spinning up any new instances

This is the answer I would like to see implemented. If they can detect the key they can put a freeze on the account so that only existing resources are available.

They should also have a timeline of all created instances in every zone so you can see what has happened in the last x hours and kill all instances at once.


Two things would make controlling these situations easier:

* Allow me to see all instances in all regions from a single screen

* When accounts are first created for AWS, set up some sane default IAM policies and/or hire some UI designers to fix that godawful mess.


This is a really good point. There was another post recently linking to a search for id_rsa files (SSH private keys) on GitHub. Instead of allowing people to publish sensitive things like that accidentally, GitHub really should require an extra confirmation.


I wonder if you actually read the story. The author displays incompetence in technology administration at multiple steps.

From committing the basic design error of using hard-coded credentials at all; to committing them to any kind of repository; to failing to check on the visibility of that repository; to giving those credentials privileges they did not need; to failing to act effectively on warning messages.

And all this, apparently, is laid at the feet of Visual Studio.

I have no love of the Microsoft ecosystem, but the author does not acquit themselves well in this incident. The arrogant claims of seniority in the opening paragraph seem ironic in this light.


> all this, apparently, is laid at the feet of Visual Studio

I wonder if you actually read the story.

He admits several faults including most if not all of those you mention.


So what? Admitting fault isn't absolution. He really does go out of his way to blame the tools.


The tool has a huge bug! Even if he wasn't putting credentials in there, it could still have led to sensitive source code getting out or something!


Article updated - VS wasn't the issue. He had installed a 3rd party github extension (from github themselves) that overrides the built-in Git support in visual studio... bug was found, confirmed and fixed yesterday though, so a quick turnaround.


Exposing an AWS key through a public repo happened to me a few years ago and AWS support called me to confirm whether I was aware of the suspicious activity, when I stated I do not know what's going on they immediately blocked the account and told me I wouldn't be responsible for the cost. Mine was actually much worse, the account was billing a thousand dollars every hour. They told me I'd have to sign up again with a different account.. I'm surprised this process was not followed in author's case.


obviously someone is good at detecting it considering 6.5k in usage got racked up with data scraped within 10 minutes of commit


The quick bitcoin mining spin-ups, and the initial email from Amazon about the key going public, are (probably) thanks to the github events firehose.

https://github.com/hootware/github-firehose-node

http://blog.scalyr.com/2013/10/exploring-the-github-firehose...


It sounds like it happened the next day, after he changed the keys.


AWS do do things. They have service limits and routinely waive charges from abuse events like this.

I do wonder if GitHub's events firehose has benefits that outweigh all the compromised credentials, though. Not just limited to AWS keys.


> They have service limits and routinely waive charges from abuse events like this.

That's almost worse though. Maybe I'm being unfair here, but the fact that they will waive charges immediately if you ask, but don't just block them pre-emptively, tells me that they're hoping at least some customers (with deep pockets and automated billing that no one pays attention to) will just pay the bill and never notice. Maybe they're making more money from that than they're losing by waiving charges on compromised accounts for customers who ask.


Devils advocate, wouldn't you rather have illegitimate charges waived after the fact that legitimate charges declined because their algorithm threw up a red flag? Something like being posted to HN could cause a huge spike in charges for your account in a similar way as leaking your keys and it would be beyond frustrating to see that opportunity disappear because Amazon saw your bill spike.


Don't be ridiculous. AWS want legitimate revenue, not brand damage.


You really want GitHub analyzing your code? REALLY? If the internet found out they were analyzing every push the backlash would be 10x what it is now.

Even if it's public... That's still a Gmail-esque situation that would immediately raise privacy concerns.

Developers should pay attention to what they're doing. Your AWS keys shouldn't even been in your private repo. You should find another way or you shouldn't be using a third-party for source control.


Looks like someone is analyzing it anyway. May as well be to your benefit.


It would probably be easy for Amazon to offer an automatic key blacklisting feature....


Natural selection at work, do not disturb.


"letting people shoot themselves in the foot like this over and over"

Stop doing it in the first place, problem solved. I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing.

Also, this guy didn't just use AWS keys...he clearly used AWS master keys...which has not been the recommended course of action for AT LEAST THREE YEARS...if he had used a proper IAM user with only the access he needed, this wouldn't have happened. Sorry, but that was just irresponsible.


"I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing."

No it wouldn't, not at all. Not unless you're one of those, "REAL MEN us {$editor}" people, in which case no one cares about your opinion.


"As a senior developer with over 12 years of experience in the financial services industry, I didn't think it was possible that I could be the victim of a data breach."

We're only one sentence in and I'm already very, very nervous about the author's hubris.


Thankfully, the author realizes later what it they really did wrong here:

> What could be done to prevent this?

> Always test new version control GUIs before using them in the wild. There could be a bug that could expose your data.

> Encrypt access keys in config files instead of just leaving them exposed in the config file.

> Better yet, move access keys to a seperate config file, and exclude this from Git deploys with a .gitignore.

The fact this supposedly security experienced and conscious developer thought it was okay to have any kind of access credentials stored in a git repository is glaring.

You should consider all data in any repository public for security reasons. Publishing to github publicly made it quite obvious - but he could have been just as screwed over by people he gave access to the private repositories and people who have broken into github and can directly access the repositories.

Sure the numbers on both those groups are lower than the numbers of people apparently scanning public github for access keys - but the severity of the risk of keeping access keys and other secrets in your revision control are the same.


He could also separate code and configuration as per the docs but I suspect the hubris excludes bothering to read them...:

http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/Developer...


> Encrypt access keys in config files instead of just leaving them exposed in the config file.

How do decrypt them when you app re-launches? How do you decrypt them automatically on every instance?


I've moved to using environment variables. I realise they have their own security challenges but at least I'm not accidentally going to git push the keys.


But you probably want to store the descriptions of the instances and what runs on them in a git repository. It doesn't matter whether you use env variables or something else. At some point you will have a git repository which stores the CM for your whole infrastructure.


I took it less as hubris and more of a way to imply to readers that no matter their experience they are not immune to the mistake he is going to tell you about.


I think it's definitely some hubris, because the mistake he made wasn't publishing to github publicly by accident. It was thinking a git repository was a good place to store secrets.


Why? It sounded like he is very security conscious. Thought he was creating a private repo in VS, immediately after he was notified deleted and recreated his keys. A pretty spot on course of action I think.


1. He stored credentials in source control rather than using configuration profiles as specified in the AWS SDK documentation.

2. He pushed code to a new repository without verifying the security configuration of the created repository (granted the tool made it easy to do this but you should create a repo, verify it, then push code).

3. He used his master AWS account key/secret in the code which gave global access to everything.

4. He didn't use IAM credentials with a restrictive policy set to just access the resources required.

No, clearly not security concious.

I'm a solution architect in the financial services industry and have been for 16 years. Never do I assume I know what the hell I'm doing.


Because anyone who ever says "I didn't think it was possible that I could be the victim of a data breach" is dangerously delusional, and in case I'm not being clear - wrong.

Also putting keys in (even a private) cloud-hosted repos is at least a step under what I'd describe as "very security conscious" for someone who self describes as working in the financial industry.


Right? Who spends 12 years in the software industry and doesn't actually test their work after doing shit?


Do you test the functionality of your git client or code editor? I personally generally expect them to work, as do most people.


On first use, i guess i have checked these parts one way or another. To assume things work the way you expect them might be the reason the bug was present in VS (if it ever actually was).


I'm fairly certain this is a bug in the GitHub extension for Visual Studio (https://visualstudio.github.com/), not in Visual Studio itself.

Here's how the sync dialog appears in Visual Studio 2015 without the GitHub extension: http://i.imgur.com/EGjYziR.png

After installing the GitHub extension: http://i.imgur.com/shRBwXM.png

Try it yourself, then complete these fun, but completely optional activities:

1. Re-read all comments on this thread using your best Andy Rooney voice.

2. Wish that the author had comments on his blog.


I logged an issue on the GitHub extension in case that's the problem: https://github.com/github/VisualStudio/issues/62


> As a senior developer with over 12 years of experience in the financial services industry, I didn't think it was possible that I could be the victim of a data breach.

Ok...

> Better yet, move access keys to a seperate config file, and exclude this from Git deploys with a .gitignore.

No shit?


Even that's a terrible solution. AWS specifically changed their tooling to look for access keys in ~/.aws and make it more difficult to load them from inside a project directory. It's really not that hard to setup MFA and require it when using STS to assume whatever role you need to make changes. For automated changes, you can give instances the necessary an IAM role and only run those from inside EC2.

He didn't get a $6.5k bill because of a bug in VisualStudio, he got one because he didn't finish setting up his AWS account. Regardless of the public/private mixup, checking in AWS credentials to any repository is just bad practice and creates a huge risk of the exact situation that happened in this story.


Albeit not on AWS, I typically just set environment vars to reflect configs I would have set in a .env (or .local.env, or .testing.env), or have some sort of config manager daemon type thing running that keeps that sort of stuff locked away.

> He didn't get a $6.5k bill because of a bug in VisualStudio, he got one because he didn't finish setting up his AWS account. Regardless of the public/private mixup, checking in AWS credentials to any repository is just bad practice and creates a huge risk of the exact situation that happened in this story.

I agree


Visual studio does not integrate with Github. This was a 3rd party extension created by Github that this user installed and used.


Most UIs for programmers still seem to lag behind the command line tools a lot. I wonder if it will be possible again some day to create apps like in Delphi without some black magic happening in the background.

P.S.: never ever put keys into repositories. Except those repos are deployment config only.


Amazon are well known for refunding erroneous charges like this, they've done so many times in the past - get in contact and ask for some leeway.


I still find it amazing that AWS doesn't offer any way to hardcap your billing and cut off service.


There's really no financial incentive to do so.


There is when they keep refunding people for having their account hijacked and on the other end are giving away CPU resources to the bitcoin miners.


Here's a hypothetical, what is the real cost, he said they were using spot instances too. A huge spike in demand might have an effect on price, pushing the cost up for all legitimate users. As far as the reserved instances, they were there, not being used presumably. They don't really lose anything except possibly some electricity and bandwidth costs?


When people run up a huge bill and don't pay it, that's an incentive.


The profit margin on AWS is stupid high. It costs Amazon nothing but power and opportunity cost to refund unintended use.


What about the potential customers they lose?


Side question: is there any similar cloud service with hard billing caps?


I'd really love to know. Because if I could get AWS-like service with the ability to have it cut off if my bill gets to say... three times my usual monthly... That'd be worth switching for the peace of mind.


If you're keeping your keys in source control, you're only one mistake away from a disaster. It could be your own mistake, a bug in a VS, an error in GitHub, a mistake by a teammate, anything, really.

It's similar to driving your car too close to the car in front - you will get away with it most of the time and get lulled into a false sense of security, but eventually it will catch up with you.

Don't be one mistake away from a disaster. Make room for at least two.


Does Amazon actually end up charging you when this type of thing happens or do they eventually wipe the bill after you talk to support? If Amazon tried to charge me, I would definitely be in touch with the credit card company I have attached to the account. This is rather straightforward fraud/identity theft so you should be covered by the standard consumer protections most credit card's provide.


I accidentally posted AWS credentials to the AWS EC2 forum, and got > $10k (US) of charges very quickly. They removed all the charges immediately, without me pushing at all. They offered it.


That is good to hear, but it also means the headline is a little on the click-baity side. This is a bug that caused a lot of headaches and cost time to fix, not one that cost $6,500.


The most important part of this story seems to be glossed over – how did an IAM key that the author insists only had access to the Alexa service get used to spin up EC2 instances?

Either the author is just mistaken/lying about that point, or there's another leak he doesn't know about, or there's a serious privilege escalation vulnerability in AWS.


Someone else in a thread above supposes that they actually published amazon master keys, rather than an AWS particular IAM role. The author of the post states they didn't even use AWS prior, so they are most likely not aware of such things.


Why does somebody pushes AWS Keys into a Git Repository? Why should somebody ever push some Authentications? Currently when I code an app I create a database with name of the app and have new credentials with just name:name so that the only thing that I check into my config if the app needs a application.conf or resource.conf are default username:passwords for development, so that bootstrapping is easier. However EVERYTHING else gets read from the Environment Variables that even Works on Windows since Visual Studio has a Feature for that, but be aware the sln file we keep the values. So however never ever put anything into git / source code repo which shouldn't be read.


> Why does somebody pushes AWS Keys into a Git Repository?

If one wants to use version control to track credentials, then this makes sense. Why would one want to do that? In order to track versions of one's production system. This enables things like one-shot rebuilds.

Now, it's obvious that such a production repository should be separate from one's development repo(s). And one could make a decent argument that it should never be pushed someplace like GitHub anyway, even as a private repository.


> Why does somebody pushes AWS Keys into a Git Repository?

Wild speculation, but this is a side effect of "developer" centric operations. Everything is done via code, and developers are instinctively trained to check in code.


http://ghcc.io - Tool used to continuously monitor a Github org for mistaken public commits (& lets you monitor repos you care about but might not be able to enforce commit-hooks for).


pre commit hooks could have prevented this: https://gist.github.com/stevendanna/4db970716cabb8f5def4


Hopefully he posts updates about his unanswered questions. Since it sounds like the compromised key should not have been able to create ec2 instances.

And I don't blame him for making that mistake. That kind of bug should have been caught in QA.

On the other hand, I'm paranoid enough about things that I wouldn't push unsecured access keys to any server I didn't control. Even if I was paying for a private repo.


im impressed how fast a bot fetched the key from github, set up a trillion instances and did what exactly on them?

edit: my guess, some kind of crypto currency mining?


Mine bitcoin?


Try reading the article


Maybe secure keys are a lot secure than passwords?


1) Seems like an inherent risk of closed source tools.

2) Someone other than the OP should pickup/waive the tab.


If he is using the GitHub Extension then it's open source see https://visualstudio.github.com/



Are we really going to run this story every month?

"I posted my private AWS keys by mistake on github, a spider got them, I got a huge bill"

Yeah, you don't say


Don't be so quick to dismiss this one. This is notable for several reasons, including (1) it wasn't his mistake, but rather that of a Visual Studio bug not following his instructions to make a private repo on GitHub; (2) the speed in which this happened (minutes), and (3) it has useful analysis into some sorely lacking functionality in AWS that lets this continue to happen.


I don't know. Even for private repos, it's bad practice to commit private keys to source control.


> Even for private repos, it's bad practice to commit private keys to source control.

Is it, though? Committing them to the same repository as one's code lives in, sure, but committing them to a separate production-deploy repo seems okay to me (although I'd much prefer that private repo never to hit a centralised service like GitHub).


> Is it, though?

Yes, it is. Period. Full Stop. Don't ever check access keys into any repo, public, private or even self-hosted. AWS needs to do a better job of making you realize that access keys are like 100-year-old sticks of dynamite and should be handled with an equal amount of care considering they can cause a similar amount of damage.

To their credit, they basically noted this when they changed the way access keys are handled in the command line tool[1]. Quoting:

  An important point is that the default location for the credentials file is a user
  directory. It's no longer part of a project file structure, such as an app.config file
  (.NET) or .properties file (Java). This can enhance security by allowing you to keep the
  credentials in a location that's accessible only to you, and it makes it less likely that
  you'll inadvertently upload credentials if you upload a project to a developer sharing
  site like GitHub.
[1] http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-...


> Yes, it is. Period. Full Stop. Don't ever check access keys into any repo, public, private or even self-hosted.

Generalizing here (and for the record I agree with your point), is that to a developer, source code is as valuable as anything else, and if a private repo is secure enough for source code, it should be secure enough for access credentials.

I speculate that this is a by product to moving towards a developer-centric engineering culture.


> Don't ever check access keys into any repo, public, private or even self-hosted.

So, how do you propose to handle version control of your exact production configuration? Do you think that recording one's production configuration is a valuable (or even useful) technique? Do you propose to have a single file with production configuration relying only on point-in-time backups for history? That gets us back to the bad old days…

A local or self-hosted repo is 'a local location that's accessible only to youation that's accessible only to you.'


> So, how do you propose to handle version control of your exact production configuration?

Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration. I personally don’t allow my team to have long-lived AWS access keys that have access to production. Production is only changed via an assumed role requiring MFA or an IAM instance role.

If this kind of setup is too complex to create on your own, solutions like Vault (https://vaultproject.io) can make it much easier to get right.


> Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration.

Why not? If your production configuration is stored as static-file in version control, then either those credentials have to live in that version control, or they have to live outside it, as a separately-backed up file of configuration variables, which are then fed to a static-file-generating tool, which means a) that one has to select a templating system and maintain those separate templates and b) that there's another step involved in developing the prototypes for those templates. This isn't a bad course of action, of course, but it does have its costs (just like storing production configuration—including credentials—in version control has its costs as well as its benefits).

> I personally don’t allow my team to have long-lived AWS access keys that have access to production.

Of course! Developers shouldn't have access to production systems, period. But operators can, and must, and I've no problem with them having access to their own version-controlled repository of information that no other team can access.

> Production is only changed via an assumed role requiring MFA or an IAM instance role.

Of course! I'm certainly not arguing for personal credentials to be stored in an operations repo.


I use Ansible Vault if it's imperative to store keys in a repo, and I've been really happy with it. StackExchange has something similar called Blackbox[0]. I remember reading about a Ruby gem too, but I can't find it now.

[0] https://github.com/StackExchange/blackbox


> https://github.com/StackExchange/blackbox

That looks pretty cool, at least at first. I'll have to take a deeper dive into it.

I like that it looks to be all shell scripts. Wish it were sh rather than bash, but c'est la vie.


AWS SDK for .Net supports named profiles for VS so your source check in only contains a profile name. If you hard coded your IAM or account keys or stuck them in the app.config you're simply doing it wrong. There is no excuse. It's all here: http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/Developer...

Also if you use a proper IAM profile locked down to specific resources then you wouldn't expose your entire account. The author stated he didn't use EC2 so why wasn't the key/secret pair an IAM account with a policy set for minimal access?

What I'd worry more about is that the VS bug exposed private source code, data and proprietary intellectual property.

We use github and I worry every day someone will public fork one of our repos by accident. That would be a grave fuck up but it's waiting to happen. We should have stuck with centrally controlled active-directory integrated SVN from a security perspective (even if it is a pain in the ass).


Well, it is partially his mistake, which he acknowledges:

> I am certainly not innocent here and some mistakes were made on my part. [...] To this end, having encrypted access keys or excluding configuration settings from GitHub would have prevented the AWS charges - and this is certainly the approach I would take from now on.


In this case it wasn't Visual studio but an extension created by Github that caused the problem.


He didn't post it by mistake. He asked Visual Studio to create a private repo and it created a public one. A bug which he reproduced.


It wasn't even visual studio but an extension created by github that caused the issue. Visual studio does not come with github integration by default


Sure, but he shouldn't have checked keys in to any repo, public or private!




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: