I set up a project[1] to automatically leak AWS secrets published to the Python package index, which then triggers the secret scanning process and quarantines the keys[2]
It is really silly that the only easy way to report a leaked secret from most services is to push it to GitHub. I don't understand how the industry failed to standardize any secret-reporting scheme.
This is a misunderstanding of how standards come about. Standards don't come about because someone theorizes pushing the envelope and then everyone gets together to form a convention to decide the best way to push the envelope.
Standards describe existing behaviour of value that benefits from a single interface. For that, the behaviour has to be first demonstrated. That's how the envelope is pushed. You make an implementation, then you show how that solves something worthwhile, and then if there are multiple implementations that could benefit from a single interface, you do that.
Totally agree, and I do hope that at some point this will be standardised.
Ideally you’d have an endpoint that returns a set of patterns to check for and a way to submit a set of tokens for checking, along with some kind of context. It would probably be something like what GitHub requires[1].
Unfortunately that has somewhat limited value when GitHub can do most of the legwork for you, and as the de-facto place to publish code the majority of “accidental secrets exposed in code” are leaked there.
That's right, the behavior exist, on GitHub. Multiple services, including open-source ones like PyPI, integrated with it because they find it worthwhile. Yet it remains impossible for anyone to use it except through GitHub.
How would this work in practice? In order to effectively filter for tokens you need a set of pretty specific regular expressions.
These specific expressions may contain non-public information about the structure of the tokens.
But imagine we live in a world where companies are OK with exposing this information publicly and investing time in keeping the expressions up to date.
Now what? How do you report it to services whilst stopping abuse? Like, if the code isn’t public then all this is an endpoint to let anyone to disable another users (or companies) key without any explanation or recourse.
Pushing to GitHub is actually quite an elegant solution to these issues I think, even if I do think it is a bit too centralized.
I mean yes — that’s the point. If untrusted parties have access to your keys, it’s already game over. You’ve lost. Disabling them is the nicest thing an attacker can do for you.
Not really: it depends on the permissions assigned to the keys.
I wouldn’t like to wake up to an email that says “your key has been disabled because someone anonymously reported is as leaked, sorry if this has broken your entire system”.
What do you do with this, outside of obviously quarantining and/or disabling the key? How was it leaked? What’s the context?
Indeed, so you’re building all this tooling and complexity and introducing more issues for the very small intersection of people that:
1. Are not malicious
2. Have access to a key
3. Are unable or unwilling to commit it to GitHub
It would be great if this stuff was public and available without a central authority. But after working on it for a while it seems like a fairly good compromise.
There’s already an entire pipeline that handles a key being compromised when it is found on GitHub. All the “tooling and complexity” you need is a simple HTML form to ask for a key and where you found it, and some server-side code to trigger the same pipeline when somebody submits the form.
There are three issues with the use of GitHub here:
1. Not everybody knows that AWS will invalidate tokens committed into a public GitHub repository.
2. There is a window (67 seconds according to OP) in which the compromised token is public but working. For the “small intersection of people”, you could bring it down to 0.
3. GitHub protects GitHub keys, and apparently AWS keys, but does it protect Azure keys? Or GCP keys?
Not necessarily people but systems. Your code hosting platform, your mailing-list host, your chat app, ... are all examples of systems who are hopefully not malicious and could easily add this auto-revoke-publicly-leaked-secret feature, if it was a simple consistent scheme like an HTTP DELETE to a URL.
I'd like a standardized token syntax that contains a domain (e.g. `secret:example.com/token`) Then you could fetch an url like `https://example.com/.well-known/secret-leak` which returns machine readable information about how to report a leaked token.
A few cycles ago [4] sudhirj pointed out that you could just use an actual URL as your secret key and have that be the URL you visit to revoke it, and I think that is genius.
We're currently working on follow up work to make secret management better (with NSF funding[1]), and we're always looking for research collaborators[2].
A few years ago this wasn't in place. I goofed and leaked a credential via Github, and woke up to $50k and climbing in AWS fees (Monero miners). AWS worked with us, and once we proved we had adequately locked our account down and added additional fail-safes, they took it off of our bill.
> In the email they also give guidance on how to secure the account. They do this in four steps: 1. Rotate and delete the exposed AWS Access Key...
I work at a very small shop (at a non-profit institution).
In the past, when we've accidentally leaked a credential, AWS has required us to rotate all of our AWS Access Keys in the account, not just the exposed one.
Which I know ideally rotation would be easy/automated and this wouldn't matter, but at the time it was disruptive for us. And a bit frustrating when we had intentionally created separate AWS credentials with least privs needed for the service, etc. But thinking on it, I guess any given credential might give permissions to create other credentials or other devious things, rotating all of them might be necessary to be safe.
It's been a couple years since we had a leaked credential now (hooray). I wonder if the policy has changed, or if I misunderstood, or if in fact you still need to rotate all credentials in the account, maybe OP just only had one credential in the account they were intentionally doing insecure things so the compromised key was in fact all keys. (I don't specifically remember the `AWSCompromisedKeyQuarantineV2` policy, but probably am just not remembering it, I had other things to worry about at the times!)
This used to not be possible. AWS only added support for OIDC Credentials in 2014, which is 10 years ago but well after AWS launched. Convenient support in client libraries was only added in 2019[1][2].
Ever wondered what would happen when you leak AWS credentials to GitHub? I was, so I leaked AWS credentials to GitHub to see what would happen.
Both AWS and malicious actors respond very quickly and start taking actions on these leaked credentials.
Has anyone got a script or git hook that will suggest you not to push anything that looks like a credential? Could be really useful.
It would need to identify strings that look like a key/password, and I guess also it would need an override, perhaps a comment like "// Demo password".
I think it will be great if AWS allowed us to use the AWS keys to be used from certain IPs only. It will add additional layer of security which can help to prevent a lot of misuse. Just like AWS best practices of putting DB instances in a private VPC can prevent a lot of attack vectors, ability to use AWS Keys only from certain IPs or IP ranges, can also prevent from misuse, even if one accidentally leaks certain keys.
I don't think AWS wants to imply in any way that you need to be on a VPN to interact with AWS resources. It's not the direction they are going (Zero trust / RBA https://aws.amazon.com/security/zero-trust/).
Zero trust doesn’t mean no VPNs, it means not just VPNs. In other words you can put your control plane behind ACLs (and you should!) but you should still require strong auth from the private network.
Basically, rotate as soon as you can* and start looking through your AWS logs and setting to see if any services you don't recognize have been spun up. Is you think you have been attacked or see stuff you did not spin up, contact AWS support ASAP!
*Do NOT just revoke keys if it is in a production system where other people are working or are depending on. Talk to your team and figure out what the remediation process is internally and follow that!
If you are working by yourself and no one is relying on services this key is associated with, then yeah, just revoke and replace ASAP.
Please be aware of leaking AWS credentials in any log output, especially Jenkins console output. That's usually protected by some layer of auth, but if I get some chump's account info, and something is leaking AWS creds to Jenkins console output, it's probably gonna end up with full account take-over.
OB: There is never a reason to have access keys anywhere in your source code or a config file in your source control directory.
All of the AWS SDKs will get your credentials from
- hardcoded credentials in your code -> don’t do this
- environment variables - you get this when running locally from your control tower interface that most companies access via SSO via AD/Okta etc
- your global configuration file.
When running on AWS, your credentials are retrieved via the attached IAM Role for Lambda, ECS, EC2, etc.
I have done one third party integration where I used Azure DevOps to integrate with AWS using the AWS tools for Azure DevOps. Even then you store your credentials in an Azure vault separate from your source code.
It says in the docs this API is delayed 5 minutes though. So no idea how they did things this quickly unless you can add a webhook to every repository somehow as it is created?
This is one of those things that I was somewhat aware existed since we had seen various reports of it, but I was never fully aware of the speed (and obviously automated) of it.
Is this actually documented by aws anywhere? I assume other cloud providers must be doing something similar.
OpenAi does the same. Can this somehow used to quarantine other credentials? F.e. is it checking the key and the secret? Or is exposing the key sufficient?
It's not clear to me that we have proof that it was a malicious actor. Was it perhaps AWS automation testing out the validity of the credentials before applying the leaked credential policy?
The AWS process is hopefully "does this match a secret we know? invalidate!" and nothing more. Relying on "can this key do x?" would be a problem, because it might be able to do x in the future, and it doesn't appear they're testing all possible capabilities either.
1. https://github.com/pypi-data/pypi-aws-secrets
2. https://github.com/pypi-data/pypi-aws-secrets/blob/main/keys...