I work on secret scanning at GitHub. When token issuers use easily identifiable formats for their tokens we can easily spot them when they're accidentally committed. We can then work with the token issuer to automatically alert them of those leaks. A good example is AWS - if you commit an AWS key and secret to a public GitHub repo we will tell AWS about it and they will tell you about it (and quarantine the exposed keys) within a few seconds. We work with dozens of other token issuers too, though - some of the latest we added were Linear, PlanetScale and Ionic.
The above relies on tokens being identifiable - we can't send hundreds of partners everything that looks like 32 hex chars. In future we want to be able to do even more sophisticated things, like ask users for confirmation before the push code that contains secrets. We recently changed our own token pattern for that reason.
GitHub secret scanning program: https://docs.github.com/en/developers/overview/secret-scanni...
GitHub's updated token format: https://github.blog/2021-04-05-behind-githubs-new-authentica...
If sendgrid tokens were `secret:sendgrid.com/91on9SIkbUfSs` instead of `SG.91on9SIkbUfSs`, or Amazon keys looked like `amazon.com/JGUIERHT` instead of `AKIAJGUIERHT`, we wouldn't need a database of regexes and endpoints to report secret leaks.
See also the last time I ranted about this: https://news.ycombinator.com/item?id=26568651
One big challenge is that it's hard to get service providers to change their token formats. Very few have this at the top of their priority list - they're busy with other things. Here's an example playing out in OSS that is pretty typical: I tried to persuade the (excellent) team at Sentry to update their format, and they essentially told me "we have other priorities" https://github.com/getsentry/sentry/pull/26313. And that's a relatively simple changes, not the adoption of a whole standard.
In addition, as Thomas points out in this article, there are a lot of different token types for someone thinking about minting API tokens to choose between. They might rationally have different preferences over them. A standard that is prescriptive of format and approach is likely to struggle given that diversity.
With that said, I do see an opportunity here for a more modest standard targeted at service providers that already use JWTs or Macaroons. Generic tokens of those types are relatively easy for scanning providers to identify, and it's easy (and hopefully uncontroversial) for service providers to encode more information in them, like an "if found" link. I think a standard that defines the attribute name there, and the API for reporting / responding, would be a good start that might see adoption.
I thought so too but then https://www.rfc-editor.org/rfc/rfc8959.html came out this year.
Importantly I am not proposing a big change at all: The tokens can stay exactly the same (in the database and crypto code), you can still use UUID or Macaroons or JWT, you only change the frontend to add this prefix. Apologies if this wasn't clear in the the two examples I posted without explanations. The benefits would also be a bit higher than the PR you reference, which seems to help with scanning on GitHub (you mention that it would already work without the change).
As you note in your PR, many tokens are already identifiable, so standardizing a way to put the reporting domain in there shouldn't reduce security (by obscurity).
I mean your core idea is decent but that's just really funny.
There's some amount of practicality being lost if your secrets start growing massively. There's also potentially restrictions to what you can put in them, and a prefix with an underscore or colon might be easier than something that has slashes in it.
Your idea is probably living as a queriable dns record on the domain in question. Or a standard subdomain, or even a .well-known path.
Could require a POST or maybe even HTTP DELETE to guard against woopsies. The latter has semantic niceness about it.
It's also up to each provider what actually happens when the "revoke" action is triggered. Maybe they just warn you immediately, which is still better than nothing.
If it is a URL, it opens in a web browser, and now you have problems.
My professional email also mangles URLs, turning them into urldefense.proofpoint.com/... Such solutions are sure to interfere with tokens looking like URLs.
Also, you couldn’t send such a secret to, say, Gmail.
But yeah, auto-link followers will invalidate them immediately. There's a case to be made for that being a good thing, but don't want to get into that.
And it's a noodly and somewhat incoherent notion of "safety" I'm using here, because of course, random tokens are unconstrained bearer tokens --- authenticated requests, CATs, Macaroons, and Biscuits all address that weakness. I'm biased by my concern over cryptographic implementation mistakes.
It's a neat property of Macaroons (and maybe Biscuits) that you can come up with sane configurations where checking a Macaroon into source code can be, if not totally safe, at least not a major incident. I wish I'd thought of that, too, since I think "checking the token into source control" is a more vivid example than "emailing tokens" or "passing them around".
Nobody's asking for royalties, so "Shout out to @SomeoneOnTwitter for reminding me" is enough.
If you all have thought about it, do you imagine you'd only warn in the presence of some generic token identifier, like `secret-token` a la https://datatracker.ietf.org/doc/html/rfc8959 ? Or, would you be able to warn on everything that matches the regular expressions your partners give you to identify their API tokens?
One thing that's worth remembering about randomly generated tokens is that it's important to always use safe comparison methods when comparing them to the stored one - otherwise you could be vulnerable to timing attacks.
In Python you can use secrets.compare_digest(a, b) for this: https://docs.python.org/3/library/secrets.html#secrets.compa...
The trick I've used for this is to have tokens that look like this:
Then you have database entries like this:
id = 234523
token = 7a561002f780e23853bdfbd89ae79bf2
When a token comes in you use the bit before the : to look up the database record by its indexed ID, then perform a safe comparison between the rest of the token and the value you retrieved from the database.
That way you’re hashing the user input, which sanitizes it. It also protects against timing attacks.*
And most importantly it protects against credentials leaking. Even insiders who can read a token from the database can’t use it to authenticate, because the app logic expects a pre-image of that value.
*HMAC resists timing attacks because the attacker can no longer control the bit pattern on either side of the equality check.
Semantically these are very different objects/concepts
- tightly couple authentication (I am kortex) and authorization (I can do everything logged-in kortex can do)
- not meant to be rotated unless breached (forced password rotation mandates weaken passwords, imho)
- are passwords meant for human transport, if even briefly
- hard to revoke (once expired, still valid until user rotates)
- usually insufficient entropy, dependent on user habits
- one flavor: a secret string
- decouple authentication and authorization
- enable automatic rotation (session tokens vs auth tokens)
- can have metadata
- as much entropy as you want
- many flavors
If there were one login system/site in the whole world, you'd still want to hash/encrypt them.
I mean hashing is one of the easiest things to do to add a layer of security, so even if the db is encrypted, why not? Maybe I'm missing your point.
Totally agree with your main point of "tokens, like passwords, should be kept discrete."
Passwords are secrets.
Ed25519 private keys are secrets.
We hash passwords.
Thus we must hash tokens.
Should we therefore hash our Ed25519 private keys?
No, but I'm of the opinion that you should encrypt private keys before storing them in a database.
The only thing the post explains that there is a difference between checking a secret with the database vs message validation, also based on a secret. But for both, if you have the secret, you're doomed.
Expiration / immediate revocation when doing message validation is more difficult than having a distributed cache, which supports key eviction/expiry/removal.
Also, an asymmetric solution has similar problems: if you get the private key from the client, you're doomed. So it does protect the server part, but if people have access there, you're even more doomed. But at least it'll be easier to identify who had a security leak./
Passwords are usually one-per-resource, e.g. a user has a single password. If that password is compromised you can reset it, but all the consumers that used it will need the new credential.
Whereas tokens are typically one-per-consumer, so if one is compromised you can revoke/expire just that token, without affecting other consumers.
Same with expiry times -- you can set that on a token without it expiring all access to that resource.
Not sure if that's what the author had in mind, but it's a difference in how these things are often used (even if fundamentally they are otherwise very similar).
Frustratingly, different tokens have different levels of exposure to all of these problems. A well-confined Macaroon has essentially none of the secret storage problems of a password. An API request authenticating key has none of the bearer token problems. No API token has the human-recall, reuse, and brute-force problems.
People hyperfixate on the secret storage problem, I think because they feel like they can get their heads around it. As you can probably tell, I'm loath to relitigate the debate about whether we should store passwords with secret salts. Even when the discussion is legitimately focussed on that problem, I find discussions about it go to weird, incoherent places that aren't really informed by real threat models, and are really hair-splitting arguments about the security of environment variables vs. the security of filesystems, just dressed up as top-3 security considerations.
Even for trivial basic-auth API keys, the secret storage problem is not the same as that of a password. Part of the problem with passwords is that they effectively cannot be revoked; "revoked" passwords creep back into systems, and, worse, infect other systems. That's not how API keys work; the threat model is different and so are the countermeasures that are profitable to deploy for them.
Another thing that I feel happens in discussions like this is that there's no notion of cost-benefit. There is option A, and option B, and option B is on some axis superior to option A, even if that benefit is marginal. Since there's no engineering cost to consider, there's no meaningful discussion. But in reality, there's always a cost: deploying a countermeasure at a minimum incurs the opportunity cost of not deploying some other countermeasure that could have been built with the same (finite) engineering resources. Since these discussions always seem to get mired in secret-salt double-hashing hmac(hmac(root-secret, measurement_1 || measurement_2), secret) stuff, I hope you can see the concern: the discussion essentially advocates for ever-more-marginal wins that "fit" the message board thread, rather than seriously considering the real problem.
But of course, the other problem is that the arc of the thread bends towards HMAC'ing your Biscuit token. "Tokens are just passwords", after all.
At any rate: this thread says, "passwords are secrets, tokens are secrets, ergo passwords are tokens". Signing keys are also secrets. Nobody HMACs them. Something is wrong with the syllogism. I'm not that interested in picking apart what.
It can, but the practicality of exploiting this timing leak isn't at all a settled issue.
Sure, better be safe and use a safe comparison method and that's simple enough, but still, how realistic is such an attack over the public internet?
I'm aware of "practical" results:
But the experiment setup was on a LAN, with AFAICT a single server, no other traffic to that server, and other such conditions.
Of course, I'm not in the field and am happy to learn more. However, my intuition also tells me we should have seen more realistic experiments and results over the years to get a clear picture of the extent at which such an attack is actually feasible and when it seizes to be.
I’d estimate a naive comparison loop to be around 20-30 cycles, for compare and control? And let’s wrap it in 250 cycles more because someone decided to use Python. 300 cycles, then, are about 100ns (tenth of a microsecond).
Not saying it can’t be done with a timing attack over the internet but you’d need a huge sample size.
The key idea there is to build your own custom proxy for the GitHub API, then issue tokens for it which are actually encrypted bundles of the full-permission API token plus a set of rules about what the proxy should allow it to do - only allow a GET to paths that match "/gists/.*" for example.
It's somewhat similar to Biscuits storing a Datalog program "to evaluate whether a token allows an operation."
One thing that I wish was addressed more was language/library support. It gets casual references a couple of times, but for an average developer (as I consider myself) a set of robust, supported open source libraries that help me use a token is so important (not write an implementation, but use in a project that just wants to use the tokens safely).
I don't have anything but anecdata, but I feel like most software is going to be in the 'just want to use it' category, rather than the 'need to implement it'.
This is where the standards like OAuth and JWT win right now. That doesn't mean they always will, but in my experience, that's the current situation.
I turned the post into a handy chart. Let me know if you want a poster of it.
I would imagine that if the entire policy is encoded in the biscuit, it is very easy to evaluate without needing to call external services. And it can be extended like macaroons without needing a central authority, assuming I groked your blog post correctly. The only issue I can see is revocation.
To top it all off, my dumb meta tags didn't even work; they needed to be in the <head> of the page, and I'll be damned if I'm going to figure out how to do that in our static site generator configuration.
I just wanted the Carl Yastrzemski with the big sideburns.
I hope you have learned your lesson in adding pictures to a long blog post :)
Jokes aside - I did enjoy reading through it and thank you for educating me on macaroons, CATs, and biscuits.
* The v1 local tokens used a novel nonce construction (I'm doing this from memory) and CFRG's take was "standard constructions or GTFO".
* The HMAC/RSA thing, which PASETO noted and documented but didn't fix.
* The fact that PASETO is basically a restricted profile of JWTs, begging the question of why it didn't just specify a restricted JWT profile.
I don't think this feedback was especially valuable.
I think there are subjects on which CFRG discussions shed a fair bit of light, when they're high-profile enough to drag academic cryptographers into the fray. But the other thing that happens in CFRG is that bad stuff (like the Dragonfly PAKE) gets blessed (because there's no outcome besides "this is fine" and "this is trivially broken" --- and even "this is trivially broken" can get laundered back to "this is fine" if the threads get tedious enough).
In the worst case, you get people proposing bikeshed changes to constructions that are already de facto standards, which (if I remember right) happened with Curve25519, though thankfully not successfully.
I think the whole practice of standards based cryptography is mostly discredited at this point. Signal Protocol isn't a standard despite being the reference model for most secure messaging systems. WireGuard isn't a standard either. The original ethos of the IETF was that things get popular, and then they get standardized. IETF does a lot of stuff de novo now, which is how we end up with stuff like Heartbleed and JWT.
What is the difference between red and black? Thick circle? Half circle? No circle?
Original harvey balls might be better here: https://en.wikipedia.org/wiki/Harvey_balls
PS - thanks for putting together.
I still don't know where the black with white dot goes. I guess it's one better than the empty circle in the middle.
If the idea was "LOL, joke scorecard" then, I guess joke is on me, otherwise, if you revise this I'd recommend to decide what one thing the scorecard is supposed to communicate, then visualise only that and accept that people will need to read the rest of the text to know more.
"Consumer Reports graphs formerly used a modified form of Harvey balls for qualitative comparison. The round ideograms were arranged from best to worst. On the left of the diagram, the red circle indicated the highest rating, the half red and white circle was the second highest rating, the white circle was neutral, the half black circle was the second-lowest rating, and the entirely black circle was the lowest rating possible"
For example, we had a pentest done on a website and the pentester got all excited because they found some AWS tokens.
Trouble is, they would be worthless to anyone external because we were making good use of AWS IAM to lock them down with ACLs, Roles etc.
So it was effectively a non-event.
What happened to the old concept of layered security ? Why should discovery or leakage of an API key automatically give the attacker all the keys to the castle ?
In my ideal world, all cloud and API service operators would have the equivalent to AWS IAM and preferably would enforce its usage (i.e. "here's your API key, but it won't work until you set some layered security")
But no amount of layering makes the problem go away. Sometimes god-like keys are unavoidable. Those needed by Terraform etc are an example that comes to mind.
In practice, I'm talking about (1) "SSH public-key authentication to GitHub" (or to an EC2 or GCE instance), or (2) using FIDO2/WebAuthn (or TLS client certs...) to authenticate an HTTP connection or browsing session.
One major pro is that this is a lot more widely implemented and standardized than "authenticated requests" -- there are lots of reasonable implementations of SSH pubkey authentication or FIDO2/WebAuthn, compared with trying to sign an individual HTTP request. And that maturity brings robustness and security, e.g., I like that I can have my SSH private key (or my FIDO2 private key) on a tamper-resistant hardware device and not worry about some rogue software stealing my private key, whereas trying to find a hardware device to hold my AWS credentials and do the AWSv4 signature is a different story. Any bearer-token-based scheme seems... a lot scarier in that sense.
I'm less clear on the practical cons (assuming "authenticated requests" is not available). With SSH keys or TLS client certs, I could imagine it's annoying because only the terminus of the secure connection can verify that the client is who it claims to be, and maybe it's annoying to have your SSH/TLS termination component directly talk to a user database to authenticate individual clients and then attest that it did so to the rest of the backend. But... somehow GitHub/GitLab/Bitbucket all manage to do this at scale, and with WebAuthn/FIDO2, any component can be the one to make the challenge. (I suppose only recently can you make a "user presence not required" challenge, so maybe that's why nobody uses this in the non-presence setting...?)
I am generally a fan of the mTLS approach for very simple topologies, like "central Consul cluster nobody can talk to without the right client certificate". I am much less a fan of it for the general inter-service authentication problem.
The authn mechanism we use is closer to Keybase's NIST (non-interactive session tokens) that are a mix of AWS-style Bearer Tokens and the usual Random Tokens. Of course, the problems around "logistics" (public-key cryptography) are a real nightmare as the post points out.
We exchange these tokens between devices (if needed) over password-authenticated channels (using CPACE ).
 As examples, see what goes on when a Keybase user associates a new device: https://book.keybase.io/docs/crypto/key-exchange or when SQRL user revokes compromised keys: https://www.grc.com/sqrl/idlock.htm
Then why did you decide to go with macaroons rather than random tokens? Do you know for sure that fly.io needs that statelessness?
Logs are a good example. We expose logs as a NATS service, people auth and subscribe to their logs. It's useful to terminate auth right there both because it's faster and because it continues to work when the internet burps.
They show how far you can drive Protocol Buffer Tokens.
AuthN is the next big challenge especially in a multi-tenant/multi-enterprise SaaS type platform
Why if you don't mind? I'm actually looking at a major proprietary protocol used by hundreds of millions of users that use Kerberos as the fundamental cryptographic attestation of identity and roles. Like clients all connect to a VPN who's bridge uses the Kerberos ticket to whitelist backend services accessible from that connection. They're basically being used the same as if an API gateway understood oath claims and could stop whole classes of client calls at the perimeter.
Disclosure: I'm the author
The reason you hash and store passwords with a salt is that there's a very good chance that a user will have used that same password somewhere else. As such, it's important that you make it as hard as possible for you - or a co-worker - or someone who gets hold of a dump of your database - to gain access to the original password.
Random tokens were generated by you. They are only valid against your own service. If someone bad gets hold of them, all they can do is make API calls against your service until that token is revoked. So salting and hashing them doesn't win you much.
That said, if a lot of people at your company have unfettered access to run queries against your database it may be a good idea to store your random tokens in a way that prevents people from copying them out and abusing them. But if you're storing any private data at all and you have that kind of a culture you have bigger problems you need to solve.
Different revocation techniques like periodically distributing a revocation list to your auth services can resolve that part of the issue.
Just pick a crypto scheme and the JWT is just an encoding that makes it easier to use.
(It's just a more convenient way of rolling your own scheme)
That said: random tokens have a lot going for them :)
> Just pick a crypto scheme and the JWT is just an encoding that makes it easier to use.
That's not what JWT is, but I can understand why someone would be misled into believing that.
JWT isn't just an encoding format, it also includes a crypto algorithm negotiation protocol that lets the attacker choose the algorithm. Even if you strictly allow-list which algorithm you want to support, you can accidentally bypass this control in many libraries if you support the `kid` (key ID) header. 
It also allows attackers to completely strip the security.  
Put shortly, JWT is a gun aimed directly at your foot. That's why there's so much hate for JWTs.
This is the norm.
As for the key ID attack, this sounds like it's just a trick to know where the private key is located? It shouldn't be publicly accessible.
Every JWT proponent says that, but it's a misuse that shows up in multiple libraries, in multiple languages, and isn't explicitly called out in the JWT Best Practices RFC at all.
I'm going to blame the standard for being error-prone.
There's nothing in any JWT RFC, to date, that calls out the need for cryptographic keys to be the raw key material in addition to its parameter choices, rather than just the raw key material. That's a fault of the standard.
That's not a single library's fault. That's the standard's fault.
PASETO has this to say: https://github.com/paseto-standard/paseto-spec/blob/master/d...
> As for the key ID attack, this sounds like it's just a trick to know where the private key is located? It shouldn't be publicly accessible.
This doesn't involve private keys at all. Look at the proof of concept code. https://github.com/firebase/php-jwt/files/6966712/php-jwt-po...
"my-super-cool-key-id": "some hs256 secret key string goes here",
"another-key-id": "-----BEGIN RSA PUBLIC KEY-----\n ... snip ...",
"yet-another-key-id": "----BEGIN EC PUBLIC KEY-----\n... snip ..."
What stops you from swapping out the kid in a JWT's header and getting the underlying library to use the wrong key type for the endpoint that accepts HS256?
- "kid": "my-super-cool-key-id",
+ "kid": "yet-another-key-id",
Strictly listing the keys does not, at all, hard-code the algorithm those keys are used with in every possible programming language and runtime.
Some languages (Java) accidentally prevent this through type safety in the low-level crypto APIs. Others accidentally prevent this by not supporting the kid header.
I have yet to see a JWT library that deliberately prevents this misuse potential. Is that the library's fault? Or their defaults' fault?
The fix, by the way, requires doing this:
"key": "some hs256 secret key string goes here"
"key": "-----BEGIN RSA PUBLIC KEY-----\n ... snip ..."
"key": "----BEGIN EC PUBLIC KEY-----\n... snip ..."
If you control both sides, then you can ignore this part or do it out-of-band.
Though, if you control both sides, then you can use literally anything else too.
https://groups.google.com/g/django-developers/c/6oS9R2GwO4k/... - on the Django mailing list
https://www.zofrex.com/blog/2020/10/20/alg-none-jwt-nhs-cont... - where the punchline is "Writing the code to sign data with a private key and verify it with a public key would have been easier to get correct than correctly invoking the JWT library. In fact, the iOS app (which gets this right) doesn’t use a JWT library at all, but manages to verify using a public key in fewer lines of code than the Android app takes to incorrectly use a JWT library!"