I think it would be interesting to do an art project with this data - some of these passwords are funny and/or revealing. Some examples:
pooplasagna - 3 times
eggsarebad - 3 times
eggsaregood - 25 times
myhusbandcheats - 4 times
icheatonmywife - 1 time
ihatemyneighbors - 2 times
iamanalcoholic - 6 times
1yearsober - 31 times
imissmykids - 51 times
imissmyparents - 6 times
893 people like turtles, 170 love turtles, 155 love everyone (363 people hate everyone though and 428 hate us all).
4301 people love their dog, 3 fuck their dog, 3 killed their dog (only one person killed their cat)
110 people are killers, 4 kill for money, 1 is a murderer. 68 want to kill, 24 kill for fun :-/
2781 love their wife, 552 love their husband. 68 people hate their wife, 38 hate their husband. 3823 people love their son, 17 hate their son. 212 love their daughter, 3 hate their daughter. 11 siblings are having sex with one another. 3 have sex with their dad, 5 have sex with their mom.
6 people save the cheerleader, 11 people save the world. 1 person is the president. 4 people are obama, only one person is trump. The password donaldtrump is almost 5 times more popular than barrackobama.
339 people love sun, 559 love rain.
114 people are healthy, 216 are sick. 108 people are old, 3 people are too old. 15 people sleep well. 80 people run fast, 6 people run faster. 1 person is the fastest (obviously).
All this is quite facinating and fun .. and partly disquieting.
jesus - 123279 times, lordjesus - 8187, satan - 11662 times, iamgod - 8834, iamjesus - 345 times, jesussucks - 100 times
Some actual expert was on TV talking about space, a European space probe I think, wearing his Hawaiian shirt with ludicrous SciFi babes on it, and a certain type of person kicked off declaring that somehow SciFi babe Hawaiian shirt is misogyny incarnate or something.
The main outcome for me was that now two different friends own sofa cushions with the identical fabric pattern, they're a bit... garish? Both these friends are women, one of them is a bona fide scientist with a PhD and everything, so evidently the "No to SciFi babes on Hawaiian shirts" misogyny claims were less "mainstream feminism" and more "I'm just looking for reasons to be angry", but whatever, that's culture for you.
this one is quite amazing, I expected it to be one of the weakest around today, I certainly would include it in one of my first attempts if I tried to brute force any password.
There’s no guarantee those accounts hold any value, though. A throwaway account is the perfect chance for such a password — you won’t forget it and won’t care if you lose it.
Also I wonder if Randall Monroe knows this somewhat ironic fact? It would make a nice addendum to the alt text on https://xkcd.com/936/ (which BTW is the origin of correcthorsebatterystaple for those wondering).
It's indeed weird that "00000" would be the hash prefix with the highest number of entries. I think it must be a hidden variable. Like some sources put an all-zeroed-out hash in the database for testing or in case of a registration error or for deleted users, and these show up here.
Great thought, but it doesn't seem to be the case - as the number of unique suffixes is the large number here -- in fact, none of the values in the range are simply all zeroes.
I wonder if the hidden variable is something to do with how the passwords are leaked. First, let's suppose that a very commonly used broken password hash is plain SHA-1 (I think that's a valid assumption-- unfortunately!). Then, let's figure that amongst the many data dumps / extracts done by hackers, some of them are only able to extract part of the database, or save part of the database, or whatever....and they are fetched / saved / uploaded in lexical order?
Can't think of anything else.
EDIT: Ooops. The other thing is, that these actually are sha-1 hashes of real plaintext passwords. So it's definitely not a test-row in that sense.
Maybe crypto people who have brute-forced up some typeable passwords that hash to low numbers on the first SHA-1 pass, for a fun-and-games equivalent to a Proof of Work? (It'd only show up in actual DB dumps for backends that use "SHA-1 with no salting" for password hashing, which might also serve as a useful canary value.)
Great idea! I ran a quick hashcat against the range00000 list on my laptop. In 1 minute I cracked 79 of them, and not too many of them look very odd - that is, they look sorta like normal cracked passwords.
I'm asking my friend to run a more thorough crack on his dedicated GPU, especially for hash value 000DD7F2A1C68A35673713783CA390C9E93:630
which does stick out to me!
Note that in the description below, I refer to any keyed involution as a block cipher. One may make a semantic distinction, but any keyed involution could be used as a block cipher (though, of course, most involutions would contain trivial cryptographic weaknesses).
SHA-1 is based around a 160-bit unbalanced Feistel block cipher. The input in broken into blocks, where the final block contains padding and a final count of the amount of data processed. A copy of the 160-bit state is made, the 160 bit state is encrypted using a block of the input as a key, and the initial copy is added back (without carries between 32-bit words) to the original copy. This is repeated for each input block in turn. This is called a Davie-Meyer construction for making a hash function out of a block cipher.
For any Davies-Meyer hash function, the block cipher is invertible and therefore unbiased. The addition is invertible and unbiased. Any bias would therefore have to come from non-zero correlation between addition and encryption. For any moderately complex block cipher, these correlations would be very complex. Real world design of Davies-Meyer hash functions focuses on absolutely minimizing any patterns present, and cryptanalysis focuses on characterizing and approximating any and all minute patterns that escape the design process.
There are some patterns (weaknesses) in SHA-1, but all known weaknesses are way more complex (and minuscule) than could explain the sort of bias seen in this data set, so the bias must be coming from a higher-level source than SHA-1 itself.
On a side note, the addition in Davies-Meyer is to intentionally make the round function non-invertible. If the round function were invertible, there's a trivial birthday attack on the intermediate state between rounds that square roots the strength of the hash function. MD4, MD5, SHA-224, SHA-256, SHA-384, and SHA-512 are all Davies-Meyer constructions using unbalanced Feistel ciphers. RIPEMD-160 is a parallel application of two Davie-Meyers hashes with different initial values, followed by XORing the two outputs to obtain the final output. SHA-3 is the most notable cryptographic hash function that's not a Davie-Meyer construction.
In case you're wondering, one could make a Davies-Meyer hash function using AES. The designers of AES took AES, doubled the word size, doubled the number of words, and fixed a deficiency discovered in the nonlinear byte substitution. The resulting hash function is called Whirlpool, and the underlying block cipher is called Anubis. I'm not aware of any use of Anubis outside of Whirlpool.
The Salsa/ChaCha families of stream ciphers and the Blake family of hash functions are all very similar to each other. They all use a very similar family of (unnamed) block ciphers internally that are twice the size of the desired output. They achieve non-invetibility by breaking the block cipher output into two halves and XORing the two halves together.
Before MD5 was broken, I did read briefly about an attempt (not by Ron Rivest) to use the inner block cipher from MD5 for encryption, but the performance wasn't competitive. Now we've characterized the hidden patterns in the block cipher well enough to break it relatively easily. I forget the name the authors retroactively gave to Rivest's inner block cipher from MD5.
Thanks for the correction. Note that a couple of times I spelled Davies as Davie. Also note that after editing, my description of the Davies-Meyer addition step got mangled. The original copy is added to the result of the encryption.
I'm a bit confused - why not distribute a serialized Bloom filter representing these passwords? That would seem to enable a compact representation (low Azure bill) and client-side querying (maximally preserving privacy).
You could do a Bloom filter on each bucket, each of which has about 500 items. This would reduce the size of the response from about 16k to < 1k. But it would be a lot harder to use since all clients would have to use the Bloom filter code correctly.
A Bloom filter with >500M items, even when allowing for a comparatively high rate of false positives such as 1 in 100, is still in the hundreds of MBs, which would not be that much more accessible than the actual dump files.
The compressed archive here is over 8 GB. An uncompressed 2 GB Bloom filter with 24 hash functions and half a billion entries has a false positive rate of less than 1 in 14 million.
75% space savings, with no decompression necessary for use, and a 1 in 14 million false positive rate is nothing to sneeze at.
Counting bloom filters are only marginally more difficult to implement. To increment a key, find the minimum value stored in all of the slots for the key, and then increment all of the stored values for that key that are equal to the minimum value. To read, return the minimum value for all of the values stored in slots for the key.
For these purposes, however, you probably instead want to store just separate Bloom filters for counts above different thresholds, since the common use case would be accept/reject decisions based upon a single threshold.
If it doesn't return anything than your password isn't in the list. You should probably start your line with a space so that it isn't recorded in your bash_history.
If someone else can make it better or shorter, be my guest.
Does anyone else not get results when searching for 'asdf' and 'hunter2', and 'lauragpe'(which appears in the article) not return results using the shell script provided?
edit: Ok, so my `openssl sha1` (version 1.0.x) outputs '(stdin) <hash>', whereas the script expects just <hash>. add ' | cut -f2 -d" "' after the 'openssl sha1' call to fix this if you have the same problem.
You don't submit to the API either a full password or a full hash (which, since Troy produced the hashes is identical). A hash is pretty much perfect for K-anonymity because if you use a prefix like this then it's extremely likely your data will be spread across the buckets, so no 5-character prefix is close to uniquely identifying a password.
As stated in the post, it's a simple solution to help with anonymity.
"The password has been hashed client side and just the first 5 characters passed to the API As mentioned earlier, there are 475 hashes beginning with "21BD1", but only 1 which matches the remainder of the hash for "P@ssw0rd" and that record indicates that the password has previously been seen 47,205 times."
But's it's not always that hash. The password you're checking may not be on the list. This is just a quick check to see if the password in question is on the list, in which case it may be a poor choice depending on how often it's seen.
For example, say I want to check "gSAey27tgGsaEG". That hashes to c2e5dfb023cd42df94751581cba33b24bc011027. https://api.pwnedpasswords.com/range/c2e5d has no entry for fb023cd42df94751581cba33b24bc011027, so it's not even in the list of passwords.
Put another way, it averages a few hundred hashes per prefix based on the total password list size (~500M), but there's 2^136 possible has suffixes per prefix. There's no point in guessing that.
Forgive my ignorance but why is submitting a hash a problem? Because Troy knows which passwords have been checked? Why should I care about that? I get that it’s like submitting your password in the clear if it’s in the DB, but in that case surely you have bigger problems.
One way that sites can use this service is to check whether a password has been leaked when users sign up. By handing over the SHA-1 hash of the password you're effectively trusting this service (and anyone who might have compromised it) with all your user's clear text passwords. Connecting the right password with the right user can be trivial in some circumstances, say because a site has a publicly visible sign-up date on profiles, or even if it just hands out sequential IDs to users.
You cannot access their support in any way without logging in. Trying to contact them via their contact/sales page won't work. They won't respond.
This means that if you lose your phone (2FA) and can't log in, you're royally screwed and will have to go to your registrar to recover access to your domains/DNS.
All of that is a good thing in my book. I've been the victim of the "customer service backdoor" on Amazon multiple times. It's ridiculous that someone can just about credentialize as you without even having to log in. They made off with whatever sensitive data the customer service rep had in front of them just from chatting to someone on that anonymous support chat widget.
Meanwhile, all you have to do is backup your 2FA secrets. Why not make it a part of your regular computer backup routine?
You should never use only 2FA for something you don't want to be locked out of. You need a 3rd authentication method to replace the 2nd when you lose it, such as backup codes, that as well as a 4th one to recover a lost password.
> You should never use only 2FA for something you don't want to be locked out of.
Tell that to... everyone.
> You need a 3rd authentication method to replace the 2nd when you lose it, such as backup codes, that as well as a 4th one to recover a lost password.
That's on Cloudflare. If they don't offer backup codes, what can an end user do about that?
Manually record the seed key when you set up 2FA (usually this is contained in a QR code). Keep it somewhere safe and offline. It can be used to recreate your 2FA setup.
That's not an inherent problem, that's poor implementation.
Procedures like this could work:
Person contacts support requesting a bypass of the 2FA due to whatever reason.
1. Cloudflare sends email to persons account notifying of the request.
2. Person is required to upload photographic proof of two govt-issued id's.
3. Cloudflare calls person (phone number on file from 2FA or account setup).
4. 30 day delay initiated.
5. 30 days layer, Cloudflare emails and calls person to confirm they requested 2FA bypass.
6. Access is granted.
With procedures like this, it's no longer about convincing a support rep.
That moment when you test an old, but still highly valued and securely used, password that you think isn't super obscure but not likely to be used much and see a 4000+ count...
And that's exactly why using this entire corpus - or even more than the first few tens of thousands - as a blacklist would be an extremely user-hostile choice [1].
Password psychology is remarkably consistent across a given demographic, because most people start by modifying of one or more base tokens that are already stored in memory because of their personal significance.
So unless the implementation gives specific, real-time UX feedback to teach users how to pick a password that is very unlikely to be in this (and future, ever-growing) versions of this corpus, using a large blacklist is "gotcha infosec". It creates UX where the user cannot possibly come up with a "good" password using most of their previous strategies. It's the worst kind of "gotcha infosec".
Same experience here, except mine was current until shortly after I checked it.
The weird part is that I only used it on internal systems at work. With an overly paranoid security department. Either they’re paranoid in the wrong ways, or I have an evil twin somewhere.
Am I the only one here who thinks typing your password to a stranger's website is a risk? How do you know he does not log it? how do you know he was not hacked and someone is not logging all passwords that are not on the list YET.
If you don't trust troy hunt / haveibeenpwned.com you can always download the data and analyze your password yourself. But if this is the case you should not trust any website with your password anwhere ever, and should not create accounts anywhere. Troy Hunt has shown himself a responsible security professional, and I trust him more to create a secure password query than some other security organizations.
Yes, with unique passwords for each services, you narrow the attack surface to compromise other accounts. But you still have to trust the operator to store and process this one - unique - password on this one service/website. It does not make any difference for the argument, if one or many accounts are potentially compromised. And you have to trust your password-manager software, since it is next to impossible to remember all the different passwords for all the different services you use.
This is absurd and impossible to remember, you should instead have at least 3 levels of password strenght, one high strenght for base services that are used to retrieve other accounts like facebook and e-mail, other for important services, and another for crap.
You're not expected to remember them all, you're expected to either wrote them down or use a password manager. That way you only really need to remember one very strong password.
Except that my 3 level system failed ages ago.
Originally I had one, then with more sites coming - several with bogus or recless implementation - it was extended to the aforementioned 3 tier one just to get f*cked up by 'knowing it better' god complex but stupid enforcers requesting or forbidding (!! how stupid is that!) characters. Not to mention leaks forcing me to introduce new ones, having eventually 5 layers with variations on each level because of the highly arbitrary rules of enforcers blocking my well thought of secure passwords.
All led to the situation that I have an encoded file on my computer with passwords (most just referrals/reminders/instructions not the actual password characters).
How stupid is that!
Writing down passwords!
Even into secured files, still, increased level of risk. A method with doubtful protection when someone is targeted for his/her secrets personally.
Stupid but that is reality.
Made necessary by recless developers.
The whole password infrastructure is dead as means of protection. It does not work against serious attackers, only agains random wanderers. And more and more against rightful users!
And the most was done to ruin it by those enforced the users to solve the problem on the user side that in fact lying in the system side.
Passwords will not fullfill their task if:
- allowing parties without permission to enter
- locking out righful parties
Very strict enforcers corrupt the system through the second point. Narrowmindedly focusing on not letting in unwanted elements cause the whole system to case working as intended, locking out and disallowing users to use it, defying the very purpose of existence.
Encouraging users not to use passwords that ever used by someone is just an extremely very radical level of enforcing and again trying to make users fix the inadequacy of the system developers.....
This is not solving systematic problems just conserving a bad habit plus making a bad situation even worse.
No, you shouldn't trust this site with your password, like you shouldn't trust any site with your password.
Choose a new password for every site folks, and if you want to use a site like this, make sure the original places you used the password have been updated to a new one.
The article explains in the "Cloudflare, Privacy and k-Anonymity" section how you can use k-Anonymity to query the API with (at least some) privacy, by just submit the first 5 characters of your password's hash.
An old password (12 char numbers and letters) I've since stopped using (but used to use everywhere) appears as pwned in this list (3 times!). I'd love to know who exposed it. Any chance I can find out?
As a policy Troy Hunt won't reveal which breach he found your data in. I considered setting up a series of 'canary' emails so that I could track who's selling what but ... well never got round to it.
You used to be able to adjust your email address to check. For example if you email was bill@gmail.com, you could sign up for HN with bill+hackernews@gmail.com. Gmail ignores the part after the + sign. Therefore if you noticed emails coming to that address, you would know that HN sold their list. However, I've found that most forms reject that as a non-valid email address now.
People have had their Google accounts suspended (especially if it's associated to say, a YouTube or AdWords account), and now you have no access to your address anymore, and Googles free tier support won't help you.
With domains there's an ICANN process to get your domain back if it gets hijacked.
(a) Using your own domain doesn't require using your own mail server, you can point your mx server at Google apps if you're comfortable or your registrar probably supports mail forward if you don't want to pay for G Suite.
(B) it means you can keep your email address if you ever leave Gmail.
Don't any half decent sites strip that out anyways? Some ecommerce sites have actually failed to accept that string, inadvertently thinking it's invalid. The rest, or any marketing CMS, would simply remove it.
Many sites reject valid email addresses. One character it is common for forms to reject is a "+" in the left hand side of an email address. The email RFCs allow this character, so denying it is bogus. Nevertheless, they do.
A combination of public leaks and private contributions. He does a pretty good job of vetting them and determining what era they're from (by polling some users to see when that password was used/changed on that service, etc.)
The UX of a blacklist with a half billion entries would be so crippling that it would cause a user revolt.
Most people's password-selection strategies are similar enough to other people's (like kbenson's 4000+ hit) that they could spend hours trying to come up with a password that has never been leaked before.
I tried to encourage Troy to suggest to implementors that blacklisting all passwords was a Bad Idea. Instead, he doubled down:
Please don't use the entire list for blacklisting unless you actively also guide the user in how to generate a random passphrase (if a human must remember it) or a random password (if it will be stored in a password manager).
Instead:
1. Use a high-level password-strength assessment widget like zxcvbn:
2. Configure a blacklist with, say, 10K or 20K of the most common passwords.
3. Hash passwords with bcrypt cost 12 (adjusted to your platform's hashrate capabilities), scrypt, or the appropriate method from the Argon2 family.
But for all that is holy, please don't use Troy's entire corpus - or even the first million - as a blacklist. To quote the old NANOG saw, I encourage all of my competitors to use it. ;)
Edit: And since Troy's API at this writing does not support only querying the top X passwords, there's no way to use the API while avoiding the UX nightmare. So if you want to use this data in a professional manner, but don't want to download the entire corpus, here are the first 20K from his list (of which I've only personally cracked 19965 so far, interestingly; gist will be updated once I get all 20K):
Edit 2: Preliminary results indicate that this data may be dirty. The 273rd most common password, according to Troy, is '$HEX'. This is almost certainly an import/conversion artifact, since the '$HEX' prefix is how most cracking suites escape non-ASCII or passwords that contain colons. I expect that there will be more artifacts. Use the data with caution.
> I tried to encourage Troy to suggest to implementors that blacklisting all passwords was a Bad Idea. Instead, he doubled down
> Please don't use the entire list for blacklisting unless you actively also guide the user in how to generate a random passphrase (if a human must remember it) or a random password (if it will be stored in a password manager).
I think he did the right thing, and think you are correct as well. I think we have the best of both worlds with this, in that it includes the count, so API users can determine what the correct cut-off is for them. Once you get into the thousands (or maybe less) might be a good indicator that your password is not only relatively common, but also likely to be on (and maybe even fairly high on) many dictionary lists. More secure services that cater to more technically savvy users (or security conscious companies) may decide to blacklist any password on the list period, and that may be okay because those sites either trust their users to deal with it or can dictate conditions for a captive audience.
As a corpus to download for password research, this is indeed useful. But for providing a blacklist -- his stated purpose - it is not.
The crucial tell: his API does not allow the implementor to specify a frequency threshold (by top X in the list, or by Y number of unique uses of the password or higher).
By both API and explicit language in the announcement, he is promulgating the idea that checking the entire blacklist is useful, and "the larger the blacklist, the better." This is exactly what I'm arguing against.
> his API does not allow the implementor to specify a frequency threshold
Yes it does. The output contains the number of matching passwords. It's just client side instead of server side. The reason for not doing so on the server is also obvious taking into account his explanation of cost and caching, which informed much of the API design itself.
> By both API and explicit language in the announcement, he is promulgating the idea that checking the entire blacklist is useful
Because the entire blacklist is useful. He's given all the relevant information to the client to do with as they may. It's up to them to choose how to utilize it. I'm not sure why you seem to think some narrower use case is necessarily better, using end use cases as arguments, given it's an API and needs a client implementation fore being usable anyway.
It's a fair point that raw password count is available.
But that value is an absolute number, without any in-API context of the total size of the corpus. This makes expressing relative rarity only possible by hard-coding the total size of the corpus into a calculation.
Put another way: the 20,000th position has a frequency value of "7889". But what does that mean? Where is that in the distribution of password frequency? It's impossible to tell, without manually constructed context that will change over time as the total number of passwords in his corpus expands.
But more crucially, there is no way to tell relative rank ("is this password in the top 20k?") using the API that I can see. That would make using the top X much easier. But with the K-anonymity "feature", there's no way to do that that I can see.
I don't follow - how is the relative rarity better than absolute frequency? What really matters is how common your password is - not how highly it's ranked in a compromised password list, which has no relevance to how common it may be.
You want to filter on users choosing a password that's been re-used across all compromised more than N times.
Filtering users on choosing a password that ranks N of M on a list of compromised passwords doesn't tell the user how bad that password is.
In fact, once you get to the rail, the ranking is basically based on sort order and become irrelevant?
The third field indicates how common the password is in the corpus (for example, the single most common password - "123456" - appears in the corpus 20,760,366 times).
So ... based on this data ... what is a reasonable value for that count, such that if the value is exceeded, the user should be disallowed from using the password? How much real-world online or offline resistance is provided by disallowing, say, passwords used at least 186 times in the corpus (roughly a million passwords, though 5201 passwords are at the 186 mark)? (The answer should be self-evident; if it isn't, I can provide more background).
Put another way ... if the corpus was only 1M in size, those right-hand values would be much smaller. How could you determine the threshold then? What I'm trying to illustrate here is that it's not the absolute value of that commonality number that matters; it's the relative rank. But that relative rank can't be determined via the API; you must analyze the entire corpus directly - and then discard the vast majority of it for blacklisting purposes.
I totally get that the threshold might vary per implementation. But it varies much less once the hash is slow enough, and the authentication service is suitable rate-limited. In other words, any system that would get real benefit from a 1-million-word blacklist is one that needs to be improved elsewhere instead.
But Troy didn't provide any guidance about that, or even how to judge for yourself what the threshold might be. He just provided an API to blacklist a corpus of passwords that is three orders of magnitude larger than a properly designed system would ever need.
1) In an online attack, against a properly-configured service, even if password spraying is used, only the first few thousand passwords can be tried before rate-limiting, CAPTCHAs, etc. kick in.
Would a user with a known leaked password at a different site be vulnerable to an online correlation attack? Yes. And that's why some big services supplement their approach by proactively searching for those leaks and forcing a password reset for those specific users.
2) In an offline attack, when the passwords are properly hashed with a modern slow hash, even an expensive GPU or FPGA cluster would take weeks to exhaust a 10,000 word dictionary against a large user corpus, and a significant amount of time even when a single user is targeted.
Would users with '123456' get cracked pretty quickly? Yes. And that's why the top X are forbidden - to make offline attackers have to dig deeper into their wordlists (and thereby also their pocketbooks) to crack a password in a useful amount of time.
Eh, "We've got lots of users so it will take a long time to crack them all" isn't much of a defence.
I mean, if you've got Obama or Snowden or Taylor Swift or Logan Paul or whoever as a user, you think hackers wouldn't spent 2 hours of GPU time per account to crack their passwords?
I'm quite familiar with password attack scenarios.
If high-value targets are selecting passwords that would be vulnerable to a targeted cracking attack, the solution isn't to blacklist a half-billion passwords (when they could just as easily come up with literally trillions of other passwords that would also be bad, yet are not included in the blacklist). The solution is to show them how to manage their specialized threat model - 2FA, creating strong passphrases, using a password manager, etc.
I find it hard to believe that you could set a cutoff of passwords that have been leaked but that you could rely on an attacker not to try. These passwords are more useful guesses than anything a password cracker would make up out of components.
XKCD considers a password that's one of 2^28 possibilities "easy" to guess, and provides a well-regarded strategy [1] for coming up with a password that's one of 2^44. Passwords in this list are one of 2^29.
Random passphrases are indeed a good idea. XKCD #936 advocates for 4 words randomly selected from a 6000-word dictionary, which is 6000^4, or ~1.296 × 10^15, which isn't actually that strong if the service in question has chosen a weak password hashing algorithm. When using pure bruteforce or masks (not a dictionary or hybrid attack) against a large-ish corpus of passwords (say, a few million) a system with 6 GTX 1080s can realistically try 8.2 billion SHA1-hashed passwords per second, which would exhaust the entire XKCD 936 keyspace in about 45 hours. (If you bump it up to five words from a 20,000 word dictionary, you get ~3.2x10^21 possibilities, which is better.). And if you focus on a single hash, that SHA1 rate jumps to ~32 billion hashes per second, which would take less than 15 hours.
At that speed, processing the entire Pwned Passwords list would almost take longer to read from disk and into memory than it would take to exhaust against a single password. Password cracking specialists would of course try raw wordlists first (And therefore "more useful", in a way) ... but we many other tools in their arsenal that generate far more than a half a billion candidate passwords. And at that rate, you can exhaust all 8-character passwords made up of printable ASCII - 95^8, ~6x10^15 - in a couple of days. Other techniques (mask, hybrid, rules) can achieve similar rates, and combinator attacks are slower but still pretty efficient.
By contrast, attacking bcrypt cost 12 on the same system can only try ~660 hashes per second - against a single hash. At that rate, if you just tell the attacker "it's somewhere in the Pwned Passwords list", it would take about 210 hours to exhaust the raw list, and 36 years to exhaust all 6-character passwords made up of printable ASCII.
In other words, if a service is storing passwords poorly, that service should be fixing that long before they should be trying to blacklist a half billion passwords. The purpose of blacklisting up front in the password-changing UI isn't to forbid a half-billion passwords. It's a way to reduce risk of online attack - and an opportunity to guide users towards better selection methods. There's a reason why Dropbox only blacklists the top 30K.
> which isn't actually that strong if the service in question has chosen a weak password hashing algorithm.
That only matters if you re-use the password in multiple sites.
If an attacker has access to the hash, that means they cracked the site already at the admin level and got into its user database.
They don't need to crack your password to gain any more access to that same site. (And they already have all the plain text personal info from your account.)
Your only additional problem now is if that password gives them access to your account on other sites that they haven't broken into yet.
The ultimate protection against that is not to have reused that password. That beats the stupidity of "password strength".
If a password is not reused, it has to be only strong enough to survive the five guesses before an account is locked out.
Password strength matters when hashes are public (like in classic Unix non-shadowed /etc/password files). Well, that's a bad idea, which is why we have shadowed password files. Shadowed password files may as well store passwords in cleartext; if those passwords are not reused anywhere, the situation is safe. Anyone who can see the cleartext is already root. If those cleartext passwords don't work on any other system, they are worthless to the attacker.
Thus password strength --- all the fussing with how we properly store passwords with a decently strong hashing function and salting --- is just a fallback strategy to protect password re-users.
> Shadowed password files may as well store passwords in cleartext; if those passwords are not reused anywhere, the situation is safe
Wait, what?
If they were randomly generated and of sufficient length, yes.
If they weren't randomly generated, even if not exactly reused, they are very likely to reveal the psychology of that user's password selection habits. This is of definite value to a focused attacker. Not only could it inform guessing passwords on other systems, it could also inform guessing that user's _next_ password on _this_ system.
> They don't need to crack your password to gain any more access to that same site.
Just because they have the hashes doesn't mean that they have other access. Hash lists are bought, sold, traded, and stolen all the time. Someone who possesses that particular hash may be multiple hops away from the group that originally acquired them.
Also, just because the database layer that the passwords are stored in is owned, does not mean that a particular target level of access has been acquired. Password storage can be abstracted into an entirely standalone subsystem, for which knowing, say, an admin of that system's password would be quite valuable.
It means that suppose the attacker can look in /etc/shadow (due to having root privs) and sees, in plain text, that the password of user "bob" is "correct-horse" (not anything fancy like "correct-battery-horse-staple"). But Bob doesn't use that password anywhere else. So what good is that piece of information to the attacker? On this system, attacker can just "su bob". On systems where attacker is not root, "correct-horse" doesn't get into bob's account.
> If they were randomly generated and of sufficient length, yes.
What does that buy you, if they are in plain text?
(Well, randomness quasi-guarantees that they are not re-used; I covered that.)
If we have passwords in plain text, issues about length related to cracking hashes is moot; the cracking that still matters is someone guessing at the login prompt, where we can lock out accounts after N attempts.
What about the other case - when they're not random, but also not reused ... such that the psychology of the user's password-selection methodology might be exposed?
If you have a password selection methodology that you do not change when hashed passwords are compromised, then it doesn't help you. The methodology will be uncovered once the password is cracked, even if that specific password doesn't itself work anywhere anymore. It's somewhat better if the methodology is discovered later than earlier, I suppose.
Right! So (from the perspective of the password alone) the owner doesn't have to care if that password is used only on that site where the horse has left the barn.
If the password is used on other sites, then of course all that protects them its strength relative to the compute resources thrown at it, relative to the time between the breach and discovery.
(From other perspectives, the user does care: like their credit card number was stolen and is being misused.)
This list is small for the purposes of password-cracking. Enumerating 500 million things is something a computer can do very quickly.
Consider this: if you store the hash of one of these passwords in your login database, you have stored something that can quickly be turned back into the plaintext password, just by enumerating the list.
Passwords that have been leaked can't become good passwords again.
> Passwords that have been leaked can't become good passwords again.
This doesn't make any sense. We already know all possible passwords: the set of all the permutations of the set of legal password characters of a given length. Your argument applies equally well to these. You can't just use these passwords and check the hash, since the database (hopefully) at least salted their hash to prevent rainbow attacks like that. But even if they didn't salt their passwords, nothing prevents you from hashing all possible passwords and checking. I think I read somewhere that making a rainbow table for all possible 8 character passwords took little time and space; thus all 8 character passwords are already broken. Need to crack a password? Generate the rainbow table and just look it up.
However, your assumption that computers can check passwords quickly is not necessarily correct: that's why bcrypt exists.
No. Let me illustrate with a different security situation where we use blacklisting.
Many years ago Debian mistakenly shipped a version of openssl that didn't use good entropy to pick RSA keys. As a result, everybody with that Debian would get one from a relatively small pool of keys when they asked for a new one. There's nothing special about these RSA keys, other than the fact that Debian systems from a particular era would always pick them.
A good CA (e.g. Let's Encrypt) blacklists the public halves of those key pairs. Again, there's nothing special about them, no reason they're worse than any other random key _except_ Debian always picked those, and since it did bad guys can trivially find out the corresponding _private_ key for each value and so they're useless.
If you propose to use one of these blacklisted public keys, there is a near certainty that it's because you have a broken Debian system making the keys, and so refusing you keeps you safe. Even though there's nothing special about these keys.
Now, if I have a system that generates RSA keys in a known secure way, I needn't check for Debian weak keys myself. Why not? Because there is statistically no chance I'd ever pick one at random, it's a total waste of engineering effort to check. But if I ask somebody _else_ to make a key pair and send me the public half, I should check against the Debian weak keys, because I shouldn't trust that they're smart enough not to use the broken code.
These passwords are crap. They wouldn't necessarily be crap if nobody had ever known what they were, but now they do, so they're crap now. Pick a different password.
Okay, I may be overstating just how quick it is -- you might have to spend a few days of CPU time to go through the whole list, based on estimates I'm seeing. (Parallelize it however you want.)
But don't act like I don't know there are a finite number of passwords. The number of possible passwords grows exponentially with length, and the number of leaked passwords grows linearly with leaks.
It's about the number of possible passwords you have to check. There's a huge difference in magnitude between having to hash "all 10-character passwords" and "all 10-character passwords that are definitely someone's actual password".
Dropbox's zxcvbn password-strength estimator already incorporates a list of 250k+ common passwords and words. No need to make a separate blacklist check unless you want to check more than what zxcvbn already does. (and you could enforce it more strongly on the server instead of locally in JS like zxcvbn)
And with zxcvbn it'll still flag a password as low entropy/low security if the password it's checking is just a simple modification of something on the common password list.
Indeed. One of the authors of 800-63B is actively involved in the password-research community, and is already aware that the guidance places no restrictions on blacklist size.
Keep in mind a lot of these passwords are associated with email addresses in the actual dumps. By allowing a user to use one of these passwords, there's a non-negligible chance you're knowingly allowing them to use a username/password combo that is publicly available, and that any hacker who wanted to compromise their account could do so _on their first try_.
Besides, when it comes to passwords... half a billion really isn't all that much. There are (26*2+10)^8 = ~218 _trillion_ possible 8-character alphanumeric passwords. So even if every single one of these half a billion passwords were 8-characters long, you'd still only be disallowing ~0.0002% of them.
Some large services do use the actual dumps, and correlate them with the email address associated with the current user, in order to give users a personalized warning that they're reusing a leaked password that's already associated with that specific email address. This is a much different proposition from forbidding that specific user from using a half a billion passwords.
A full 80% of the v1 corpus can be avoided by simply requiring a minimum password length of 12. As Troy has pointed out elsewhere, this wouldn't be great UX, either. While it would dramatically increase the chances that they came up with a word that would be A) not in the existing blacklist, and B) harder to attack offline ... it would still be significantly bad UX compared to the best-practice alternative that I lay out in a separate thread branch.
But it would still be much better UX than use of the full blacklist.
Reusing username/email/password can cost your users hundreds per day on a gambling site. And users don't care about a password-gen guide. For example, in that case, you'd want to consider just generating passwords for them.
But of course this would be silly for the run of the mill website.
That’s funny because coral.co.uk posts its login over HTTP. In fact if you try to login via https it redirects you to http. It would be fun to setup a hotspot called _The_Cloud outside a Coral and see what you find on the wire!
> The UX of a blacklist with a half billion entries would be so crippling that it would cause a user revolt.
The situation doesn't need to get as bad as you think. If you suggest XKCD's four random common word method[1] to your users as part of the user interface, you'll be fine. As a test, I tried putting together 2 random, common, unrelated words together:
- yak elephant -> yakelephant
- crowd brown -> crowdbrown
- plastic envy -> plasticenvy
- colon spanish -> colonspanish
- jogging adhesive -> joggingadhesive
Even with two words, none of the above appear in the Pwned Passwords database of a half a billion passwords. It's not difficult to choose memorable passwords and avoid entries from Pwned Passwords if you suggest this method to your users (preferably recommending four words, but three might be OK depending on the threat model).
You may also want to consider running `unset HISTFILE` before that to ensure that the line containing your password doesn't end up sitting around in your bash history.
(It'll emit the line from the API response matching your pass; if it does, then that password was compromised. Bash isn't real good at error handling though, so my biggest concern would be what this might do if an HTTP/TCP error happened. I've attempted to throw -S there to catch that, but use with your head screwed on.)
You could skip the "tr" part, as the API to query by hash prefixes not case sensitive, and once you have its results, you can use "grep -i" with the hash.
I wouldn't recommend using this. Even if you were to read the page source and see that it is in fact not sending data back to a third party, either a) the site owner could change this in the future, or b) you could be man-in-the-middle'd since they're using http, or c) one of the third party scripts they run on that page could either accidentally or intentionally take your password. Don't put your password in there.
Another table, another database or another storage system in general.
If an attacker SQL injections your database don’t go spilling every hashed or unhashed password you’ve got.
I tend to store passwords in a separate keyvalue store from where my authentication identifier is (email, “username”).
If someone gets into my network they need to get into my servers with the email addresses and then get into a secondary system where the passwords are stored.
I like my password systems to be a k/v store because there is no need to “query” it. I usually store the password under a key that isnt the identifier. Instead using something like a database surrogate key.
Have a secondary system (microservice, private subnet) that simply returns a boolean representing if the provided non-email (key) and password (value) match.
Have that secondary system take the plain text password so it can do the hashing without letting the dependent service know what algorithm, salt or stretches you’re doing. This will also allow you to easily roll over to new hashing algorithms over time without affecting the service that is doing th authenticating.
Edit: I’m not trying to be a know it all or a crabby old tinfoil hat a-hole. But it’s passwords, man. When you leak them you ruin people’s days/year/life. Building that system above takes a middle of the road engineer a day or two. Put the effort in. Every password leak makes all of our jobs harder. It’s your companies responsibility to keep that safe. If you know that already, be the annoying guy that brings it up in every stand up. Make that debt known.
> don’t go spilling every hashed or unhashed password you’ve got.
Please don't store unhashed passwords. By now even PHP gives us the tools to do this right. You truly don't need secondary systems, or separate tables, or separate anything. Just hash the passwords.
I would argue that the benefit of putting all the hashes into a separate table is not really worth it. A separate service just to verify passwords sounds an awful lot like reinventing LDAP or AD/Kerberos with less features.
It should be good enough to simply encrypt the password hashes with an application-side key, a simple database dump won't leak passwords anymore.
If your passwords are properly hashed and stretched with appropriate and modern functions (SHA2/3 and Argon2) and encrypted-at-rest (Chacha20 or AES256) then you should be sufficiently equipped to secure your customers passwords against most attacks.
The point is not putting them in a system that can leak them right alongside the login identifier.
If someone wants to go exercise 9384828388 GPUs on your list. Fantastic, at least the other piece of their auth username/etc isn’t sitting right next to it.
Putting them in a separate system can be a simple REST server that sits in front of another database, LDAP or even something like Vault.
Don’t set them right alongside your ecommerce app you’ve got 30 juniors hacking on trying to get something out ASAP.
Why are your juniors hacking in production? They get a testing environment, if lucky they can play with staging but production should not be hacked upon.
I don't see the benefit of a seperate system still, if you really want to, LDAP already does all this. As does AD. Why reinvent the wheel and built a rest service for it?
I also don't know why it's harmful if a hash sits next to the username, if the database has been breached, password hashes are probably the least valuable information in such a leak. Mail addresses are more valuable.
They aren’t “hacking on production.” They’re occassionlly under pressure to release features. If they don’t get someone that catches a SQL injection or something else silly in a PR it’s nice to know we cant possibly bleed our password hashes.
Do you jam all your tables into one public/default schema? Or do you break them up by domain across schemas?
A very simple implementation of this in Postgres is an auth schema. Only one role can read from the table with the credentials, no app has access to that role. You write a function to verify a credential and use a security definer to give a role that can use that function access to read a single result.
In this case you’ve mitigated two “attacks”:
- a sql injection selecting all from users
- some random engineer taking a schema dump for whatever purpose, accidentally grabbing auth details and then having that get leaked somehow down the line.
I literally takes a few lines of SQL to set that up. It’s not an extra server to manage. Just a different schema with higher security constraints.
From your app with the credentials to call that compare function instead of doing a SELECT FROM you can call that function with the credentials and just accept Postgres telling you yes or no the credentials matched.
PG has a suite of hashing functions. Take your pick. That function can do your salting or you could add it as a parameter to the call.
SQL Injection is to my knowledge not a problem if you use literally any query building method but string concatenation.
Any sane SQL driver offers queries with parameters and at work and in private I enforce the usage of parameters in queries entirely. If I find myself needing string concat anyway then I will do it very carefully and isolated from user input.
That mitigates SQL injection.
I also see no benefit in breaking it up into multiple schemas or pushing authentication entirely into SQL. At best that means you complicated the database definition unnecessarily and secondly it means your database gets raw password strings which is contra to what I learned about password security, namely that the raw password does not touch the DB or the DB driver or any DB related code. Ever.
I also don't see how engineers dumping the DB and then accidentally leaking it is a tangible risk. I'd put that on the bottom of my list of things I have to worry about when I verify passwords.
Not everyone works with PG either or even MySQL or there is no possibility of setting up functions or triggers in the database.
>If they don’t get someone that catches a SQL injection or something else silly in a PR it’s nice to know we cant possibly bleed our password hashes.
Which is totally irrelevant if you properly secure the application, do proper code reviews and don't put juniors under pressure to push out code at all costs.
Pushing crappy code because deadline is asking for it. End of story.
The fantasy world you work in must be great. I’m jealous.
I don’t know how many times I’ve seen someone jam a gigantic pile of string into a query builders select method or done some whack string interpolation trying to make a clever search form instead of using a tool like ransack or the equivalent.
Like I said, that was a for instance using Postgres, you can do it plenty of ways. Hashi Vault has been an awesome tool for this recently.
Getting the password near your database isn’t the problem. It’s persisting it unhashed that is. I’d conjectire a fair amount of sites terminate SSL at the edge of their network (nginx, ALB) and they have a password floating through the stack in plain text until it hits a compare method. What does it matter if that method is in your app code versus pg or vault or any tool you’d use for credential storage? Are you sure your app is scrubbing all the json keys or http field names that could contain a password to make sure it’s not going into your rails/node/whatever logs or into your nginx access logs?
Having multiple schemas isn’t complication, it’s organization.
As far as people doing dumb shit with files they’ve dumped or downloaded. It happens.
When you’re designing a system for storing credentials you don’t know who will work on this system or what practices will be in place around code quality, reviews and deadlines when you’re gone. You can’t literally watch every PR or guarantee that ever code review has someone as smart as Zaarn presiding over it.
You should be designing something people can’t shoot themselves in the foot with.
An example of a totally reasonable mistake to make i saw recently in a PR in a client I work with.
They had designed a pretty solid API using the JSONAPI spec. JSONAPI if you aren’t familiar allows an API client to request what fields they need from the endpoint instead of just getting all the fields the API returns. This is nice for mobile clients.
They had the forethought to exclude allowing people to ask for “password” through the /users API. Nice. Nailed it.
A year passes.
They add a feature for changing your password that required you to confirm by email that’s the password change was you and then it swapped the password from the “new_password” column to the “password” column. While this PR was in staging you could do /users?include=new_password and it’d return their new password (hashed, but still).
Password resetting was a feature they were adding to their registration and auth code.
The people that worked on that front end didn’t necessarily work on the user API and weren’t familiar with how it worked. But by adding this flow and this field the created a vulnerability in a tangentially related system.
Now I’m sure you could crap on JSONAPI but it serves a purpose. If they hadn’t been using that someone just as lazily could have designed the regular JSON endpoint to return all fields but password and they’d have ended up in the same boat.
The takeaway from that (which has been known for a decade at least) is that your passwords need to be hashed and salted, and you need to use a hashing algorithm designed for security (meaning it is slow). Not that you should complicate your system architecture and create more potential points of attack.
Absolutely, Im not stating otherwise. Im saying you shouldnt have your authentication inside your business logic. It should be a separate service. Limited access, and transparent to the application that is calling on it.
If you want that to be a REST service, great.
If you want it to be another table/schema/database/server, great.
If you want it to be something else, great.
Just stop putting it right inside the application that is taking user input.
Don't roll your own hashing, use something solid, like Argon2, salt your stuff. Store your salt someplace secure, not in your frigging rails config.yml or inside env on your webservers or anything else that is publicly addressable.
> Im saying you shouldnt have your authentication inside your business logic. It should be a separate service.
And I am saying this is a wrong, counterproductive idea. It complicates your authetication for no substantial gain, and will only result in additional vulnerabilities.
> Store your salt someplace secure, not in your frigging rails config.yml or inside env on your webservers or anything else that is publicly addressable.
So you don't even understand how a salt works. Why should we take security advice from you again?
Using something like Hashi Vault, Gluu, or Shiro does not overly complicate things. They’re stable, trusted solutions.
It can also greatly simplify things. In the common scenario of having a web app for customers and a web app for admins, instead of them each having their own authentication baked in, you can choose an open source solution and deploy it twice, once for each service.
I know how salt works. Don't belittle me.
If someone gets into your web servers they’ve pretty much got carte blanche at your database the web server has access to. They also now have your code, which in a lot of cases is probably plainly readable. So they can see your salt, see how you salt and see the hashing algorithm you've chosen.
I’m saying don’t store the salt in your apps config right along with the credentials to access all of your login identifiers and hashed passwords.
You gave them a piece of the puzzle. If someone wants to grind through and crack all the salted/hashed passwords, I'd prefer them not to have the salt to help in that endeavor.
He's talking about how each password should use a separate salt. This is normally stored next to the password hash. Many hashing algo implementations will even do this for you.
All security decisions boil down to your threat model, doesn't it? At some point there are decreasing marginal returns with increasing security. What threat models are you satisfying with the strategies your proposing? Do you think these apply to everyone? Or even everyone who stores hashed passwords? I personally don't think so, but I'm interested in hearing about things I haven't come across yet.
A day of engineering effort to make exposing passwords much more difficult. Yes. If you’re a company that stores someone’s password. You know they probably use it elsewhere so it’s your responsibility to help keep it safe.
Sorry, but this is convoluted nonsense that can only achieve one thing: make yourself more vulnerable.
You want your security system to be as simple as possible, and to involve as little custom code as possible. Because you can and will fuck it up if you try to be clever.
Hash and salt your passwords using a library designed exactly for that purpose (which means it will use a slow hash). That's it, end of story.
Rule n1: don't roll your own security.
Rule n2: goto 1
You are overcomplicating your authentication system by oversimplifying security problems and the result is that you have solved nothing.
Security always seems very easy to solve and usually non-security engineers tends towards solutions like yours that doesn't provide extra security, they just add a few extra steps for a hacker to obtain you database and as a result you need to maintain extra databases, there are more error points... Do you remember that thing about "each extra system exponentiates complexity"?
go a step further and you will get into key management devices like the HSMs and/or the Amazon KMS. KMS cost almost next to nothing and it is pretty neat since its a web service, especially coming from the world of $40k+ Thales/Safenet HSM devices which are a pain to deal with (backups, rehash, redundancy).
It was to meet the PCI-DSS Level-1 security standards for banking compliance. We'd store encrypted cards in one place and store the master keys in the AWS KSM to later decrypt it. But to retrieve the master keys, it goes through another layer of encryption.
[Pasting an old comment of mine on password managers, since I see people talking about starting to use Keepass. I hope this helps someone]
----
If you're just starting, here's some guidance on setting up a password manager.
First of all: Don't be afraid of using one. It's not just more secure, it's super convenient. Never again will you ask yourself: Did I make an account for this website/service? What email did I use? Never again will you have to remember a password. Using a password manager is a quality of life improvement.
KeepassXC is what I recommend to people at this point. It's free and you own your data (your passwords). They live wherever you want them to live. There are plenty of online services that are supposedly more convenient but I have to say I trust them less -- YMMV (1Password is the best I'm aware of).
If you do use keepassxc, you get the added benefit of being able to store 2FA settings in it as well (if you store them in the same database as your passwords, be aware that you lose the security benefit of a second factor, however it is still more secure than not having 2FA enabled due to the One-time password component).
Put every account you ever made and ever make into keepass. Enable 2fa wherever you don't have it enabled. Add login URLs and notes. Generate your passwords from keepass itself; the password generator is really powerful and lets you very easily deal with site-specific shitty password limitations. I'm telling you this because, seriously, it's incredibly convenient to have this stuff as long as you're rigorous about maintaining it.
Oh, also, keepass has the full history of all your passwords. Need to look up an old password? Go into details and look at "History". You can also attach files to items (items don't have to be accounts at all, you can use keepassxc as a simple encrypted storage db).
Mobile support: Keepass2Android. Best android client, with google drive support. iOS I have no idea, suggestions welcome.
IMPORTANT: BE STUPIDLY PARANOID AND RIGOROUSLY CAREFUL ABOUT YOUR MASTER PASSWORD. That thing, together with your keepass database, unlocks all your accounts ever. Use a really long passphrase that you will never have to write down (if you do decide to write it down because you don't trust yourself, store it in a safety deposit box, don't put it in a bloody drawer). Make sure the device you unlock the database on is malware-free.
PS: Wondering what's up with Keepass vs. KeepassX vs. KeepassXC? Keepass is the original app, written in .NET but with poor multi-platform support. KeepassX is a rewrite in Qt and is a fantastic password manager, but has gone unmaintained recently. The open source community picked up the slack in the KeepassXC fork (after continuing countless attempts to upstream the patches) and has implemented lots of powerful features. I've switched to it and at this point I strongly believe it's the better client.
I've just switched from 1password to keepassxc in the past few weeks. The only reason I did so was because 1password was trying to force me into their subscription service as I switched from macOS to linux mint. I looked at a few work-arounds on github, but eventually just decided to move over to keepassXC.
The export / import and overall setup was pretty painless. I am still able to sync through dropbox just like with 1password. There is also an ios app called keepasstouch which was a breeze to get going and syncs with my dropbox password vault. Finally, the browser extension works in a very similar way to 1password's.
Overall, I definitely recommend it. I haven't lost any functionality or security to my knowledge. I was happy to pay up front for 1password (probably dropped 60-80$ or so for their apps), but, after doing so, just couldn't stomach being forced into their SaaS model. Especially when a similar free, open-source alternative exists.
I have to thank you for bringing this to my attention!
Although I have not seen any reports of keepasstouch being compromised, it is true that it is not open source ... I had meant to download minikeepass, which is. I've remedied the problem. 2FA keeps me pretty safe on most of my important accounts, but I have still changed pw's on many accounts thanks to your comment. Much appreciated!
> you get the added benefit of being able to store 2FA settings
Don't do this. If you use a password manager with all the benefits this entails (long, random passwords, each only used for a single site), the only benefit 2FA really gives you is if your password manager is compromised somehow. If your second factor is in your password manager, you're screwed.
I use Authy with a long, secure password printed on a piece of paper. Yes, it is cloud and third party and everything, but it's on a completely orthogonal chain from my Keepass DB, so dual compromises are significantly more difficult.
On the other hand, do do this, but be aware of the tradeoffs.
I hate telling people not to do something. Most people just end up not turning 2FA on at all. My approach has converted many people from "one password reused everywhere, at best with variations" to KeepassXC unique passwords everywhere + 2FA and I classify that as a big win.
The biggest benefit of TOTP 2FA isn't the "second factor" part, it's the OTP part. This removes many forms of phishing, keylogging and database leaks as a threat to your account. You do not lose these benefits when you have it all in one factor.
If you read my comment, you'll see I address this concern. If this is a real threat for you, then you can always simply use a separate Keepass database for your OTP settings.
I grant that it protects against phishing, but I would cautiously suggest that sites that are smart enough to enable 2FA are smart enough to salt/hash/bcrypt/whatever best practice their passwords, so leaks are neutered. It doesn't not protect, so to speak, but the protection is likely to be redundant.
But it emphatically does not protect against keylogging, anyone who can install a keylogger on your computer can grab your password DB and your master password. This is exactly the scenario where you need actual 2FA.
Anyway, broader point: yes, it's a tradeoff, but the kind of people who needs explaining why a password manager is a good idea, do not understand enough to make an informed decision about these tradeoffs. And so, the responsible advice is to not use it.
I do know enough to understand these tradeoffs, and my conclusion is to keep password management and 2FA strictly seperate.
> But it emphatically does not protect against keylogging
1. A keylogger on your password db is useless if it does not also upload the db (at which point you're looking at a targeted attack, and you have far bigger problems than that).
2. Keyloggers are more and more often browser-based. KeepassXC is immune to those.
3. KeepassXC supports 2FA for the database encryption itself. If you're that paranoid, use that. There's always more you can do.
> And so, the responsible advice is to not use it.
No.
Just as you see in the article where Troy has to make the difficult decision not to include a "Do not put your password anywhere not even here" disclaimer, the same holds in my message: I weigh the pros of someone turning 2FA on as far more important than the cons that come with the less-than-ideal security 2FA adds.
Your advice keeps people from turning 2FA on. 2FA is a pain in the ass for most people.
You are one of the lucky few who understands the tradeoffs involved, as you yourself said. So use that knowledge of yours to actually get people to secure their accounts.
My goal isn't to keep Edward Snowden's accounts secure. It's to keep the bored HN user's account secure. The average HN user has medium-to-high technical literacy and low-to-medium security literacy. A lot of people on here reuse passwords, I'm sure. This is what I'm trying to fix, and I won't advise Ed to keep his TOTP seeds in the same database.
I'm not sure that 2FA is going to give the average person as much protection as you assume. You have to keep your 2FA key somewhere. So instead of needing your master password and password DB, you need the master password, password DB and 2FA key. But the question is how hard is it really to get that key? Certainly harder than having it in the password DB, but in practice, not really any harder than getting the password DB in the first place.
There are lots of options: using memorable passwords/passphrases, using random passwords, using cloud based password managers, using password managers on your own devices, putting the 2FA on the same device, putting it on a different device, putting the 2FA on an air-gapped device.
IMHO, only putting your 2FA on an air-gapped separate device gives you dramatically better protection in the areas you are concerned about. The rest of the conversation is really about where "good enough" lies -- and that depends entirely on what you are doing.
> If you use a password manager with all the benefits this entails (long, random passwords, each only used for a single site), the only benefit 2FA really gives you is if your password manager is compromised somehow.
I don't think this is true. Your long, random password can still be compromised, for example if ever type it on an insecure device (a friend's computer? public or shared computer? etc). 2FA keeps that single compromised password from being useful.
It is undeniably true that storing 2FA secrets in the same vault as your passwords reduces security. However, it's still way, way better than not using 2FA at all.
I use KeePass 2.x (can't be arsed switching to KeepassXC), but I absolutely second Keepass2Android. It supports a whole host of backends, which includes SFTP - great it you want to securely keep your password DB online without using a 3rd party service (e.g. Dropbox).
I use keepass, and I whipped up a quick script to look all the passwords in my keepass file against the hashes in the list. You have to download and unpack the file first.
I would also recommend KeepassXC. I personally store the password file in Dropbox allowing it to be used on all of my devices. I use a second KeePass file with a different password for storing 2FA backup codes. I open that so rarely that I have to sometimes test that I still can!
On iOS I would recommend the free MiniKeePass app. It isn't as easy to use as some KeePass app(s) I used on Android years ago, but it gets the job done.
I have a old article[0] on how to use KeePass effectively which makes the workflow more seamless. Ugh - really old, I really should update that!
That’s good info with one exception. You are probably ok with Keepass, but for most people I’d MUCH sooner recommend a solution that can not be done wrong.
LastPass is my go to for most people. It simply can not be set up wrong or confusing. The same can’t really be said for keepass.
As a sysadmin who is pretty critical of password managers, I can vouch for keepassxc as well. I started out with keepass, then keepassx, and ended up on keepassxc. I refuse to use an online pw manager.
I use it regularly. The only real drawback is that the database has to be updated manually (e.g. from DropBox or wherever). Outside of that issue it has lots of features and config options and is generally easy to use.
I just tried a 11 character password without special chars, that I’ve used on over 50 sites, over the last decade. It’s my password for throaway websites. Some pretty dodgy.
Not in the database.
Makes me feel pretty good about password security overall!
I have a password that’s a pair of words in two languages with some number substitution that I use a lot on websites I don’t care about and it’s not in the dB. And I’m sure I’ve used it on sites that have been hacked, so I dunno.
He specifically tells you that you shouldn't do that. But as the winning tool shows, perhaps people who have passwords in HIBP will changed them after finding this out?
I'm going to have to disagree with the premise that sites should stop users from choosing a password which happens to have been cracked offline at some point in the past -- to the tune of blacklisting half a billion potential secrets.
What exactly is the end goal, and at what cost? Well, there are 3 ways to steal a password. You can steal it from the user -- either by phishing or with malware -- in which case it matters not a bit how complex the password is. You could attempt to crack it online, that is, by attempting to login as the user through the front door. In this case, a simple counter should limit the number of attempts before a second factor is required, such as clicking a link in an email, and a list of half a billion candidates isn't going to help here either. Finally, you can steal the password verifier database and attempt to crack the password offline.
So, the theory must be that passwords in a known attacker's dictionary are more likely to be used as candidates in an offline attack. This is likely true. But once the verifier database is stolen, if an attacker is able to run the password hashing function, then every password which is not raw entropy already must be assumed to be cracked. Regardless of how hostile your password policy is. So what exactly is this policy saving you?
On the flip side, it's reasonable to ask, what would such a policy cost you? It's hard to say without actual data, but I'd love to see some data on what percentage of candidate passwords offered by a user trying to signup on their mobile device would be rejected under this policy? How many attempts on average would it take a user to find a password which was not rejected? And what's the increase in bounce rate, and therefore lost signups, that would result? How many increased password resets would be required from users choosing passwords that they inevitably don't remember? How many additional lock-outs which require customer support capital to resolve?
Password policies on average are pretty horrendous. But password policies which are arbitrary black boxes to the end user are about the worst you can find. Sitting on my mobile device, not knowing if a chosen password will be accepted, having to type it twice each time, is a wretched user experience which would need extremely lofty benefits to outweigh the cost. I fail to see any benefits to this approach which couldn't be solved with better hashing which wouldn't impact the user experience whatsoever.
I'll say one more thing on the idea of blacklists. Users just trivially work around them. Password quality (entropy, guessability) does not generally increase. Bad password policies often decrease password quality, particularly in the case of password expiry. But a frustrating opaque blacklist could be just as bad. (I'm not aware of any studies on this).
An attacker who knows a particular blacklist was in place will use munging rules to find derivatives from the master list which are not on the blacklist. How much will their crack rate (percentage of clears recovered from an offline attack of a given magnitude) be affected? But more importantly, potentially driving down the crack rate through user hostile password policies is a game which has huge dividends at first (getting a password which can't be attacked online) and very little dividends after that point.
Disclaimer: Founder of BlindHash, which is the "better hashing" I refer to above.
> I'm going to have to disagree with the premise that sites should stop users from choosing a password which happens to have been cracked offline at some point in the past
Well, NIST, NCSC and Microsoft all seem to be on the same page:
The end goal is to prevent password spraying attacks. If the attacker can get in on the first try because the user has re-used credentials in between two services, account lockout policies don't help and neither does BlindHash (if the compromised service wasn't using it). For it to bring any significant security benefit you would need everyone to use it (which will obviously never happen).
NIST has unfortunately been the source of a lot of bad advice which has actively harmed password security the last decade. (e.g. [1])
Cargo culting is generally a good thing in crypto because, you know, don’t roll your own. But in this case we’re talking about policy. And this policy is as user hostile (if not worse) than the prior NIST advice on password expiry.
If you want to stop password spraying, protect your hashes. There’s no proof that blacklisting half a billion specific secrets will make cracking any more difficult. Making it neigh impossible for users to register with your service, well I guess if you have no users you have no passwords to lose.
But the point is a blacklist this extensive is just as likely to make passwords easier to crack, not harder, and will come with a direct cost to the company implementing it. I understand well the goal, I’m entirely unconvinced this helps achieve it.
I would be interested to hear Gosney’s (cracker extraordinaire) and Cormac’s (Microsoft Research) take on this.
> If you want to stop password spraying, protect your hashes.
Again, it's not about your hashes, it's about the attacker having access to your users' credentials.
Users re-use credentials accross services and you have no control on how (in)securely they are stored there.
Blacklisting (I don't have an opinion on how big the blacklist should be) what is known to be widely used accross services sounds sensible... and there is definitely an argument to be made about blacklisting what is known to be widely available/effective for attackers.
> How many attempts on average would it take a user to find a password which was not rejected? [...] I'll say one more thing on the idea of blacklists. Users just trivially work around them. Password quality (entropy, guessability) does not generally increase. Bad password policies often decrease password quality, particularly in the case of password expiry.
That was my concern - if this blacklist is used widely,could it _encourage_ password reuse? If it takes multiple attempts to find a valid password, will the user just find one that is accepted and then use it everywhere?
Hmm this is a pretty great list to use for any service that has user signups -- disallow signup when using a password that is known to have been "pwned"! :)
Take the first 5 characters, in this case "5baa6" and use at the end of the API endpoint in your browser. E.g.
https://api.pwnedpasswords.com/range/5baa6
Then take the all the rest of the hash after the first 5 characters, in this case "1e4c9b93f3f0682250b6cf8331b7ee68fd8" and ctrl-f search the results page for it.
Yeah, but you would need two commands then, because you need to search for bytes 6-40 in the resulting output. I made a one-liner farther down the comments.[1]
I guess you can draw some conclusions there... maybe
- pwned accounts were created between 2000-2016
- Democrats do worse at secure passwords?
- In 2000-2008 people chosed worse passwords?
If you spend the time building a system to search those half billion passwords when you’re users are signing up, you should focus on building a login rate limiting system so it’s not possible to brute force someone’s password.
Though I agree rate limiting should be done (and done carefully), it is not very effective in all cases. As just 1 example, a determined attacker who wants to pop any account can make 3 attempts on hundreds of thousands of accounts, using a unique IPv4 address per account, thanks to Windows & IoT botnets.
WAF, IP blacklists, naive bot detection[0] and why would a decent thresholding system allow for a single IP to fail multiple accounts in a short time period.
If you hit two valid accounts [1] with bad passwords in a few ACCEPTABLE_UNIT_OF_TIME, it’s captcha time.
Thresholding isn’t just action per IP it’s being smart about how people are going to attack your system. It requires thought and upkeep.
The purpose of a blacklist is multifold - to reduce the efficiency of an offline attack, in which the hashes are stolen and can be attacked at high speeds without rate limiting - as well as an online attack.
I think this is intended for the case where a database has been compromised, but then again, I thought that hash algorithms and salting were supposed to deal with that.
This is a tangent; but I had a 'pwned' password that I've used for years on steam that started getting hacked like 4-5x a week; I would just ignore the 2 factor attempts for several months.
I finally changed the password to a slight variation that is not in this list (nor likely any others, 9 random alphanumerics); and within a week the two factor notifications started back up!
I was really surprised; admittedly the modification was trivial, but that is pretty thorough for a steam account I've spent like $100 in.
Did you have any items in the account (TF2, DOTA2 etc.)? Some of them sell for silly prices (and it isn't always immediately obvious which), which can result in your account being targeted.
> However, I got a lot of feedback from V1 along the lines of "simply blocking 320M passwords is a usability nightmare". Blocking half a billion, even more so.
Because normal humans will only be able to generate so many passwords on their own before they give up.
He also later makes the example of understanding the frequency matters — abc123 is mathematically equivalently bad to mno678, except one is far more likely than the other.
This dataset would be a fun project to implement on IPFS -- content-addressed, distributed database of leaked passwords using the same hash range protocol.
It would be great if the many password managers out there like keepass for example, use this data to filter out any password that exists in the list. I know it would be very rare, but still, why not filter them out?
What would be the point of filtering out passwords that are in the list, when using keepassX or other password managers? Isn't the point of password manager that you don't have to choose or come up with passwords yourself?
I thought it was funny that toepoke thought "People won't know what 'pwned' means." So instead they inform the user that they have a "Pawned password."
The article explains how the site has been designed not to send passwords to the server. Of course, it's up to you to decide if you trust them to keep it that way.
What would be nice is if we could improve the process of generating per device client side certificates that can be associated with a user account. Then we could just use certificate based authentication (and add on password based authentication if we want a second authentication factor).
Bit off topic, but I was searching for a better way to manage passwords a few weeks ago (rather than have 1 or 2 master passwords across all websites).
I found KeePass through an old ask HN thread. It's a great little free, open source key/password storage app that works across all my devices (iOS, macOS, windows). https://keepass.info/
I'd be interested to hear any suggestions for similar apps I could recommend to my parents, who expressed concerns about their online passwords. KeePass would be the ideal solution, but I don't think it would hold their hand through download, setup, and password generation enough to be 100% ideal. Any suggestions?
It allows you to login to multiple workspaces so you can have your family plan and your work team plan at the same time.
There is a cost but I can guarantee that it’s saved me more in time than the cost could ever amount to.
I’ve never used an insecure password since using it. All completely random and it’s so easy to use.
The different vaults means I can grant my little boy access to the stuff he uses whilst having my credit card and server details in other vaults without fear that he can access them (more the credit card details). I just need to get the wife into using it now...
But be aware that they are moving to cloud storage for password data. At this point, you're not forced in to it (and they've made no concrete announcement yet), but that's pretty clearly where they want to go.
Perhaps that's acceptable to you; I'll be moving off at that point. But in any case, it is something to be aware of when choosing something like this - moving between password managers is a hassle.
Have you read their security white paper[1]? As much as I tend to freak out about cloud storage of password data (one reason I moved off of lastpass), they do seem to take fairly strong steps to host-proof the data (such as having an on-device generated secret key that they never receive).
I did read it. For me, it doesn't matter for two reasons. Professionally, I have a policy to comply with. And personally, I don't use other peoples' machines for personal storage, period. So why would I want my (even well protected, assuming 1PW didn't make a mistake) passwords to be an exception?
Have a look at 1password.com and their family plan, so you can admin your parents too, in case they forget their master password.
Can’t recommend this enough.
Edit: If that helps, Troy Hunt and other security guys are also using 1password (even the cloud solution).
Keepass works great, but one disadvantage is that you cannot you additional credentials providers if you want to open your database on other devices. For instance, if I would use a Yubikey provider, I cannot open the database on my iPhone.
It would be great if the authentication providers can be configured in an 'or' instead of an 'AND' (this means convenience vs security). So on Windows, with my Windows account or perhaps Hello it unlocks my Keepass database and perhaps use a Yubikey, but on my iPhone I'm still required to enter a password. But unfortunately, the current implementation combines the authentication providers to create a combined encryption key.
But still, Keepass is easy to setup and maintain. Just put your database on your favorite cloud provider (if you want to take the risk) or manually sync it to your phone and you got the righty credentials everywhere at hand.
I've used Lastpass for awhile, and so does my work. Recently I switched to Bitwarden partly because of security concerns, partly because it's open source, but mostly due to Firefox moving to WebExtensions.
My biggest frustration with lastpass is that it doesn't seem to know the difference between subdomains, so it suggests several passwords for the same domain even though I am on different subdomains.
There was an option somewhere with which you could disable that one url at a time. I never bothered with it though and recently switched to keepass, so can't verify anymore.
Just keep in mind anything simpler likely has additional attack surfaces. e.g. Anything that's cloud based, browser plug-in, centralized, etc. all bring the standard attack surfaces in that domain. Still, it's probably better than what most people do, same user name/password combo everywhere.
I'm a heavy KeePass user in private and I like it. Recently, I was pleasently surprised that my new corporate IT overlords allow me to use it at work too.
Password generation is fine. Whenever you create a new entry it prepopulates it with a generated password. Just type in domain and username and hit ok.
On the negative side, KeePass UI is old school. Lots of buttons and options. Not streamlined for the common use cases.
(No comment on download and setup, since I'm not a Windows user)
I was looking at Dashlane which seemed to have some good features, including some ability to do password rotation automatically, and some level of yubikey support. That said I'm still using keepass. I discovered it when I needed a solution that worked on an original Surface RT, as well as an iphone, and I've remained happy.
I love Dashlane, and the Linux support is solid through the browser extension. Dashlane has the best interface across devices, of all the password managers I've used. The password sharing feature is great for business.
Sure, it's a cloud service, but it's polished enough that I can get non-tech people like my family and coworkers to actually use it. Lastpass did not pass that test.
I use Dashlane every day on Mac, Linux, Windows, and Android.
Thanks. One question I have - can you turn off auto-login globally or just on a per website basis? Docs seem to imply on a per site basis only but that seems a bit odd.
I'm wondering how DashLane handles multiple logins for the same service (Google/GitHub etc etc).
Hello and thank you for your comment. I am Thibault, from Dashlane's User Support team
If you mean having several accounts for the same service (aka. Google/Github), then yes. All you will need to do is create a Dashlane "credential" (id+password) per account.
If you mean Federated sign-ins (such as this one https://i.imgur.com/se5toyn.png), then it should be possible as long as the button opens a browser window on the right domain (Facebook, Google, etc).
Please don't hesitate to contact us directly through our Help Center if you have other questions about Dashlane !
Thanks for your message. I am Thibault, from Dahslane's User Support team, and am here to provide you with a bit of information.
Here are a few things to know about Dashlane!
* Dashlane does integrate with Yubikeys for Two-factor Authentication. All keys are compatible except for "FIDO U2F Special SECURITY KEY". Please see this link http://bit.ly/2FnNxka for more information on the subject
* Dashlane has been consistently rated the easiest password manager to use, which is extremely important since the last thing you want is a headache every time you’re trying to store or access your passwords. Don’t just take our word for it - Dashlane has over 50,000 5-star reviews.
* Dashlane is the only password manager with a U.S. patent for its security architecture. And your Master Password is never stored on our servers. Never!
* Dashlane’s auto-fill technology, which reads the fields on the page to store or enter data on your behalf, is the most accurate auto-fill in the industry. We’ve benchmarked our auto-fill tech against the competition and have found Dashlane is up to 40% more accurate than other password managers.
* Dashlane invented Password Changer and is the only Password Manager that allows you to change up to 500 passwords in seconds without ever leaving the Dashlane app.
* Dashlane’s VIP customer support is the best in the industry. Super-quick response times, availability via live chat, and premium support keep our customers happy.
You can check out our great review in the New York Times by following this link : http://nyti.ms/2CCUGKr. You will also find Dashlane in first position of PCMag's list of Best Password Managers for 2017 here : http://bit.ly/2BIdv1S.
Please note that although we propose the Password Changer feature, the automatic rotation is yet to be developed. I will pass you request down to the Development team for further consideration.
I confirm you Dashlane is indeed compatible with iOS devices, and you can access your Dashlane account trough the Web App with your Surface RT!
If I can be of any further help, please let me know!
I've tried to get my mom to use KeePass with limited success. To her having her easily guessable passwords written in a little notebook next to the computer is fine.
I think a prerequisite is getting buy in on the idea that passwords need to be treated like the keys to your house or your bank account numbers.
As far as I'm concerned writing passwords in a notebook in a draw or something is fine. The easily guessable part not so much. If someone has physical access to your computer in the first place there's lots of things they could do.
Algorithmic password in your head, with a google spreadsheet to keep track of usernames, password rules, etc. on a per site basis (i.e. "bankofamerica, ythn.smith@gmail.com, 4-12 characters - no special).
Edit: downvote away, but so far no one has ever been able to give a compelling reason why algorithmic passwords are bad. It has tradeoffs, sure, but so do password managers.
If a couple of your algorithmic passwords get leaked, attached to same email, then the algo could be guessed and you potentially lost all your accounts.
That's a lot of unlikely ifs. And that also assumes my algo is easy to reverse from 2 digests. My counter to that is that it is unlikely anyone would specifically target me and waste resources trying to crack my algorithm when there is so much low hanging fruit elsewhere. If I am specifically being targeted, I've probably already lost, even if I use a PW manager.
Compare to PW managers: "If your password db and master password get leaked, attached to the same email, then you lost all your accounts." Also with PW manager, losing your phone can mean Denial-of-Service to all of your accounts if you are i.e. travelling.
Other methods being poorer doesn't make your method objectively better. You're probably right on low-hanging fruit, all depends on your threat model I suppose.
Don't password managers have 2FA, and alternate phone numbers?
There is no method that is objectively best. Every single current method has tradeoffs. Being cumbersome to setup/use/recover is a big tradeoff of PW Managers.
I don't trust myself enough to keep track of a local database of passwords, because of that for years I sticked to the tactic of repeating the same 2~3 passwords over and over on all websites.
Now I'm using https://lesspass.com/, a deterministic password generator that requires no setup besides installing a browser extension[1] and doesn't requires me to keep a database or run any other custom programs.
[1]: There's the official extension for all browsers, but I've developed my own, https://lesspass.alhur.es/, which solves 99% of the problems associated with the whole LessPass idea.
I've been trialing it recently on sites of lesser importance, and in the configuration OP described, it doesn't - everything is deterministic from login+site+master password.
It does however you set password profiles, which alter how the password is generated. Here's an abridged example from their FAQ[0] with my notes about what each option does:
"login": "contact@lesspass.com", # User name
"site": "example.org", # Domain name
"lowercase": false, # Site accepts lower case characters (Default: true)
"uppercase": false, # Site accepts upper case characters (Default: true)
"symbols": false, # Site accepts symbols (Default: true)
"numbers": true, # Site accepts numbers (Default: true)
"counter": 1, # Increment e.g. on compromise and the generated password is changed (Default: 1)
# Password entropy is derived from pbkdf2, counter is used as part of the salt
"length": 8, # Password length to be generated (Default: 16)
In the default configuration, these profiles are stored in the browser's local storage, so lost if/when that is cleared. LessPass provide a service to store these profiles on their server (log-in is generated from the master password using a default password profile), and provide the scripts (Docker) that let you self-host this data (the browser extensions include a field to set server domain).
For me, it's a useful balance of convenience and security (each site gets a strong unique password, but I can regenerated them on devices where I haven't synced a password DB) vs. KeyPass which I was previously using.
I'm probably going to switch to 1Password thought - it gives the same "any device" benefits, has basically the same risk model (compromise of the master password is a full-compromise, and because I need the synced profiles I need to put _some_ trust in a third-party anyway[1]), but it gives me the full flexibility of a real password manager (store associated data, credit cards, set specific passwords if needed etc.)
[0] https://github.com/lesspass/lesspass/wiki/FAQ
[1] 1Password: need to trust their client isn't malicious/capturing the password at login and is secure; LessPass: need to trust the webclient is never changed to exfil. the master password, (theoretical?) risk of brute-forcing master password from a generated password, and sends sites/username metadata to a largely unknown third party (could run my own server, but it would likely get neglected and fall behind on security updates etc.).
It's written in Rust. Install the Rust toolchain installer from https://rustup.rs/
curl https://sh.rustup.rs -sSf | sh
And remember to add ~/.cargo/bin to your PATH.
Then install my command-line utility
cargo install pgen
Usage is described in detail in the README on GitHub. Additionally you can ask the program itself for a brief help summary.
pgen --help
Eventually pgen will be available in some package manager repos so that you can use your preferred package manager to install pgen but until then it must be built from source following the steps above.
Mine makes use of a friendly wordlist so that is my selling point, though I forgot to mention that because it already says so in my README but I should have said it in my comment too.
Picking the bit you like makes it hard to reason about how secure that is. It could be your preferences are much narrower than you realise. If you're going for something you find aesthetically pleasing or memorable then that's probably going to seriously limit password entropy. On the other hand if you're trying to pick something that "looks random" then you should know humans are terrible at that.
20 perfectly random characters from a base64 stream have 20*6 = 120 bits of entropy. If you pick 20 characters from out of 1000 choices you can see on the screen, that would seem to remove at most 10 bits of entropy, no? (2^10 ~~ 1000)
"picked a bit I liked" means selecting a length suitable for the site without any weird characters that the site will reject, not trying to find my place of birth in the output!
I dunno, I wouldn't discount this entirely. An advanced targeted attack could likely profile a user for their "likes" in a random string (e.g., favorite letters appearing often), but in the case of anonymous, brute-force-style attacks, "I just click things and scroll around until I see something I like" is probably really useful. It could, for example, protect against a somewhat predictable generator, since you won't know "the user always uses the first password, and the first password is always seeded incorrectly", or some other specific implementation flaw.
I do similar filtering to :graph:, or :alnum:, and dropping some characters.
What I keep meaning to do is optimise for different entry mechanisms, I know it reduces strength but going from lower to UPPER to specials to UPPER to lower to additional-specials, etc., is a huge pain on a 5-key entry system (eg FireTV).
FWIW, 'cargo install' is designed to be for distributing dev-tools, rather than for end-user tools. I believe https://github.com/japaric/trust is designed to make it very easy to publish Github Releases from CI systems.
I don't want to be distributing pre-compiled binaries because I don't want others to trust random pre-compiled binaries.
The plan as I said is to get pgen into some package manager repos. Until then building from source will be the only way I will encourage anyone to get a copy of my tool.
I don't think encouraging people to download the precompiled Rust binaries and then blindly compile code downloaded from crates.io is that much different in practice to providing your binaries.
If you're trying to encourage people to implicitly trust as few things as possible, I'm not sure your suggested steps are enough: there's no connection between the code in pgen on crates.io and that repository. Even setting aside the prebuilt rust and cargo, I think there would have to be a guarantee that the code being built is the code the suspicious user actually inspected:
git clone https://github.com/ctsrc/pgen
cd pgen
less src/main.rs build.rs # etc.
cargo install # (installs the current package)
And, one would have to somehow do the same for the dependencies clap and rand, to ensure the code that is built is the code that is inspected.
It's true that avoiding pre-built binaries does avoid issues with the computer that builds them, and problems with the distribution mechanism, but instead distributing as source from external package repositories (and packages maintained by others) seems like it's losing convenience without gaining much security.
In any case, neat project! I like the option to throw physical dice.
That looks like a decent tool but I wonder about its use case. The passwords are far too long to be entered on anything without a dedicated keyboard and you'd not want to be typing them in too often.
Sure, there are some use cases that fit but there's many more that don't.
Rust is still undergoing changes frequently enough that most package manager repos have a very old version of the Rust toolchain in terms of what it is capable of doing.
For example the version of rustc that you get from Ubuntu default repositories was too old to compile my pgen when I checked some weeks ago.
And exactly because the tool is for generating passwords I don't want to distribute pre-compiled binaries of my tool myself, and therefore until I get pgen itself into package manager repos I tell people to download the Rust toolchain and to build my tool from source themselves as I did above.
You wouldn't manage that with a one-line shell script, assuming that you want to format it reasonably. :)
I do agree that having to install loads of things for a simple tool is overkill, but I'd wager the actual binary produced doesn't have many dependencies (I'd expect just libc, in fact); so would this at some point land in a package manager, your life will improve.
I wrote this as a replacement for what was previously a function in my .bashrc because of two things:
1) My shell function was too slow. There was a noticeable delay and it became annoying. The tool I wrote in Rust is insanely fast. Life improved!
2) The wordlist I used to use would use /usr/share/dict/words. These words are not good for typing because there are a lot of weird and arcane words in that list. The new tool I wrote has an optimized wordlist made by the EFF (read about it in the README) compiled right into it.
Regarding your question about installing other dependencies, like I said I will eventually get it into package managers. The pgen utility is a single binary. My assumption until then is that the people interested in my tool also happen to be interested in Rust and that therefore installing the Rust toolchain to use my tool will also give them motivation to get back into learning Rust like they at some point started doing. This was intentionally unstated but now you made me say it :)
