For my private keys, I export them into PEM format, convert that into a QR code, and print them out. This goes in my fire-resistant Fort Knox vault [1] for safe keeping. The vault is bolted to the cement floor and cannot be carted away.
I use intermediate certs for my internal CA and do keep those private keys online for quick and easy generation of new certs but the root cert private key only exists in physical form under my control. It's a bit of a pain to convert the root key back into digital form but I only have to do this every couple of years when the intermediate cert expires.
I got into a habit of writing shell scripts for every certificate related task to make it easier. I use the dialog(1) utility to prompt me for input when that's needed. If things are easy, you're more likely to do them.
How do you check that your printer isn't secretly storing your key somewhere? I've looked for open source/open hardware printers some time ago, unsuccessfully. I've written my main key down on paper by hand as a backup, but it's a tedious task.
As you mentioned fire-resistance, do you also use some sort of fire resistant paper? Because I don't think anything in made of paper would survive an actual fire.
> I've looked for open source/open hardware printers some time ago
You want a pen plotter. Or its cheaper, more expensive cousin, the Axidraw.
> As you mentioned fire-resistance, do you also use some sort of fire resistant paper? Because I don't think anything in made of paper would survive an actual fire.
Actually there are document safes with built-in insulation which will keep regular paper documents safe for time X under fire conditions Y (it's some EU standard classification). Certain ways of mounting safes makes them intrinsically fire-safe(r) as well.
Vintage pen plotters (or X-Y recorders) are essentially zero-price items and tend to be built like actual tanks. The ones you can buy today (and afford) are way more expensive than them, yet cheaper (in quality). However, vintage pen plotters tend to not have USB plugs.
Speaking of plotters, consider engraving the QR code in aluminum instead of having it on paper. If there's any hacker space nearby where you live they might have CNC equipment that they will let you borrow and instruct you how to use.
> I've written my main key down on paper by hand as a backup, but it's a tedious task.
Elliptic curves are nice because they allow short keys, for example curve25519 uses 32 byte keys that are pretty reasonable to copy manually (44 base64 chars or 64 hexs).
Meaning you could use PBKDF2 of your favorite passphrase as your private key, then keep it in organic memory.
Personally, I would love to see a strategy for generating a private from an analog picture.. But I don't see a good way to do it reliably.. Maybe one could generate it from error correction bits in a QR code.
I'm not sure you can use a KDF because it requires a salt.
I'm working on a blog post on deterministic passwords (i.e. not keys for ECC), but I guess that if you really want determinism you can do something similar, maybe sha3.512 and "cut" a 256-bit ECC key.
> Personally, I would love to see a strategy for generating a private from an analog picture.. But I don't see a good way to do it reliably.. Maybe one could generate it from error correction bits in a QR code
Not clear exactly what you meant, but but deriving a key from a photograph shouldn't be that difficult. A good perceptual hash should work, maybe some preprocessing is needed (straighten/crop/normalize/resize). It helps that the amount of data in an image is absolutely massive compared to typical key size.
Prime number generation is typically done like this. Pick a random number, then add 2 until you get a prime.
So you can do something similar. Pick a deterministically generated bit string, and use as key the next prime number (you need to do it twice, for p and q).
You certainly get less security and I unfortunately don't have any reference to quantify that.
Even if there is, such a key would be much less safe (assuming the attacker knew the algorithm, which must be assumed, otherwise why keep the key in a safe?)
Of course, because that drastically reduces the key space.
Before: 2^2048, After: 256^20 = 2^160 (when using a 20 key password. Obviously you could use looong random generated passphrases, but then you could just base64encode your key?)
What's the point of encrypting it? It seems like just another thing to mess up, and much more likely that you'll forget the password versus someone breaks into your house, goes into your safe, and knows what a private key is.
What's the point of storing data inside a combination safe? Isn't that equivalent to encrypting the data with the combination to the safe and storing it wherever?
Love that vault suggestion, but for a lot of folks, a safety deposit box might be sufficient (with your own enclosure within it using tamper evident stickers).
Having moved 5+ times in the last decade, I'd dread thinking about having to move that safe you linked to.
We are a military family so we typically move every two or three years. My vault weighs about 900 pounds but the movers never have any trouble with it. In fact, the guy that installed it was an old man driving a pick up truck. He used an amazing motorized dolly that had no problem pushing it around and could climb stairs.
My vault is bolted 4" into reinforced concrete using bolts that are only accessible from inside the vault. Unless you have the combination to open the vault first, you're going to have to rip out the slab to get this thing out of my garage.
For your PEM formatted keys, why a vault? Don't vaults just advertise "what is in here is SUPER important"? Police love vaults. Wouldn't it be safer on a sd hiding case in a few places (mom's house in attic, one under your oven?)
I should have explained this better: the vault is not to keep the private key safe from home intruders, it’s to keep the key safe from loss and fire and anyone that might compromise my CA machine. It acts as an air gap of sorts.
Once in a blue moon when the intermediate cert expires, I pull out the QR code and scan it and convert it to a PEM on a burner laptop, VM, or phone. There I generate a new intermediate and securely delete the root PEM file and put the QR back in the vault. Then I copy the inermediate off the burner machine and go about my business.
Glue the QR code onto an old boarding pass and keep it in your photo album... then stick a broken yubikey into your grandmothers coffin for plausible deniability.
- the latter is a joke ofcourse! :)
For plausible deniability it's a lot less controversial to mix a broken yubikey in concrete and cast it as pavement.
I have tried printing QR code of keys but with no success (the phone does not recognise the image) what do you use to generate the image that is sufficiently resolved to print and be functional ?
"Optar stands for OPTical ARchiver. It's a codec for encoding data on paper or free software 2D barcode in other words. Optar fits 200kB on an A4 page, then you print it with a laser printer. If you want to read the recording, scan it with a scanner and feed into the decoder program. A practical level of reliability is ensured using forward error correction code (FEC). Automated processing of page batches facilitates storage of files larger than 200kB."
Do QR codes provide any error correction, or is the a crease, fold or spec of dirt on the paper you printed on going to render your QR-encoded key unreadable?
GPG keys are not that long, maybe 2 pages (ascii armored). Just print the private key out and store that in a safe or deposit box. You can physically see the key, so you know the backup worked.
If you ever need to restore it, yes it will take ~45 minutes to hand type the key back into a new computer. Oh well, that's the tax you pay for restoring your backup or making a new subkey.
Civilization has long figured out how to store paper. I've grown to respect how powerful a printer and some patience can be for key backups.
EDIT: I think most of the replies are missing the point. I want to keep things barebones. The only thing that should be required to recover is a copy of GPG. No qr decoder, script, or other nonsense. If you depend on something else for recovery, you are putting your trust in that. I trust GPG, paper, and my eyeballs.
I've been working, on and off, on a little side project to solve this exact issue.
It's just a python script that converts a piece of text of arbitrary length, into a series of QR codes.
I then plan to work on the "decoder" side, which takes a series of QR codes and outputs the original text. Basically just a concat of the QRs.
The idea is that I'll print my private keys in both text and QRs so that I can easily recover the digital form if needed (and if that doesn't work, then I'll type it... Ugh).
Hopefully I'll be able to finish it and do a Show HN sometime this year.. (or maybe the next).
So I'm guessing there's definitely some need for a tool like this, right? I mean, being able to "QR-codify" any text and later getting it back could simplify some of these offline-storage actions, right?
On several separate occasions I tried to use Tesseract to OCR base64 and similar text that was printed in a normal monospace font (i.e. not a special OCR font) and scanned.
I never got even close to getting useful results. I tried limiting the alphabet, disabling language models as far as I could and at most I could get a few recognizable character sequences right out of the whole page. I got the impression that the whole thing very much depends on being able to split text into English words and have easily separated paragraphs.
When you print it, you can add a column for each line with a few characters of the hash of that line for quicker location of typos when typing it in (provided that you write a script that does the check).
As others already hinted at in the lwn comments, a keycard without an external keypad or confirmation system gives you a lot less protection than you probably expect: usually your computer will be no more protected from keyloggers able to record that pin and from the execution of requests to the keycard then from the theft of locally stored private keys. Actually in what's probably still most cases a keylogger will be trivial to set up (https://theinvisiblethings.blogspot.com/2011/04/linux-securi...).
A keycard will avoid (with very high probability) the theft of the private keys, but in most cases this is not that important, if an attacker can do individual sign/decrypt/encrypt operations on demand it will be more than enough for him.
Unfortunately for some reason it looks like almost all the recent affordable key-handling devices lack an external keypad. There's either some misconception about their security or some wilful scheme to give a false sense of security to the people who use them (I wrote it tongue-in-cheek but actually we've seen worse...).
Also, it's not true that "a key generated on a keycard cannot be backed up", most devices permit the export and re-import of keys by first encrypting them with a key known only by the device (and not exportable); these will effectively be back-ups, although they're only useful against the accidental deletion of the keys, not against the loss/damage of the device.
By the way, this feature is also notable for being the source of vulnerabilities in many devices that made it possible to extract the raw keys.
P.S. I don't know why the author uses that unusual "keycard" term, I assumed he means smartcards or dongles
Probably because smart cards as they exist today are intended as end-user credentials to be issued by an institution, not secure key storage for individuals. They’re meant to authenticate to enterprise managed endpoints. The vendors seem to treat HSMs for key management separately.
If you're speaking strictly of smartcards (in the form of cards) you're probably right, but I meant all devices with cryptoprocessors, especially usb tokens.
For what I can remember when I looked into them, a few months ago, keys/data storage, signing and decrytpion where at least as much advertised as authentication.
And in most cases external confirmation is useful/strongly advised for authentication as well.
Case in point, I personally needed a token for code signing, it's almost impossible to find one with a pinpad.
It's easier if you settle on a smartcard + smartcard reader (which I don't think many developers do).
Yubikeys carries a single button, so you can configure them to require a manual touch for each operation.
It's not perfect, but security is always a balance between the risks involved in practical solutions and then risk that users side-step convoluted solutions :)
I was just (re-thinking) about this at work.. Just bought a Yubikey and was basically going to start from scratch with my PGP setup.
I really don't like the idea of storing anything really critical on a usb drive or an airgapped system.
I don't have an airgapped computer just laying around that I can store secrets on (and keep alive), and I don't trust a usb drive to last.
I really wish there was something like a clean way to store an encrypted printout that could be scanned years later if neccesary, ie a method of storage that I actually have faith would reliably survive for a decade or more.
The issues don't seem too problematic and looking at the code it's just wrapping qrencode. So it should be trivial to fix the three issues mentioned. I'd just halve the chunk size and post-process the generated QR codes manually in a image editing program to put multiple of them on the same piece of paper, including some textual information. Then laminate that.
I'll actually be doing this soon I think. Previously I used http://www.jabberwocky.com/software/paperkey/ which is quite good, but it's like two pieces of A4 filled with text which would be a huge pain to restore.
I was quite surprised how much text you can fit into a single QR code nowadays. 2778 bytes of lorem ipsum generates a massive code (1000x1000), but it can still be read passively with an iPhone's camera. For comparison, a Google CA cert is only 1363 bytes. You could theoretically dump your cert and private key onto their own (big) single page codes, then just scan them back in when you later need it. That just leaves the possibility of your printer holding onto something it shouldn't.
Why does it need to be scanned years later? I suppose if you are mitigating the risk of going to prison or something like that, then it might make sense.
But either way, you can print to paper and store it in a fireproof safe (or probably better still -- a safety deposit box). There are lots of methods for ensuring printed paper survives for a long time -- if you google it, I'm sure you'll find more than you want to know.
My personal method is storing passphrase encrypted on multiple USB drives (they're cheap) and replacing them every year or so (they're cheap).
I think a more interesting question is: how do I provide access to my non-technical wife in case I am incapacitated or dead? Especially, how do I convince her not to put the passphrase on a sticky note on the fridge?
Encryption. If you encrypt files with the public key you are going to want to keep private key around long after the signature key is expired. The alternative is to decrypt and re-encrypt your files when you rotate keys out, but there is always a worry of forgetting something.
What about actual punch cards? Maybe not the most efficient given the lack of equipment but should not be harder to print or punch than QR. And standards were around for much longer than QR or DM so you can be sure to find some designs online or in a library if you lose all of of your equipment in 10-50 years (and they can always be scanned or read by eye if everything else fails). I'm looking for a simple design for a RS232 puncher, preferably one that would support some non-biodegradable plastic cards (they say PET bottles do not degrade in centuries, ain't that a perfect backup material?) But even paper cards should be more durable than anything printed on laser printer.
Using IBM 80-column cards you'd need just about 10 cards to store a 4096-bit key with three subkeys in binary mode. More if using char mode (might be preferred if you have to type them using keyboard). The key can be condensed for backup if needed. OTP and/or Shamir's method could be added to the mix to improve security, increasing the number of cards required, but even without Shamir you can split the deck in half and store in different places.
Longer, possibly foldable, punch tape is yet another option. If it has the same width as IBM 80-column cards then most of DYI equipment designed for the latter can still be used. It can't be metal though, only paper or plastic.
A bit of obscurity helps too, it's not like a deck of cards screams "this is an important secret" or possible attackers have card scanners with them when they invade your backup facility (although they can make photos and decipher them later so it's not real protection, just a small bonus.)
Now, how would we solve a problem of rubber-hose cryptoanalysis with it?
An air-gapped system is really just for key generation (or generating new subkeys). You really shouldn't store your key on the air-gapped system when you are done.
With that script, a 2048 bits key in PEM is about 4 pages (+ 5 others if I print it in text format), which I can store in a safe, and conveniently rescan using zbarcam anytime.
There are fairly affordable FIPS 140-2 Level 2 USB HSMs.
Kingston makes a fairly decent one. It's how we manage our root of trust right now. Two of those with root identity and reciprocally signed exec identities. All of the artifacts stored in git RSL repos on the HSM, the two HSMs sync'd via signed commits and merges so we have an audit trail, one HSM is stored on-site and the other off-site, and one can be checked against the other to measure for tampering. All of the initial provisioning happens on an air-gapped machine with intermediate artifacts that only live in a temporary RAM disk that itself is encrypted with a 4096 byte key that is never known to anyone (it's fed straight into the ecrypt tooling and discarded).
The next layer out from that is all Yubikey based.
It's an extremely cumbersome process to do normally, but we invested a fair bit of time in creating automated key ceremonies of different shapes to handle different parts of the process.
Nope, you're absolutely right. We'd just adopted the colloquialism internally compared to the Yubikeys, and I'd lost context for the whole purpose of our key ceremonies was originally to be able to treat those IronKey devices tacitly like HSMs.
As the other users said if you're using one the drives listed at https://www.kingston.com/en/usb/encrypted_security they seem to be just "encrypted drives", without any support for internal signing operations.
If that's so you're doing all the key operations on the computers at which they're connected, and you've come up with an extremely cumbersome process that gives you very little protection.
Adding to the insult, there actually exist many cheap USB proper cryptographic tokens, that really do all the operations internally and have even FIPS 140 level 3 certifications. They are just slower and have less physical protection, features and storage than "proper" thousands-dollars HSMs.
Note that the FIPS certification of those Kingston drives means just that the key with which the data is encrypted cannot (easily) be extracted.
I stumbled again at this post; "the key cannot be extracted"???
Why on earth would a portable drive need to store the decryption key for its data, however protected??
Actually I've never looked before into "encrypted drives" and I don't know what do they do or what purpose exactly do they serve, I must have written that thinking about HSMs and other signing devices.
It does seem indeed that at least those Kingston drives do not store the encryption key (actually, they store it but encrypted with the password).
I'm not sure what purpose do the FIPS certification and all those physical protections serve, then.
Yeah, I don't think those are really HSMs. Just a very secure USB stick.
The whole idea behind an actual HSM is that you never trust any computer it is plugged into, it is designed to be impossible to retrieve the key stored in the device via any means. All signing operations are done internally in the device itself - you simply can't mess up and accidentally leak key information.
> I have personally discarded that approach because I feel air-gapped systems provide a false sense of security: data eventually does need to come in and out of the system, somehow, even if only to propagate signatures out of the system, which exposes the system to attacks.
This need not be a problem if the machines are truly air-gapped, and all interactions are conducted by typing & reading the screen. This is impractical for OpenPGP or X.509 keys & certificates, and generally for any system using RSA, but it's quite practical for SPKI certificates[0] and systems using Ed25519.
Updates aren't nearly so important with an air-gapped system: after all, there are no network attacks to worry about, and physical attacks can be dealt with physically.
Note that a truly airgapped machine does not have data transferred to or from it, even via a USB key. The whole point of an air gap is an air … gap.
Typing data in is not inherently different from transferring it on a USB drive. Your "truly airgapped" machine is still parsing input from an untrusted and potentially malicious source. Yes, you're protected against malicious filesystems, but I'm a lot more worried about buffer overflows in the S-expression parser and SPKI interpreter that you're typing into than e.g. the ext2 or FAT implementation in the Linux kernel, simply because no SPKI implementation has been subject to nearly the same level of attack or scrutiny as Linux kernel filesystem drivers.
I think you could probably do it for something like Ed25519, which has the nice property that any 32-bit string is a valid key, so all you have to do is type exactly 64 hex characters. But SPKI seems like a huge attack surface.
You would not accidentally type, with your own hands, some string that would attack the parser. People do not actually name their children Robert"); drop table.
An SPKI parser + a manual keyboard operated by a trusted user is an incredibly small attack surface.
Ok but it is a lot more "gapped".
A buffer overflow vulnerability is a lot more unlikely to be exploitable if you can't extend arbitrarily the amount of data passed and you have very little room to manipulate it (and it is so if the target has at least a vague knowledge of the format of the data).
The process is surprisingly involved, and there's a lot of opportunity for error given gpg's dreadful user interface (not saving to keep the key on both the host and the keycard? wtf?).
I think there would be a lot of value in creating tooling around the "best-practice" setup, even if it's just wrappers around gpg commands. Scripts to setup the main key & subkeys, revocation certs, smartcards and backups, all that jazz. A Raspberry Pi image to act as an "air-gapped" computer for sensitive operations might also be part of this.
I recently did my GPG setup.
For the day to day operations, I have subkeys stored a yubikey.
For the storage and backup, I use the nitrokey storage (https://www.nitrokey.com/), open hardware, open source usb key with hardware encryption. It's compatible with all os and there is a non encrypted partition (on which I put the app needed for main OSes)
As a bonus, the nitrokey is also a smartcard openpgp (so I have a smartcard backup) and also has slots for 2-FA (So I have a backup of that)
I plan to buy a second one that I would store outside of my home.
If you're speaking of something that you have to actively use, that in order to use it your computer will have to decrypt it, rendering it vulnerable to all attacks that a locally stored key would be.
I'm a fan of the Apricorn Aegis encrypted USB drives (FIPS 140-2). In assessing the risks around critical offline storage, my primary concerns are that the data has an adequate backup, that no one vendor is entrusted solely with data protection, and low barrier of accessibility as that the solution will actually get used.
The 8GB Aegis drives are around $80. Unlock is performed via PIN entry. The drives are small and have a sliding case to protect the PIN pad, making them pocketable. The hardware is capable of wipe upon failed unlock attempts. Pairing these drives with a software-encrypted filesystem reduces the likelihood that a single-vendor fault could allow bypass and data access. This is a strong option for primary always-on-hand instances of offline data, which could be paired with some other secondary backup option from another vendor (like paper in a safe, HSM or additional encrypted USB keys).
It's a shame that keysafe[1] wasn't mentioned. It's a fairly interesting project, where it splits up the key into separate parts (using Shamir secret sharing) and it's encrypted and sent to separate servers. Brute forcing is quite hard because even with the right password it takes more than an hour to recover your key (due to the difficulty chosen for the Argon2 hash). Joey Hess gave a talk about it at LCA[2].
I use SSSS to store physical copies of the key I use to encrypt my backups with ssss http://point-at-infinity.org/ssss/ . Basically print a bunch of copies and give them to people I trust.
I use a Trezor to store my PGP key. I never have access to the private key, it's generated by a seed which I backup in a cryptosteel protected by a password in my brain (to prevent physical theft). The device only provides an interface to sign and decrypt. This keeps the key safe even if your computer is compromised. See: https://github.com/romanz/trezor-agent
all of these need a false/fake entry so when adversaries compel information and you give it to them the result is useless and they have "the truth" and nothing at the same time.
You can do this with Trezor. It allows you to enter a 25th word to your seed when you unlock it (it looks like you're just entering your password, but you could enter any password), generating a whole new set of private keys. Although this makes less sense in the context of a PGP key...
I typically create a new key pair specifically for managing my keys, and when I want to store or transport a key I encrypt it with the pubkey of my 'manager' key pair, so that if the process is interrupted somehow I can revoke the manager key and fail closed. And then for offline I would just print out the encrypted key.
It's easier to find smartcard readers with keypads, and use a smartcard. For example see here: https://en.cryptoshop.com/products.html. There are also keyboards (e.g. from Cherry) with an integrated smartcard reader.
Note that I haven't tried any of them.
But really it's largely enough to have a confirmation system, and that is supported also by the YubiKey 4, for example.
Side question about LWN, which maybe someone would have some opinion on.
I don't read a lot of LWN, but I LOVE the work they're doing. I maybe read an article a month, maybe even less, but I think that their model is great and appreciate their work. I used to subscribe at the regular rate, but never read LWN much and let it lapse when it was up for renewal and my history showed that I hadn't visited in 3+ months.
Would it be frowned upon to buy the starving hacker subscription, even if I am probably not a starving hacker, given that I rarely read it and just want to support them? $42/year just seems easier to stomach for something I know I'll never use, than $84. First world problem, I know, but curious what others think.
IMHO, better something than nothing, and marketing names should be treated only as suggestions. Even if you haven't read in a long time, aren't currently subscribed, and see an article you find super valuable, it's a good reminder to chip in something.
The LWN model seems an awful lot like public radio in the US, but without the annoying pledge drives. Like you, I both read and subscribe in bursts of activity and inactivity. If it's something that you feel is valuable, and needs support, don't let the names get in the way of donating at the level that you feel you use it.
This was my thoughts exactly, but felt a slight bit guilty, and wanted some external validation. :)
As a note to anyone reading this, as a former LWN subscriber a couple years ago, and as someone who will probably renew at the $40/year level here today again, definitely consider throwing them a subscription. It's one of the best tech publications on the web.
I think that giving them something is better than giving them nothing. I doubt the intention of the "starving hacker" level is to only apply for hackers that are literally starving, more just for people who cannot afford it or who feel that higher levels are too much.
I use intermediate certs for my internal CA and do keep those private keys online for quick and easy generation of new certs but the root cert private key only exists in physical form under my control. It's a bit of a pain to convert the root key back into digital form but I only have to do this every couple of years when the intermediate cert expires.
I got into a habit of writing shell scripts for every certificate related task to make it easier. I use the dialog(1) utility to prompt me for input when that's needed. If things are easy, you're more likely to do them.
[1] https://www.ftknox.com/vaults/legend-vault/