Most people are focusing on modern reasons. The historical reason is that when the internet was being developed, public key encryption hadn't been invented yet. The 1960s and early 1970s was when the internet was created and it was only until the mid 1970s that public key encryption came about. The problems of secure key exchange hadn't been solved at the time. Even if those problems had been solved, clock speeds were measured in megahertz [edit: sometimes kilohertz]; every clock cycle mattered.
Fast forward to today and this legacy has remained. Many protocols are being retrofitted to support encryption, but it's similar to why railroad tracks are the width they are: Roman chariots used the same width.
Additionally, don't discount developer laziness. It's much easier to debug a plaintext stream than an encrypted one.
Read the Snopes article in it's entirety. It's still "technically true;" it just didn't happen with deliberate or accidental intent. "Inertia" is the perfect word in such a case.
Putting aside the whole issue of delegating trust via certificates, the simpler answer is that encryption is hard, and thereby expensive. For many purposes, both service provider and client continue to consider it an unnecessary expense.
Only recently, with the spread of wireless networks, has the threat of en masse MITM and eavesdropping attacks become realistic. For a long time, it took serious knowledge to pull these things off on a wired LAN or within the walls of an ISP.
You could say the same about phone calls--why isn't the POTS network encrypted?--but that's because it's well constrained within wires run by trusted people and such cost is not justifiable; but with the advent of cell phones, protocols like GSM had encryption built into them, since telcos couldn't ignore that data sent over the air is readily captured and manipulated.
Oh, I wouldn't say it took expert knowledge in the past. I've been able to download ettercap for years. It'll sniff switched networks. I just need to plug into your LAN.
>why isn't the POTS network encrypted?
Because I can't pick up the receiver and listen to my neighbor's calls. That's exactly how ethernet doesn't work. I hear all, even on switched networks with a simple tool. Wireless? Even worse, it's the equivalent of using a non-switched hub. I'll even toss in that POTS should be encrypted point to point, but if you would have attempted it before Phil Zimmerman trail-blazed strong encryption for us commoners, you would most likely have ended up in a federal prison if you used non-trivial encryption.
So, to address the original question:
1. Encryption really was expensive in 1996 or so, but in the age of multicores, its questionable if there's a real cost now.
2. Performance hit. SSL handshake is long and annoying. We could use a replacement here.
3. Legacy. The hackers who slapped together networks and the various protocols we use weren't interested or couldn't properly secure this stuff. Thus the legacy of spam, plaintext everything, etc. Retro fitting security is hard. Its good if its there from the start like GSM.
4. Standards are hard to change. Imagine telling all the browser makers and open source projects that we should all switch to something that won't add to the bottom lines/download rate/market penetration, but instead cost them time and complexity as well as add server load. Google can't get anyone to care about SPDY. Who is going to re-architect this stuff? We're essentially living in a world convcieved in the 70s, implemented in the 80s, and made popular in the 90s.
The easy answer is to stop allowing non-encrypted wifi. I'd love to see the wifi consortium just default to a random key for non-authenticated guests in whatever the newest version of wifi will be. It'll be transparent to the end user and probably stop all these local attacks. It won't help you past the gateway, but right now between the client and the gateway is the problem area. Heck, why not make all networking gear do this, even on wired networks? We have the processing power.
> 1. Encryption really was expensive in 1996 or so, but in the age of multicores, its questionable if there's a real cost now.
A lot of "budget-priced" hosting services are near-free for http and heavily-surcharged for https. - Pricing it to make it seem as if there were some huge cost difference. Makes a difference when shoestring clubs/nonprofits want to throw up a cheap static site.
That's because SSL is a shitty protocol, it has nothing to do with CPU.
Turns out that you can only have one SSL applied to one IP because browsers* can't tell the webserver hosting 1,000 sites which particular site you're requesting, because of, you guessed it, encryption.
Essentially, you're paying for a dedicated IP not CPU.
*theres a fix for this supported by several browsers, but it will never be backported to legacy browsers and the millions who won't upgrade for many years so its unsafe to assume you can use this method, thus one IP per SSL for the foreseeable future.
The problem is really only with IE on XP; of course, that's still a huge share. We can thank MS for that, since if other browsers support in on XP, there's no reason IE couldn't, especially considering 7+ already does in Vista and 7.
The problem with that is that systems like Skype require strong encryption to work at all, and Skype manages to provide high quality voice transmission despite using encryption on a wide variety of OSs and hardware.
It will probably be more technically difficult to make Skype friendly to snooping than it was to make it secure.
WiFi traffic has nothing to do with internet traffic per se. The problem is that the only popular WiFi protocol that is anonymous (doesn't require passwords/user credentials) is also not encrypted. This problem could elegantly be solved with a ssh-like tunnel between the consumer device and the WiFi router, the rest of the traffic could still be plain text.
The user's device would somehow have to be sure it's talking to the legitimate router, and not one an attacker set up with a deceptively similar SSID. I can't think of a way this would be possible, unless popular access points were signed by CAs.
This is the classic excuse to not implement encryption: "but we need a web of trust and certificate authorities and central management of identities and a million complex things that'll never work right even if implemented."
Encryption without identity validation is a worthy goal in itself, especially when we're talking consumer grade stuff.
That's really just another way of saying that users shouldn't connect to routers they do not trust.
There's this new feature of SSH key generators, where the user is shown an ASCII art of the generated SSH key, to be able to visually remember and recognize it. A standardized graphical representation of certificates could be used to confirm the identity of the router. Imagine walking into Starbucks or any WiFi enabled place, you see a sign on the wall that gives you the network name and the "VisiSign", you connect to the router, your computer shows you the VisiSign of the router, and you can visually confirm the identity of the router.
I think it's legitimate to want encryption even if you can't verify identity. Man in the middle may be trivial in that case, but it's still harder than just sniffing packets.
The problem here is that encryption is an overloaded term. There are two different things happening when one encrypts, and both are useful independently.
The two parts are identity and obfuscation.
Obfuscation protects you from listeners who do not control the network, and identity protects you from listeners who do control the network.
Together, you know only your intended recipient is getting your message, but apart, at least you are protected from certain threats.
So even if we just had obfuscation, that would protect us from most threats. When people say "we need encryption everywhere", I think generally what they mean is "we need obfuscation everywhere" to protect from the worst threats, which are those of the uncontrolling listener.
The flip side is that the uninformed will get complacent and assume that obfuscation is giving them identity protection as well.
If we're going to overhaul the entire structure of the internet, it wont be because of coffee shop eavesdroppers that haven't learned how to inject traffic.
I think that's it in a nutshell. When we talk at the level of the design of protocols, we generally concede everything we can reasonably concede to the attacker, and then design against that well-armed attacker. Denying attackers the ability to redirect or inject traffic ties their hands behind their back.
There's really no such thing as an "purely passive" attacker. Since the 1990s, probably starting with the deployment of switched ethernet, active attacks have gotten steadily easier and more popular, and passive attacks less so.
The frustrating thing here is, we are so close to having this problem licked, but so many smart people are spending time concentrating on problems we don't have. What we need to do is to come up with a credible replacement for the Mozilla/Microsoft/Google-controlled CA system. It's probably no harder to solve that problem than it is to push adoption of weak unauthenticated protocols.
I assume by obfuscation you mean browsers accepting self-signed certificates in some cases. To do that would require everyone to be using a browser that accepts them, nearly all don't. On the other hand, StartSSL is free and already accepted in some browsers. It seems more promising that eventually nearly all browsers in popular use will have StartCom as a root CA.
Of course, there are other problems preventing small and large sites from using SSL, like the requirement of a dedicated IP address (which SNI fixed, but is again not supported in all popular browsers) and the overhead.
The technical issues are real and deep but they can, were, and continue to be tackled. The bigger issue is that successfully addressing and deploying robust encryption solutions is not a political priority (quite the opposite).
Keep well in mind that for many online in the 90s, the assumption was that eventually standards for email encryption, as one example, would become well deployed. The fact that email is today routinely not encrypted (disregard ssl connections for the moment, a whole different barrel of fish) is a demonstration that interested power structures (governments, US primarily) were successful in limiting uptake.
There are real challenges to successful distributed crypto, but the political forces in play had the effect of early work withering on the vine for the most part.
I hear it all the time, but I think this is a pretty silly sentiment.
Nobody wants the entire Internet encrypted more than the giant banks that Redditors believe control the political system.
What malign forces do you believe the government is actually wielding to retard adoption of crypto? Who controls how much of the Internet is encrypted? Because the Internet is an end-to-end system design, the two parties most responsible for crypto adoption are your browser vendor (which vigorously supports crypto adoption), and the people who run servers (most of whom vigorously support crypto adoption).
The onus is on you to provide evidence to back your silly argument up.
It was absolutely the case in the 1990s that much of the government actively retarded crypto deployment. The whole ITAR/export control thing (40 bit as a compromise) was a HUGE factor in keeping crypto out by default. Then there was the clipper thing and proposals for key escrow. And, A5 on GSM being weak.
Outside of payments, where the government and industry have pushed for strong integrity and authentication (although not really confidentiality), government and big corps have hindered the deployment of crypto.
There has been some improvement in the past decade or so, at least in other regulated areas, both in regulation and in industry self-regulation/compliance standards.
I think the majority of the reason cryptography isn't more widely deployed is that it's 1) hard to do well and 2) most people writing applications have a hard enough time getting a non-encrypted form working and 3) few people make it a requirement as a customer. However, government has definitely hindered ubiquitous crypto deployment.
You are absolutely right that the government had an overt, irrational, and hostile reaction to encryption in the '90s. I dealt with it firsthand writing security code for a Canadian company.
But that was the 1990s. The government does not in 2011 believe you are shipping "munitions" when you allow open downoads of software that incidentally encrypts traffic. People do it all the time now. And, to be fair to the government: nobody saw the mainstream Internet coming, and prior to that, crypto basically was a munition.
I agree with your (1) (2) and (3) reasons. I just don't see anything the government is doing today, or in the last 10 years or so, to hold back an encrypted Internet. The people I know in government who think about this stuff would dearly like to see a more secure Internet.
Indeed, Yahoo, Hotmail, and Gmail could get the ball rolling here completely transparently to users and with relative ease, thereby preventing thousands of phishing attacks in the process. If they did, then webmail would suddenly become useful for talking to banks, stockbrokers, foreign dissidents, avoiding craigslist scams, etc. etc.. But not only have Gmail et al not implemented any features like these, but they've not even uttered a peep as to why.
OK, I'll bite. :) I belong the camp which says Gmail's lack of encryption is "conspicuous".
Look at Ubuntu: they're constantly improving the user experience of encrypted home directories. In the latest release, this is as simple as checking a box during account creation (it creates the keys under the hood and uses your login password as the passphrase for the data key etc).
Also re: gmail has "no secure way". I read the white paper written by your company on the problems with encryption and javascript. AFAIK, if all traffic is exchanged only over SSL, I don't see any reason why encryption can't be done in Gmail in javascript (unless you're going invoke quality of the random number generators of browsers).
Here's an UX I can think of: (1) generate key pairs in the background in the browser for all gmail users. (2) transparently use this to encrypt to users within gmail.
Start from there. Then add the ability to import keys of contacts.
Its feasibility is dubious. Its UX complexity is immense (the comparison to encrypted home directories is tellingly unfair --- if that's the best example of end-user-visible encryption gou can come up with, &c &c). Nobody is asking for it. Few would use it. No other major email vendor does it. You find its absence "conspicuous"?
Hey tptacek, I know you must busy, it's late in the night etc. But it doesn't seem like you really responded to my points: you just seem to be repeating your original assertion that (paraphrasing) "it's too difficult".
What aspect of the UX I described do you think is confusing? In the step 0 that I proposed (encryption only within gmail), there is virtually NO UX.
Wrt "nobody is asking for it". Then why is ubuntu/windows adding the feature to encrypt dirs? Btw, at a company I worked, they were rolling out SSL certificates to sign email. So I don't agree that "nobody is asking for it".
Wrt your point about the comparison being "unfair": I don't think anybody disputes that the "usual" way of doing public-key encryption (rigs of trust etc) is too complex. At the same time, I remember the past when encrypting data dirs was an immense chore. If that UX could be improved, why not the email encryption UX?
Go ahead and work on it! You have my blessing. The problem of encrypting a home directory is nothing like the problem of implementing a secure, persistent, long-term group messaging system that has to scale up to the entire world.
Demand is what's holding back encrypted email. Nobody cares about it.
I think it's actually a chicken-and-egg problem. The people who care a lot about it can't have it because they need to communicate with everyone else, the people who care a little don't want to bother with using encryption for some contacts and no encryption for others, and the people who don't know and/or don't care don't get any of the benefits.
I think two things would make a big difference: 1. marketing end-to-end e-mail encryption as an anti-spam measure, and 2. someone who cares a lot about it implementing an end-to-end, non-government-controllable system that's undeniably easy to use, then releases it for free and gets all the major e-mail platforms to adopt it (obviously much easier said than done).
Your mom couldn't recognize that a message was "verified" as having come from her Doctor?
Or click the "encrypt" box (which causes Gmail to determine whether it knows the public key of every one of an email's recipients).
Or click the "sign" box, to add her signature to the email before it sends?
Sure, sometimes Gmail would have to say "this email cannot be encrypted because of insufficient information about recipient X". But if anything this might lead to more people using large email providers.
Sure, consumers don't know that they want PGP. But when they suddenly discover that can trust that all their messages are seen by only a particular recipient, and that they'll always be able to tell when a message is or isn't from their coworker, they'll quickly start insisting on it the added security.
Have you ever used PGP for a prolonged period with a changing group of people also using it? I'm inclined to think that someone who'd write a comment like this hasn't. But I could be wrong!
Virtually every intra-company mail I send or receive is PGP-encrypted (we're getting to be a fairly big company by HN standards, too). I assure you: my mom could not deal with PGP's (constantly evident) corner cases. How could I possibly think it's a credible solution for the whole world?
Dunno if by PGP you literally mean the PGP/GPG toolkits or something else. I think most of us agree that PGP/GPG suite is too difficult to use.
What are the challenges with using public-key encryption with a changing group of people? Key exchange? Remember we're in a more-centralized, always-connected world now wrt email than we were in 1995 (remember offline email writing?). The problem should be simpler now if anything.
People have false notions about their email and web traffic in general. The closest previous analogy is shipping packages and letters. There're laws preventing USPS from snooping. People seem to reasonably assuming that that's the case with electronic letters too.
And nobody in the mainstream is disabusing of this notion (encryption is mostly shown as being used by terrorists and swindlers). When they become aware of how naked their communications really are, they'll learn how to cope with encryption (including your mom :)).
No, I haven't used PGP extensively, but my reasons for not doing so (as probably with everyone else) have had to do with the fact that no one else uses it and it hasn't been cleanly incorporated into existing online identity frameworks.
I'm imagining that the webmail providers could implement it piecemeal, and though I'm speculating here, I think this slightly-less-secure-than-complete-PGP approach would address a lot of the problems you're referring to (I'd be eager to hear why you disagree, though!). Corner cases should only be a problem if the approach is all-or-none, right? And if done properly, a piecemeal PGP implementation would still be more secure than no implementation at all.
From the user's/your-mom's perspective:
a) conversation threads would be kept separate from each other depending on whether or not they were secure
b) some messages would be signed, explaining what is known about the sender and why (e.g. "this message has been verified as originating from xx@gg.com", or "message is known to have come from John Smith at 2nd National Bank, with email address xx@gg.com", or even "John Smith, with the following verified profile".
c) some messages would be encrypted, telling the user "this message was securely sent to you, and person X, and person Y, by xx@gg.com."
d) when a user sends a message it would default to the most secure mechanism that all of the conversation participants allow, given what it knows about the recipient
Even if Gmail was the only provider that implemented this, especially given two-factor authentication, users would immediately get secure conversations at least in conversations that included only other Gmail users, which would go a long way to dealing with the fact that it's the lack of other people using PGP that makes adopting PGP kind of pointless.
Given that Google provides webmail to many universities and businesses, implementing PGP would immediately turn on secure and verified communication within those organizations: the benefits of PGP there would be instant and enormous: identity verification, timestamps, and signatures are the only reasons people still shuffle paper around. No more clumsy and expensive university-stamped transcripts; no more running around trying to find people to get their signature; no more running signed documents between various offices, etc.
Furthermore, your mom might already use Facebook (or another social website) and if so is already experiencing the benefits of verified identity and higher-trust communication. On fb you have absolute confidence that you're actually posting to friend X's wall; and when you're messaged by friend X you're absolutely sure it's X (unless someone gained access to X's account). This implies that similar such UI-based cues could be used for webmail, no?
Indeed, one could characterize part of the success of facebook (and other social networks) in terms of identity verification: you know that information posted by person X has actually been posted by person X (vulnerabilities notwithstanding); you know that what you post will not be seen by people whom you don't want to see it; you know when person X posts on your wall that it was actually the X-that-you-know and not some other one or a fraud; you know by virtue of X's existing relationships to your other friends that it's the X-you-know rather than someone else.
Ultimately, I think Google hasn't implemented PGP, in spite of all the incredible benefits, because it doesn't want to encourage its users to be more private. Is this evil?
tl;dr partial PGP implementation on webmail would avoid corner case problems while providing huge benefits to millions of people and organizations
If you can't even explain what the system would appear to do from a user's perspective in less than (hold a sec...) 11 paragraphs, I don't think you get to call Google "evil" for not doing it.
> Nobody wants the entire Internet encrypted more than the giant banks
Why is so much bank security nothing more than ridiculous security theatre? I agree they have the money, they have the need, they have enough smart programmers; so why do they fail so hard at security?
1) They don't have enough smart programmers. Nobody does, since "enough" means every single one. It only takes one CRUD screen with an SQL injection or buffer overrun exploit to get owned. By definition, a large amount of code must tend towards average quality.
2) Selective perception. We only notice when they fail. On the whole maybe they do do a pretty good job and nobody notices when things aren't wrong.
3) Sometimes it's an insider job.
4) Dumb users; no amount of company security can stop a successful phish or keylogger or whatever.
5) Cost-benefit. Sometimes it really isn't worthwhile to pour as much security as you possibly can at a problem. What if a bank guarded every transaction by having a live operator call the customer at a pre-known number? Sure that's pretty stiff security, and completely impractical in a competitive capitalist marketplace. That's an extreme example but it goes to prove the point. Sometimes it's cheaper just to clean up aftewards.
Security is a thorny problem; you can't always just magically have more security by throwing more money at it.
These are great points. Regarding #2: most of the banks we've all heard of are huge companies with hundreds of apps. A breach in any of them, even a brochureware app, is reported as if it was a full-blown bank heist. Serious incidents do happen at banks, but when you hear about them, odds are it isn't the retail account management system that broke
I think it would be paranoid to encrypt all Internet traffic. A lot of traffic isn't sensitive enough to warrant disabling that benevolent man in the middle, the proxy cache.
Bruce Schneier was a great proponent of cryptography. In he lat 90's, he touted encryption as the end-all of security measures. Since then, he has backed off his claim. (He later went on and continues to be a proponent of legislative laws for proper surveillance and forcing companies to be more security compliant). So, even Bruce doesn't think encryption is a good idea.
Excerpt from a 1998 essay, "Security Pitfalls in Cryptography" that sums it all up:
"Strong cryptography is very powerful when it is done right, but it is not a panacea. Focusing on the cryptographic algorithms while ignoring other aspects of security is like defending your house not by building a fence around it, but by putting an immense stake into the ground and hoping that the adversary runs right into it. Smart attackers will just go around the algorithms."
I agree. Encryption is a good idea, but I just don't think it's all that practical. If people really wanted to crack encryption, the technology is there (and if the person has the patience and time) it will be cracked. And they don't want to spend the time and effort to get that encrypted info, there are other ways to really get at it. The user still remains the unpredictable variable and weakest link in network security. I just think there are better ways to secure networks and the internet...
Fast forward to today and this legacy has remained. Many protocols are being retrofitted to support encryption, but it's similar to why railroad tracks are the width they are: Roman chariots used the same width.
Additionally, don't discount developer laziness. It's much easier to debug a plaintext stream than an encrypted one.