
Milestone: 100M Certificates Issued - okket
https://letsencrypt.org/2017/06/28/hundred-million-certs.html
======
koolba
SSL certificate from a traditional provider valid for a year: $10.

SSL certificate from a traditional provider valid for two years: $20.

Automated SSL certificate generation and deployment via LetsEncrypt with zero
human intervention and more importantly zero human intervention to renew it
going forward - _priceless_.

\---

That's the real value for me. At $10/cert, that's not even a rounding error.
But manually generating a new CSR, uploading it via crappy web form, waiting a
random amount of time, proving domain ownership by responding to an email
(sent in plaintext), waiting a different random amount of time, downloading
the new cert (again usually sent via plaintext email), and finally copying it
over the old cert and reloading the SSL conf ... now that costs some serious
time and time _is_ money.

~~~
Asdfbla
What's the justification of traditional providers to charge for the
certificates? Do they offer services or extended certificates that "Let's
encrypt" doesn't or is it just a matter of "no one tried to offer it for free
before"?

~~~
cyphar
Currently their only upside is that they provide wildcard (and EV)
certificates. EV certs can't be automated for obvious reasons, but LE doesn't
support wildcard certs because they don't believe there's a secure way of
providing an automated way of getting them.

Given that traditional CA models don't actually have much more security than
LE (in fact from personal experience they're far less secure), I wonder
whether they should be in the business of providing wildcard certificates at
all.

~~~
giobox
One annoying thing I've seen less technical stakeholders ask about with
letsencrypt is "why the padlock isn't green" in Safari/Edge. Unlike Chrome and
I think Firefox, those browsers only display a green padlock for SSL certs
with EV, which of course letsencrypt doesn't (and arguably can't really) do.

The "grey padlock" for an otherwise perfectly valid certificate seems to
confuse a very small subset of users, especially when there's a ton of badly
written articles out there advising users only to trust "green padlocks",
given the behavior in almost all other browsers. Thankfully showing google.com
not using an EV certificate is usually easiest way to shut these conversations
down.

~~~
nlawalker
>> only to trust "green padlocks"

It's not so much about browser-specific behavior, it's about using the padlock
to ensure that the _company or other entity_ you're interacting with is the
one you wanted, as opposed to only ensuring that the server you're talking to
is the one indicated in the address bar.

A gray padlock is better than none, but telling less technically-inclined,
clicks-all-the-links, fat-fingered users to trust them outright may not be the
best idea now that you can go get bankofomerica.com, dress it up to look like
BoA's site, and put a gray padlock on it for free.

~~~
giobox
Given how piecemeal the use of EV certs is by big companies, I'm not sure EV
as implemented today delivers any 'real' benefits. It absolutely is about
browser specific behaviors. How can you tell users only to interact with
companies if they have an EV certificate, when huge internet properties like
Amazon, Google and Facebook don't use them? In reality you can't, which means
Safari/Edge's behavior arguably doesn't really help anyone. I'm not saying
Safari/Edge shouldn't tell the user an EV certificate is present, they
absolutely should, but that the approaches used by rival browsers arguably
make more sense.

I think Chrome/Firefox got this one right by using a green padlock for both
valid EV _and_ DV certificates like Letsencrypt, displaying the extra EV
information clearly inline in the URL bar if it is present. The changing color
in Safari/Edge can give the impression that the the non-EV certificate is
somehow less secure, which in technical terms is not the case. Questions of
identity verification and security of data transmission are two separate
issues which the Safari/Edge approach arguably conflates - both EV and DV can
provide equal security of transmission, even if only EV provides some degree
of identity confirmation.

As a user, it's simply not possible to make the distinction too - if I visit
your fake Bank Of America site with a grey padlock, how can I even be sure the
'real' site had an EV certificate and a green one, when there's so many
companies that don't? As a user you are effectively forced to treat the grey
and green padlocks the same, so why not make them the same color and display
extra EV info if it is there, like the other browsers?

~~~
nlawalker
I agree with you. My point - which I admittedly didn't state very well - is
that the best-practice messaging to non-technical users needs an update. In
the earlier days of the web, it was as simple as "look for the padlock,"
because the state of SSL was such that generally only legit sites had it. With
the presence of LE and the overall failure of the "web of trust", this is no
longer good advice - they need to carefully double-check the address bar as
well.

------
tyingq
This is an interesting situation where "public good" happened to align well
with business goals of some deep pockets.

Particularly, Google and Akamai...two of the biggest LE sponsors. They both
retain good visibility to user behavior (like specific urls visited) because
of things like GA,MITM proxying, etc. But, ubiquitous availability of that is
taken away from ISP operators.

Which is a good thing. Makes me curious if there's anything else like this
that could be achieved. Are there other net public good projects that align
well with deep pocket potential sponsors?

~~~
okket
What also may have motivated Google & Co was that ISPs and, further down, hot
spots like hotels etc. started to insert/replace ads on unencrypted
connections...

~~~
skybrian
Security professionals tend to be motivated to work on improving security
because they think it's important to protect users from the bad guys. And it's
not like people have a sudden change of heart when joining Google.

~~~
mirko22
But it is also not like you get to do what you want in big corporations.
Rather you do what you have been told, and usually you might not even know the
real reason.

Also security professional is a person that know a lot about security, his
intentions are not necessarily "good" for him to be considered a security
professional in my opinion.

Very good black hat hacker are also professionals.

~~~
skybrian
That's rather more top-down than I've seen. There are large corporate
initiatives, but many projects happen because someone thought it was a good
idea and sold it to management.

------
jagermo
I think they nail their point with "it illustrates the strong demand for our
services."

Letsencrypt is cheap (free) and easy to use.

Even people with not a lot experience can secure their sites and apps, and it
just works. Yes, you have to update it every three months, but that's worth
the price and the excellent documentation.

Before letsencrypt I always wanted to secure my blog with https but never got
around to it or it just looked super complicated and error prone.

Then, my provider built its own tool for LE and its just so easy to implement.

~~~
the_common_man
> Even people with not a lot experience can secure their sites and apps, and
> it just works. Yes, you have to update it every three months, but that's
> worth the price and the excellent documentation.

This is just a cronjob, no?

~~~
m12k
If you're comfortable with said cronjob having access to your private key

~~~
tskaiser
If the private key is only used for this purpose, and the cronjob as well as
the key only reside on the server, is the security of the key then not a moot
point if the server is breached? Genuinely curious.

~~~
m12k
If you update the certificates manually, then the private key would just
reside on your local machine and not need to be exposed on the server.

~~~
tskaiser
Indeed, but I cannot see how this addresses my question?

~~~
codehenge
because your private key never needs to be compromised, even if your server is
breached, because it never needs to be on the server. That said, I agree that
if the PK is only used for this single server, you have bigger problems than
its loss if your server is breached.

~~~
stephenr
If the key isn't on the server how does software use the certificate?

------
TekMol
I still wonder how Let's Encrypt works.

I understand that the problem they solve: A user wants to get the public key
for a certain domain. So he knows he is talking to a server by the domain
owner and not some man in the middle.

So he asks a third party whos public key he already has. In this case Let's
Encrypt. Ok.

But how did Let's Encrypt get the public key from the domain owner?

I know they make the domain owner install some software on his server. But how
does that make sure they talk to a server by the domain owner and not a man in
the middle? Does the software include some private key so they can send them a
"hey, encrypt this" message and to prove the server in fact has that private
key?

If so, why is the Let's Encrypt software so complicated and not just a 5 line
script or something?

~~~
JshWright
They have a variety of "challenges" used to prove control of the domain. They
basically boil down to:

\- LE makes a request to example.com, to a known URL that contains a key that
LE told you to put there

\- LE does a DNS lookup for a specific TXT record (again, containing a key
they told you to put there).

The complete answer to your question can be found in the ACME spec (the
protocol LE uses):

[https://ietf-wg-acme.github.io/acme/draft-ietf-acme-
acme.htm...](https://ietf-wg-acme.github.io/acme/draft-ietf-acme-
acme.html#rfc.section.8)

~~~
theandrewbailey
LE probably does that multiple times from a few places around the internet, to
reduce the likelihood of something unexpected going on.

~~~
j_s
Ain't nobody got time for that!

They might have " _a few places around the internet_ " to verify from, but I
doubt it is very diverse location-wise. And there's no way they're going to
bother with doing it multiple times, for free.

~~~
dylz
> they're going to bother with doing it multiple times, for free

You are aware you are talking about the cost of a single HTTP GET?

~~~
j_s
> You are aware you are talking about the cost of a single HTTP GET?

First, the conversation is about the possibility of multiple (not single)
interactions, including protocols other than HTTP such as DNS.

Perhaps you're confusing a subset of what they _currently_ do vs. coordinating
the initiation of requests _multiple times from a few places around the
internet_ (per OP) supporting multiple protocols, aggregating and verifying
the responses, etc. -- all within a reasonable timeframe at scale with
Internet infrastructure level high availability (while protecting against
malicious misuse)?

I hope this is enough to explain that no, that's not what I was talking about.
But thanks for taking the time to reply!

------
gator-io
The biggest issue we've had is the short expirations. We have 51 certificates
in our organization and do not want to rely on auto-renew.

As our community project, we built a totally free to the public service to
monitor certs and alert you when they get close to expiration or are invalid,
etc:

[https://letsmonitor.org](https://letsmonitor.org)

Feel free to use it for any certs.

~~~
j_jochem
Out of interest, what makes you not want to rely on auto-renew?

~~~
gator-io
It has never worked right on Windows servers (don't know why). And it is
simply a check of last resort if something goes wrong.

------
asenna
Genuine question: Are the other smaller cert-issuing services going out of
business? If not, what has been their response to LetsEncrypt?

Not that all of them should survive, there are a lot of crappy services that
deserved this. But I'm just trying to place myself in their CEOs position and
wondering how the game plan should be.

~~~
collinmanderson
StartSSL effectively died, and I assume LetsEncrypt was part of the reason.

~~~
duskwuff
No... the real reason StartSSL died was because they were acquired by another
CA, Wosign, which had a number of serious security and management issues.
These issues led to both CAs being distrusted by all major browsers.

[https://wiki.mozilla.org/CA:WoSign_Issues](https://wiki.mozilla.org/CA:WoSign_Issues)

~~~
collinmanderson
Right, but why did Eddy Nigg sell StartSSL to WoSign?

My guess is Heartbleed made him re-consider whether he wanted to keep doing
ssl/tls certs, and once Let's Encrypt came along he started looking for a way
to get out of it.

------
rmc
Wasn't the Snowden releases part of the motivation for Let's Encrypt? If the
web is "encryption for everything by default", then dragnet survellance from
the NSA (etc) is much harder.

------
doublerebel
Shameless open-source plug: if you need a automatic LetsEncrypt module that
works with your containerized environment, try ten-ply-crest [0]. Works best
with Consul where it automatically registers based on service tag, and can
securely store certs in Vault, which makes it a great match for Fabio [1].

This way you can run multiple load-balancers and keep your existing
infrastructure while still enjoying all the benefits of LE. Written in
JavaScript but works with any stack.

Based on the great rawacme, and we'll keep this updated as ACME hits 2.0! [2]

[0]: [https://github.com/nextorigin/ten-ply-
crest](https://github.com/nextorigin/ten-ply-crest)

[1]: [https://github.com/nextorigin/ten-ply-
crest/issues/27](https://github.com/nextorigin/ten-ply-crest/issues/27)

[2]: [https://github.com/AngryBytes/rawacme-
node/issues/2](https://github.com/AngryBytes/rawacme-node/issues/2)

------
linkmotif
How timely. I got my first LE cert two days ago! I used caddy, a really nice
http server and proxy that's HTTPS first [0].

[0] [https://caddyserver.com](https://caddyserver.com)

------
tscizzle
The graph of HTTPS percentage has some sudden massive dips a little bit ago.
Does anyone know the reason for those?

------
corford
I love letsencrypt but I wish they would hurry up with deterministic dns
challenges. It would make securely automating certificate renewal vastly
simpler and easier (especially for certs that are being used for non www
serving endpoints e.g. mail servers etc).

I worry that at the moment there are probably a lot of systems out there that
leave DNS API keys lying around on endpoints because it's easier to automate
the renewal than write a convoluted two stage ansible role/play that delegates
sensitive actions to the controller running the play.

~~~
pfg
This is due to the CA/B Forum Baseline Requirements stating that domain
validation via DNS records (and pretty much all other methods) has to use
either a random value or a single-use request token.

There's a neat trick with CNAMEs and a separate DNS zone that works kind of as
a workaround, if you're okay with introducing another piece of
infrastructure[1].

[1]: [https://github.com/joohoi/acme-dns](https://github.com/joohoi/acme-dns)

~~~
corford
Yeah I read somewhere that they were working to try and get the CA/B rules
changed to allow it (no idea if this is true).

Will check out the CNAME trick - thanks!

------
michaelbuckbee
LE really fills a big need for cert issuance for massive web hosts with custom
domains (WPEngine, Hubspot, Shopify are all on LE) as it both greatly reduces
the support burden of setting up certs for all of their sites and sidesteps
some of the limitations of the legacy cert structure.

------
binocarlos
I've been using kube-lego ([https://github.com/jetstack/kube-
lego](https://github.com/jetstack/kube-lego)) to automate certs for Kubernetes
ingress for the past 9 months. It is a joy to just add a domain to a manifest,
kubectl apply and then hit a browser with a certificate working. Thank you
LetsEncrypt for making the Internet more secure.

------
pas
And most of that is because LE [still] doesn't issue wildcard certs, nor does
it really plan to :(

~~~
eropple
And that is a without-exception good thing. Wildcard certificates are
dangerous and encourage bad practices around SSL. You shouldn't use them.
Instead, you should fix your workflow to know exactly what domains you have
and have active and use automated tooling to provision them correctly.

~~~
F_r_k
I don't think so.

I have an industrial VPN with ~300 embedded devices behind them. For some
reason the user wants to access to each one of it via https.

I thus set up an Apache proxy which proxies
[https://device_n.vpn.cust.tld](https://device_n.vpn.cust.tld) to the VPN
device (and some auth)

LE cant issue 300 certificates and it would be very ugly.

~~~
eropple
I don't believe LE has an upper limit on its number of SANs.

For systems not explicitly public, whitelists are almost always better than
blacklists.

~~~
twothamendment
> I don't believe LE has an upper limit on its number of SANs.

The limit is 100, unless they removed that when I wasn't looking. Back in the
beta I tried one domain, a handful and then, because I couldn't find a
technical reason why SAN should be limited, I tried 500. It fails. I have many
certs with up to 100 and would be happy if this changed.

~~~
eropple
The highest I've seen in the wild is 60 or so, so I'll bow to your superior
getting-bitten-in-the-ass-by-it. =)

(And if so, in the prior commenter's situation--okay, create three certs. LE
might not support 300 certs, but 3 with 100 SANs should work just fine.)

------
id122015
I'd like to have the list with those 100 million websites, I didnt know there
are so many.

~~~
schoen
It's not 100 million web sites, because Let's Encrypt certificates expire
every 90 days, and you have to get a new certificate if you want to add or
remove a domain name, and you can get duplicative certificates as long as
they're within the rate limits.

As I said in my post at

[https://www.eff.org/deeplinks/2017/06/lets-encrypt-has-
issue...](https://www.eff.org/deeplinks/2017/06/lets-encrypt-has-
issued-100-million-certificates)

it's probably between 17 million and 46 million web sites, although it could
be a bit smaller than 17 million at the most restrictive notion of "web site"
if you don't consider subdomains to be a separate web site even if they have
different content and are operated by a different person.

You can see all of the certificates at

[https://crt.sh/?Identity=%25&iCAID=7395](https://crt.sh/?Identity=%25&iCAID=7395)
[https://crt.sh/?Identity=%25&iCAID=16418](https://crt.sh/?Identity=%25&iCAID=16418)

and you can also get a lot of the data from

[https://scans.io/](https://scans.io/)

or [https://www.censys.io/](https://www.censys.io/) (registration required),
although the people running it told me in connection with this that their
Certificate Transparency data is not as current at the moment as crt.sh's
data.

------
halloij
Shame they can't issue wildcard certs. I don't see why they cannot.

~~~
techmagus
It was, and I think it still is, a very much requested feature. I for one will
benefit from it in one particular setup. But so far, can live without it.

------
pulse7
I would like to get a certificate with 3-years lifetime. I know that 90-days
will limit the damage from key compromise, but I don't want to automate...

~~~
simias
I was in the same boat, I didn't like having to run python on my server just
to renew a SSL certificate. Still, a few months ago I decided to give it a try
and I don't regret it. The certbot script is packaged in FreeBSD, it wasn't
painful at all to setup and a crontask later it was over. Definitely beats the
crappy free "startssl" certificates that I used previously.

~~~
weavie
acme.sh runs in pure shell, no need to install Python. It's my goto client and
it just works.. everywhere.

[https://github.com/Neilpang/acme.sh](https://github.com/Neilpang/acme.sh)

~~~
theandrewbailey
Seconded. I love that it supports custom port numbers, so I can use my not
privileged web server port.

------
mavdi
Nearly 20K of them for Paypal phishing sites and who knows how many for
others. While a noble intention, one can't ignore the damage they've done.

~~~
mixedbit
What prevented Paypal phishing sites from buying certificates from other
providers?

~~~
mavdi
checks they're supposed to be performing. Some are ignoring them but being
punished for it.
[http://www.bbc.com/news/technology-39365315](http://www.bbc.com/news/technology-39365315)

~~~
pricechild
I'd definitely recommend clicking through that article and reading the source
of the announcement:

[https://groups.google.com/a/chromium.org/forum/m/#!msg/blink...](https://groups.google.com/a/chromium.org/forum/m/#!msg/blink-
dev/eUAKwjihhBs/rpxMXjZHCQAJ)

> As captured in Chrome’s Root Certificate Policy, root certificate
> authorities are expected to perform a number of critical functions
> commensurate with the trust granted to them. This includes properly ensuring
> that domain control validation is performed for server certificates, to
> audit logs frequently for evidence of unauthorized issuance, and to protect
> their infrastructure in order to minimize the ability for the issuance of
> fraudulent certs.

> On the basis of the details publicly provided by Symantec, we do not believe
> that they have properly upheld these principles, and as such, have created
> significant risk for Google Chrome users. Symantec allowed at least four
> parties access to their infrastructure in a way to cause certificate
> issuance, did not sufficiently oversee these capabilities as required and
> expected, and when presented with evidence of these organizations’ failure
> to abide to the appropriate standard of care, failed to disclose such
> information in a timely manner or to identify the significance of the issues
> reported to them.

i.e. they apparently weren't even checking requesters controlled the domain
they requested a certificate for in some instances!! And that's pretty much
the only requirement for DV certificates?

My point is that the legacy ssl providers you're referring to were doing less
checks than Lets Encrypt!

