
Let's Encrypt and Nginx – State of the art secure web deployment - llambiel
https://letsecure.me/secure-web-deployment-with-lets-encrypt-and-nginx/
======
schoen
It scared me to see that the author recommended running

    
    
      curl http://nginx.org/keys/nginx_signing.key | sudo apt-key add -
    

(This adds a key or keys downloaded over an unauthenticated http connection to
one's Debian keyring, allowing whatever keys the network sends back to
authenticate any future package updates.) I wrote to the author with a note
expressing my concern.

~~~
icebraining
Unfortunately, it seems there's no secure way to fetch the key. The nginx team
recommends one checks the "web-of-trust" to check if the key is signed by
others.

~~~
rajivm
At the least though it could be https.

~~~
schoen
I also suggested that in the meantime the author of the article can provide a
SHA256 checksum, so you can see if you get a different key than he does.

------
diakritikal
Just using caddy server seems a lot simpler...

~~~
tlrobinson
Neat, I like that when started with no arguments/config it just serves the
files in the current directory, but then you can customize it from there.

I have "alias webserver='python -m SimpleHTTPServer'" in my shell config, but
I think I'll switch to Caddy.

~~~
callahad
For local development, consider looking into devd
([https://github.com/cortesi/devd](https://github.com/cortesi/devd)). It's a
single binary that supports things like livereload, network throttling,
routing, and reverse proxying.

~~~
StavrosK
Ooh, that looks pretty nice. I was using Caddy to serve the current directory,
but this seems even nicer.

------
IgorPartola
I have been happy with
[https://github.com/lukas2511/letsencrypt.sh](https://github.com/lukas2511/letsencrypt.sh).
I am trying to get it packaged for Debian/Ubuntu and either get it into
Debian-proper or at least host the repo myself to make it easier to use for
the common case. Since nginx reloads the cert on a SIGHUP it makes it really
easy to have zero downtime renews.

As for getting notified if something goes wrong I use the following in my
crontab:

    
    
        10 5 * * *  root    test -e /usr/local/bin/letsencrypt.sh && /usr/local/bin/letsencrypt.sh -c > /dev/null
    

letsencrypt.sh outputs errors to stderr, so any errors will be sent to the
root account. To get that working, do:

    
    
        apt-get install postfix
        echo 'postmaster:     root' > /etc/aliases
        echo 'root:           igor@example.com' >> /etc/aliases
        newaliases
    

Problem solved.

------
feylikurds
Ewwww, that renewCerts.sh is pretty crappy. Who the hell is going to check the
/var/log/letsencrypt/renew.log everyday to see if renewing failed?

Could not they do something nicer with systemd and email?

~~~
pfg
The default behaviour of cron is to email the user if a job finishes with a
non-zero exit code, which seems to apply here in case of renewal failure.

~~~
feylikurds
But is not the default account that it would email root? I run Debian and
almost never log in as root. Would all admin sudoers receive the email?

~~~
jcrawfordor
Without judgment intended, as a Linux sysadmin you should absolutely be
monitoring mail to root. That is the standard place to deliver error output
from unattended processes. You can easily /etc/alias it to something else if
that's more convenient.

~~~
toomuchtodo
Sysadmin/Devops here. I send all root mail to Graylog.

~~~
StavrosK
Graylog looks fantastic, thanks for the mention.

~~~
toomuchtodo
You'll love it. I'm pushing tens of thousands of messages per second into a
cluster, and it works like a champ.

------
nickpsecurity
It's a nice tutorial. The title tripped me out, though: a common webserver +
HTTPS + free certificate on Windows/Linux is "state of the art secure web
deployment?" I'd hate to see what passes for average or (shudders) ancient.

In my mind, I'm seeing "state of the art" being more like a combo of Ur/Web
for apps, robust implementation of OP2 web browser for client, lighttpd
rewritten in Haskell, HTTPS component written in SPARK or Rust, all running on
GenodeOS or CheriBSD in isolated partitions, C parts compiled with CompCert
extended with Softbound + CETS, anti-fuse FPGA doing I/O offloading/mediation,
and hardware done in Bluespec. _That_ is state of the art with probably badass
results. This submission is... more run of the mill. Immediately useful,
though. :)

------
velox_io
Thanks for the info on the headers, I can't believe they've issued certs for
over a million domains!

Here's my notes on setting up LE on IIS if anyone one is interested, it's done
by using Powershell/ Package manager.

//1\. Install (you will get some security prompts) Install-Module -Name
ACMESharp

Import-Module ACMESharp

Initialize-ACMEVault

New-ACMERegistration -Contacts mailto:somebody@example.org -AcceptTos

//2\. Request the challange, this is for a website currently running on IIS.
'WebSiteRef ' refers to the name of the site within IIS

New-ACMEIdentifier -Dns demo.velox.io -Alias demo Complete-ACMEChallenge demo
-ChallengeType http-01 -Handler iis -HandlerParameters @{ WebSiteRef = 'Demo'
}

Submit-ACMEChallenge demo -ChallengeType http-01

//3\. Create & download the certificate

New-ACMECertificate demo -Generate -Alias demoCert

Submit-ACMECertificate demoCert

Update-ACMECertificate demoCert

Get-ACMECertificate demoCert -ExportPkcs12
"C:\Users\USER\desktop\demoCert.pfx"

You can now install this on your server.

------
sschueller
I just use
[https://github.com/lukas2511/letsencrypt.sh/](https://github.com/lukas2511/letsencrypt.sh/)
single bash script.

Add a config.sh and setup nginx alias. Then just add domains to the
domains.txt and have the script run via cron daily.

Finished

~~~
X-Istence
This is the script I use too. I have a hook that automatically restarts nginx,
which fires only if a cert has changed. Very simple. Works very well.

------
ruslo

      > sed -i 's|PasswordAuthentication yes|PasswordAuthentication no|g' /etc/ssh/sshd_config
    

Will not work if string is commented out:

    
    
      > grep PasswordAuthentication /etc/ssh/sshd_config
      # PasswordAuthentication yes
      > sed -i 's|PasswordAuthentication yes|PasswordAuthentication no|g' /etc/ssh/sshd_config
      > grep PasswordAuthentication /etc/ssh/sshd_config
      # PasswordAuthentication no

------
ivan_ah
The --webroot option doesn't work for my setup, so I need to shutdown nginx
for 2-3 seconds and use the --standalone option. I set this as a CRON job that
will run every two months. It's not elegant, but it's done.

Here's the modified script using certonly and the --force-renew flag.

    
    
        #!/bin/bash
        # Force-renew the "Let's Encrypt" certificates for a given domain
        # Run this as root as a BI-MONTHLY cron job
        export DOMAINS="yourdomain.com,www.yourdomain.com"
        export LOGFILE="/var/log/letsencrypt/renewal_yourdomain.log"
    
        echo "Stopping nginx temporarily to renvew certificates for $DOMAINS ..."
        service nginx stop
    
        echo "Calling /opt/letsencrypt/letsencrypt-auto certonly --standalone --force-renew -d $DOMAINS"
        if ! /opt/letsencrypt/letsencrypt-auto certonly --standalone --force-renew -d $DOMAINS > $LOGFILE 2>&1 ; then
            echo "certonly call failed, restarting nginx"
            service nginx start
            echo "LOG info:"
            cat $LOGFILE
            # TODO: email administrator...
            exit 1
        fi
    
        echo "certonly call succeeded, restarting nginx"
        service nginx start
    

Note: don't run this as a daily cron job since this has --force-renew...

~~~
schoen
Do you ever get problems with the socket still being in use after nginx is
shut down?

~~~
ivan_ah
Not on the N=1 times I've run the script, but will look out for this in the
future.

------
tbrock
Lets encrypt fixes the encryption problem sure but does anyone else feel that
all we really needed was really great documentation on what to do instead of
an intrusive set of scripts?

~~~
doublerebel
Yes, the API documentation is lacking especially with what we've gotten used
to from Swagger markup and Stripe's API doc style. I have scoured for such an
easy breakdown and found none. As a result I actually just implemented a new,
clear, client for LetsEncrypt and have been documenting as I go.

It's made me think we should have a Swagger or API Blueprint of the spec on
github that everyone can keep up to date. What do you think?

~~~
pfg
Are you referring to the server-side API the client is communicating with, or
the internal API the client exposes?

The former is documented in the ACME specification[1], currently being worked
on by the IETF. There are _many_ low-level ACME libraries for basically every
language[2], and a pretty decent guide on writing your own client as well[3].

[1]: [https://ietf-wg-acme.github.io/acme/](https://ietf-wg-
acme.github.io/acme/)

[2]:
[https://github.com/letsencrypt/letsencrypt/wiki/Links#librar...](https://github.com/letsencrypt/letsencrypt/wiki/Links#libraries)

[3]: [https://github.com/alexpeattie/letsencrypt-
fromscratch](https://github.com/alexpeattie/letsencrypt-fromscratch)

~~~
doublerebel
Thank you very much for these links. 3) is the closest to what I'm looking
for, and really good! but is still an implementation and not a spec. 1) is
fine for a spec for an internet committee who has to delve into every detail
for standardization. But if a client can be written in 150 lines of code,
there should be a much shorter version of the spec (only a couple pages) in a
standard format. I should be able to easily write a client from a 3-page spec
without looking at all the implementations.

All due respect to the client authors, but only a few clients are good. Many
are very poorly written, which I do not trust for security. I believe the
cause is not having a clear, short, standard modern spec.

~~~
simoncion
> But if a client can be written in 150 lines of code, there should be a much
> shorter version of the spec (only a couple pages) in a standard format. I
> should be able to easily write a client from a 3-page spec without looking
> at all the implementations.

Right, but even though the protocol is simple, there are pretty much always
subtleties and potential ambiguities that _need_ to be resolved by the spec so
that one can write good implementations.

> I believe the [problem] is not having a clear, short, standard modern spec.

The IETF ACME draft spec _is_ (like many IETF specs) clear, short, standard,
and modern. The entire document is only 50 pages (fewer if you reduce the font
size), and (from skimming the ToC) the last ~10 of those pages are largely
optional material for someone who's just reading to implement the protocol.
That document shouldn't take you more than an hour to read and digest.

If you've never actually _read_ an IETF spec they can be intimidating, but -if
you're a programmer, network guy, or backend web dev- you _really_ , _REALLY_
owe it to yourself to learn how to read them:

* Use the IETF's HTML RFC viewer rather than the plain text viewer.

* Until you become familiar with the way IETF standards documents are written, don't skim! They're generally information-dense documents that do _NOT_ repeat themselves.

* Start from the beginning of the document and read through the end.

* If the spec references another document, and then starts to talk about things _from_ that document that you don't understand and can't figure out, go read the relevant parts of the referenced document.

* If the spec starts presuming knowledge of things that you're _sure_ it hasn't mentioned yet, backtrack a bit... you probably overlooked something.

* The ASCII-art diagrams present in some specs aren't there for fun; they're important information.

In regards to shorter documents, I'm not sure what you're looking for...
_just_ a listing of the HTTP conversations and their payloads?

~~~
doublerebel
My point is, the clients are generally pretty bad _despite_ the IETF spec.
Lots of edge cases ignored, poor security practices. I understand the
intention but the effect is that the clients are just as opaque as the spec
but often more incorrect. Who would try to launch a startup API for wide use
without a Stripe-style spec these days?

Thanks for the advice, but it's not that _I_ don't understand how to read it,
it's that I can tell _other_ devs don't understand it despite the good
intentions of the authors.

Plenty of good clients are written for plenty of other tools, based on a much
more straightforward call-and-response API spec. For example, the Hashicorp
tools have a simple spec and proper clients in many languages.

~~~
pfg
The spec contains sample payloads for pretty much every resource. In fact, you
can build a functional client for http-01 just by looking at the examples.

On top of that, if you're using a programming language that's at least close
to mainstream, there's a very good chance someone has already written a
library which handles _most_ of the nitty-gritty details of ACME. As an
example, this is all the code you need with the acme-client ruby gem in order
to solve a http-01 challenge and get a cert ( _slightly_ abbreviated):

    
    
        require 'acme/client'
        client = Acme::Client.new(private_key: private_key, endpoint: endpoint)
        registration = client.register(contact: 'mailto:contact@example.com')
        registration.agree_terms
        authorization = client.authorize(domain: 'example.org')
        # serve challenge.filename with content challenge.file_content
        challenge.request_verification
        # loop/sleep until challenge.verify_status == 'valid'
        csr = Acme::Client::CertificateRequest.new(names: ['example.org'])
        certificate = client.new_certificate(csr)
        # certificate.to_pem contains your signed cert. done!

------
sleepychu
I'm really not a fan of this domain grab to write a single article with no(t a
lot of?) new information aimed at selling services from a single host. You're
not the only person guilty of this but it feels quite misleading like the
article is coming from a 3rd party.

------
alexpeattie
You can create a more hardened setup by using a 4096 bit RSA key:

    
    
      /opt/letsencrypt/letsencrypt-auto certonly --rsa-key-size 4096 --server https://acme-v01.api.letsencrypt.org/directory -a webroot --webroot-path=$DIR -d $DOMAINS 
    

...and using the secp384r1 curve for ECDHE key exchange:

    
    
      # in your nginx.conf
      ssl_ecdh_curve secp384r1;
    

Arguably, the real state of the art is to use an ECDSA certificate. Let's
Encrypt recently started supported them, they offer a equivalent level of
security to RSA at much lower bit lengths (a 384 bit ECDSA key is considered
equivalent to a 7680 bit RSA key) and a few recent TLS vulnerabilities (like
DROWN) have targeted implementation details of RSA.

~~~
lorenzhs
4096 bit RSA keys offer very little additional security (2048 is plenty for at
least the next few years, and with a certificate that's valid for 90 days,
there's practically no risk - you can rotate the key rather easily if
something bad comes along), but has a fairly big impact on performance and
battery life, especially on mobile devices.

------
realusername
Here is also my config if anyone is interested (also A+ on ssllabs.com):
[https://gist.github.com/alex-
min/158f35f604b24e163ae9](https://gist.github.com/alex-
min/158f35f604b24e163ae9), feel free to copy it. (or suggest improvements !)

I also recommand [https://sslcatch.com](https://sslcatch.com) which sends you
a warning email if your certificate is about to expire. I have a crontab to
renew it but this can be also helpful just in case.

------
ivan_ah
Isn't running this as a `@daily` CRON job too much? I thought Let's Encrypt
certs were good for 3 months? Why not @monthly or months 0,2,4,6,8,10 ?

~~~
teraflop
If something breaks, you might as well find out about it as soon as possible.
That way you have the full 90 days to figure it out at your leisure, instead
of 60 or 30.

~~~
ivan_ah
Right. Also, I just saw that `letsencrypt-auto renew` will only issue new
certs if < 30 days left on current cert.

------
smithclay
Recently went through a similar setup, but used Docker and some existing
h2-friendly images. Think it's a nice way forward for deploying to production
environments.

Wrote about the process here: [https://clay.fail/posts/hip-http2-using-
docker/](https://clay.fail/posts/hip-http2-using-docker/)

~~~
mikewhy
I recently built docker-gen-letsencrypt[1]. It's the same concept as what
you're using, but fully automated for getting certs.

[1]: [https://github.com/mikew/docker-gen-
letsencrypt](https://github.com/mikew/docker-gen-letsencrypt)

~~~
smithclay
That's awesome, will be updating the site to use your image this weekend. Like
the support for docker-compose and the staging servers, too.

------
andersonmvd
Look at how many lines we need to secure tls connections on nginx. We need
better defaults.

------
mrits
Interesting choice of cryptos. My latest client isn't letting us use anything
besides GCM right now.

------
jzelinskie
"State of the art" and "cron" should probably never be in the same article.

------
emilevauge
[https://github.com/containous/traefik](https://github.com/containous/traefik)
now has native Let's Encrypt support ;)

------
homero
Can someone tell medium? They're still buying comodo certs for their custom
domains

~~~
iancarroll
They probably do not want to deal with LE's rate limiting and shorter renewal
periods.

