
Introducing OpenBSD's new httpd [pdf] - fcambus
http://www.openbsd.org/papers/httpd-asiabsdcon2015.pdf
======
xiaq
So... Apache was removed from base on Mar 14 2014 in favor of nginx, and nginx
on Aug 27 2014 in favor of OpenBSD httpd.

For sysadmins who closely follow the "recommended" way, having to migrate the
configurations of the http server twice within half a year must have been a
frustrating experience.

Also, I wonder what "removal from base" means exactly - can you still install
them (the OpenBSD-patched versions) from the ports collection or something
like that?

~~~
sarnowski
That's correct. They are still available in the ports. If were operating a
complex setup, you probably already used nginx from ports before.

Not being in the base means that it doesn't get the same security attention as
its not officially part of OpenBSD anymore.

~~~
xiaq
> If were operating a complex setup, you probably already used nginx from
> ports before.

Does "before" refer to the period before Apache was dropped from and nginx was
imported to base? (Since apparently you can just use nginx from the base when
it is there.)

EDIT: It just occurred to me that "complex setup" likely indicates that you
would like to change some build flags, so you have to install from the ports.
Stupid me.

------
jacquesm
I've been going through the code for the last half hour and I really hope this
isn't representative of what the OpenBSD group considers to be defensive C
programming.

Stack allocated buffers, questionable logic and a generally terrible style as
well as a complete lack of comments.

Don't take my word for it, see for yourself:

[https://github.com/reyk/httpd/blob/master/httpd/server.c](https://github.com/reyk/httpd/blob/master/httpd/server.c)

The "new" is a bit off too, the copyright runs 2006-2015.

~~~
nly
Don't forget yet another hand-written HTTP parser, and the complete lack of a
test suite.

On a more flamebaity note, I don't know why you'd even want to write something
like this in C. Writing a server for one of the most prevalent network
protocols on the Internet in C, in 2015, just seems like masochism. This code
reinvents so many wheels for the 1000th time in C code history, it's just
tiresome to read. C++ would've reduced and simplified the code substantially
and there are a growing number of other fine choices these days.

~~~
tedunangst
The compilers for any number of other fine choices aren't included in OpenBSD,
making the addition of software written using them more complicated.

~~~
Dewie
What other languages does the OpenBSD people use? Shell scripts? Maybe
something else?

~~~
tedunangst
There are some larger perl programs (pkg) and some smaller awk/sh/etc.
floating around. Not sure about using them for a web server though.

~~~
slurry
Awk for a web server? It's been done.

[http://awk.info/?tools/server](http://awk.info/?tools/server)

------
marc_omorain
Is there a technical reason why you would implement HTTPS in a HTTP server? If
you ran a separate process on port 443 to terminate SSL connections, and then
proxy that request to a HTTP server running locally, there would be better
separation of concerns.

For example, this setup would mean that a security flaw in the HTTP server
that allowed a user to read memory would not be able to read any private keys
used in the HTTPS server.

I guess some downsides would be some extra latency while the request is
proxied, and some extra memory overhead for the second process.

I'm interested in anyones thoughts on this.

~~~
throwaway2048
OpenBSD has added support into libressl for privilege separated processes that
hold SSL keys, any operation requiring the use of private keys such as the
creation of session keys, and signing things are shuttled off via a small api
to a separate process. This is somewhat analogous to what ssh-agent does for
openssh clients.

OpenBSD's TLS private key consuming daemons have moved to this model or are in
the process of doing so. This helps to mitigate the problem of access to
process memory results in disclosed private keys, also the requirement of the
daemon's user facing bits to have access to the keyfiles.

[http://article.gmane.org/gmane.os.openbsd.cvs/139527/](http://article.gmane.org/gmane.os.openbsd.cvs/139527/)

------
zx2c4
I love these guys:

[https://github.com/reyk/httpd/blob/master/httpd/server_http....](https://github.com/reyk/httpd/blob/master/httpd/server_http.c#L802-L804)

~~~
fafner
At least they added Comic Neue. I'm a bit disappointed that a project like
OpenBSD that is so vocal about free software is promoting a proprietary font
like Comic Sans MS.

~~~
termie
Not sure if whooshed, but just in case anyone else doesn't get it, you're not
going to see Comic Sans unless you load it from a Windows machine or a Unix
distribution that's installed the font. IOW, it's just a gentle jab at
Microsoft and perhaps those distributions that install the MS fonts.
Technically that is promotion in name but not spirit.

~~~
fafner
Well the OpenBSD devs seem to use Comic Sans in their presentations though

E.g.,
[http://www.openbsd.org/papers/ru13-deraadt/index.html](http://www.openbsd.org/papers/ru13-deraadt/index.html)

~~~
termie
You're right, they use it as a weapon of snark and for fund raising:

[http://www.openbsd.org/papers/bsdcan14-libressl/mgp00025.htm...](http://www.openbsd.org/papers/bsdcan14-libressl/mgp00025.html)

------
detaro
Supports TLS using LibreSSL, serves static files and FastCGI.

[https://github.com/reyk/httpd/issues?q=label%3Afeaturitis+is...](https://github.com/reyk/httpd/issues?q=label%3Afeaturitis+is%3Aclosed)

featuritis tag in die bugtracker for currently denied features. Clearly aiming
for as simple as possible while being useful.

~~~
bluetech
What is wrong with compressing static files?

~~~
falcolas
That's a good question. Though a better question might be "why not serve pre-
compressed static files alongside regular files?"

Compression will change your server from being network throttled to being CPU
throttled. And in this day and age, we can scale CPUs more easily than we can
bandwidth.

~~~
bluetech
Pre-compressing is possible, but why shouldn't the http server do it for you?
The server can cache the compressed files in memory and/or disk as it sees fit
(although gzip was pretty cheap last time I checked). It can also deal with
cache invalidation better than humans - I speak from experience :).

~~~
falcolas
Very true. The difference lay in the benefit of not having to implement the
compression logic as part of the server, making the server implementation much
simpler.

------
ezequiel-garzon
If I may take this opportunity... Does anybody know what I'm supposed to put
in /etc/ssl/server.crt for SSL encryption? I have concatenated all six
possible permutations of my own certificate ssl.crt, the intermediate
certificate sub.class1.server.ca.pem and the root certificate ca.pem, but this
gives me the error _The certificate is not trusted because no issuer chain was
provided. (Error code: sec_error_unknown_issuer)_ (my Ubuntu Chrome gives me a
green lock, though). Feel free to visit my blank site [https://ezequiel-
garzon.net](https://ezequiel-garzon.net)

Thanks!

~~~
cesarb
In cases like this, turn to Qualys' SSL tester:
[https://www.ssllabs.com/ssltest/analyze.html?d=ezequiel-
garz...](https://www.ssllabs.com/ssltest/analyze.html?d=ezequiel-garzon.net)

It shows your server as sending only one certificate, the one with
"CN=ns.ezequiel-garzon.net". It's missing the next one in the chain,
"CN=StartCom Class 1 Primary Intermediate Server CA". I don't know the
configuration details for the server you're using, but many servers use a
separate "chain" file for the intermediates; if that's the case, you should
put the main certificate in one file and the "StartCom Class 1 Primary
Intermediate Server CA" in the other file.

And why it works in some browsers? Notice that Qualys listed the intermediate
as "Extra download"; some browsers can download the intermediate certificate
directly from the CA's web server. Some browsers _cache_ the intermediate
certificates they've seen, so if you've visited a properly-configured server
with the same intermediate before, the browser will use the copy from its
cache. But it's not recommended to depend on this; you should always include
all intermediates.

~~~
ezequiel-garzon
Thanks for pointing to this useful resource. It's clear now that the question
is how to make httpd send all three things. I'll bring this up in the mailing
lists. Thanks again!

~~~
Tiksi
You can probably do it with a bundle, the same way it's done for nginx; append
the CA cert to yours:

    
    
        cat your_cert.crt CA_cert.crt >> cert_bundle.crt

------
fcambus
Slides are available here : [http://www.openbsd.org/papers/httpd-slides-
asiabsdcon2015.pd...](http://www.openbsd.org/papers/httpd-slides-
asiabsdcon2015.pdf)

------
andor
_FastCGI: The protocol provides the single and fast interface to serve dynamic
content_

That's a bad choice in my opinion. Without reverse proxy functionality httpd
can't match the flexibility of nginx.

~~~
skissane
Why bother with FastCGI? Why not just use reverse proxy HTTP instead? Why
introduce yet another protocol when HTTP works fine?

I didn't write this, but I agree with it - [https://ef.gy/fastcgi-is-
pointless](https://ef.gy/fastcgi-is-pointless)

~~~
tveita
Unsurprisingly his "400-ish line C++ header that implements an HTTP server"
falls far short of supporting the full HTTP spec.

For instance it looks like multiple occurrences of the same header will be
ignored, and it will just close the connection instead of returning status 400
on protocol errors - in the best case.

Sure, you can try to implement just the parts of the HTTP spec that you think
you'll need, and hope for the best. But wouldn't you rather use a simpler
protocol that you could implement fully without having to worry about subtle
errors sneaking in because you parsed the request line slightly wrong, or
because your proxy decides to send two X-Forwarded-For headers instead of one.

If you're writing in Java or C++ you'll be able to find a mature HTTP library
that handles all of this for you, but if for whatever reason I were writing
from scratch, I'd take a limited-purpose protocol like FastCGI any day.

~~~
jyujin
That's because the code was never meant to be used in production. It was meant
to be used on a unix socket behind an nginx instance, which pretty reliably
fixes up almost all the slight nuances and omissions in the protocol.

That said, closing the connection when you get rubbish input is generally a
perfectly reasonable strategy, especially behind a reverse proxy that'll clean
up after you. And in the code that is only done at all if asio.hpp claims that
the underlying (tcp or unix) socket is in an error state. At which point it is
impossible to reply.

As for repeated headers, according to
[http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2](http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2)
all multi-line headers must be representable as a single header by
concatenating the parts with commas. So there was simply no use case for
implementing this.

And FastCGI is definitely not simpler or "more limited-purpose" in any meaning
of the word. At least not if you insist on implementing the full spec, which
includes fun things like the authorisation and filter roles. And I highly
doubt you can convince any conformant implementation to parse duplicate
headers for you.

So implementing a reasonable subset - e.g. clean queries for HTTP or merely
the responder role for FastCGI - is a perfectly reasonable strategy. And given
all that, I will take a text-based protocol that in a pinch I can query
against directly with a web browser or telnet any day.

------
vacri
Why use a name that's already in use as a general descriptor? At least the
other httpds have names that can be used to differentiate them:
[http://en.wikipedia.org/wiki/Httpd](http://en.wikipedia.org/wiki/Httpd)

~~~
wongarsu
In practice, everybody will just say "OpenBSD httpd" and there will be little
to no name confusion.

~~~
vacri
Ugh, there's 10 syllables in that. Maybe they should call it o10d?

------
captaintod
Surprised that nobody has mentioned ashd yet:
[http://www.dolda2000.com/~fredrik/ashd/](http://www.dolda2000.com/~fredrik/ashd/)

------
jalfresi
Does anyone know if the FastCGI implementation is complete i.e. it supports
FastCGI processes in all three roles; Responder, Authorizer and Filter? I've
always wanted to use FastCGI more but most implementations (in Apache and
Nginx at least) only support some of those roles (or require work arounds
using server specific features; i.e. apache filters rather than FastCGI
filters)

~~~
tedunangst
I think it's "it runs the software we tested with" complete.

------
jnazario
[update - i read the back story elsewhere and the reason is less boneheaded
than i had assumed. still, i think the community needs to focus on higher
priority needs and gaps]

this is the sort of thing that makes me happy i'm no longer involved in the
OpenBSD world. httpd & previously smtpd are two replacements that (in my
opinion) have little additive value beyond existing, community-adopted
solutions (e.g. nginx and postfix), diluting effort where it is needed.

does the world need a new httpd? maybe. but the world needs other replacement
software to be done first because it'll have a greater impact.

for example, OpenBSD could invest time and effort in maturing static code
analyzers to assist in code audits (especially of ports).

i suspect this new httpd was done less because it was needed and more because
it could be done. that's the attitude i disagree with.

~~~
floatboth
OpenSMTPD is much, much simpler than Postfix. I can't stand Postfix's fucking
1000 line multi-file configuration.

Nginx is simple, so, yeah – no need for a replacement there.

~~~
clarry
> Nginx is simple, so, yeah – no need for a replacement there.

Nginx is approximately twice the size of the old Apache 1.3 based httpd
OpenBSD had in base.

Nginx is an order of magnitude larger than the new httpd.

Perhaps nginx is simple from a user's perspective. But from a code/complexity
viewpoint (think of all that must be read and verified to make sure it is
correct, clean, simple and secure?) it is not quite so simple.

------
joosters
OpenBSD seems to have caught a bad case of the 'not invented here' sickness.
If they didn't like where nginx was going, why not just fork it and have a
working web server with a known codebase? The forks would diverge but they
could still grab fixes from nginx whenever they wanted to.

~~~
stsp
This wasn't a question about having a functional webserver or not. It was a
conscious decision against running code without reading it first. The "run but
don't read" strategy belongs into ports, not base. Hence nginx moved to ports.

Forking nginx was actually discussed when a proposed nginx update diff was too
large for proper review. Tons of complex regex parsing code was added with
nobody willing to go through it all in detail.

The forking option was quickly dismissed. The tipping point happened when
someone (reyk?) pointed out that relayd had most of the guts of a complete web
server anyway, including OpenBSD-style privsep which has to be bolted on to
virtually all software imported from elsewhere, including nginx. It took about
a week from that discussion happening to having a relayd-based functional and
peer-reviewed web server in base (+ a couple of months of shaking out a few
bugs and adding minor features).

~~~
jimktrains2
> he "run but don't read" strategy belongs into ports, not base. Hence nginx
> moved to ports.

Why is there a webserver in base?

~~~
jquast
For the same reason the other 20+ net-daemons are:
[http://cvsweb.openbsd.org/cgi-
bin/cvsweb/~checkout~/src/etc/...](http://cvsweb.openbsd.org/cgi-
bin/cvsweb/~checkout~/src/etc/rc.conf?content-type=text/plain)

With OpenBSD, you can actually avoid the ports tree all together and still
play a critical role in your infrastructure.

If you're going to distribute a system that doesn't do much of anything on its
own, and delegate the responsibility of security (and choice!) to "the ports
tree" you're not doing any better than Linux or FreeBSD.

~~~
jimktrains2
The majority of users are not serving static pages, so unless they bundle
their own JDK, Python, PHP, Ruby installs you have to go to the ports tree
anyway?

------
davidgerard
I read the config file format and I fell in love.

I really hope this gets the portable treatment.

------
cturner
What would be the elegant way to implement websockets on the new openbsd
arrangement? Would it be to use relayd instead of httpd? Or is websocketd
suitable for the openbsd base?

~~~
detaro
Probably to install nginx from the ports and use that. I don't think
supporting web sockets would fit the scope of the base system.

------
floatboth
Why do they even have an httpd in base? They like to say they're smaller and
simpler than FreeBSD, but FreeBSD doesn't include a web server in base!

~~~
stsp
One reason is that network installs require http. The base system would be
unable to self-replicate across network installs without httpd.

~~~
floatboth
Why not FTP? Or even TFTP?

~~~
tedunangst
FTP is dead.

~~~
wglb
How I wish this were true.

~~~
protomyth
Yeah, don't you just love how "Enterprise" vendors make it more secure with
going with ftps instead of picking a different protocol?

------
dyoder
Did OpenBSD just standardize on an HTTP server they wrote in 2 weeks, has no
tests, doesn't fully implement the spec…and then brag about it?

~~~
tedunangst
No.

------
mdekkers
any performance benchmarks in the wild?

------
kymywho
Https authentication support for Subversion could be the killer feature.

------
amelius
I'm amazed we still use configuration files like that. Why not have some
capable programming language (python, javascript) handle the configuration?
This would let one more naturally describe things, and will eventually reduce
the possibility of obscure configuration commands that do one very specific
thing, and hence lead to frustration when the user needs something slightly
different. Using a programming language, one could even configure using
callbacks, etc. and is much more flexible.

~~~
m0th87
I'm not a lisper, but this makes me pine for some universal, maybe lisp-like
language. It's a total waste of time to learn a new config syntax for each
application.

~~~
gjggkjasdf
the ironic thing here is that Lispers fucking _love_ sticking new DSLs
wherever they possibly can. And I imagine most developers of these pet config
files would argue the same thing. Unfortunately.

It was Microsoft of the 1990s that surprisingly brought some sanity to the
table, with INI files. I'm glad tools like Git use this format.

~~~
ams6110
Then they threw sanity out the window with the registry.

