
A fast and static web server for serving web apps - timothy89
http://shortfin.io/
======
jfb

      Just run the following command as root to install the server.
    
      # wget http://shortfin.io/install.sh && sh install.sh
    

What? Are they serious? Is this becoming a thing? Please don't tell me that
this is a thing.

~~~
tlrobinson
Yes, it's a thing, and aside from the lack of HTTPS it's not much worse than
your standard "./configure", "make", "make install" install flow of the last
20-whatever years of open source. If you don't trust the developer to serve
you a safe install.sh script you shouldn't trust the rest of their code
either. If you don't trust them you should audit the code no matter what the
install process looks like.

That said, if you're going to do this, you should do the following:

1\. HTTPS only. No excuses.

2\. No URL shorteners, which make or may not use HTTPS, and introduce another
point of failure.

3\. Ideally serve directly from Github repo. Eliminate another point of
failure. The user can be reasonably confident that the install script is the
same one that's in the Github repo.

~~~
ebbv
It's a lot worse than the standard configure, make and make install.

For one, it's only requiring one simple script to get compromised and replaced
to compromise your entire machine. At least when I download the tarball,
unpack it, etc. I've got several steps before I even get to the make install
that I would run as root. And I might not even do that depending on my install
target. If I do, I have ample opportunity to verify the package in question
and examine the script I'll be running via sudo before I do so.

With the instructions provided by this site, not only is there no opportunity
to verify the script I've received is the one I wanted to (HTTPS alone is not
enough for this, the site or domain may have been compromised) but they're
instructing me to just run it immediately as root without any examination.

This is a terrible practice, and while yes it has been around for a long time
in the past it was something only done and encouraged by idiots, it seems to
be gaining popularity.

This is a bad idea, stop doing this people.

Provide an install tarball and provide a checksum of it that we can verify.

~~~
skrebbel
I'm a Windows user. Virtually every piece of software I install comes in the
form of an executable, that I download-and-run directly from the browser. I
often give this executable administrator priviledges. The executable can then
do whatever it wants on my computer. Usually it will install said piece of
software (except when it was made by Oracle, it'll install said software _and_
the Ask.com toolbar)

This has been standard practice for decades. It's very comprable to that wget
oneliner. Why is it bad? Or, why is the wget oneliner bad but not Windows
installers? Do Windows sysadmins compile tarballs and verify checksums? Or are
Windows sysadmins simply dumber/less paranoid than Linux sysadmins?

~~~
ebbv
You're giving those executables full control over your computer. Yeah, you
won't get burned very often (depending on where you're getting these
executables.) But when you do get burned it's going to be catastrophic. That's
the problem.

It's a bad practice.

~~~
skrebbel
Hm, I see your point.

How would you then recommend installing non open source software on a
production system? E.g. a freeware (but proprietary) FTP server or a database
engine.

~~~
Arnor
1\. Don't install software from untrusted sources. 2\. The trusted sources
should provide checksums so you can verify the package before installing it.

This protects you from a compromised package so long as the page you accessed
the package from has not been compromised (if that page is compromised, the
checksum could be changed to the new one).

I have no idea how you would verify closed source software if the checksum and
package are both corrupted...

~~~
donutz
"2\. The trusted sources should provide checksums so you can verify the
package before installing it."

If the checksum is published to another trusted location, in addition to where
you download the files, that would help in the case that an attacker
compromises both the checksum and the package on the download site.

The developer could post the checksum to Twitter at the same time as making
the package available for download. An attacker would need to compromise both
Twitter and the download site.

------
c-a
The request parsing code seems to be of dubious quality
[https://github.com/timothyej/Shortfin/blob/master/src/reques...](https://github.com/timothyej/Shortfin/blob/master/src/request.c#L39)

From a cursory glance:

1\.
[https://github.com/timothyej/Shortfin/blob/master/src/reques...](https://github.com/timothyej/Shortfin/blob/master/src/request.c#L69)

should be (data_len - i >= 4) since it accesses data[i+3]

2\.
[https://github.com/timothyej/Shortfin/blob/master/src/reques...](https://github.com/timothyej/Shortfin/blob/master/src/request.c#L125)

shouldn't headers[header_count]->key also be null terminated?

3\.
[https://github.com/timothyej/Shortfin/blob/master/src/reques...](https://github.com/timothyej/Shortfin/blob/master/src/request.c#L144)

header_count can become greater than 49 which causes a heap overflow at
[https://github.com/timothyej/Shortfin/blob/master/src/reques...](https://github.com/timothyej/Shortfin/blob/master/src/request.c#L148)
and
[https://github.com/timothyej/Shortfin/blob/master/src/reques...](https://github.com/timothyej/Shortfin/blob/master/src/request.c#L154)

~~~
timothy89
I'll look them up, thanks for noticing!

------
ds9
What's a "static web server"? I haven't seen this term.

A webserver for only static pages, i.e. it won't talk to a runtime? But it's
"for serving web apps", this means all the dynamic content is pulled from
client side JS? Then you would still need programs on the server to answer
AJAX requests.

I did not find an explanation on the page or by Googling.

~~~
sehrope
A server that only serves static files. It doesn't enable the use of dynamic
content like CGI[1]. Compare this to something like Apache or nginx. Both can
serve static files but also support a number of ways of either directly
running scripts (ex: mod_perl or mod_php in Apache) or proxying to other
servers that handle dynamic content.

[1]:
[https://en.wikipedia.org/wiki/Common_Gateway_Interface](https://en.wikipedia.org/wiki/Common_Gateway_Interface)

~~~
ds9
OK, then, sorry if I'm being dense, but how can it be "for serving web apps"?
App seems an odd term for a site consisting of only static files.

~~~
packetslave
Perhaps you've heard of this new language, JavaScript?

~~~
ds9
Still not much of an app unless the JS does AJAX, but then it's talking to a
server that's running some dynamic process, maybe with DB access etc. - right?

~~~
Isofarro
Not necessarily. You can make an ajax request to a static resource. The server
doesn't have to be a dynamic process. Static resources are just files.

A client-side JavaScript Twitter could be done this way. Every tweet, or
timeline can be pre-baked into a static json file, and ajaxed in.

Twitter can then be simplified to a backend application that updates these
files.

Granted, tweeting is a separate matter; that requires an endpoint that
processes data.

~~~
WhitneyLand
Cross-domain REST is the big win. MongoLab, APIs, Widgets, etc.

------
ChuckMcM
Close.

Can add this to the list like thttpd.

Something I keep thinking I'll build is a fast, high connection count,
_limited_ HTTP server, something that is essentially a wrapper around a
program that works like 'regular' and emits HTML. It is a corner case in a
custom corner, but the target it something which is essentially a
'transponder.'

In the 'Internet of Things' I want to build a wrapper/environment such that my
program can be

    
    
      main(int argc, char *argv[]) {
        uint16_t sense;
        uint16_t chan;
       
        sense = adc_read(atoi(argv[1]));
        printf("Sensor %d reads : %d\n", chan, sense);
      }
    

And then link my 'wrap around web server on it' and then it can be accessed
with wget '[http://ip:port/?1'](http://ip:port/?1') have it do the right
thing.

The keys are low memory footprint, lots of connections, easily wrapped around
a 'regular' program.

~~~
tlrobinson
_" easily wrapped around a 'regular' program"_

Sounds like you want CGI :)

~~~
ChuckMcM
Yes, "static CGI" if such a thing was a thing :-). I've got a thttpd tree that
I do this sort of hack in (these are great for monitoring large numbers of
servers for example).

~~~
skrebbel
I don't understand (and am interested). What's "static" about it? With Apache
and mod_rewrite and CGI, I could make the functionality you want (which is, of
course, completely not low footprint, but I don't understand how what you want
is different from CGI).

~~~
ChuckMcM
Static in the sense that it always executes the one function (doesn't load it
from disk, its part of the executable) and so there is never any risk that
some other path might get you 'out' of the docs directory and into the cgi_bin
directory. So in this case static is code for 'compiled in' versus
'dynamically loaded.'

~~~
mbreese
Like a statically linked HTML document? I know you're talking C code, but I
could imagine some kind of HTML to C "compiler" that could you then statically
link into an executable. Then you could have a full website that ran as a
single static executable, including CGI-esque calls.

~~~
icebraining
It's actually rather easy: [http://www.linuxjournal.com/content/embedding-
file-executabl...](http://www.linuxjournal.com/content/embedding-file-
executable-aka-hello-world-version-5967)

~~~
mbreese
That's cool! I wonder if this would be (much) faster than just letting nginx
or this embedded httpd read the files directly and serve them. I would hope
that the OS or server would cache things enough to make it unnecessary, but it
would be interesting to find out.

HTML isn't that big, so if you used linked resources, it would be pretty
simple to make a C program and compile it to a binary for any purely "static"
site.

~~~
damien
That's how the Kore web server works ( [https://kore.io/](https://kore.io/) );
it makes you generate a static.h file at compile time that contains all of you
your static content.

But like you say, I have to wonder if it actually makes a difference
performance-wise. The kernel will cache the static file data anyways. The only
improvement I can see is maybe to avoid having to look up file metadata
(length and mtime)? Or to avoid a syscall?

------
babuskov
I don't get it. You create a server to solve C10K problem, and then host its
own website with nginx? Why?

Having HN crowd coming and testing your server would be a great test, no?

~~~
timothy89
I host more websites on the same server and as kajecounterhack said I'm using
nginx as a reverse proxy. The server is fast and have no problems at all with
the HM crowd ;)

~~~
babuskov
What are advantages over nginx? Memory footprint, higher through-output?

I mean, what's the main "selling point". Why should one use this instead of
battle tested nginx?

Don't get me wrong, I'm not criticizing, I'm just curious.

~~~
timothy89
On my tests it's a few 100 requests faster per second than nginx but the
average should be around the same. The kernel is the bottleneck in this case I
believe.

There isn't any "selling point" ;) Anyone could use it as they like. The main
usage for me has been in other projects where I needed a fast and lightweight
http server. Such as tracking servers etc. It's pretty "experimental" but I
think I will use the "original" server to serve a web app that I will be
launching soon. The problem for me has been the lack of a reverse proxy.

------
reidrac
I was going to say that there are *~ files in the repository, they should tidy
a little bit because things like that don't look good.

Then I thought may be I was being a little bit to picky and unfair, so I
checked the code of some random files and... well, it looks like it was part
of a learning experience but it definitely needs work before being considered
a viable option.

------
dictum
>wget [http://shortfin.io/install.sh](http://shortfin.io/install.sh) && sh
install.sh

Isn't it recommended that any project doing one-line installs use HTTPS?

~~~
bru
Side-note for the author: the download can be made more elegant that way

    
    
        $ curl http://shortfin.io/install.sh | sh
    

I definitely agree with the HTTPS. For example this is what meteor proposes:

    
    
        $ curl https://install.meteor.com | sh

~~~
pnathan
asdfsdfgkjlsdfg!

Streaming your install script directly into computer execution without even
giving it a once-over, md5 compare, etc is atrociously insecure.

~~~
chc
That's completely orthogonal to what we're talking about here. If you're
already not doing it, you're not doing it any less by being more efficient.
And it doesn't really matter if you're running it in a throwaway VM like I do.

------
joeblau
I just ran two speed tests since there aren't any benchmarks provided

1\. First Test is using
[http://www.webpagetest.org/](http://www.webpagetest.org/)

Results shortfin.io:88 -
[http://www.webpagetest.org/result/130806_71_13HK/](http://www.webpagetest.org/result/130806_71_13HK/)

    
    
      First View: 1.945s
      Repeat View: 1.632s
    

Results shortfin.io -
[http://www.webpagetest.org/result/130806_QV_13F7/](http://www.webpagetest.org/result/130806_QV_13F7/)

    
    
      First View	2.099s	
      Repeat View	0.084s
    

2\. Second Test is using Apache Bench

Results (Best results of 3 runs): ab -n 100 -c 100
[http://shortfin.io:88/](http://shortfin.io:88/)

    
    
      Time taken for tests:   12.125 seconds
      Requests per second:    8.25 [#/sec] (mean)
      Time per request:       12124.501 [ms] (mean)
      Time per request:       121.245 [ms] (mean, across all concurrent requests)
      Transfer rate:          38.46 [Kbytes/sec] received
    
      Connection Times (ms)
                  min  mean[+/-sd] median   max
      Connect:      210 1114 1517.2    360    4041
      Processing:  5017 8407 2034.0   8715   11762
      Waiting:      169 1084 3054.8    188   11408
      Total:       9057 9521 1068.7   9058   12123
    

Results (Best results of 3 runs): ab -n 100 -c 100
[http://shortfin.io/](http://shortfin.io/)

    
    
      Time taken for tests:   5.790 seconds
      Requests per second:    17.27 [#/sec] (mean)
      Time per request:       5789.949 [ms] (mean)
      Time per request:       57.899 [ms] (mean, across all concurrent requests)
      Transfer rate:          82.97 [Kbytes/sec] received
    
      Connection Times (ms)
                    min  mean[+/-sd] median   max
      Connect:      171  249  35.7    263     290
      Processing:   172 1798 2345.5    212    5524
      Waiting:      170 1055 1825.1    212    4949
      Total:        344 2047 2368.2    469    5789

~~~
rarestblog
You need some better hardware

$ ab -n 100 -c 100 [http://shortfin.io/](http://shortfin.io/)

Requests per second: 1363.85 [#/sec] (mean)

~~~
rorrr2
Benchmarking a fast remote static server is pointless.

You're basically benchmarking your internet connection.

------
nine_k
I think the landing page would greatly benefit from two short paragraphs:

 _Why Shortfin when we have Nginx /Lighttpd?_

 _Why Shortfin when we have thttpd?_

~~~
timothy89
1\. Shortfin is not a replacement for nginx. I use it myself to create other
projects, e.g. fast API servers with one purpose only and a REST server that
uses redis as storage.

2\. See #1.

3\. I don't try to sell anything, it's just an open-source project that I
think maybe someone could benefit from. I've learned a lot while coding it.

~~~
nine_k
OK, Shortfin is just trying to be an efficient HTTP server, without a radical
differentiating feature or implementation approach.

Well, let a hundred flowers blossom.

------
eksith
While I appreciate the time and effort that went into this, I'm more concerned
with security than just speed alone. It's still relatively easy to throw
hardware at the load problem (up to a point, naturally), but safety shouldn't
take a back seat to performance.

The wget shell script installation does make me nervous and I'm glad the
source is available separately. Blind installation of scripts was never "a
thing" with me.

Of course, I'm still glad people are writing proper web servers (as opposed to
simple 2-10 line ones). That creates an opportunity to explore the field with
fresh ideas.

------
zzzcpan
I find it very hard to trust any C code without tests. Is this a thing for C
code? Why most of the new projects in C lack tests?

------
el-mapache
My favorite part is this, under the config section: # vim
/etc/shortfin/shortfin.conf # /etc/init.d/shortfind restart

No explanation of what the configs are, just that "its important to configure
it right."

Why not put this information on the site?

------
riledhel
I can't find anything related to the response headers the server can send to
the client, and the OP site just serves responses with content-length,
content-type and server. Anyone?

------
halayli
This code is far off from being production ready. Skimming through the code
quickly, there's barely any error handling. HTTP parsing is not compliant
either.

[https://github.com/timothyej/Shortfin/blob/master/src/respon...](https://github.com/timothyej/Shortfin/blob/master/src/response.c#L5)

I am not sure what's the point of using it instead of nginx.

------
sciurus
There's a sample configuration at
[https://raw.github.com/timothyej/Shortfin/master/config/shor...](https://raw.github.com/timothyej/Shortfin/master/config/shortfin.conf)

------
tunnuz
You're from Lund! I've been studying in Lund!

By the way, can you show some statistics and performance metrics with respect
to nginx, Apache and such?

To put it another way: why did you choose to write your own webserver instead
of using, say, nginx?

~~~
davidcollantes
I am curious too, on the performance metrics. He is still using Nginx though.

~~~
timothy89
Nginx is just acting as a reverse proxy because I host other sites on the same
server. Benchmarks will come!

------
ahknight
okay ... why? what's special about this? features? benchmarks? anything?

~~~
grey-area
Why not?

The author claims it's 'high-performance and open-source' \- sounds good to
me. It's not as if we have a hundred web servers all competing as we do static
blogging systems, the more web servers the better in my opinion, just as with
web browsers.

The zip file is 312KB as linked from here, so it's pretty compact, possibly
suitable for embedding in apps, etc:

[https://github.com/timothyej/Shortfin](https://github.com/timothyej/Shortfin)

------
timothy89
A lot of you wanted benchmarks so here's one comparing Shortfin with Nginx.
The tests was performed with a 56.1 kB PNG image with keep-alive turned off on
my laptop. The best result out of 3 tests is shown below.

 _tl;dr:_

Shortfin: 18 914 req/sec

Nginx: 15 603 req/sec

 _SHORTFIN_

    
    
      sudo ab -n 100 -c 100 http://127.0.0.1:40/timothy-johansson.png
    
      Server Software:        shortfin/0.9.5
      Server Hostname:        127.0.0.1
      Server Port:            40
    
      Document Path:          /timothy-johansson.png
      Document Length:        56089 bytes
    
      Concurrency Level:      100
      Time taken for tests:   0.005 seconds
      Complete requests:      100
      Failed requests:        0
      Write errors:           0
      Total transferred:      5618000 bytes
      HTML transferred:       5608900 bytes
      Requests per second:    18914.32 [#/sec] (mean)
      Time per request:       5.287 [ms] (mean)
      Time per request:       0.053 [ms] (mean, across all concurrent requests)
      Transfer rate:          1037701.56 [Kbytes/sec] received
    
      Connection Times (ms)
                  min  mean[+/-sd] median   max
      Connect:        1    2   0.0      1       2
      Processing:     2    2   0.1      2       2
      Waiting:        1    1   0.2      1       2
      Total:          4    4   0.1      4       4
      ERROR: The median and mean for the initial connection time are more than twice the standard
           deviation apart. These results are NOT reliable.
    
      Percentage of the requests served within a certain time (ms)
      50%      4
      66%      4
      75%      4
      80%      4
      90%      4
      95%      4
      98%      4
      99%      4
      100%     4 (longest request)
    
    
    

_NGINX_

    
    
      sudo ab -n 100 -c 100 http://127.0.0.1:41/timothy-johansson.png
    
      Server Software:        nginx/1.2.6
      Server Hostname:        127.0.0.1
      Server Port:            41
    
      Document Path:          /timothy-johansson.png
      Document Length:        56089 bytes
    
      Concurrency Level:      100
      Time taken for tests:   0.006 seconds
      Complete requests:      100
      Failed requests:        0
      Write errors:           0
      Total transferred:      5631000 bytes
      HTML transferred:       5608900 bytes
      Requests per second:    15603.06 [#/sec] (mean)
      Time per request:       6.409 [ms] (mean)
      Time per request:       0.064 [ms] (mean, across all concurrent requests)
      Transfer rate:          858015.83 [Kbytes/sec] received
    
      Connection Times (ms)
                  min  mean[+/-sd] median   max
      Connect:        1    2   0.7      2       3
      Processing:     1    2   0.5      1       3
      Waiting:        0    1   0.7      1       3
      Total:          2    4   0.9      4       5
      WARNING: The median and mean for the processing time are not within a normal deviation
            These results are probably not that reliable.
    
      Percentage of the requests served within a certain time (ms)
      50%      4
      66%      4
      75%      4
      80%      4
      90%      5
      95%      5
      98%      5
      99%      5
      100%     5 (longest request)

~~~
corresation
I don't know how valid a test of five milliseconds is to one of six
milliseconds (much less one requesting a single known resource with no keep-
alive). Though practically, even if those results held in a realistic test, is
it still a viable alternative to nginx? You eliminate an _enormous_ amount of
flexibility (I admit -- I ♥ nginx) and proven trust for a margin-of-error
theoretical speed advantage?

I understand the desire to get software out there, but webserver is a
ridiculously hard nut to crack. nginx broke in through a new architectural
paradigm.

~~~
timothy89
Nope, I'm not trying to replace nginx. I love nginx too ;) The "challenge" was
to build a very lightweight server that was faster than nginx - which I did.
Then I've used it as a base for a various of projects (see my other comments).

------
BlackDeath3
Very cool. I've been doing the same thing lately, writing my own little web
server that serves static files. It's been a fun learning experience!

~~~
timothy89
Nice, in what language? I've really learned a lot from creating shortin!
Shortfin is written in C so it's a nice project to spend some time on when I
get tired of writing javascript and html ;)

~~~
BlackDeath3
I'm writing mine in C as well. I've never been too into web programming, but
the lower-level stuff is quite interesting to me.

Keep it up!

------
nickzoic
Lots of people have asked "Why not Nginx? Why not thttpd?" ... but if all you
want to host is static files, why not AWS S3 or similar CDN?

------
tuananh
author should add a benchmark, comparison. or maybe it's just a fun, personal
project and not ready for production.

------
simon_vetter
I would recommend adding ipv6 support. Should be fairly easy to do in
socket.{c,h} and in your configuration parser.

------
ksec
Waiting For Benchmarks.

