Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Servasm – Literate web server in x86 assembler (zarkzork.com)
117 points by zrkzrk on June 19, 2015 | hide | past | favorite | 37 comments

If anyone finds this interesting, and would like to drill down into this type of low level code, I highly recomment "Programming from the Ground Up" by Jonathan Bartlett [1]. It is an excellent introduction to assembler, and programming in general.

[1] http://freecomputerbooks.com/Programming-from-the-Ground-Up....

The non-spam link to download is here: http://download.savannah.gnu.org/releases/pgubook/

That's really fascinating and interesting and educating.

Now every other http server implementation should need to explain every byte it is larger and every ms it is slower it terms of "why?" and "what for?" and "who gains from this?" ;)

I don't know, for longer already I don't buy this tales about "it needs to be this large because..." ... mostly legacy, abstractions and ease of code maintenance etc. This took us all into the world in which the very smallest app on the phone reacting to a click with a "beep" takes how much memory? And the software development craft accepts unbelievable inefficiencies. Memory, Cpu manufactures for long time added to this fires by essentially mis-nurturing devs by optimizing in the background and by creating an environment of limitless virtual resources. I like how "battery-life" enforces, brings back some old ideas on efficiency and software craftsmanship.

Humanity starts to deal with limits of its planet, its own limits and maybe this kind of thinking will bring back some level of limits into the virtual realms as well? I think we would gain from it.

Are you being serious, or just trying to sound super philosophical about why "less is more"? I assume it's the latter, because it shouldn't take very much thought to see why HTTP servers are fairly complex pieces of software, given the purpose they serve - a class of software that is ruthlessly optimized in today's world - nor should it take very much thought to see this HTTP server is actually ridiculously inefficient compared to any modern one. You know, wasting all those precious CPU cycles you opine so much about?

(Now, if you want to talk about the actual efficacy of HTTP vs other stateless protocols, that's a different story... But I doubt we're going to go there in a thread about an assembly HTTP server where people are ooh'ing over the achivement. Not that it isn't cool, TBQF.)

Programmers at large really need to stop being pretend philosophers clutching at straws about "why things are so darn bad today!!!" (among other psuedo philosophical positions). They're mostly terrible at it, and it's almost always just so damn hamfisted and full of itself.

Please be a little more charitable when commenting on HN.

There is a long (albeit minority) tradition of thinking this way in computing, one that values small, intelligible systems as the best way for humans to work with computers. An example of this philosophy surfaced here recently (https://news.ycombinator.com/item?id=9689800) and there are countless others.

HN itself has a rich history with this model. It was created by a practitioner of it, is written in a language inspired by it, and the smallness and intelligibility of the code are always on our minds when we work on it.

We need more projects like this. They are deeply satisfying systems to build and work with, because they're human-scale in the way that behemoth software is not.

>There is a long (albeit minority) tradition of thinking this way in computing...

It seems close to the Occam's razor.


I like this quote from Chuck Moore (from Forth).

"We need dedicated programmers who commit their careers to single applications. Rewriting them over and over until they’re perfect. Such people will never exist. The world is too full of more interesting things to do. The only hope is to abandon complex software. Embrace simple."


Thank you for helping fight the Dijkstras of this world.

thoughtpolice: "because it shouldn't take very much thought to see why HTTP servers are fairly complex pieces of software"

Dijkstra: "He was the first to make the claim that programming is so inherently complex that (...)" (from Wikipedia)

Down with industry pessimists and "astronauts" that think everything must be complex.

Wow, chillax boy!

You did overlook the ";)", right? I did not intend to pluck that string of yours. Sorry about that.

I chided thoughtpolice for being a little uncharitable, but this comment breaks the HN guidelines outright. It's unsubstantive and uncivil (though I don't think you meant it to be). Please don't post such things.

Very nice.

However, I wonder who started this trend of bundling "better commented code" with "literate programming" though?

I appreciate the layout and hyperlinks etc - but this really is just a well-structured assembly program laid out in a way that it won't assemble until after it's been through a pre-processor (I'm talking about the program as presented with html/css etc). It's pretty far from "literate programming".

I suppose one could argue that if you manage to simplify the structure of your program to the point that it reads like prose, one has a "literate program". But it's a strange use of the term. The core idea is to have the code be incidental to the commentary, so that, among other things, one would update the commentary whenever one change the program.

Laying out the comments in a funny way doesn't quite do that. I first saw this with the "literate" re-write of coffee script, but perhaps it's older?

Perhaps the difference between "old" literate programming, and this style ("prose programming"?) is similar to the difference between unit testing and TDD, or between TDD and BDD?

One functional feature that is missing is that of having blocks in different order. Jeremy Ashkenas argues that because of functions, we do not need that reordering.

I disagree with that. I do think a large part of what makes literate programming powerful is that one can easily sculpt the order of the presentation. It elevates the code and comment to telling a story for humans.

I think the code is not incidental, but is part of the aspect of the story, much like in a fictional story one has descriptive passages and dialog passages. If one updates the story, both may change or perhaps just one of them.

My take on literate programming, using markdown: * minimal client at https://www.npmjs.com/package/litpro * core library and docs at https://github.com/jostylr/literate-programming-lib

I agree that functions are not usually a good one-to-one fit for abstraction.

If they were, we wouldn't need variables, loops, blocks etc. Now you could argue that we don't, we should just write code in lambda calculus -- but we don't do that.

Especially for languages like assembler and C-like languages, it can be very nice to single out smaller sections of code. And while for eg: C, or I suppose a macro assembler, one might be able to in-line a lot of such blocks -- having to stay at the "block-semantic"-level of the host language can make some things pretty hard to communicate to the human reader in a good way.

Ruby might actually be a good candidate for "simple" literate programming, in the sense that one probably could, and perhaps sometimes should, program ruby much like smalltalk -- no method/function longer than five lines or so, except for the most exceptional circumstances.

Even then, I think one would find patterns that would make sense to abstract out of the "function level", or "language level".

At the other end of the spectrum is too much magic, just as with any meta-programming technique, such as proper macros.

I had a most peculiar experience trying to write some (mostly procedural) java with noweb. It was typical intro programming stuff, basically some very simple data structures/algorithms.

It was a very nice fit for literate programming, but not such a good fit for java -- while the literate program read nicely, and was well structured, the mangled java code was unwieldy -- it turned out I'd ended up generating a java program that did what I wanted.

Now, that's not really a problem, we generate assembly programs that are unwieldy all the time -- but the point is it took some discipline and thought to write a literate program that was also readable in it's tangled form.

Only an issue if people are expected to read/modify it in that state, obviously. And they are, as they'll have to debug it at some point ;-)

> I think the code is not incidental

That was a bit tongue-in-cheek -- I meant more in the sense that comments are incidental to code in most programming languages.

> https://github.com/jostylr/literate-programming-lib

Very nice, thank you for sharing.

My version of literate-programming may have a bit too much possibility of that magic. I agree it can be dangerous and is something that takes good discipline, like any programming technique. But at least there is something to fall back on in terms of the narrative.

On the flip side of the tangled form, I do appreciate the "flattening" of potential functions. Imagine code that uses a lot of functions, scattered about. It can be hard to follow all that, rather a bit like the GOTO of old (better, to be sure, but still hard to follow). With literate programming one can get the conceptual separation of writing in different blocks, but then they get put back together and the flattened version can be read quite easily, one would hope. So I agree that a locally readable tangled form is very important.

I'd be curious to see your example's code, if you are willing to share it.

> My version of literate-programming may have a bit too much possibility of that magic.

To be honest, that was (part of) my initial reaction. But at the same time, playing with noweb, reading a bit of literate C etc -- I think some tooling on the literate side can be good. Sometimes one wishes for too much, and too late realize that the beautiful unique snowflake of a poem one has wrought is, while splendid in its simplicity, as brittle and hard as ice.

It just not something one will be entirely comfortable handing over to someone else to modify -- because even if they could figure out what it did, they'd be hard press to modify it in any meaningful way. Too dense isn't very good either.

Sometimes I think that literate programming and APL stand at opposite corners of some kind of 2d graph of program complexity/simplicity (not necessarily diagonally opposed). I'm not entirely sure what would be in the other corners.

I came across your blog post, and I think we are in agreement on a lot of things:


But I also think a very real problem with literate programming (to quote Knuth?) is that it demands people to be both good (technical) writers and good programmers. Some are both - more are one or the other.

I think the best way to get a simple, yet featureful literate programming system, is to combine a simple markup language, like RST/Markdown/etc with a language that lends itself to be pulled apart and rearranged. I think something simple, like Smalltalk might be a good candidate.

So far the only real literate programming I do, is with doctests in python -- and to a lesser extent, ipython notebooks.

I remember I looked at Leo: http://leoeditor.com/ -- and have been toying with moving from vim to Emacs+evil partially for the benefit of org-mode -- but these still feel like very heavy solutions to something that I feel should be a rather simple problem. That feeling might be wrong, though.

Another tool I've come across (which might be abandoned, I'm not sure) is: http://pywebtool.sourceforge.net/

Regardless of the state of the tool itself, the page has some interesting points on literate programming.

I must admit, getting something like proper LaTeX typsetting of the code is nice though. I'm still looking for a tool that doesn't botch up the (La)TeX conversion and html+css conversion of simple RST/md-documents -- it seems everyone tries to be way too clever, and bundle the weirdest little themes/template leaving one to pick apart everything just to get some straightforward html/TeX. Not to mention trying to output HTML from (La)TeX.

At least for html output we now have a few half-decent alternatives thanks to everyone writing a static blog engine. And pandoc. Pandoc is great.

[ed: I feel I should preface this to say that this is very old code, and something I'd normally show to anyone. But since you asked... :-) ]

Unfortunately it looks like the code that was the best (worst) example of what I'm talking about isn't readily available (I have it some backup-archive or other...) -- and my other code is partially restructured based on what I learnt, and most significantly in Norwegian.

However, one example, while not breaking the function-gap, is a small sub-section from the noweb document that reads (this is java 1.4):

  import junit.framework.*;
  import junit.extensions.*;

  public class AllTests extends TestCase
    public AllTests(String name) 
       super (name);
    public static void main(String[] args) 

And the testSucceed-section is just a simplistic sanity check to make sure tests can be run/pass:

  /** Testing the test harness */
  public void testSucceed() throws Exception 
    assertEquals(10, 10);
These two blocks, the test setup, and the meta-test go together in a neglected corner of the document, given only a little more priority than the build.xml-file -- while the more functional tests are moved close to the various interfaces (in this case) they are testing (rather than having them too close to the implementations they're actually testing).

So one could say that this gives us injection without an injection framework. Or something. This is basically fixing a naming issue in java, where you need methods to be bound to a class (except for static stuff etc) -- and is an "abstraction free" (at the language level) way of moving things that go better together -- closer together.

Note that the blocks pretty much lacks comments, as they are a bit gnarly to get to work with both java and noweb and javadoc -- but that is more of a tooling issue than anything else.

In this case (imagine 10s of test cases) the tests might be a bit under-documented when seen in isolation. Note that each block actually holds a couple of test-cases applicable to whatever class/interface-pair it tests.

But it gets even more interesting when you're sharing a block of code that implements a binary search on an array-like structure... does make it a little too easy to trip over local variables though.

The full code (in Norwegian) is at: http://folk.uib.no/st05861/inf101/oblig1/ -- it's not very good, but the noweb source and pdf (oblig1.nw, oblig1.pdf) might be of some very limited interest -- and could be contrasted with the java code under src.

This is all from an entry level programming course, the task was given as a pdf-document with some specs and a few stub interfaces -- the resulting pdf essentially interleaves the questions/specs as given, along with the interface stubs -- and builds up answers to each sub-section/point.

The other code I mentioned was more along the lines of implementing binary search for a an array-like interface etc -- basically Abstract Data Structures.

At first I was thinking "For the love of bog why?", but then I started reading, and couldn't stop reading.

It is very interesting.

I like the simplicity of the system calls.

With DynASM (a subproject of LuaJIT), it is possible to target more platforms with this kind beauty.


Annotations aside, this is actually more readable than a lot of C code I've seen.

Thats funny because C is a one-to-one mapping over assembly. And frankly its not using any esoteric instructions such as "PUNPCKHQDQ" or "VFNMADD231PS"

It used to, but it's nowhere near as correspondent with modern architectures that incorporate SIMD/SSE, complicated branching and pipelining strategies, SMT and so on. Nowadays C is a register machine model of its own, with plenty of gotchas and UB that are quite specific to it.

The spec isn't terse for a reason.

*With -O0

No, C does not expose a 1:1 mapping to assembly in any architecture, at -O0 or otherwise. To give one simple example, the C pointer types correspond to nothing meaningful in any commonly used ISA.

asm("xorl %eax, %eax");

And I'm not sure about your example, a C pointer type would be stored as a number that will be used with "lea" or "mov [pointer], foo". Elaborate.

What's wrong with "Unpack and interleave high-order quadwords from xmm1 and xmm2/m128 into xmm1." and "Multiply packed single-precision floating-point val-ues from ymm1 and ymm2/mem, negate the multi-plication result and add to ymm0 and put result in ymm0."?

It's simple, right?

(This is sarcasm, by the way.)

Although I will point out that C has its esoterics also.

Both "esoteric" examples were just picked from a larger set of data packing and FMA (fused multiply add) operations that high performance real life code needs and uses. Intel doesn't generally add instructions no one needs.

Unfortunately, compilers aren't indeed very good at picking optimal instructions when it could take good advantage of instructions such as these. No wonder, though.

Packing and unpacking are often needed in SIMD context. There are a lot of such instructions, including shuffles and permutes. Individually they may sound esoteric, but actually cover a nice number of real life data shuffling needs and are extremely fast.

Fused-multiply-add instructions can double effective FLOPS.

I am well aware of the potential advantages of such operations. And the difficulties associated with trying to use them well and identify when they can be an advantage.

However, just because something is useful does not preclude it from being esoteric.

In particular, there are an astounding number of such miscellaneous and less-often-used instructions in x86 and extensions, and trying to remember which ones exist, and which ones have which limitations, is... a fair feat. Hence, esoteric. Understood only by a few with special knowledge or interest.

Interesting read and I like it. I'll have to read it more on my Desktop. I get a HTTPS error on my android phone however.

You might be missing an intermediate certificate. (Just a heads up.)

As do I on my Android lollipop 5.1.

Thanks, I will look into this. After running ssl test from ssl labs and I some alarming issues https://www.ssllabs.com/ssltest/analyze.html?d=zarkzork.com I should have done it earlier

Surprisingly, I get this fairly often on my mobile. I'm not quite sure what's changed with the last version (or couple) of Android to trigger this. Over-secure, perhaps?

Android mobile browser requires the full signature chain of authorities who signed the site certificate[0]. It is done to avoid calls to CA servers and reduce network use on mobile devices.

If a person does not concatenate authorities signatures to their site certificate, Firefox and mobile Chrome will alert, desktop Chrome and Safari won't.

[0] https://en.wikipedia.org/wiki/OCSP_stapling

Simply beautiful and elegant, probably very fast and with a very small memory footprint.

Would love to see benchmark on this versus nginx for fun!

Other than needing to fork for every request, this should be really fast because it uses the efficient sendfile() mechanism in the kernel to handle returning the file and most importantly only implements a bare minimum of support for the protocol. nginx is almost certainly going to be slower because it has to worry about different types of requests, different protocols, IPv4 vs. V6 sockets, CGI, etc...

It's easy to make something fast when you only implement a very small number of features.

IIRC, Nginx can use sendfile too, it's just not there by default because the transfer happens entirely in kernel space and hence you cannot apply filters like gzip on the response (correct me if I'm wrong).

Yes, that's what my sense is: it would, within its very limited domain, beat nginx and co. fairly easily.. its interesting because this sort of thing could be made to be modular and thus feature extendable without the overhead if done in a smart enough fashion - many webservices today are sufficiently specialized that they only use a fraction of the features that modern webservers include.

nginx is highly configurable at compile time. You can strip out a considerable amount of functionality (modules, in nginx lingo) for performance.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact