
When Haskell Is Not faster than C - DanielRibeiro
http://jacquesmattheij.com/when-haskell-is-not-faster-than-c
======
simias
This reminds me that when I was interested in learning haskell I was
completely put off by their "introduction" page on their homepage. It was a
couple of years ago but it doesn't seem to have changed much:

<http://www.haskell.org/haskellwiki/Introduction>

It looks like a very pretentious sales pitch to me. You have quotes like
"Writing large software systems that work is difficult and expensive. [...]
Functional programming languages, such as Haskell, can make it easier and
cheaper." and a whole bunch of very vague and poorly backed up assertions.

After a couple of paragraphs brainwashing you about how good functional
programming is, you finally get to some code. The _first_ code you get is (of
all things) a quicksort which is not used as an introduction to the language
but as a way to boast that haskell is so much better and more expressive than
C. I mean, look at that C code just after it, disgusting isn't it.

What they don't say is how much slower their implementation is, of course.
They also give a link to a more direct (and faster) C-to-haskell version of
the quicksort[1]. Hey, it's actually longer than the C version! And it uses 3
imports. And it's still slower than the original C. Oops. I wonder why they
didn't put that one in the tutorial. I went back to common lisp after that, I
have yet to write 1 line of haskell.

TL;DR: when I want an introduction to your language, I want to know what I can
do with it and how I can do it. Give me some code to chew on, show off some
features. Don't spend 10 pages telling me you're the best of the best and
everything else is crap. This is the kind of article I expect:
<http://perldoc.perl.org/perlintro.html>

[1]
[http://www.haskell.org/haskellwiki/Introduction/Direct_Trans...](http://www.haskell.org/haskellwiki/Introduction/Direct_Translation)

~~~
revelation
That wiki link to an actual implementation of quicksort in Haskell is pure
comedy gold (1):

" _Unfortunately none of the above "real" quicksorts seems to compile as
given_ "

" _The program below is working very very slowly. It's probably slowsort._ "

" _A more specific/direct translation (neither this nor the C version is
polymorphic) is offered by Daniel Fischer, who reports that this version runs
within 2x of the C version_ "

(and it's a completely unreadable mess that is clearly a line by line copy of
the C code given)

What a complete and utter trainwreck.

[1]:
[http://www.haskell.org/haskellwiki/Introduction/Direct_Trans...](http://www.haskell.org/haskellwiki/Introduction/Direct_Translation)

~~~
sltkr
To be fair, QuickSort is particularly suitable for imperative languages that
allow in-place modification of arrays. For functional languages, merge sort is
a much more natural solution that is simple to implement, guaranteed to be
efficient, and probably reasonably fast (though still unlikely to beat a
QuickSort implementation in C).

~~~
ajross
To be fair, QuickSort is particularly suitable for _actual computation
hardware_ which has hardware features like numerically addressed, stateful
random access memory. Imperative arrays happen to feature "in-place
modification of arrays" not because it's a useful language feature but because
it's a straightforward abstraction of the actual machine being programmed.

That distinction is really important, and something I think many language
nerds completely fail to understand. Sure, often all that's "really" important
is the abstraction of the "Human Design Space" idea being implemented, and
that's a regime where high level language abstractions have lots of value.

But sometimes you want to actually direct the hardware. And there, as
everywhere, it helps to use a language with the proper abstractions. C has
them, Haskell doesn't. There's absolutely no shame in that!

But somehow people get fooled into thinking that somehow Haskell (or whatever
language is involved in the argument, but it seems like Haskell people fall
into this hole more than most) doesn't _need_ those abstractions to be "as
fast a C". And that's just insane.

~~~
bcoates
To be fair to mergesort, it's particularly suitable for actual computation
hardware too. It's mostly linear reads and writes, which almost all i/o
systems heavily favor. GNU sort uses mergesort (in C).

~~~
ajross
GNU sort is designed to sort on disk though. If you actually watch it working
on a giant file, you'll see it generating lots and lots of files in /tmp
containing the merge lists.

This is because quicksort obviously requires fast random access to the whole
arrays, and (at least when it was written) /usr/bin/sort would be expected to
be able to quickly sort data that can't fit in memory.

So... if anything I think this is evidence to my point: in the performance
regime, you have to match the algorithm choice to the hardware. Specifically
here: sort is making up for the lack of memory by exploiting the comparatively
(relative to virtual memory) fast _streaming_ behavior of storage media.

Notably, stateful filesystem I/O is _also_ an area where Haskell tends to fall
down. I'm not at all sure a Haskell merge sort of the form used in GNU sort
would do well at all, though I'm willing to be proven wrong.

~~~
sltkr
> GNU sort is designed to sort on disk though. If you actually watch it
> working on a giant file, you'll see it generating lots and lots of files in
> /tmp containing the merge lists.

The number of files created is just an implementation detail; you could easily
do everything in one file if you wanted to, or at most 2N files if you're not
allowed to seek or overwrite files at all, and you want to do N-way merges.

Maybe GNU sort is just trying to avoid creating files larger than 2 GB (which
is a problem on legacy file systems).

> Specifically here: sort is making up for the lack of memory by exploiting
> the comparatively (relative to virtual memory) fast streaming behavior of
> storage media.

Yes, also the fact that GNU sort can sort multi-terabyte files on 32-bit
platforms, which otherwise wouldn't fit in memory. (Though systems with
smaller process memory space than disk storage space are an oddity that will
soon be outdated, I suppose.)

> I'm not at all sure a Haskell merge sort of the form used in GNU sort would
> do well at all

Well, even the on-disk sort starts by sorting chunks in-memory, which we have
already established Haskell is bad at. However, the merge sorting part of the
algorithm may well be I/O bound in practice, in which case Haskell should be
able to do it just fine.

The only good reason I can think of why C could be faster, is that if you can
merge more efficiently, you can choose a higher value of N for your N-way
merge,

> stateful filesystem I/O is also an area where Haskell tends to fall down

What do you mean by this? AFAIK Haskell can read/write files just fine through
the I/O monad.

------
gnuvince
> This article is in response to an earlier one comparing Haskell and C, which
> made the claim that Haskell beats out C when it comes to speed.

Perhaps my reading comprehension of the original post is different from
Jacques' (or I'm just wrong), but I don't think that the original article made
such a claim. Here's the TL;DR of the original article:

> TL;DR: Conventional wisdom is wrong. Nothing can beat highly micro-optimised
> C, but real everyday C code is not like that, and is often several times
> slower than the micro-optimised version would be. Meanwhile the high level
> of Haskell means that the compiler has lots of scope for doing micro-
> optimisation of its own. As a result it is quite common for everyday Haskell
> code to run faster than everyday C. Not always, of course, but enough to
> make the speed difference moot unless you actually plan on doing lots of
> micro-optimisation.

From this, I understood that in a larger program, most programmers wouldn't be
doing the kind of micro-optimizations that they do for the Benchmarks Game. I
figure that most code would be written following this pattern:

* Write code to follow the spec (usually without thinking too much about performance)

* Run the code, and evaluate its performance

* If the performance is good enough, you're done

* If the performance needs to be better, run a profiler

* Find the biggest bottleneck and optimize it

* Re-run the code and re-evaluate its performance

The original article took a micro-benchmark (a mistake in my opinion, because
it's easy to micro-optimize every single detail of that code) and showed that
in the case of Haskell, the first version was too slow, but that with the help
of a profiler, finding the cause was easy, and the fix was trivial, while in C
the profiler didn't show any problem with the code of the user, so it must be
a problem in the standard library's code, and to fix it required changing the
design of the program and making it more different than the spec. And I felt
this was the author's real point; that to get the maximum performance from a C
program, you'll need to code it not like a human would imagine it, but like a
computer would, and it makes the code harder to maintain later on.

~~~
jacquesm
> Perhaps my reading comprehension of the original post is different from
> Jacques' (or I'm just wrong), but I don't think that the original article
> made such a claim. Here's the TL;DR of the original article:

The TL;DR is not representative of the whole article. It would be nice if it
were but it isn't, so just reading the TL;DR is not enough, better do the
short-enough;did-read version of that. Another commenter in this thread said
more or less the same thing, I've responded to him here:

<http://news.ycombinator.com/item?id=5091270>

~~~
papsosouid
The whole article didn't make such a claim either. Your hate boner is clouding
your judgement. Your article is far more biased and far less useful than the
original. Especially all the random off-topic nonsense like "you should
totally add error checking" and "lets change the style for no reason because I
want to pad the length of this".

~~~
stcredzero
_> Your hate boner is clouding your judgement._

Should be a button or T-shirt.

------
VeejayRampay
I still wonder to this day why Haskell programmers want their language to be
loved so much.

At times it feels like that kid at the playground that spends half his time
telling everyone how he's the best thing since sliced bread and cries himself
to sleep at night wondering why no one will play with him and his monads.

Don't get me wrong, Haskell looks like a great language with obvious qualities
and I don't knock anyone for using it, to each his own, it's just the never
ending publicity and proselytism that really rubs me wrong. People adopt new
languages, not the other way around.

~~~
Swizec
Two things I love about Haskell.

1\. It is very terse/expressive. I can look at code I haven't touched in 6
months and still understand what it's doing. Perhaps because my code is newbie
code and thus somewhat simple.

2\. It is the only language that brings me back to the days of Turbo Pascal
when the joy of coding came from coding itself, not from building cool things.
A small but vital distinction.

That said, I rarely get to use Haskell. For most of my tasks javascript is
better suited, but haskell has affected how I write javascript to a great
extent.

~~~
abraxasz
Maybe it is because I'm not a Haskell Guru, but although I love Haskell's
terseness, it tends to make it very hard for me to read code I've written a
while ago. I tend to get excited about cool Haskell features like Arrows,
using them whenever I can, then I forget about them, and when I read my code 6
months later, it's like gibberish..

~~~
evincarofautumn
When I find myself using arrows, usually I realise that I got lazy and used
tuples because my data model wasn’t good enough. Good code flows from good
data.

But really, you should use the “tricks” when they make your code more
expressive, not just because they’re cool. It’s the same in any language.

------
sharpneli
I ran into similar issue few years ago where lack of knowledge combined with
large amount of fanboyism clouded someones mind. In a popular language
comparisons site there was a threading benchmark. It was simple, just start
256 threads. Haskell beat C by far.

I did strace. The C benchmark actually spawned 256 threads and the Haskell one
spawned 4. The response I got was that it's a builtin feature in Haskell that
it uses lightweight threads internally. And because pthread doesn't do that
you cannot do it in C and therefore the comparison is valid. Even though C
doesn't have the concept of threading and everything must be done with an
external library in anycase.

I could dig the old thread up but I've given up with those. I have nothing
against high level languages, I use them a lot. However I find it very weird
that some true fans feel the need of creating enormously biased benchmarks and
then being proud that those benchmarks show that their favourite language is
the best.

~~~
jeremyjh
I've never worked on a project where my job was to create 256 threads.
Instead, I would be tasked with processing lots of requests or datasets in
parallel. If Haskell provides convenient and idiomatic ways to do this with
lightweight threads that C does not, then C is effectively slower. If C has a
commonly used and available library to implement this same approach - then
maybe its faster. The whole point is we are comparing implementations of a
programming task.

~~~
pekk
C doesn't impose a specific threading model. So a C threading benchmark is
specific to the library you are using. And if you choose a heavyweight thread
library to compare to a language which is not using heavyweight threads, it is
not an apples to apples comparison. Arguably it could be, if C imposed a
specific threading model, which it does not.

You are comparing implementations, not languages

~~~
jeremyjh
Yes and that was my point. The benchmark is a comparison of implementations -
compilers, run times and libraries. I am not sure how you would benchmark a
language.

------
benbataille
Could everyone on HN just take a course in languages theory so we can all stop
with these stupid trolls about the best languages which have been emerging for
a week.

Hopefully it would allow everyone to realize that a language is just some
syntax and semantic and that a compiler is just a program like another.
Nothing sacred here. Hell you can even do imperative programming in Haskell if
you wish. Coding an interpreter to do so is not particularly difficult.

With a bit of luck, everyone would understand at the same time that the
compiler is building what is executed therefore the expression"speed of a
language" is utter nonsense. You can only speak of the speed of the
implementation which, yes, vary widely amongst compilers for the same
language.

So, yes, Haskell semantics encourage functional programming, C semantics
imperative programming, both are Turing complete and yes GCC is currently
better at optimising C code than any Haskell compiler, nothing new under the
sun. Can we all go back to a normal activity now ?

~~~
norswap
Some people like bickering. But I think the point of the article is that C is,
by nature of the language, easier to optimize than Haskell, since it is lower
level.

~~~
Shish2k
> C is, by nature of the language, easier to optimize than Haskell, since it
> is lower level.

Depends what level you mean - if you want to do all the optimisations
yourself, C is better; but if you want to write code and let the compiler do
the details, I would think higher-level is better (you can just write
doWhatIMean() and the compiler will automatically choose the optimal
implementation for your current problem and platform - where if you'd
specified the implementation details yourself, you'd be sub-optimal in many
cases)

~~~
zxcdw
> _"(you can just write doWhatIMean() and the compiler will automatically
> choose the optimal implementation for your current problem and platform -
> where if you'd specified the implementation details yourself, you'd be sub-
> optimal in many cases)"_

How far are we with declarative programming in sense of being expressive and
expecting the compiler to understand what we mean? I'd believe that without
strong AI human will always be better off with imperative programming than a
compiler with declarative, for anythig beyond relatively simple isolated
pieces of code/algorithms. _In general_ we have so much more experience and
knowledge about the target system and environment than the compiler does, this
is less true for JIT compilers, but they come with their own overhead.

I'd really love to see _superoptimization_ [1] to be done as a service for
compilers. Say you have a function with certain semantics the compiler is
fully certain about. The compiler fingerprints this function and uploads a
fingerprint + behavioral analysis to < _somewhere in cloud_ > where it's being
exhaustively optimized by bruteforcing all the meaningful instruction
sequences which conform to the semantics of the function. After the most
optimal piece of instructions is being found, it's associated with the
finerprint and stored in a database and then returned to the compiler. Now,
when ever someone else writes the exact same function(or code with exact same
semantics) the compiler queries the < _some cloud service_ > for the optimal
piece of code. Of course, a system like this would need more information about
the _actual target_ of the code(CPU architecture, cost of things like memory
access, cache miss, branch mispreciction etc.).

[1]: <https://en.wikipedia.org/wiki/Superoptimization> (The system I roughly
described is far more extensively analyzed in some of the papers. _Really_
great read!)

------
dllthomas
_"Third, a comparison on speed between Haskell and C is almost meaningless,
speed is not Haskells strongest point and it would be folly to bench it
against C for that particular reason."_

I disagree wholeheartedly. If I am choosing Haskell over C, I am giving up
some (possibility of) performance. The question of how much is an important
piece of information, entirely relevant to that decision.

As I observed in a comment on that other post, the actual performance of both
the Haskell and the C depends on the amount of effort expended to make them
faster (first in general, and then possibly on a particular architecture). At
the limit, the C beats the Haskell by some margin - the size of that margin is
informative; but that's also not the whole story - what the margin is earlier
also matters, and for a particular project, it might matter more.

This is _not_ to say that the particular benchmarks in the earlier article
were good choices - I don't have a clear position on that.

------
nicholassmith
Nice refutation. The problem about doing language comparisons for speed is
that they generally require a non-trivial example, so you have to be an
_excellent_ programmer in all the languages you've compared.

Of course, language speed charts are generally as useful as PC spec charts and
YouTube videos of Nürburgring laptimes when you're buying a new car.

~~~
chii
well, what about an existing piece of software, and compare on more than just
speed, but ease of maintenance (meaning, how quickly you can add features or
fix a bug)?

There seems to be a myriad of window managers and web frameworks. Someone
should do an unbiased comparison.

~~~
nicholassmith
Same thing again, it depends entirely on really specific domain logic. How
easy it is to add a new feature on a C based web browser versus one built on
Haskell is mostly useless, it just tells you how quickly one person added one
feature on one specific thing.

Languages aren't inherently comparable, you can get useful benchmarks but they
shouldn't be taken as gospel.

------
spookylukey
This article starts by totally misrepresenting the original article
([http://paulspontifications.blogspot.co.uk/2013/01/when-
haske...](http://paulspontifications.blogspot.co.uk/2013/01/when-haskell-is-
faster-than-c.html)) by saying that it said "Haskell beats out C when it comes
to speed".

If you bother to read even the TLDR of the original article you 'll find it
says something completely different, which makes this article pretty poor as a
response.

~~~
jacquesm
From TFA:

"So here is the bottom line. If you really need your code to run as fast as
possible, and you are planning to spend half your development time doing
profiling and optimisation, then go ahead and write it in C because no other
language (except possibly Assembler) will give you the level of control you
need. _But if you aren't planning to do a significant amount of micro-
optimisation then you don't need C. Haskell will give you the same
performance, or better, and cut your development and maintenance costs by 75
to 90 percent as well._ "

Note the 'same performance or better' in there.

Maybe you missed that bit in the original article?

This wasn't a large effort by any stretch of the imagination and a factor of 5
difference compared to the Haskell code isn't even in the same ballpark as
"the same performance", and about a factor 10 difference with the C code
listed in the original article. You'll notice no micro optimizations were done
in the re-write, just some global re-arranging of the way data flows through
the program.

The rest is in response to the misleading title, that Haskell is 'faster' than
C, faster to program, faster to debug, easier to maintain and so on while
making claims about performance that are not born out by real world
comparisons.

Speed was the wrong thing to compare on here, and should not have been part of
the original article without a serious effort to become equally proficient in
both languages.

~~~
spookylukey
Compared to the amount of optimisation that many people typically do (i.e.
zero or close to zero), it was a large amount of extra work to get that
performance boost. The global re-arrangements of the way data flows through
the code are _more_ difficult than many micro-optimisations, not less, so I
don't understand how you can claim.

And you don't have to be an expert in C to make the claim he is making,
because most people are not experts in C, but might turn to it because "it's
faster". It's precisely this kind of person who needs to understand that with
their level of knowledge, and the amount of effort that they might put into
optimisations, Haskell can be just as good or better.

~~~
jacquesm
> it was a large amount of extra work to get that performance boost.

Not really, actually. It took a lot of time to document the transformations,
but actually doing the coding was short, much less than an hour total.

> The global re-arrangements of the way data flows through the code are more
> difficult than many micro-optimisations, not less, so I don't understand how
> you can claim.

Maybe more difficult for you, but not for me. Micro-optimizations are actually
harder for me, there are infinite possibilities there, flow has really only
one (or very few) 'proper' mapping(s) from problem space to solution space.

> And you don't have to be an expert in C to make the claim he is making,
> because most people are not experts in C, but might turn to it because "it's
> faster".

Every tool has its uses. If you pick C then probably you're doing so because
you are convinced that you need it. You should then study how accomplished
users of that tool use it, not stop at the first naive use that you come up
with yourself.

> It's precisely this kind of person who needs to understand that with their
> level of knowledge, and the amount of effort that they might put into
> optimisations, Haskell can be just as good or better.

That's very well possible, again, I do not know Haskell so I can't comment on
it. But I don't see any meaningful comparison between the two languages here,
speed certainly isn't it and you failed to acknowledge that the article in
fact did make that claim.

------
bborud
If Haskell, Lisp and friends are so great for building large software systems
then surely people are going to flock to them and do exactly that. And if that
didn't happen, of course the proponents of these languages would sit down and
try to figure out why this wasn't happening. Right?

~~~
wereHamster
Education is a big part of the problem. When I attended university, we weren't
even told that functional programming exists. Only later on, through one of my
friends it was brought to my attention. And it's hard to change that mindset,
particularly at the university I was. They are set to produce a constant
stream of Java/.NET developers for the companies around them (large banks, in
particular).

~~~
PommeDeTerre
Which university was this, and which program?

~~~
wereHamster
The university in Lucerne, Switzerland. Hence the reference to large banks ;)

------
mistercow
The original article did mention that switching to use lines instead of
characters would probably speed things up considerably, but argued that this
makes the program harder to understand, while Haskell makes the same
optimizations automatically on the character-based program.

And that's a fine point to make, but I do think it was a little misleading and
handwavy. The proper thing to do would be to make the change and show the
difference both in code and performance, and let the readers draw their own
conclusions.

~~~
jacquesm
The 'line reader' version of the code (not shown in the article) was actually
a lot easier to read than either one of these but the article was already over
long. I do have that version, if anybody is interested I can add it.

~~~
ibotty
i'm sure you have all your intermediate steps as well. i really liked an
example i've seen a while ago that described an optimization process in git
history. you'd simply check out the right step and have a look for yourself.

------
nathell
_Yawn._ It is not possible to directly compare languages wrt speed: we only
can compare the speed of their implementations and reason about the features
that make a language amenable to optimizations.

 __And we all know this. __

So why, despite this knowledge, do we engage in "language speed flamewars"
like this?

That's a valid question, and it belongs in the field of psychology of
hackerhood. Does anyone have a reply to wager?

~~~
zxcdw
I for one find things like these extremely interesting. The more there's
analysis about performance implications of certain ways of doing the task the
better. The only thing I'm left missing here is to actually look into the
profiling information(branches, caches, instructions per cycle etc.),
especially on the compiler generated assembly to see what exact instruction
sequences are to blame, and if there are ways to save cycles here or there.

If only there were more things like this in the internets with deep analysis
about performance of a given piece of code... :)

~~~
jacquesm
It was pretty long as it was... Tacking on a code generation section to show
_why_ certain things are inefficient (and the associated detour to how the
standard library is implemented which would be required reading to understand
all the implications) would make the article three times as long. I promised
I'd have it done by Monday.

------
JabavuAdams
Excellent. Documenting successive transformations on code is much more
instructive than showing a final snapshot. This is one reason I really enjoyed
doing live-coding when teaching programming.

------
JabavuAdams
This is a classic pattern: when advocating for one of several competing
approaches, you spend more care optimizing your favourite approach than the
competitors.

Prof. Geoffrey Hinton specifically calls this out as a pitfall in research.
It's better to compare your promising new approach versus another group's
approach (not your own implementation thereof). This, assuming that the other
group has deep knowledge in tuning their competing approach.

~~~
jacquesm
I think it's only human. The standard should be that you present your case for
your 'favorite' and then put out an open challenge to supporters of the
opposition to do their worst.

------
confluence
The problem with fringe languages has always been the "all talk - no walk"
nature they all seem to have.

Stop telling me why you are so awesome. Just show me with shipped product.

Until then it's just an intellectual circle jerk.

Talk less. Ship more.

~~~
tensor
The old appeal to popularity. The only argument that is worse is the appeal to
faith.

An intellectual circle jerk occurs when one repeats the same old mantra
without doing any actual research. There are plenty of "shipping" products in
languages like Haskell. I'm no expert on Haskell, but this wasn't hard to
find:

<http://www.haskell.org/haskellwiki/Haskell_in_industry>

A better tagline might be: Think more and don't settle for the status quo.

~~~
anon1385
That page has generated some heat in the past:
[http://flyingfrogblog.blogspot.ch/2010/05/why-is-haskell-
use...](http://flyingfrogblog.blogspot.ch/2010/05/why-is-haskell-used-so-
little-in.html) (admittedly this guy seems to have some personal grievances
with people in the Haskell community) I don't know how much the page has
changed since this rant was written, but some of the entries seem to be the
same, e.g. 'Ansemond LLC' which is a company with a single mac app that hasn't
been updated since a beta in 2009 and only mentions Haskell in an old blog
post. The site for that company is so charmingly dated that I feel kind of bad
for highlighting it, but if this is the kind of 'industry success story' they
think worth mentioning then I'm not sure the wiki page is of any value.

~~~
Sssnake
>admittedly this guy seems to have some personal grievances with people in the
Haskell community

No, he's a troll. He trolled lisp, then haskell, then ocaml. 2010 fell into
his haskell phase, that's the entire reason for that particular pile of
bullshit.

~~~
anon1385
The thing is, there is some truth in among the bullshit. That 'Haskell in
Industry' page really does have links to non existent products or long dead
projects or things that don't seem to be using Haskell at all. And even after
that blog post was widely circulated 2 years ago the page remains unchanged,
and people are still citing it as evidence for the widespread use of Haskell
in industry.

------
ColinWright
Just a quick point. He quotes me:

    
    
        You can't make a computer do stuff faster,
        but you can make it do less work.
    

That's not original to me, but I don't have (and haven't been able to find) a
good, definitive, original source.

------
Zircom
Is there a reason "Is" and "Not" are capitalized but "faster" and "than"
aren't, because it's just been bugging all day whenever I refresh the front
page and look over the list for anything interesting.

------
JoachimSchipper
This is cool. However, in the spirit of contributing, comparing one byte at a
time is not optimally efficient. It's possible to write clever optimizations
by hand, but I'd be surprised if just using
<https://github.com/rbdixon/glibc/blob/master/string/memchr.c> doesn't cause a
meaningful speedup.

~~~
jacquesm
The inner loop of those comparisons is indeed the spot where you can still
speed up as noted in the last part of the post, the kind of optimizations that
you describe are extremely effective but qualify as 'micro optimizations' and
I expressly left those out because they impact readability considerably. But,
you're right, if that's what it takes then so be it and then readability would
have to suffer in deference to the last couple of % of speed. Maximum gain
from this optimization relative to the final runtime is about 20% by my
estimation. (Inner loop will step 8 bytes at the time, but will have more
instructions).

~~~
JoachimSchipper
I concur that doing this directly in your code is extremely ugly; but note
that you can get this speedup by just dropping in a call to glibc's memchr(),
hiding the ugliness behind a well-known interface.

~~~
jacquesm
Good point about memchr, I missed an opportunity there.

edit: because it's clearer _and_ faster, that's a no-brainer, updating the
blog post with a remark to that effect and a link to parent.

------
neopallium
There is a bug that causes invalid output when the input file is larger then
the BLOCKLEN. Set BLOCKLEN to 4096 and run the program on the test input file
[1] (compare with output file [2]), to see the problem on a smaller input
file.

The bug happens when read_sequence() is called with partial data saved from
the last sequence, since (size == read) the first fread() will be asked for 0
bytes which causes the read loop to end early (n == 0).

1\.
[http://benchmarksgame.alioth.debian.org/u32/iofile.php?test=...](http://benchmarksgame.alioth.debian.org/u32/iofile.php?test=revcomp&file=input)

2\.
[http://benchmarksgame.alioth.debian.org/u32/iofile.php?test=...](http://benchmarksgame.alioth.debian.org/u32/iofile.php?test=revcomp&file=output)

~~~
jacquesm
That was a good catch! Serves me right for testing with the small file for
correctness and testing for speed with the larger one.

------
sunwooz
What do you guys think I should learn next? I've been studying javascript/ruby
and I would like to diversify what I can make. I don't want to rush learning
new languages, but it's interesting to learn about paradigms that certain
languages enforce.

What's important to me(not in order):

1) Being an omni-platform developer 2) I only know two high-level languages,
so learning a low level language will benefit me. 3) I want to get into
Natural Language Processing and deep neural networks/AI, machine learning etc.
4) I'm a self-taught programmer, so the amount of good resources online/amazon
is a big factor. 5) I need a fucking job. 6) I like making web
applications/web games, and I would like to make native applications/games as
well.

------
waxjar
I'd love to be able to reason like that about the program I'm writing. Really
cool article.

------
dschiptsov
Perfection is achieved not when there is nothing more to add, but when there
is nothing more to remove.)

When one programs a computer (in a way it suggested in TAOCP) and you know
your data, your memory layout and your CPU instruction set nothing could beat
it.

Of course, there are tasks so massive, that a decent compiler could save lots
and lots of man hours, but it is a completely different story.

No Haskell (leave alone J* ) could beat a code by people who know what they do
close to the metal.

In some languages the code could be shorter much more readable and elegant
that C, but of course, it is not Haskell, or J* .

Thoughtfully crafted Common Lisp could be close to an ideal.)

~~~
pohl
Lisp's homoiconicity comes with the cost of some extra syntactic weight that
Haskell doesn't have to bear. (It's also nice that "lambda" is considered so
important that it's given a single character in Haskell).

I'm not familiar with J*, and it seems to be difficult to search for. Do you
have a reference so I can go learn about it?

[http://rosettacode.org/wiki/Sorting_algorithms/Quicksort#Has...](http://rosettacode.org/wiki/Sorting_algorithms/Quicksort#Haskell)

[http://rosettacode.org/wiki/Sorting_algorithms/Quicksort#Com...](http://rosettacode.org/wiki/Sorting_algorithms/Quicksort#Common_Lisp)

~~~
dschiptsov
I prefer Erlang.

[http://rosettacode.org/wiki/Sorting_algorithms/Quicksort#Erl...](http://rosettacode.org/wiki/Sorting_algorithms/Quicksort#Erlang)

And your illustration is particularly good for showing how much more "sane"
(actually, in many ways) Erlang is. And how elegantly different Python could
be.

J* is my own abbreviation for Java and Javascript.)

~~~
pohl
_I prefer Erlang._

Just out of curiosity, why is that? (With reference to the linked example, I
mean.) Clearly they're in the same league, and so it's a close call. From my
perspective, I like that Haskell reserved the single vertical bar for the list
comprehension notation, so that it would be as close to familiar set-
comprehension notation as possible. I also like how Haskell takes advantage of
how the cons operator already implies in the pattern-matching that you're
working with a list, and so the square brackets become unnecessary, whereas
Erlang needs the noisy extra layer of brackets on top of the parenthesis
needed for the pattern.

But once languages get this elegant, it's a close comparison. They're both
beautiful to my eyes.

~~~
dschiptsov
I think the sytnax of Erlang is much more elegant, because it was build around
the fundamental idea of pattern matching as a core feature of the language.

When you have a few good ideas put together, you could get an elegant
solution. When you just stuff everything inside (like Clojure) or went to
extremes (like Haskell) all you got is just a mess.

Erlang syntax, however, is noisy due to all those punctuation, but it is
elegant nevertheless.

    
    
      map(Fun, [H|T]) -> [Fun(H)|map(Fun, T)];
      map(Fun, []) -> [].
     
      member(H, [H|T]) -> true;
      member(H, [_|T] -> member(H, T);
      member(H, []) -> false.
    

This is intuitive, readable, but noisy.)

~~~
pohl
I'm not sure I follow you. Is the pattern-matching in Haskell somehow not
'core' enough? Both qsort examples linked in this thread use pattern matching.

~~~
dschiptsov
All I'm trying to say that Erlang syntax is much more intuitive and readable,
being derived from Prolog.

I honestly can't see why should I use Haskell (except for being _so clever_ )
when I have Erlang, or at least one real advantage.

~~~
mwotton
well, let's look.

map f (h:t) = f h:map f t

map f [] = []

member x (h:t) = x == h || member x t

member _ [] = False

I think the only difference is that you can't match for equality directly in
the pattern.

------
rtkwe
If you're going to do a response blog post link to the original article at the
beginning when you mention the original.

I read the original but don't assume everyone has.

