
Using D to Create the World’s Fastest File System - aldacron
https://dlang.org/blog/2018/12/04/interview-liran-zvibel-of-wekaio/
======
tombert
I have wondered for awhile now why Rust is getting all this attention of being
"Like C but safe and cool!", but no one would mention D.

I personally don't do systems programming because I'm not smart enough but the
little bit I have done, I found D to be the most-natural feeling (even over
Rust), and D has been around a lot longer.

I suppose timing is everything for these kinds of things.

~~~
z3t4
The Rust community is extremely hostile, especially against other languages,
and D in particular.

~~~
z3phyr
Its not hostile. Its aggressive in selling the strong points of the language.
Having said that, I do wish newer languages spring up that adapt the good
stuff from Rust. I can never like Rust syntax.

~~~
zanny
A ludicrous amount of engineering has gone into the Rust compiler, the
extensive documentation in RFCs about its design, the huge language docs that
took a million man hours to make, RLS is another huge undertaking...

I do forsee Rust getting a Coffeescript of its own. I'm in love with the
control flow model of the language but always felt some things that were in
from day one (like double colon :: namespace delimiters, ampersand references,
asterisk pointers and dereferencing, semicolons, curly braces, etc) were just
wholesale copied from the languages de jour of the day because the focus was
on the borrow checker as an innovative feature. So the semantics and
ergonomics were completely phoned in early on, and by the time the modern
development process around the language and compiler matured the glyphs were
so totally entrenched there was no way to reconsider any of it.

A syntactic wrapper that gives you the types and behaviors of Rust with
beautification like whitespace significance over mandatory brace delimiters
and semicolons (like Python) with single colon (or maybe even dot) namespacing
and such would be an insanely hard project (because parsing Rust is already
insanely hard, and a lot of work went into minimizing compiler passes to parse
grammar so keeping that with less control characters to work with in an
intuitive way sounds like a substantial exercise in ergonomics) that compiled
to ugly normal Rust would be fantastic to help newbies avoid the "glyphic
overload" I easily see happen to anyone I try to preach Rust to.

~~~
skohan
I totally agree re the ergonomics. It really feels to me as if a lot of C/C++
conventions were copied over, with no justification beyond "this is what
system programming languages look like".

------
hardwaresofton
I would have liked to hear more about what exactly made D so great -- reading
through it it's kind of vague. He mentions generics and finally getting LLVM
support working as a target, and being able to use D at a high and low-level
but I don't see anything here that's groundbreaking.

Could someone who uses D

Hate to be that guy, but if you could compare it with rust that would be
double A++ good... I know the targets are somewhat different but I find that I
can write high level rust _and_ low level rust. Also, I get the feeling that
you could very easily have used Go for the same purposes (D was around long
before Go IIRC so maybe it was just not being present).

~~~
WalterBright
My experience, which of course is anecdotal, is that the advantage is it's
easy to change the algorithms and data structures.

I've maintained C and C++ code bases for decades, and I've found that the
first algorithm I tried has stayed there in the code. It gets tweaked,
optimized, refactored, but it's the same algorithm and data structure.

With D, when I developed the Warp preprocessor,
[https://github.com/facebookarchive/warp](https://github.com/facebookarchive/warp)
I continually tried different algorithms & data structures to see which was
faster.

One reason it's easier is that in C/C++ the . is used for value field access,
while -> for pointer field access. In D, . is used for both. So you can easily
switch between a value and a pointer to the value by changing little code.

~~~
sanxiyn
> In D, . is used for both.

Yes, this is a great feature. Rust also does this, and I attribute substantial
portion of ease of refactoring Rust to this feature.

~~~
stochastic_monk
Oddly enough, so does Cython.

------
notacoward
For those commenting on the name: weka is not only the name of a storage
company and some machine-learning software, but originally a flightless bird
native to Aotearoa / New Zealand.

[http://nzbirdsonline.org.nz/species/weka](http://nzbirdsonline.org.nz/species/weka)

Not to be confused with weta, which is a large scary looking (but actually
pretty harmless) insect in the grasshopper family and also the name of another
company that did most of the digital effects for the LotR movies.

~~~
regularfry
Weta Digital ([https://www.wetafx.co.nz/](https://www.wetafx.co.nz/)) are
named after the grasshopper, not the bird.

------
guitarbill
> Our previous company, XIV (that IBM acquired in 2007) used a private version
> of C that had polymorphism, generic programming, and integrated RPC, for
> code that was a mixture of XML, C, and weird header files.

Super interesting. XIV was a great system, but "weird" is a good description
for some of the source (IBM of course neglected/ruined it). Thought most of
the XIV folk went to Infinidat.

In the storage world, C is still king, so nice to see a challenger. But some
of the hurdles he describes in the article, ouch. No wonder people stick with
un-sexy languages and toolchains.

------
dvfjsdhgfv
I discovered D is a secret weapon at several companies, but not all of them
want to make it widely known...

~~~
thwy12321
Is this true? Mind speaking on which types of projects it was mostly used?

~~~
arunc
Congrats for Weka.IO. They managed to pull it through after the CEO visited
the DConf and asked for help.

D is a brilliant language. I got drawn into D by this article[1]. I've used D
to replace some of the test tools where Java/Python were used. GC is a friend
in such cases. I could've written them in C++ as well, but Phobos wins over
STL hands down!

Initially the "solution" test application for one of our REST services was
written in Java. It covered the functional, performance and longevity tests.
The REST service itself was written in C++. Ultimately the customers reported
that under heavy load the response time of the REST service is too long (> 400
ms) and we fixed the code as well. But during tests we were unable to really
load the application to the fullest. 48 cores & 48 threads in Java and we
still couldn't overload the REST service. We had no way to prove that our fix
actually worked. Ultimately, we rewrote the test app in D and boom, we were
finally able to achieve the load that customers created in their production
env - both in terms of response times and throughput. I could've written it in
C++. But D's std.parallelism + template mixin is a boon, thanks to David
Simcha. Based on our tests, the D test app was __20 __times faster than the
Java equivalent (written by one of the finest and the expert).

Sure, there is a lot of visual noise due to @nogc, etc, etc. But beneath that
there is a beautiful and readable language that's waiting to unfold, if and
only if the D authors realize that they stop the feature creep and revert to
sane defaults.

On the other hand, I started to convert the REST service from C++ to D last
year and it didn't go well. Blame me! There are _very_ rough edges in D as
well that are result of over engineering. One such thing is the const'ness in
D. It make it impossible to use const, ever, by library authors. I don't care
to go and fight in D forum. Manu has already screamed many a times. Lately I
found that one of the reasons for poor performance of my D port was also
because of auto-decoding! Duh! It was not even GC. (The default should be
fast, isn't it?). I spent a considerable amount of time for this port and
eventually it failed to meet the performance vs maintainability trade-off. C++
is getting better year by year, at least every 5 years :-). std::vector,
std::map, etc are safe for concurrent reads (at least on POSIX). std::vector
has erase with iterator. Phobo's std.array doesn't have an equivalent
features, albeit it is much more readable than STL.

FWIW, D doesn't deserve the negative publicity that it usually gets. It's a
fantastic language. Try it and if you like it, enjoy using it for the right
use cases. After all choose the right tool for the job at hand.

[1]
[https://wiki.dlang.org/Component_programming_with_ranges](https://wiki.dlang.org/Component_programming_with_ranges)

~~~
WalterBright
The difference between const in D and const in C++ is that D's const is
transitive, and C++'s is not.

Transitive const is necessary for things like pure functions. It's more
difficult to use because the compiler really, really means it. But when it's
there, you (the user) knows that function is not modifying the const data
structure, not no-how, not no-way.

------
z3phyr
D standard library is so well documented. Its criminally underrated.

------
fb03
Random Fact: I can't help but smile when I read that D's "STL" is called
Phobos because of my Doom 2 nostalgia :)

Have a nice day, everyone o/

~~~
jordigh
Well, Walter's company is Digital Mars so that's where he gets his naming
scheme from. Nothing to do with Doom except they both are using the same
Greco-Roman deity astronomical naming scheme.

~~~
skocznymroczny
Originally D was supposed to be named Mars. The stdlib is named Phobos and
there is a project called Deimos which is about providing bindings to C
libraries.

------
jonathanstrange
Perhaps an unfortunate name, though. For some reason my brain involuntarily
auto-corrects "WekaIO" into "WeakIO". I think advertiser sometimes even play
with this auto-correct feature of the brain to make suggestions without
stating them.

------
acd
Weka is a name of a Machine learning product from New Zealand.

The IBM system had 24xHGST N200 SSD, 830 000 random read, 200 000 random
write. that is 24 * 200 000 IOPS=4.8Million random writes.

The Matrix system had 64x1.2TB Micron 9100 SSDs. 750 000 random read, 300 0000
random writes. 64 * 300 000=19.2 Million IOPS

Ok so you have benchmarked hardware with 4.8 Million write ops vs 19.2 Million
write ops.

~~~
billforsternz
I clicked on the story in the hope that there was a New Zealand connection,
and was disappointed that it seemed to be an Israeli operation.

------
lumberjackstian
It seems this article is not accurate anymore? (the reference is from March).
It's not far off though:

[https://www.theregister.co.uk/2018/11/30/wekaio/](https://www.theregister.co.uk/2018/11/30/wekaio/)

[https://www.hpcwire.com/off-the-wire/wekaio-places-in-top-
fi...](https://www.hpcwire.com/off-the-wire/wekaio-places-in-top-five-of-the-
virtual-institutes-io-500-10-node-challenge/)

------
jpz
Ah, just looking at this. When I was looking at D back around 2005 it was a
closed shop run by one guy - I had the feeling it did not take off because it
simply wasn't an open enough project.

It is a shame, as since then we've had go, Swift, Rust all come along and it
is a lot more competitive for mindshare in that space... not to forget the
merits of modern C++ (it is a far better language post 2011)

~~~
fsloth
"it was a closed shop run by one guy"

You do realize Walter Bright is a frequent HN resident :) ?

~~~
jpz
I don't criticise him, I just think the structure of the project meant it
failed to really launch. There are people like me (20y+ C++) that when
certainly I think of D, I think of it being a closed project. I spent quite a
bit of time looking at it and thinking how great it was, but it seemed
confusingly proprietary (which is not a great place for a programming language
to be.)

------
iddan
Sounds to me like they would be better off with Rust as they are actively
avoiding GC. How are the two languages compare in this problem space?

~~~
StreamBright
You can avoid GC with a non-GC standard library that they actually wrote:

[https://github.com/weka-io/mecca](https://github.com/weka-io/mecca)

~~~
ileri62
Here's the thing though: They wrote a new compiler and a new standard library
to make the language work for them. How exactly is that good publicity for D?

D itself looked and still looks very interesting to me, as it can be very
performant, without having insanely ugly syntax such as c++ or rust (inb4 rUsT
iS VeRy ReADaBlE).

~~~
vthriller
> D itself looked and still looks very interesting to me, as it can be […]
> without having insanely ugly syntax such as c++ or rust (inb4 rUsT iS VeRy
> ReADaBlE).

I did a quick dive into aforementioned stdlib only to stumble upon something I
would rather prefer any Rust/C++ version over: [https://github.com/weka-
io/mecca/blob/f5dc6d9f71983ea7b852ad...](https://github.com/weka-
io/mecca/blob/f5dc6d9f71983ea7b852ad85a99aca2496674f98/src/mecca/lib/exception.d#L284-L293)

Some might want to exempt standard library from the "readable code" argument,
but I'd argue that readability argument was lost the moment language designers
allowed anyone to write things like `whatisthis!"whatever"({t.__ctor(args);})`
(regardless of best practices that regular devs might unanimously follow). You
can't just say that "A is bad 'cause it allows unreadable code" and follow
that with "nobody's doing bad things in B therefore it's much better".

~~~
ileri62
that's just nitpicking. I bet you can find equally ugly examples in the other
2 languages. My point is that "standard" code is infinitely more readable than
either c++ or rust.

~~~
vthriller
> My point is that "standard" code is infinitely more readable than either c++
> or rust.

Still I'm struggling to see any code that is _significantly_ more readable
_because of D 's design decisions_, or details on what makes D more readable
than $lang (especially compared to modern C++/Rust that you seem to despise so
much).

~~~
WalterBright
1\. Template syntax. For example, a struct template in D is:

    
    
        struct S(T) { ... }
    

A function template is:

    
    
        T func(T)(T t) { ... }
    

2\. No forward reference declarations required. This cuts down on a mass of
unnecessary boilerplate. Even better, it allows the coding style of ordering
the functions from most important to least important, rather than the reverse
that is typical of C/C++ source files.

3\. Terser declarations:

    
    
        ulong x;
    

instead of:

    
    
        unsigned long long x;
    

4\. No ugly #preprocessor code interspersed with your nicely formatted code.

5\. Nested functions mean they can be nestled close to where they are used
rather than much further away in the file.

6\. None of that awful

    
    
        #ifndef __INCLUDED_FOO_H
        #define __INCLUDED_FOO_H
        ...
        #endif

~~~
vthriller
I already find function template syntax (1) a bit more confusing because of
consecutive parentheses (I can get used to it pretty easily, but it will
always be a bit slower to read), and 2–6 does not make me want to convert to D
from Rust.

