
RustgreSQL - desiderantes
https://www.postgresql.org/message-id/CAASwCXdQUiuUnhycdRvrUmHuzk5PsaGxr54U4t34teQjcjb%3DAQ%40mail.gmail.com
======
solidsnack9000
Postgres's extension APIs are pretty good and Rust's interoperability with C
makes it quite feasible to extend the database without rewriting it. Some
months ago we wrote a logical decoder -- to capture and forward the write-
ahead log as JSON -- in Rust and later were able to pull out some helpful
bindings for logical decoders, foreign data wrappers and so forth.

The logical decoder:
[http://pgxn.org/dist/jsoncdc](http://pgxn.org/dist/jsoncdc)

The bindings:
[https://github.com/posix4e/rpgffi](https://github.com/posix4e/rpgffi)

The extensibility of Postgres is really profound -- there are so many things
you can do, and so much internal functionality you can access, without having
to recompile the database. The way extensions are loaded also mitigates the
risk of what can be admittedly tricky issues relating to safety. Extensions
run in backend processes, not the postmaster.

If rewriting Postgres happens, I hope it would be gradual. You can replace a
lot of the core with extensions, and the scope for extension is ever
increasing. So say you wrote a storage engine in Rust. Writing a foreign data
wrapper for it would give you a way to use it from Postgres for all your
tables; but you wouldn't have to write a SQL parser, client manager,
permissions system, or any of the rest of that stuff.

Although the C stuff is a little gritty, a lot of what goes on in a database
is gritty "because it is" and that low-level control is needed to deliver on
tail latency, fault tolerance or simple storage efficiency. For example,
Postgres has code to recover from malloc failing.

~~~
masklinn
I'm not sure there's much reason to rewrite Postgres as a whole. If there are
specific subsystems which are common sources of CVE they might be targets, but
looking at the list[0] no specific system really seems to jump out.

Extensions could be neat, there doesn't seem to be a high-level Rust API for
it (in the style of ruru or rust-cpython) yet though, only incomplete-looking
bindgen:
[https://github.com/posix4e/rpgffi](https://github.com/posix4e/rpgffi)

[0] [https://cve.mitre.org/cgi-
bin/cvekey.cgi?keyword=postgresql](https://cve.mitre.org/cgi-
bin/cvekey.cgi?keyword=postgresql)

~~~
stymaar
I'm not advocating for a complete rewrite of Postgres in C, but I'd like to
emphasize the fact that security is not the only good reason to justify the
transition of a project from C to Rust:

Nowadays most developers comes from memory-managed languages => for us, C is a
scary land where it's really hard to get things right when you don't have all
the prerequisite knowledge (what's the stack ?) => the language is really old,
you don't have all the fancy and useful tools and syntactic sugar (package
manager, proper error handling, generics) => then we don't want to invest a
lot of time to learn a difficult language that is so unproductive. => and even
if we do out of curiosity, we'll most likely never feel confident enough to
dig into an open-source project and contribute.

Rust on the other hand, is like C but with a personal IA assistant (the
compiler) and all the syntactic sugar you're used to (and even more if you're
not familiar with the ML family). That's what's attracting so many web
developers to Rust !

In my opinion it means that, in a somewhat near future, the amount of
potential contributors for projects written in Rust could be higher than for
projects written in C. And those contributor are also likely to be more
productive, thanks to the language features and tooling.

~~~
thesmallestcat
You're describing qualities that are meaningless to a database user. A Rust
implementation of PostgreSQL, with its runtime checks, would almost certainly
be slower than the existing implementation. So a Rust port is a non-starter.
Database users don't care about your syntactic sugar, that you find manual
memory management a "scary land" due to background in managed languages, or
how many contributors you have. Speaking of which, the PostgreSQL project is
healthy and has many, many contributors. And while we're discussing developer
productivity, I can't even imagine the build time of a Rust port.

What's annoying about OP's email is that they clearly did no research on the
matter. You can find many discussions on the list about "upgrading" to C++,
and there has been much wringing of hands. The last I heard, it sounded like a
faction was considering having the project build with a C++ compiler rather
than a C compiler for better type checks and as a step on that path.

~~~
stymaar
> A Rust implementation of PostgreSQL, with its runtime checks, would almost
> certainly be slower than the existing implementation. So a Rust port is a
> non-starter.

Wait, which runtime checks are you talking about ? This is a recurring FUD we
often see on HN about Rust and I'd like to clarify this point.

IIRC, there is no mandatory runtime check in idiomatic Rust except UTF8
validation for strings[0]. There is no overflow checks on numbers, and no
bounds checking when using iterators[1]. Another known overhead is the default
use of siphash in hashmaps for DoS prevention[2], but this is not really a
«runtime check».

Do you have anything else in mind ?

> Speaking of which, the PostgreSQL project is healthy and has many, many
> contributors.

Indeed, and as I said, I'm not advocating for a port of PostgreSQL in Rust.
But for other projects that suffer from a lack of attention[3], it could be a
good move.

> And while we're discussing developer productivity, I can't even imagine the
> build time of a Rust port.

touché, but compile time is one of the biggest focus of the Rust core team
atm, so I'm confident things will improve in 2017 :)

[0]: which can be worked around if need be by using &[u8] instead of &str
(see:
[https://news.ycombinator.com/item?id=13268051](https://news.ycombinator.com/item?id=13268051)
from /u/burntsushi)

[1]: there are bounds checking when you access array elements through indexes,
but most of the time you don't do that since iterators are great. If you
really need to, and this have a performance impact, you can opt-out the bound
checking in unsafe block.

[2]: But this is opt-out by using a different hash function, if you use your
hashmap in a non adversarial environment.

[3]: like librsvg which was chronically under-maintained.

~~~
thesmallestcat
[1] is great. Rephrased, "there isn't runtime bounds checking, unless you use
the language. But you can opt out in an `unsafe` block." OK? So we're back to
the usual Rust argument: At least your unsafe code is greppable. And I don't
think any competent C or C++ developer considers that a worthy tradeoff when
it comes with a more complex, constantly evolving language, far less
mindshare, slow builds without solutions (unlike C++), and so on.

Regarding [2], I think it's safe to consider any software that takes input as
operating in a hostile environment.

~~~
tupshin
That is an extreme mis-characterization of stymaar's excellent summary. in
short, there are no runtime checks for iterating through an array, and since
the general case of random access to an array can't be safe (without dependent
typing), that is the one place the runtime checks are automatically added, and
even there you can opt out of them. It is easy to get C equivalent speed with
far more safety. You are spreading FUD, indeed.

------
nl
_My motivation is primarily I don 't want to learn all the over-complicated
details of C_

If the author finds _C_ overcomplicated I look forward to finding out what
they think of a complete database system.

~~~
nathancahill
As a Postgres user, I like the idea. But their post comes off sounding like a
JS dev wanting to rewrite an app in the latest popular language.

~~~
flukus
In this case there are potential performance and safety improvements by
rewriting things in rust, the exact opposite of a JS rewrite.

~~~
SwellJoe
Why would Rust be faster than C?

And, is potential safety 5+ years in the future (that's an absurdly optimistic
estimate of a rewrite for an OSS project the size and scope of PostgreSQL)
worth effectively standing still for all that time? Is the security/stability
history of PostgreSQL poor? I'm not under that impression.

~~~
cornstalks
Rust isn't inherently faster than C, but it can be easier in some cases for
the optimizer to optimize Rust code better than C code. In particular, I have
in mind Rust's reference types and the fact that they all "follow LLVM’s
scoped noalias model"[1]. You can, of course, get this same benefit in C by
using restrict, but most of the C code I've seen makes little use of restrict
(and it's quite easy to accidentally alias a restricted pointer, so it
definitely imposes a mental burden, even if it's very slight).

Rust's safety model allows you to get the optimization of C's restrict without
the footgun. That can be a big win.

But of course, just because something is written in Rust doesn't mean it'll be
faster than the equivalent C code. Rust isn't magic, after all.

[1]: [https://doc.rust-lang.org/reference.html#behavior-
considered...](https://doc.rust-lang.org/reference.html#behavior-considered-
undefined)

~~~
Sanddancer
This has been touted, but has there been a non-trivial project that shows
this? Something like a media manipulation library where there are lots of
places for optimization. It's been stable for a few years now, so I imagine
someone's seen performance benefits if it's happened.

------
diziet
I am not sure what to make of this: the author is a cto of a 100~ person
company, but he is thinking that a project like pg can be realistically
rewritten in rust. I just don't get it. Even if you could somehow make the
'complicated' and 'scary' parts of the code base magically transform into rust
(much less handle performance and other issues due to memory management being
different between c and rust), how would you maintain that project? Ai driven
code translation from c to rust is a singularity too soon.

~~~
rtpg
What's the argument against possible Rust porting of Postgres?

Because of the interop ease, it's not hard to imagine a multi-year peicemeal
rewrite. Postgres' code has some pretty good idioms that could be nicely
encoded in Rust, and a good separation of concerns

~~~
jeffdavis
The biggest argument against it is falling into the "pure technical" trap.

Postgres has a great focus on solving real problems for real people. Losing
that focus would be a disaster.

------
protomikron
Ok, I really like the approach Rust is taking and appreciate their effort to
make low-level system programming safer, but it really is getting ridiculous.
Not that implementing a DB system in Rust would be infeasible, but advocating
to just rewrite some large projects in Rust is probably not the step forward.

Waiting for a flamewar on the LKML on this.

"Hey, Linux devs. I heard there is this nice language called Rust. Do you mind
rewriting your OS in this? Ok, might not be possible over the weekend, but
maybe over a year? Maybe use some Deep Learning to automagically transform the
code?"

I smell an upcoming Linus Rust rant.

~~~
anarazel
> Ok, I really like the approach Rust is taking and appreciate their effort to
> make low-level system programming safer, but it really is getting
> ridiculous. Not that implementing a DB system in Rust would be infeasible,
> but advocating to just rewrite some large projects in Rust is probably not
> the step forward.

Note that the OP has, afaik, no involvement in the rust project. Nor has he
done much hacking on postgres. Don't think the rust people can be blamed for
that one.

~~~
protomikron
Yeah, that was probably phrased poorly. I wanted to criticize the "just
rewrite it in Rust" evangelists, not the Rust makers.

------
iagooar
I would suggest that the author creates a clone of a "simpler" DB like SQLite
or Redis. Those have, as far as I understand, good C codebases, so
understanding the system should be much easier.

This way you could also say: hey! I have successfully ported a DB written in C
in the past, and it has been an extremely positive experience, how about
trying the same with Postgres?

~~~
nathancahill
SQLite for sure. The test suite is top tier.

~~~
dllthomas
Hopefully you could even reuse it!

~~~
rosser
If you wanted to write a drop-in SQLite replacement in Rust, using the
original project's (incredible) test suite would be on the mandatory side,
wouldn't it?

~~~
dllthomas
To a degree. Certainly the end-to-end tests should be something you can expect
to reuse. Other tests may be hit or miss. In the case of SQLite, where there's
>100x as much test code as application code, there's a large range between
"can use some of it" and "can use most of it" \- as mathw gets at I think we
should be able to expect the latter here.

------
kibwen
Come now, this is just silly. And I say this as someone who's been using Rust
for years now! Let's avoid upvoting stuff this insubstantial.

~~~
stymaar
HN is just weird …

A guy randomly asks a question on a mailing list, and then his mail goes in
the front page, and then people start insulting him: «blah blah, stupid
politic CTO … understand nothing, blah blah blah».

~~~
ww520
Because this reminds us of the damage done by technical decisions made by
ignorant people in charge. The resonance draws out the overwhelming reaction.

------
hendzen
More importantly, when are we going to see a port of PostgreSQL to React?

------
rcthompson
I was immediately reminded of this:
[http://thedailywtf.com/articles/unpythonic](http://thedailywtf.com/articles/unpythonic)

(TLDR: CTO comes in and says "Angular and TypeScript is old. Python is new,
and Google is writing everything with it. Python is the best practices for
development." Tells IT department to rewrite everything in Python. “If our
team still needs to use Angular, just use the Python version.” After failing
to convince CTO this is absurd, IT realizes they can do nothing and tell the
CTO they _totally_ rewrote everything in Python, since he won't know the
difference anyway.)

This case isn't quite that bad, obviously. The request is more reasonable, but
still seems to betray a pretty fundamental misunderstanding of the situation.

~~~
sebcat
The sender of the e-mail is also a CTO.

A CTO of a payment company, so the inquiry to see if it would be possible to
replace a large C project in their stack with a drop-in replacement written in
a more safe language makes sense from where he's standing.

------
andrewchambers
This person doesn't seem to know much about reality. Or maybe I'm just a jaded
jerk.

~~~
michaelvoz
Agreed. Not discounting that you are a jaded jerk, as I think I may be one
myself.

------
prymitive
_My motivation is primarily I don 't want to learn all the over-complicated
details of C_

I don't understand this statement. Is he saying he want to contribute to
postgres, but he doesn't know C well enough, so he kindly asks the team to
consider the idea of a rewrite in Rust to make it easy for him?

------
andrewvijay
Is the C language more complicated than building a database?! That's looking
to be way more exaggerated.

~~~
stymaar
For people coming from managed languages (and in 2017, the huge majority of us
is), contributing to a C project is way more complicated than contributing to
a database project.

For open-source projects, being written in a __scary __[1] language is a
liability in the long run.

For many tasks, we used to have no good alternative than using C (or C++ which
is even scarier), but this is no longer the case now thanks to Rust.

I'm obviously not saying we should rewrite PostgreSQL in Rust, but the ability
to grow/maintain the contributor base is an argument that should not be
underestimated.

[1]: no matter how difficult or not it is in practice, the C is scary for many
people and most of them will never feel confident enough to suggest changes in
a C code base.

~~~
thesmallestcat
Out of curiosity, if you find C and C++ to be scary and unapproachable, how on
earth can you feel confident in your assessment that Rust is a good
replacement for it?

~~~
steveklabnik
"scary" and "unapproachable" does not mean "I do not understand."

~~~
thesmallestcat
Yea, but it strongly implies it.

~~~
steveklabnik
I very much disagree, but I doubt we're going to agree in general.

------
akst
If you want to write extensions in Rust, here's a library for rust that
utilises Postgresqls own allocators

[https://github.com/michaelsproul/postgres-alloc-
rs](https://github.com/michaelsproul/postgres-alloc-rs)

------
cwyers
> Porting PostgreSQL to Rust would be a multi-year project,

That's... underselling it somewhat a little.

------
nostrademons
I assumed this was satire of the various "Let's rewrite X in Rust!" posts, but
perhaps that's just Poe's Law talking.

------
alexnewman
This isn't a port of postgresql, more of an attempt to extend it's
functionality into new spaces with the power of rust.

github.com/posaix4e/rpgffi (logical decoders and foreign data wrappers in
wrust) github.com/posix4e/jsoncdc ( an example logical decoder in rust)

We are going to be adding bigtable and s3 fdw soon.

------
myf01d
> Let's rewrite & test a 20 year old already working & popular project with
> ~1,000,000 lines of code just because I don't fucking know what I am saying.
> Please engineers try to figure that out and come up with solution before
> Monday.

------
perlgeek
FWIW if somebody were to port a DBMS like PostgreSQL to Rust, or any other
programming language, it would only be feasibly to do it component by
component. Much like Mozilla now plans to use Servo for CSS rendering first,
and maybe for more tasks later.

------
flukus
Aside from a complete OS, a database implementation that is performant seems
like it would be the ultimate proof that rust can replace c.

I did find a basic one:

[https://github.com/nukep/llamadb/blob/master/Usage.md](https://github.com/nukep/llamadb/blob/master/Usage.md)

------
timClicks
I really hope that this doesn't mean we've already hit peak Rust.

The recent hype of the language is great, but I think it's really important to
start seeing substantial large projects receiving productivity gains otherwise
it'll attract derision and then loose its momentum.

~~~
mathw
Firefox should be the poster child for this, with the Oxidation project now in
full swing. I'm hoping to see similar work appearing elsewhere though. There's
no reason why any large C or C++ project couldn't decide to write some new
component in Rust, if they felt it would benefit them.

But they would need to have a contributor and maintainer base sufficiently
motivated to learn Rust and stick with it. Which will happen more and more if
Firefox can be visibly shown to benefit from including Rust in their toolkit.

------
Jweb_Guru
It's perfectly feasible to write something like Postgres in Rust, and it's a
good idea, too! But Postgres is not going to be that database, for a huge
number of reasons. Even Mozilla isn't replacing Gecko wholesale with Servo.

------
belvoran
I'm getting sick of that Hype Driven Development.

------
bwblabs
A port to a way more strict type and memory safe language would be great, but
as Craig Ringer from 2ndquadrant replies: the postgres project is pretty
conservative & this could only happen in an incremental way.

------
purpleidea
A lot of people aren't being very nice ITT and elsewhere and criticizing the
guy for asking. If you don't ask you won't learn. Remember to be nice!

~~~
saurik
This person could have and absolutely should have ran this past some other
people before bringing it to the upstream PostgreSQL developers, not just
wasting their time but putting them in the awkward position of having to
respond for an audience.

------
titraprutr
This feels more like a troll than a genuine question.

