Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Pg-extend-rs – Easy Postgres extensions in Rust (bluejekyll.github.io)
244 points by bluejekyll 6 months ago | hide | past | web | favorite | 22 comments

Ohhh this is great! I have had a query planner extension idea for a while but dealing with C bugged me a lot. I think that this might help me try some Rust ans some Postgres

As warning, there’s still a bunch of work to translate various types from Rust to Postgres Datums.

I’m interested in feedback on the library, so if you use it and experience any issues, please let me know.

This made me curious - are there any SQL databases being actively developed in Rust? I found one on GitHub (LlamaDB) that does not appear to be maintained.

TiKV (https://github.com/tikv/tikv) is a database (KV-store to be precise) written in Rust and speaks MySQL protocol.

Toshi (https://github.com/toshi-search/Toshi) is Elasticsearch-like engine that uses tantivy (Lucene-like), both also written in Rust.

There was https://github.com/PumpkinDB/PumpkinDB but it seems dead :(

Same with https://github.com/mozilla/mentat/

https://github.com/indradb/indradb is a newer db in alpha.

https://github.com/spacejam/sled is also in alpha but is being actively developed.

Thanks for the tip to Toshi! Exactly what I was looking for.

I have a need for an ultra simple, low volume, non critical full text search system. I was looking for alternative to Elasticsearch, that I could also contribute to, and here it is!!!

Tikv is key-value. Mysql layer, tidb, is in golang.

Tidb is built on top of tikv though.

Noria is an interesting one: https://github.com/mit-pdos/noria

It's compatible with the MySQL wire protocol and offers extremely efficient, incrementally updating materialized views.

It could be adapted to postgres. It's just that no one has written the library doing so.

What's the purpose of this? Why this over a function in PL/pgSQL?

That's a funny comparison. The much more obvious contrast would be to writing a Postgres extension in C.

Or are you asking, "why have native DBMS extensions at all?" Looking at the set of native Postgres extensions that currently exist would probably be enlightening:

• PostGIS, for one. You can't really define those types—and especially, the algorithms that operate on them—efficiently in PL/pgSQL. In this case, the extension is native for low overhead operation.

• uuid-ossp, for another. To generate the data it does, this extension needs access to several pieces of data from the OS (e.g. the MAC address of an Ethernet interface) that you can't access from PL/pgSQL. In this case, the extension is native to query native OS data APIs.

• Citus, for a third. Citus basically changes Postgres's runtime "operational architecture" in several fundamental ways. It augments the Postgres Postmaster, changes how replication works, etc. None of this is possible from PL/pgSQL. In this case, the extension is native to use native OS threading and IPC APIs to override how Postgres works.

(And that's ignoring the fact that having native extension support usually gets you ABI compatibility with the embeddable runtimes of most other languages, allowing you to then support things like Python or Java—and thus get access to low-overhead shared-memory+FFI-based "IPC" with existing libraries in those languages, rather than needing to have them run as co-servers and communicate over sockets. Which, in turn, gets you the ability to cheaply and easily use client libraries written in those languages to create Postgres Foreign Data Wrapper libraries for Datomic, or Elastic, or the AWS Public Dataset API, or whatever other weird-API-with-relatively-few-library-impls you like.)

I thought uuid-ossp was just a bunch of utilities around uuid parsing and generation. It shouldn’t need anything special from the OS.


A UUIDv1 contains the host's MAC address as a component. And the uuid-ossp extension (or rather, the OSSP's libuuid, which it uses) does indeed read said MAC address when building UUIDv1s: https://github.com/sean-/ossp-uuid/blob/master/uuid_mac.c

Keep in mind, if you're using uuid-ossp rather than just relying on Postgres's built-in gen_random_uuid() function, it's very likely because you want to generate UUIDv1s instead of UUIDv4s (as they're the only two types of UUIDs that see much use.) So this functionality is pretty essential to the extension—a version of the extension without it wouldn't be worth much.

The very first function on that link is for generating uuid v1s, which require the mac address...

A lot of Postgres extensions are designed to directly integrate external information systems that PL/pgSQL would never have access too such as announcing certain types of changes to RabbitMQ, or ensuring an ElasticSearch index is fresh with a subset of the data.

Additionally extensions can be used to extend PostgreSQLs native functionality (like PostGIS does).

I expect extensions in Rust will be much faster than those written in pgSQL, which would need to be interpreted, of course this should be validated. In addition to that, this is stdlib code in Rust, and stable, so you should be able utilize anything from the Rust ecosystem in an extension built in Rust.

There's always a trade-off of adding new languages to a project, and for simple things, Rust might not be the best choice.

Curious how tantivvy performs for full text search, in comparison with the native one.

This work inspires me. I am going to try to make a fake-rs extension to enable fake data generation

The benchmarks seem to show it neck and neck with lucene, so I'd imagine faster (and with a better ranking alogrythm) than the beuild in postgres search.

PostgreSQL has several bindings, including Python and JavaScript (V8).

You might want to do complex value manipulation at the DB-level, like a regex-like operation, or processing XML.

Does it have additional overheard compared to rolling in C/c++?

Right now, the only additional overhead I can think of would be on transforms from Rust style Strings and &str's to C style CString and CStr's.

Beyond that, I think it should be nearly identical with C/C++.

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact