
Recutils – Tools and libraries to access plain text databases called Recfiles - jdemler
https://www.gnu.org/software/recutils/
======
jdemler
At CurrySoftware we use recfiles combined with git for all business-processes
(incoming and outgoing invoices, customers, etc). It allows us to automate
everything we want with simple bash scripts. But we remain flexible because we
can perform non-automated tasks manually.

~~~
weeber
Your processes made me think about a email based interface instead of a bash
script, this may allows to easily interact with the database bot without
knowing bash or python.

~~~
jdemler
We plan use a Telegam interface for many things (status checks, new invoices
etc). Its easier and faster than E-Mail and available everywhere!

~~~
aeorgnoieang
Telegram (Messenger)?

~~~
jdemler
Yes. Communicating with telegram from bash is simple. Check out
[https://www.curry-
software.com/en/blog/telegram_unit_fail/](https://www.curry-
software.com/en/blog/telegram_unit_fail/) for example.

~~~
alvil
I'm wondering if it is possible also with Signal (signal.org)

~~~
detaro
Harder, since they don't have an open API and don't want people using non-
standard clients. The clients are open-source though, so probably you can do
it.

------
Mizza
I have a growing affinity for non-database-databases for personal and
low/sparse-traffic projects. Lots less hassle.

Here's one I maintain designed for use with AWS Lambda which uses S3 as a
Pythonic data-store:
[https://github.com/Miserlou/NoDB](https://github.com/Miserlou/NoDB)

~~~
maxerickson
I'm always fascinated that people get stuck differentiating between using a
database for indexing and as a canonical data store.

Like the majority of media apps, it's either impossible or a huge pain to get
them to index without managing.

------
fimdomeio
Apparently you can get output in csv (natively) and json (python one liner)
without too much assle
([http://swick.2flub.org/recutils_JSON_output.html](http://swick.2flub.org/recutils_JSON_output.html))
which suddently makes it even more interesting.

------
ajsalminen
The link to the newer presentation video on the page is broken but this is
probably the same one: [https://fscons.org/videos/2011/gnu-recutils-changed-
title-an...](https://fscons.org/videos/2011/gnu-recutils-changed-title-and-
subject.webm)

~~~
neuland
Thanks! I was a bit dissapointed that only the older video link was working.

------
rb808
I really like the idea of plain text data files so am really interested in
this.

YAML serves this purpose too but I'm not a huge fan of indenting so recfiles
look great. Anyone compared and contrasted?

Also are there other resources on this? Would be nice to have Java/C++/Python
libraries. (As well as convert to parquet, arrow etc )

~~~
jstimpfle
I have done some prototyping on a similar idea, but I think with a more
idiomatic approach. The idea is mostly adding relational structure (schema) to
CSV, and enabling a cleaner lexical syntax (get rid of the line noise).

Might some day dust it off and try to bring it to a more serious level
(performance, tooling etc).

[http://jstimpfle.de/projects/python-
wsl/main.html](http://jstimpfle.de/projects/python-wsl/main.html)

------
majkinetor
Fro docs:

YAML 1 is an example of a hierarchical data storage format which is much more
readable than XML. The problem with YAML is that it was designed as a “data
serialization language” and thus to map the data constructs usually found in
programming languages. That makes it too complex for the simple task of
storing plain lists of items.

I dont see how this is true. Provided sample with books is almost identical in
yaml.

The main benefit over yaml looks like more control of individial fields but
again, yaml based db app could do that too.

------
ausjke
I noticed this is GPLv3, which means if you use this library, all your
application will have to be open source, however IANAL.

~~~
creatonez
They have a page explaining their reasoning:
[https://www.gnu.org/licenses/why-not-
lgpl.en.html](https://www.gnu.org/licenses/why-not-lgpl.en.html)

~~~
yellowapple
The only reasoning they really need is "we wrote the GPL and we're gonna
goddamn use it".

------
oblib
This reminds me a bit of using CGI.pm's "Save" function. I built a pretty
decent invoicing app using that and the searches for data in documents saved
in that format are pretty fast.

I won't pretend to know the ins-and-outs of that but was told on a Perl mail
list that the server created a "B-Tree" index when an initial search was made
and used that afterwards.

------
hellmann
I do something similar with toml files for simple stuff. Python for piping
around, but maybe this is more convenient on the commandline.

------
setheron
Did this come out of Amazon ? I remember a similar set of tools for passing
along "recs" through pipes.

------
rwmj
I guess these are quite slow (because no indexing) once you have a serious
number of records? That in itself isn't a problem as long as you understand
the scope of the project. I wonder why they didn't use (a well-defined subset
of) CSV as the format however.

~~~
jdemler
CSV is neither human-readable nor -writable.

And I don't think the performance issue exists. Computers are fast nowadays.
Parsing recfiles is straightforward. Also you could easily archive
historic/old/probably irrelevant records.

~~~
rwmj
This is why I was very careful to say "well-defined subset". I wrote a full
CSV library[1], and so I'm well aware of how deceptively difficult CSV is to
deal with. However with a well-defined subset (and perhaps not using "," as a
separator as well) it should be editable for at least simple changes.

[1] [https://github.com/Chris00/ocaml-csv](https://github.com/Chris00/ocaml-
csv)

------
kwhitefoot
If you want something lighter to use text files as tables you could try
TextQL:
[https://github.com/dinedal/textql](https://github.com/dinedal/textql).

------
dguaraglia
Erm... am I the only one that's a little thrown off by the "mascot" in this
project?

~~~
agumonkey
I'm not fond of the joke right there but alas.

ps: I hope they have org-mode interop.

~~~
mbrock
It says so in the feature list so I guess they probably do.

~~~
agumonkey
dammit I failed at simple search, it's indeed listed

