This is really neat. But before you integrate it into anything else you build, consider that it appears to be a very old blob of C code. It took afl-fuzz something like 2 minutes to start finding wild free crashes†. This seems like a worthwhile thing to reimplement in a higher-level language.
† I don't know, maybe they've already done this work and ruled out anything bad from the dozens of unique crashes AFL trivially finds? I wouldn't want to pretend that I've done any serious inspection here.
For anyone else reading thread and wondering if someone has reported this issue yet: as of 10:35 UTC 27 Jan, nothing seems to have been posted to the bugs mailing list [1]. I'd encourage anyone who has even minimal knowledge about fuzzing and security and AFL (I don't) to make a post there. Don't let the combination of bystander effect + impostor syndrome stop you from doing it.
I think that a blog post with the details (exactly what to install, exactly what to run) will get a few upvotes here. (At least, I promise my upvote.)
Sometimes the manuals seam clear, but when you actually want to run the program you discover that it needs a library, or that the directories must have some specific names, or ...
Honestly, I don’t think it’s fair to request from 'tptacek to do this. It was just a friendly heads-up from him. We are many here who could do it. So if anyone has some time over and feel free to dig in :) (Or feel free to refrain.)
if you can be bothered, rebuild it with debug symbols, run it, dump core and try and find exactly where the bug is.
I vaguely remember doing this with wget, there was a way to make it think the terminal's width is (unsigned)-4, then when printing the download status to stdout, it clears a buffer with a memset(ptr, ' ', -4). Of course -4 in this context is a huge number. It overwrote its whole self until segfault. (this issue was fixed, btw)
great learning experience, for anyone who knows enough C to understand what they're looking at.
If I'd done anything significant, I would, but all I did was confirm the suspicion that this old c-language GNU tool hadn't been exhaustively fuzzed. I'm sure the recutils team can do a perfectly fine job fuzzing it themselves.
Indeed it's a compliment. And also a humble request because I'm interested in this subject. Perhaps my way of expression was not the best? But I'll take some time to see what other resources are available there.
I'm serious that you can read the (excellent) Quick Start for AFL, pick a C program (try recutils!) and get afl-fuzz running very quickly, and it's really sort of self-explanatory once it's running. It's a really well-built piece of software.
Hmm. Doesn't look like very "hands on" to me (README.md). Or then I just couldn't find the document you mentioned in the previous post. But I guess one has to learn these things by trial and error then.
For those looking for tutorials; in addition to the one already linked, I’m quite sure there are quite a few decent YouTube videos about fuzzing with AFL.
https://llvm.org/docs/LibFuzzer.html might also be quite interesting due to the potentially significantly higher fuzzing speed (no fork(2) for each try).
I think you're referring to my book data? It's all from Goodreads export, not hand entered. I love my compose key and will get around to cleaning this up.
I noticed the “Yeelong” mention in one of the examples. Do you happen to have a Lemote Yeelong? Always been on the lookout for one and would love to hear about it.
Recutils is really handy when coupled with command-line tools that return structured data.
`guix search`[^] outputs data in recutils format, so if you are searching for a database driver for Python, but want to filter out "python2" variants, and ignore uninteresting fields such as versions or dependencies, you can do:
$ guix search python mysql | recsel -q 'python-' -p name,synopsis,homepage
name: python-mysqlclient
synopsis: MySQLdb is an interface to the popular MySQL database server for Python
homepage: https://github.com/PyMySQL/mysqlclient-python
name: python-pymysql
synopsis: Pure-Python MySQL driver
homepage: https://github.com/PyMySQL/PyMySQL/
name: python-peewee
synopsis: Small object-relational mapping utility
homepage: https://github.com/coleifer/peewee/
Without recsel, the output is 100 lines long, with lots of duplication between the Python 2 and 3 variants.
[^] GNU Guix is a package manager that works on top of any GNU/Linux distribution.
Ah the joy of something old as a new discovery! What a great feeling, where has this been all this time?
OP/blog author: Great post, thank you for sharing your experiences
I've been sloshing in the text soup of Tiddlywiki Sqlite3(csv) VimWiki OrgMode Mediawiki Freemind(xml) and now it looks like Rec is the next ingredient to experiment with.
Inb4 someone makes or mentions a JavaScript-based version of this, which operates on JSON files.
In fact, you can already do the querying part using https://stedolan.github.io/jq/, I believe you can make modifications with it as well, but a different front end a la recins/recdel would make that a bit more convenient.
[{"title": "Opening",
"lede": "What happens when we \"open\" a file?"},
{"title": "ltrace",
"lede": "It turns out that ltrace uses the \"ptrace\" system call."}]
or
title: Opening
lede: What happens when we "open" a file?
title: ltrace
lede: It turns out that ltrace uses the "ptrace" system call.
? When you get a parsing failure after you edit it (or after a sector on your SD card gets a read error), which one do you think will be easier to fix?
In my (oceanographic) research area, some formats are binary and others are text-based.
Binary tends to be used for big datasets recorded by instruments that are left in the field, unattended, for months to years. Since every byte counts, these instruments cram information in very tightly. The binary nature of the files makes them a pain to deal with, but it also confers an advantage: the files are very seldom corrupted by a person who thinks they are benignly viewing the data.
The text files, on the other hand, sometimes get extra junk inserted because someone in the data analysis pipeline thinks it's OK to look at information in MSword or MSexcel.
Sometimes, opaque binary data formats are superior, in terms of data integrity.
I thought of this, while reading about recutils and thinking of a contrast with sqlite. Recutils looks great, but if I started sharing data in that format, I bet it wouldn't be long before derived versions of the files had become corrupted, as someone edited with MSword.
In this forum, people will sniff at people who use MSword, and I have done so, myself. But, it's a simple fact that some people who are good at one thing are not good at another. Some of my colleagues who use MSword for every silly thing (e.g. seminar announcements) are actually very good at their subject matter (e.g. the science talked about in those seminars).
I have a CI script which builds that YAML file into a SQLite database and deploys it + Datasette + custom templates to https://www.niche-museums.com/
I've been running the site like this for a few months now and I really like it. I love having my content in source control, I find editing the YAML to be reasonably pleasant (I even edit it on my iPhone sometimes using the Working Copy app) and any YAML errors are caught by CI before they are deployed.
I especially like this idea where you distill data into its most primitive form. It isn't quite as hairy as using n3-notation, but the ability to just collect the data without regard to being CSV or SQL gets you to an interesting space where tracking these changes with GIT works better. I've worked with CSV with 400+ columns (don't ask it was a horror) and SQL is really too verbose for these kinds of collections.
GNU Recutils is a nice metalanguage for data. You can churn out CSV from it which in turn can be \copy loaded into PostgreSQL. Maybe its too many steps, but I found it an important format for creating the logical model of representing data without predisposing it to any other technology or encoding.
Unfortunately its org-mode integration with emacs is broken when using spacemacs-- to far for me to fix as I'm no Elisp wizard.
Looks really interesting. Could imagine someone can build a open source Notion alternative with this and offer the Recfiles as export option, would enable all types of crazy use cases. Add a read/write API and now you can have bots acting on your knowledge base. Basically MediaWiki but with a long-standing underlying format and compatibility with some unix tools already.
I couldn't find much more, which is why I decided to write this intro. I may do a video exploration of the various tools and features as well. It seems like others are as excited by this as I am.
Earlier this weekend I was doing experimental performance testing of an idea I have for a wiki solution[0].
This, or SQLite could be really useful to embed data into pages.
[0]: yeah, I've had to dig beneath the surface of Confluence lately and Confluence has this weird property of immediately making me motivated to try writing a decent wiki
Looks really nice except agpl license which I used to think meant that any integration towards any existing solution.
At this point I've recently read someone from FSF(?) saying this is only MongoDB and others misinterpretation so at this point I am just utterly confused.
Also, when somebody uses AGPL that usually means: we found the scariest license we could find while still calling it open source, but, we have a commercial license to sell you.
However I couldn't find any licensing option. Does this mean it isn't just a way to sell commercial licenses?
I'm completely honest here. I really don't get it, but then again it took a while before I really understood the GPL as well so I'm ready to be enlightened :-)
TBH I hadn’t noticed that it’s licensed under the AGPL. I haven’t studied that license closely but yeah, I guess for many companies it’s considered a liability.
I went ahead to study the FSF FAQ but they don't really answer it completely as far as I can see. The clear cut answers are:
- if you combine your program with an AGPL program then your program has to become AGPL as well, just like the GPL.
- if you use an AGPL program unmodified it doesn't seem like you have to distribute kt to users who use it over the network
But as far as I can see the FAQ doesn't say what happens if the AGPL program reaches out to other applications to get data. For some reason I always though anything that was touched by the AGPL program, either over the network or otherwise would have to become AGPL.
If that isn't the case - and there is more and more to suggest that, then I think FSF should point that out clearly.
Notion is a free-form information space, Recutils is more about a collection of entries with a specific format. If anything Recutils could be the import/export format of Airtable instead.
Why haven’t I ever encountered this before? Seems cool and better than all the bespoke config files lying around on any given unix machine (passwd, hostnames, fstab, etc).
I wonder sometimes if, every 1-2 years, I should review the full catalog of well-maintained libraries and utilities available on Linux / GCC / LLVM / Python. I forget about many of them because they weren't relevant at the time I came across them.
Then I forget about that idea 10 minutes later when a new episode of my anime watchlist gets dubbed. I wonder how much that has cost me.
This is a fantastic idea! I might start a checklist (hmm, or a recutils db?)
I've been using emacs for a couple of years now and every couple of months I find something new and huge buried in it that surprises the heck out of me... how can these things hide for so long!
I used to do something similar, namely; every 6-9 weeks I would open up the Linux Kernel menuconfig and keep up to date with "everything", reading the helptext on each "(NEW!)" item.
Off topic, but do you have any good anime suggestions? My favorites are Deathnote, Attack on Titan, Full Metal Alchemist, etc. Trying to find something as amazing as Deathnote is challenging.
> a cat/mouse detective story with well defined rules and Sherlock vs. Moriarty levels of intelligence that _actually convince_ you that the characters are geniuses
Deathnote is pretty exceptional, so it's not surprising that you're having trouble finding other series of that caliber.
From your list, I'm guessing that you like anime that makes you think, and aren't turned off by story arcs that are overall dark and depressing.
The first thing that comes to mind is "Death Parade". It's currently available on Funimation, not sure where else.
A few more suggestions are "Elfen Lied" (although it can be a bit gory) and "Black Lagoon" (not super dark, but very entertaining imho).
Note: I only watch English dubs because I can't follow the action while also reading subtitles. If you're open to subtitled anime, I'm sure someone else could give you a much bigger list of suggestions.
I'm pretty much with you on that. I would prefer an English manga, but the translations are weaker in my experience when they're in the respective show. Although to be fair I've only both read and seem Deathnote.
I did watch the subs for the latest Attack on Titan because the dubs weren't out yet. But I was already invested enough in getting answers to that shows many questions as soon as possible.
It's a shame there's not much Deathnote level stuff. I'm so invested in that particular execution of a cat/mouse detective story with well defined rules and Sherlock vs. Moriarty levels of intelligence that _actually convince_ you that the characters are geniuses. Watchmen also comes to mind.
Any recommendations for non-anime that fits that genre? Novels, shows, comic books, etc?
I'm not sure. Are we talking about stories where the author is a genius, or where one of the characters is a genius?
For example, the film "A Beautiful Mind" is about character who's clearly a genius, but I don't recall the script showing signs of its writer having rare intelligence.
Whereas stories like Deathnote have some plot twists that make me think the author is highly intelligent, regardless of the IQ of most of the characters. I'm guessing I'd put a story into this category if there were plot revelations at the end which in retrospect I could have figured out my self, but didn't until the moment chosen by the author.
Like you might expect, this is quite poor. CRC32 for authenticity, mac-then-encrypt, no binding between keys and values, using low-entropy passwords directly as AES keys, and a fairly trivial looking read overflow in the decrypt function. That's just two minutes looking at one source file.
Just for answering how and where it could be used: GNU Guix uses recutils format to display search results and more, for example use the recsel command to select sessions of interest
I think this gets really interesting when combined with the bash builtin that supports reading records into variables[0].
recsel contacts.rec | while readrec
do
if [ $Checked = "no" ]
then
mail -s "You are being checked." ${Email[0]} < email.txt
recset -e "Email = '$Email'" -f Checked -S yes contacts.rec
sleep 1
fi
done
I like the idea. But I love sqlite. I think the perfect match of this idea and sqlite is a serialization format to and from Sqlite, in a nice text format. It could look something like this, but probably closer to the "line mode" that the sqlite3 tool implements.
So what I don't understand about recutils is in what category of problems does it really excel at over other solutions?
If I want human readable files I think I would opt for just a bunch of markdown files in a directory structure before I go the recutils route. If I want a lightweight database I think I would go for SQLite like others have already mentioned. In what situation do you really run into that you need human readable/editable referential integrity?
Some people might prefer human readable databases. For example if your database is not too large, you could use it with Recutils and version control the database itself.
You can do this with SQLite but it would require CSV export:
So maybe the key technical benefit is human readable diffs of the database where the tool enforces referential integrity? Otherwise, it feels like a subjective personal preference thing. (e.g. I like the color green, but you like the color red)
Just like markdown provides a quick way in that to renderize something pretty, but is still readable in plain text, recfiles do the same for data, it has data types, keys, integrity, and is easily queryable with its tools, and also easily read in plain text.
Sure, you could use some type CSV, but that is not pretty to read as recfiles. The point is the same as markdown, usable as just text, but with good tooling around it.
with markdown files in a dirs, you would have to provide you own tooling.
Okay, but markdown is data (perhaps not well structured, but data nonetheless) + formatting. If I'm editing and going into the files directly anyway why do I need to be able to do queries? I guess I'm asking what's the point? Can't I find the info I want just using find and grep on a bunch of markdown files? Maybe I don't even have to do that if I've organized my markdown files sufficiently well.
Let me be clear, I get that you don't have db operations with a bunch of markdown files with a directory structure, but on small scale data repositories do you really need that? For example, if I have a bunch of recipes in my "personal database of markdown files" I can quickly find the chili recipe I'm looking for by going to the directory "Soups & Salads > Chili > Slow Cooker Chili Recipe" or something along those lines. Or I could have grepped for "slow cooker chili". Either way I'm going to find that recipe with no extra tooling on my part.
Where do rec files features add value? Plus, if I use rec files now it seems I have to build my own formatting because I can't rely on markdown editors to build it automatically for me. Or is there a way to specify formatting for rec files?
I never used 'recutils' before, so, before I start investigating, did anyone try to pair this with Markdown? Could this be made an extension to Markdown, so I can encode data fields within markdown?
I had an idea a while back to to this end. Markdown files as a database of sorts with queries, validations, actions/plugins, etc. written in Javascript. I documented the interfaces but haven't had time to implement it.
I imagine you could use recfmt templates [0] which generate valid Markdown, and use (pp [1] and) pandoc [2] to (pre)process into the desired final format.
I'm not sure what you mean "data fields within markdown".
You could definitely use markdown within fields of a recfile. There would be some multi-line syntax you wouldn't be able to use, because it would conflict (e.g. '+' unordered lists), but other than that you could mark up text with Markdown, no worries.
It's practical for many use cases where that doesn't matter. Keeping track of your personal book collection is good, your bank's online transaction processing system probably not.
I think the most current version is this one [0]; I submitted a bug report to request the GitHub repository be updated to point to the Savannah repository. Also, the Python bindings are here [1]
I do not think that there was ever a period when "the only option for computing for quite a while" was text files. It certainly wasn't the 1960s, 1970s, or 1980s. dBase databases were not text files, for just one example.
I was refering to plain text as the interface to the machine, not the file formats. When your primary interface is through text, it makes sense that good tooling would develop.
In the 90's there was rdb aka nosql. It was packaged in debian too at that time. I was a happy user. Then lost track of it. Thanks to this post I tracked it down again
your database is a human-readable text file that you can grep/awk/sed freely, and a line-oriented structure makes it perfect for version control systems.
But in fact it's not great for grep/awk/sed/, etc. For those tools to work well, you'll find you need to keep each record on its own line.
Is recutils available as a library? Are there language bindings? Specifically asking for Go support.
Never heard of recutils before but the on-disk format looks compelling. There has been limited (no?) takeup however which makes me fear recutils are intrinsically broken
> (yes, the offical package image is two gay turtles)
It's weird that the FAQ page linked there [1] doesn't seem to link back in any way to recutils' own page. The only way to get there [2] seems to be to click on the "Software" header link and Ctrl-F for recutils.
Given this is human-readable, I'm curious to learn what you're looking for in a specialised GUI that any text editor wouldn't provide? (vim, nano, notepad++ VS Code, whatever)
I'm thinking on using this for me, but being able to interact with non-programmers used to Excel and similar GUIs. But I enjoy working only with the text formats :)
While I agree with the general awesomeness of sqlite, I worry about the blanket statement that you've claimed.
If you read the documentation 'Purpose' [1], it says, the issues with databases, like SQLite:
- The stored data is not directly human readable.
- The stored data is definitely not directly writable by humans.
- They are program dependent.
- They are not easily managed by version control systems.
I just want to say that I also love sqlite, but I cannot avoid worrying that when the asteroid strikes and the ocean men invade, I will not be able to program a sqlite driver on a homemade 8 bit computer.
Sure, text files are awesome, I get it and generally I agree. But sqlite is in every nook and cranny of our computer and mobile systems, it is in Android, it is in the browsers. I find it hard to imagine a postapocalyptic world where one would be able to read a file somehow, yet not be able to get access to some sqlite processing software.
I like the rectools for educational purposes, but beyond this for any however small purpose project I'd rather switch to sqlite pretty soon.
My ambivalence exactly. The user in likes plain text for its accessibility and flexibility, but the programmer in me dislike it for its gross inefficiency.
Does some sort of general data compiler/decompiler exist? Please don't say Binary XML.
There are so many use-cases where the compute inefficiency of this just doesn't matter but the human efficiency of being able to read and edit the files is a huge win.
Consider the gross inefficiency in the other direction. To simply edit done simple data, you have to have how much code and infrastructure? And how much of that is plagued by bitrot?
It is a "scaling" problem. At small scales, yes, human time can be more valuable than computer time. At large scales, humans will sacrifice their own time to make computers work faster. The problem is with intermediate scale, where the computer takes a bit too long to do simple things.
Case: You really, really enjoy your text editor and have even built special tools that work within it, but it doesn't support SQLite and you generally enjoy using text tools anyway.
I wrote a SQLite <-> Json converter for this reason. Ended up coming up with a weird convention in the Json to represent relations. It might be cool to instead write a recfiles <-> SQLite converter. I want to use SQLite from within my app, but want to check in readable data to a source control system.
Sometimes you want data under version-control. You want to track changes and maybe you want to be able to reverse them. An example is human-supervised data-imports.
Example: we operate a pipline that pulls datasets from multiple government agencies and merges them. The sources are updated periodically and updates frequently introduce new inconsistences. To reconciliate, we use auxiliary datasets that record tweaks to each source, such as entries to add, mutate or delete.
In our implementation of the pipeline the source datasets, the tweak tables, and the resulting merged dataset are kept in Sqlite for ease of processing. The pipeline writes out dumps of each table during processing, even for intermediate stages. When running the pipeline, I can scan the diff and decide whether the changes are reasonable or tweaks must be introduced or removed. Once I'm satisifed with the result, the dumps are then committed to record the current known-good state.
When somebody wants to know why one record in the result is the way it is, I can determine how it changed in the source data, how it was tweaked, and what the result was before and after. It's really easy to produce diffs between revisions. Code-reviews are conducted over changes in pipeline code, validation logic, source and result all in one step.
In GNU Emacs SqlMode is not as nice to work in, whereas recfiles integrate out of the box with org mode. Doing the same with sqlite works, but is not nearly as nice.
I've never heard of Recutils before, but I've been using grep, sed, awk, cut, etc. to work with data for years. sqlite is just too much of a hassle for version control and sharing. If I use sqlite, I have to export to text, commit and push, pull on the new machine, then import into sqlite. I've never run into performance issues even with thousands of records.
And where you need a reasonably performant relational data store, all those cumbersome gymnastics might be worth it.
But if you're working with a modest dataset where volume and performance aren't a primary concern, working straight in text is infinitely more convenient. There's a reason people do so much bulk data processing in CSV, JSON, or other easily manipulated formats.
It would be simple to implement an append-only writer in Recutils. I don't know where I'd start if I wanted that for Sqlite. I don't think the format allows it.
I recall a type of XML-representation that looked like this, i.e. was line based. And there was a Python (?) tool to convert XML to and from the line based format. Would be nice to compare with recutils but I can't find it now
I'm not sure exactly what I'm imagining or how it would work, but reading this post I can't help thinking some kind of Wikidata integration could be really interesting.
This is cool in a kind of "neat idea" way but please god nobody use this for anything that anybody else might ever use! I feel like it's almost irresponsible to tell people that this exists.
† I don't know, maybe they've already done this work and ruled out anything bad from the dozens of unique crashes AFL trivially finds? I wouldn't want to pretend that I've done any serious inspection here.