If anyone reading this builds software like this, and releases it publicly, whether free or for sale, please let me know. I'd be happy to send people your way.
Because of https://sive.rs/dbt I get a few emails a week from strangers, asking if my software is public yet, or if not, what else I would recommend. (And I might never make my software public. It's too tied-in to my now-complex PostgreSQL system of everything.)
I'm definitely going to send people to Jakob's post here now. But if interested in this subject, please email me here:
My implementation of your system is just a bunch of markdown files. The title of the file is a friend's name, the frontmatter at the top keeps info about them that doesn't change too often (ex: birthday, address, kids names), and each time I chat with them I add a section with an H1 title that's the date we chatted (ex: "# 2022-02-14\n"). My notes about the chat go under that heading.
Based off of that structure I can do a bunch of cool things with simple scripts:
- print out a list of upcoming birthdays, half birthdays, birthdays on Jupiter, etc. of all my friends
- print out a list of friends I haven't contacted within my desired `frequency` (the subject of this post)
- create or open the friend file containing a particular name from the command line (ex: `,people Sarah` prints out all the Sarah files I have and asks me which I'd like to open)
I think plain text is the way to go with something as personal as a system for keeping track of your friends. I was inspired to go this route by your post on journaling[1]:
> If digital, use only plain text. It’s a standard format not owned by any company. It will be readable in 50 years on devices we haven’t even imagined yet. Don’t use formats that can only be read by one program, because that program won’t be around in 50 years. Don’t use the cloud, unless you’re also going to download it weekly and back it up in plain text outside that cloud. (Companies shut down. Clouds disappear. Think long-term.)
I've been chewing on how to make this system something that might be used by other people who don't know how to use the command line. Something like the way Obsidian[2] works might make sense.
In 25+ years of keeping people in my database, and writing plain text all day, I'd never once considered keeping my list of people in plain text files.
The tooling (or lack thereof) is the only problem I have with recfiles. It quickly gets unpleasant entering the (repetitive) metadata every time (and the CLI entry isn't much better). I get the impression from the recfiles page that the Emacs plugin is the main supported one, and maybe that one has a good UI flow that makes it usable, but I don't use Emacs.
I thought/hoped visidata would solve the problem. It has decent support for recfiles display - just being able to get a tabular view of the data is very useful - but the editing support is pretty unintuitive and full of footguns.
I'm glad you mentioned it because it's the format I would like to see get more attention, and I agree that tooling around it is holding it back.
The repetition is genuinely useful if editing or adding one or two entries by hand, but tools for ingesting data into recfiles format, working with it in bulk and so on, I agree with you, they're primitive.
My daydream is a tool like jq for recfiles, with something in front of it like pandoc for structured data, so that I can put my choice of JSON/TOML/YAML/TSV/CSV through the filter and work with it in recfile format.
The real prize would be a solid library for importing and exporting SQLite data in recfiles form. The mapping is obvious and the metadata would allow for preservation of more of the underlying schema than the standard outputs, which are CSV and a collection of literal SQLite commands which you can execute to insert the data.
I think pairing SQLite with a standard recfiles protocol would be exceptionally powerful, a workflow where SQLite reads a collection of recfiles into an in-memory database, slices and dices at leisure, and exports the changeset as patches over the recfiles? Yes please.
Sorry to butt in, but this is a fallacy that I see expressed a lot (and shares a lot of similarities to the "computers can execute improperly-typed programs" fallacy), and that has caused a non-trivial amount of harm.
>> If digital, use only plain text. It’s a standard format not owned by any company.
This is fine for anything where you don't care about the computer processing your data (e.g. a journal where you don't care about tagging or sorting by date or whatever), but it sounds like you do (based on "Based off of that structure I can do a bunch of cool things with simple scripts"), so:
"Plain text" is mutually exclusive with machine-parsing, because "plain text" is not a format. If I create a text file for each of my friends, and somewhere in each file is an English-language description of my friend's name, then that is both (a) plain-text and (b) not machine-parseable at all.
A "format" is necessarily structured - something like "each line of this file represents a "row", which is divided into "fields" by commas". But then you don't have a plain-text file - you have CSV, or JSON, or something that's a subset of "plain text file", but is still a structured, machine-parseable format.
Even if you do what I did and create your own ad-hoc format like "text files are broken into "blobs", where each blob is separated from others by a blank line, and has a header consisting of the blob name..." - that's still not plain text. You've still invented your own structured-object-embedded-in-text format - except that because you aren't using JSON or XML or CSV, you're incompatible with every tool in existence, and have to write tools to parse, lint, generate, and process these files yourself.
(and, hence, it's not a "standard format" - each ad-hoc plain-text-embedded format is different, except for the ones that are either accidentally exactly the same, or the ones that are standardized and have names - at which point plain-text purists claim that they're no longer plain-text)
You use Markdown - which means that you have all of the problems above, because Markdown encodes formatting, not structure. You even alluded to it yourself - you say that "...I can do a bunch of cool things with simple scripts" because you had to write those scripts to handle your own custom format, instead of just using JSON and writing data["first-name"] in Python.
It's your system, so you can do what you want - just don't fool yourself (or others) into believing that you're saving effort, because you're not - you're just reinventing a wheel that has been reinvented millions of times before.
(this isn't meant to be a random rant - I'm currently in the middle of rewriting my own system from using my own custom, ad-hoc structured text format to using s-expressions and typed objects. there's hundreds of thousands of objects spanning thousands of files, and the process is absolutely miserable - I want to save as many other people from having to go through this as I can)
I think "plain text" in most cases really just means that it can be read and written by a human in a basic text editor. It is not mutually exclusive because parsers can be written to handle more freeform text than just csv and markdown.
"Plain text" is like a carrier wave that can hold a format like CSV or markdown, or loosely conventioned prose that the right kind of parser can make reasonable guesses about.
Your semi structured blob format and mine are still plain text. They are comprised of the basic set of characters from 32 to 126 maybe and 13 and 10, and maybe multi byte things these days. Plain text just ain't what it used to be anymore!
They may not be standard formats, but they can be read and learned about in standard tools.
But your cautions still ring true, as I have created several such formats and have yet to hit on something that is easy to write, and somewhat easy to process, and ticks all the other boxes.
I see your point. But as long as you're consistent with the format for each value, I don't foresee an issue. I consider everything that has a readable representation plain text, but your definition may be different. You can go with a standard format, like JSON or YAML, but I think you can probably define something that is easier to write and still processable. I agree with you, but someone can likely get by with adding markers and being consistent with them.
I've been working on one of these on and off for a while now!
It'll eventually be up on gosayhito.com. I'll try to remind you when it's done, but you can also probably put the website in as a monthly thing to check on.
If anyone reading this builds software like this, and releases it publicly, whether free or for sale, please let me know. I'd be happy to send people your way.
Because of https://sive.rs/dbt I get a few emails a week from strangers, asking if my software is public yet, or if not, what else I would recommend. (And I might never make my software public. It's too tied-in to my now-complex PostgreSQL system of everything.)
I'm definitely going to send people to Jakob's post here now. But if interested in this subject, please email me here:
https://sive.rs/contact