Since Joplin's developer decided to put all Markdown files into a database and thus give up all the advantages of a flat file system, I have uninstalled Joplin. In my view, the last discussion in Joplin's forum on this topic further discredited the developer.
Thanks for bringing it up though, I went for spicy programmer drama but instead found a somewhat interesting and incredibly polite disagreement. I am curious about what happened that they have such a solid opinion about acceptable use of file metadata.
Not sure I agree. Here's one comment by the developer (laurent):
"Ok, there's so much wrong in what you write about that it's hard to know where to start. I guess it would help if you had a clue about what you're talking about."
I'm not proud of that answer but to be fair there are also more balanced views in this thread. It's also an old thread and some of what's in there is no longer relevant. For example someone recently created a prototype of the app that saves the notes to disk, and although we can't merge his work as it is, we plan to investigate and see whether it can be somehow integrated.
It's one of these features that, all things being equal, we'd rather have than not have, but it's obviously not trivial to add it when the app wasn't originally designed for it.
I completely understand. It wasn't my intention to blow this out of proportion. It's a single expression of frustration that every developer can relate to.
Sure - but it is a departure in tone and sandwiched by more normal and productive comments (from that dev). It isnt ideal but it is also a pretty normal and minor human mistake (growing frustrated and then short to be specific).
And the previous comments were claiming that this project's data was 'proprietary' and not 'open' and the situation was inexcusable, basically because they assumed a database was this big scary thing (as opposed to an open book). That is a frustrating experience, and yeah maybe indicates that the person telling you how to write software doesn't know what they are talking about (for the folks following along at home, those frivolous cries of foul play were distracting from the real need they had for plain text over a database, someone else describes it quite succinctly a little later)
Sure, it's no big deal and it's completely understandable. I just wouldn't characterise it as "incredibly polite".
I think file system vs database storage is very much about trade-offs. Unfortunately the debate descends too quickly into who is right and who is wrong.
It's always difficult for developers to talk to users who often really don't know what they are talking about. I know that very well.
The only thing you can do as a developer (if you have time and patience) is to translate what users say into the closest thing that does make sense in terms of trade-offs.
I applause Laurent's answer.
I'm so tired of self-entitled users that don't contribute anything and have no skin in the game and just complain "this should be done as I say because I'm smart". Yes, obviously sometimes maintainer can get something bad and a better solution is easily available, but as app complexity goes up, it's unfair (to not say naive or a worse word) for users assume that developers are clueless and hadn't thought about the problem space before taking the design decision.
Thanks for bringing this one up. It's almost strange to see a civilized discussion about something that clearly polarizes a lot of people in the thread. Makes me wonder exactly what kind of social dynamic results in a bunch of people actually talking through their differences, instead of the these days usual trench warfare...
I loved the TinaCMS blog post on how they still use flat files as their source of truth, but have a level.js database that powers a bunch of their functionality. https://tina.io/blog/Tina-CMS--Leveljs/
You can blow up the db & regenerate it very quickly. All the data is still flat files. But the advanced functions harvest & are derived from the source of truth. Seems like a perfect compromise.
I haven't done it yet but I have a git-auto-cimmit project I intend to use with logseq, that'll help me better keep a record of changes over time. That ability to have a flat file source of truth feels hugely hugely empowering to me, and I have a hard time imagining having anything else.
You can blow one database (level), but not the second one (git) which you also need to sync, and this doesn't cover the issues of changing the data layout, git isn't always great in tracking those (doesn't store renames). Then also all your data is duplicated in a git db and a flat file db
I'm not sure what you are talking about vis a vie "changing the data layout". You can do whatever you want, it's just flat files. Git is great at tracking file renames.
Trying to say data is duplicated is basically wrong, as far as I'm concerned. There's a materialized view, which very few people I think would argue breaks the idea of data-nornamization. There is only one data source, the other is a generated view which can be regarded as ephemeral data.
I sympathize a lot with the parents perspective that Joplin switching to database backed world is a pain. I think the appeal of well managed flat files, controlled by git, is much more directly manageable & has much broader tooling. I appreciate attempts to raise dissent against this system, but none of your points carry much water for me personally. It strikes me as pretty weird having these litany of basically wrong non-issues to swat away?
But you don't need two, one in git (which stores the full copy) and another one in flat files when you can have only one in the database and then store just the diffs (and have an easier way to discard history when needed vs git, which would be especially painful with many clients)
Git uses heuristics to identify renames, and not a great one (and also limited in number of files), which is a known limitation of this "great" tracking
And data structure means renaming/moving your flat assets, even if git catches them, your note app will not. Or breaking cross-references due to that. Ability to regenerate broken wouldn't help you here
What "broader tooling" do you mean for notes when you don't even have a real system, just something all you could imagine as the best?
Also wild to ignore smartphones as yet another non-issue
Feels exceeding weird to me to draw a distinction between git and the flat files it has.
Git does come with some challenges (deleting a file from history is a good one; the filter-branch command for doing so can be a beast!), but overall there's so many git tools available. Gitlense, GitHub, magit, gitk, and hundreds more ("broader tooling"). And is imminently google-able. Love it or hate it, it's a well known system, and TinaCMS using git seems like a huge advantage to me (with my reasonably competent modern developer hat on) in any circumstance.
Tools like logseq and TinaCMS have ways to rename pages, without breaking references in their markdown. Seems solved.
Smartphones are pretty good at using web apps I've found so not sure why you once again are leaping to attack again.
"you can have only one in the database and then store just the diffs" has big "draw the rest of the owl" vibes to me. Maybe it is really easy. But I have to learn that system & figure out how to work with that history. I see very little benefit in a simple system, that is custom tailored to one particular use, that will have a range of limitations I can't assess or judge well ahead of time. We have a great tool for handling control of different versions of text over time. I'm here for it. It seems wild to me to imagine DIY'ing for your specific app.
But I don't care about "overall", I've asked you about note-specific workflow, and "hundreds" of bad/irrelevant tools hurts googlability, not helps
> "It seems solved"
Care to share specifics? You've changed file names/folder structure/image file name/format etc. in your favorite flat file tools
How does a system reconcile those edits happening outside its garden?
> Good at using web apps
A single good example for this domain would be nice. Say, you have your flat files in Dropbox with .git? Which web apps could I use?
> "draw the rest of owl" vibes
More like "imagine someone can draw it" just like you had an image of the owl in this meme in your mind.
By the way, if you plan to use autocommit, so basically have just a linear series of diffs added automatically, what does git buy you?
> but I have to learn
Well, yeah, doing something better requires learning how to do better
Well, if what you need is a flat file system, the json-as-database approach that penx is taking isn't really an improvement.
I'm interested in what advantages you have lost. Joplin allows you to export your entire notebook as a directory of markdown files (with front matter), and from a performance, reliability, and replication perspective, a database seems like a much better choice.
I won a lot and didn't lose anything. Obsidian plus a few plugins beats Joplin on every level. And I can edit all notes without redundancy by exporting from a Joplin database with nvim, iA Writer, find, grep, cat and millions of other tools.
> Obsidian plus a few plugins beats Joplin on every level.
Not really. Obsidian has its shares of problems too, and most of them originate from using Markdown. Markdown is a freeform text-format, and works very well for writing text, but it really sucks for data and structured content. Most plugins and features in that area are very brittle and overspecialized, working only well enough in their specific use case. And gosh, Obsidian has really a huge amount of plugins for data-handling. At some point, it was so bad that there were multiple competing task-plugins which broke each other just because they had different formatting for dates.
> And gosh, Obsidian has really a huge amount of plugins for data-handling. At some point, it was so bad that there were multiple competing task-plugins which broke each other just because they had different formatting for dates.
Depends on how you see it. I think it's great that they have a free market of open-sourced plug-ins that you can donate to. I'm actually impressed of how rarely they break each other.
> Not really. Obsidian has its shares of problems too, and most of them originate from using Markdown.
Aha. Which problems do you mean?
> Markdown is a freeform text-format, and works very well for writing text, but it really sucks for data and structured content.
Joplin is using md to. And if Joplin does a good job on "data" and "structured content" (whatever you mean by that) by separating that in their DB, it's a big NO for me since it's a closed silo.
> Most plugins and features in that area are very brittle and overspecialized, working only well enough in their specific use case.
Aha. I don't think so. Which authority says that? And even if It's like that, my markdown files would survive everything, since they are a) in git. https://github.com/denolehov/obsidian-git and b) easy to fix since it's a text file. Gosh!
> And gosh, Obsidian has really a huge amount of plugins for data-handling.
And gosh, this is a good thing!
> At some point, it was so bad that there were multiple competing task-plugins which broke each other just because they had different formatting for dates.
Installing multiple task plugins shows that something is "broke" on the user side. It's not the fault of Markdown or Obsidian.
The way it's handled can make the difference in control.
> by separating that in their DB, it's a big NO for me since it's a closed silo.
Joplin is using a popular open database with a healthy community and good tooling. It's as open as markdown. Maybe not for you, when you lack the knowledge, but markdown is similar closed for anyone not understanding filesystems and editors.
Good for you, but that is very low level in terms of data-handling. Dataview is really just an elaborated search, there is no good level of interaction. Datacore, the next project of the Dataview is supposed to bring this, but it's not even usable yet AFAIK. Coincidental, the Obsidian-devs are also working on that front, but nothing is finished yet.
That's useless when the app itself is not working. And even worse if you are not realizing the errors early.
> Aha. I don't think so. Which authority says that?
My own experience. I've tested enough plugins over the years to know their dark corners.
> And even if It's like that, my markdown files would survive everything
The thing is, technically you are not even having proper markdown, but a fork with some extensions of Obsidian. So some features of your parts might break when switching away from Obsidian. And the reason for all this is also because markdown is lacking definitions for what obsidian-people are doing with it. Coincidentally, this seems also one of the reasons why Joplin is using a database.
> And gosh, this is a good thing!
Not if they all suck.
> Installing multiple task plugins shows that something is "broke" on the user side.
Sure, because the plugins are lacking features, its the users fault... Maybe some users have just very different levels of requirements from you.
