I just released the first version of a project I’ve been working on solves a very specific problem that perhaps only I have. I welcome any and all feedback, even if you just want to drop in to say that this is a hot piece of garbage!
I am excited to see more solutions that work with Apple Notes database directly. This is my go-to application to capture unstructured memos while researching various topics. Last year I got carried away and built a simple site generator for Notes [0]. As a fun exercise I implemented ORM models to query Sqlite database in Ruby:
Apple Notes' database structure is a fascinating product story. Notes started as a simple note-taking application that stored its content as raw HTML in a few simple tables. However, as the product grew and needed more features like collaboration, they had to make changes and add new columns and tables. They also implemented two flavors of CRDT-like algorithms that evolved over time: one for plaintext chunks and the other one for embedded objects like tables. On top of that, every change that Apple makes must be backward compatible to support legacy devices and systems; but backward-compatibility was not always possible and users had to go through “Migrate your Notes” steps, but it is again a product story.
I wish that one day, someone from the development team would write an article about it.
Yeah, I'd like to know the story behind it. topotext is weird and unlike anything I've found in the literature. It is kind of like RGA, but with more links. And I haven't seen anyone else layer attributes on top of text.
It looked like the other stuff came later and was somewhat orthogonal. (Although their ordered set construct was abusing topotext to represent ordered lists of ids.)
I don't know how much has changed since 2018. I think my table decoding may be broken now.
Hey! I remember reading your blog post a few months ago, and it inspired me to try to create a CMS using Apple Notes + the IMAP syncing features[0]. But everything felt like a hack all the time. Many features are not supported when syncing over IMAP. I wanted to have access to the SQLite database, so I can't have a non-macOS VM syncing and processing notes continuously (or at least I couldn't find a tool that can do that). Parsing the data seems like a lot of work, I was also a bit worried that Apple could introduce a change in the format that would break the current tools. Linking between notes is not well-supported, etc.
At some point, I also had some issues syncing notes between different iCloud accounts when the folder had 100s of notes, and that was probably the tipping point. As I was struggling to make it work, I kept thinking that if I just had a bunch of Markdown files I could move around, the whole process would have been easier.
I eventually realized I didn't need most features of Apple Notes, and moved to Obsidian[1]. It feels quite snappy too, and now I can easily publish some of my notes. It can also use iCloud Drive for syncing between Apple devices. It's missing some collaborative features, but I also didn't need those.
I second your sentiment that it is a hack, and here be dragons. At the same time, I had an honest conversation with myself about the dilemma, and whether I am worried about changes to the database or not. I ended up answering that it is a low-risk factor especially for the project of that kind.
I am sure that Apple won’t be making any substation breaking changes to a database structure unless they have to, there is a price to pay. Such changes will always be tied to a major macOS/iPhone release, and it doesn't happen very often.
That said, I had to put this project on the back burner, it was not immediately clear what is the ROI especially considering there is a long tail of databases to parse. It starts with Mountain Lion and then adds up quickly.
I need to release my source code to Github--ORM approach was worth exploring, it is a concise declaration that is relatively easy to update and manage.
I’m trying to consolidate my note taking with Obsidian but I’m with you about Notes being a go-to tool… it’s just so much better integrated into all the apple devices, I can scribble quick notes with the pencil, ask Siri and all sorts of shortcuts can easily make or add to notes… but I’m absolutely sick of being held hostage by my software and for me my notes are too important not to protect. I’m glad to see anyone work on this, because I wasn’t relishing the idea of trying to force Apple to open up using the EU digital markets directive. Its my nuclear option if the situation gets worse, the various scripts and fixes stop working and I can’t work around it, because I’m pretty blown away by how complicated the notes format is and I won’t be surprised if they add something new and the format develops another layer of complexity.
Is this an alternative to needing Remarkable’s Connect service subscription?… Because needing Connect to sync notes off my tablet pretty much killed any desire to buy a remarkable 2 … and I voted with my wallet on it too… I bought a PineNote eink tablet instead (I hope this will eventually be a useful tablet… but I’m not holding much hope given my experience with various Linux distros on the first generation PineTab…)
I think this is a fair perspective on the one hand - on the other hand, as a rm2 owner i appreciate that the software keeps getting significant updates, and i think a subscription model makes a lot of sense.
I do whish the standard software made more concessions to third party software - ssh access is great, but running any kind of gui app feels hacky.
I thought it might based on a lot of things I saw back when I was still keen on a remarkable tablet. It’s still awesome to see people developing tools to funnel data into Obsidian. It’s great additional evidence I’ve the right long term choice with Obsidian for my notes. And who knows, maybe I’ll end up getting a Remarkable as a gift or change my mind, and I’ll definitely try to remember your tool if that happens… and I’d definitely pay for a tool that automates my Apple Notes to Obsidian… for what that additional data point is worth to your product planning.
It's possible to do it without their connect API for sure, but that would require users to
1. Look up their ssh password
2. Connect tablet to their computer
3. Run a (my) program on their computer to install custom sync software on the tablet
I'm not focusing on the technically capable ReMarkable users so I opted not to do this. Besides, it would involve a lot more development time which I don't have a lot of as I have a day-job as well.
For Mac - I use Noteplan. Obsidian has some amazing feature but it is soo complicated. I find Noteplan has a more simple user interface and most of the features. They can work together because they both use local markdown files.
I was toying with this idea, and the best (and also the worst) workaround I found is to share a note with yourself; then link a note with ⌘-k. I tried doing this a couple of times--nah, it is a workaround.
I didn't actually know AppleScript before writing this tool... but it turned out ChatGPT knew it well enough to unblock me and provide me with exactly the code I needed to build the rest of the project! https://til.simonwillison.net/gpt3/chatgpt-applescript
Apple Notes Liberator creates a copy of the SQLite database and then runs queries against that directly to extract the data.
I chose not to do that (despite being all-in on SQLite for everything) because I worry about future changes to the software baking my script - I figured the AppleScript interface was less likely to have breaking changes in the future.
> I've been stubbornly refusing to learn AppleScript for nearly twenty years at this point. Thanks to ChatGPT I can continue not to learn it, but I can now use it to solve problems
The Apple scripting stuff can also use JavaScript. It might be better to continue not to learn in JavaScript instead of AppleScript, if ChatGPT is as adept at Apple scripting in JavaScript as it seems to be in AppleScript.
That way if you want to tweak what ChatGPT gives you it will probably be easier. With AppleScript both interacting with other things and control flow or computation can be weird and confusing. With JavaScript at least the control flow and computation will likely be more normal.
I was trying to do a thing with AppleScript, and the barriers I ran into were not with the language, but with discovery of the APIs that Apple surfaces to the end user. In that regard, you’re better off with AppleScript. I can’t believe just how bad the AppleScript documentation is, but the js documentation is even worse; using js is just using AppleScript but with an additional layer of indirection that doesn’t provide any productivity gain.
There's an O'Reilly AppleScript book that's pretty good.
The problem I always had with AppleScript is that it wasn't clear when I was writing AppleScript and when I was talking to an application's OSA dictionary (its "terminology" as Apple calls it in OSA).
Once I switched to Appscript (https://appscript.sourceforge.io/), that confusion went away. Of course I haven't used that in years since it's no longer supported and today I'd use JavaScript.
The issue though is that each application's terminology is isn't really documented. You can usually figure it out using Script Editor, but not always.
IggleSniggle expresses it well. As ugly as the AppleScript expressions are, they force you to write in a way that's sympathetic to the API, so, in addition to returning errors at all in more cases, they're easier to diagnose or minimize.
I did a bit of AppleScript recently (scripted changing between draw and erase in zoom annotations from the buttons on my tablet pen). In my reading I got the impression that not everything is implemented in js so you might get stuck.
"Unblock" is the perfect term for the effect of all these GPT things on me. Thank you for coining that!
Recently I wanted to have something like that but for Safari Reading list. The main difference is I tried to find a solution not for one time liberation, but for continuous 2-way exchange with some web service. It appears you HAVE to run everything locally for that. I like "privacy", but I like having options even more.
By the way it was ChatGPT that helped me to remove this several month old item from my TODO list.
It is also based on Action Script commands. I haven't played with it much yet, but it's on my TODO list (to add automatic headings for every day + archive DONE tasks automatically).
Interesting. I was curious and wanted to achieve something similar with voice memos. I used the prompt "Write applescript to loop through all of my Apple Voice Memos" and the code it produced couldn't run in applescript.
I just tried that prompt against GPT-4 and got "Note that the Voice Memos app does not have an AppleScript dictionary, so the script will use UI Scripting to interact with the app" followed by a very complicated recipe.
Nice tool! Just tried it, works great.
Interesting that Applescript exports the images as Base64, or does Apple notes store them like that as well? Maybe converting them to binary when exporting would save some space.
A feature that a lot of Apple apps have that doesn't seem talked about much is they can use standard protocols and not require iCloud.
Mail.app/Reminders.app/Notes.app can use IMAP, Cal.app with CalDAV, and Contacts.app with CardDAV [1].
It doesn't offer a lot of the features iCloud has like advanced formatting or collaboration, but is a useful for simple standards based syncing and portability.
I've done this with my iPhone/Macs and Fastmail for the last few years and it works great [2].
Yes, the IMAP sync protocol was one of my discoveries when trying to turn Apple Notes into a CMS[0]. But like you said, many features are missing. I could do without the collaboration features, but I found that tables didn't work either. A lot of the unsupported features just disappeared from the IMAP-synced note, so you can't recover them later if you move the note back to an iCloud account.
Super interesting, particularly the link through to the blog with all the research into how Notes works.
A particularly interesting thing that I don't see mentioned anywhere is that Apple Notes uses CRDTs (conflict free replicated data types) to enable both shared notes and multiple concurrent edits. With the CRDTs resolving the conflicts.
A notes app that only works with one 2-trillion-dollar company's hardware and does not have builtin functionality to export notes (e.g. a menu option to do so) cannot reasonably be described as good. One's expectations have to be through the floor for that to be an honest appraisal.
Compare it to Obsidian, which just stores data in a bunch of markdown files on disk.
I don't actually care about getting some data dump out of Notes. I use it as a way to quickly store temporary information. Any note older than a month is most likely useless. The most important feature is that it effortlessly syncs between my phone, work laptop, and home laptop. And that it does. Unlike a bunch of markdown files on disk where you have to create some janky syncing setup yourself which has no conflict resolution system.
No goal posts move when the original issue is not resolved with the proposed solution
Like, without any menu you can export your notes to a screenshot, but you wouldn't accept it as an export option, would you?
Similarly, PDF is an awful format for notes, so the fact that it exists doesn't solve the issue that you can't export your notes (which at a minimum you need to be able to do in batches, not individual notes)
Most users interpret "Export" as an operation that lossily finalizes a project from its native representation to a common third-party format. "Export" is one-way, "Save As" is bidirectional.
- Audacity "Saves" audio projects to its native .aup, but can "Export" them to .mp3 or .wav
- Photoshop "Saves" .psd files, but "Exports" them to .jpeg
- Gimp "Saves" an .xcf, but "Exports" to .png
- Apple Notes "Saves" notes to its own sqlite database, but "Exports" to txt or pdf via a menu item
This technically satisfies your comment above, which is why we speak of moved goalposts.
It would be great if apple notes could convert your note library to ENEX or a .zip file of HTML items.
This is not a project, and your examples are just the same biased multimedia project example, which is very much not rich text
Export/import is about comm/direction, it does not (technically, just as you like it) imply loss. Also, technically we're discussing a comment from a specific person here, not from "most users".
(Joplin also uses sqlite database for storage, that's another very interesting internal detail)
So even technically you're wrong, but more importantly, you're missing the forest for the technicality tree
Thank you for posting about CRDTs! I remembered Apple Notes used some type of versioning that was good and I’d wanted to use in my own app design but I had forgotten the name.
I have published an app [Exporter](https://apps.apple.com/us/app/exporter/id1099120373?mt=12) that does something similar - exports Apple Notes to HTML and markdown with the attachments. Give it a try, let me know if it solves your issue.
Also, why did you select JSON as the output format?
Thanks for making your app. I’ve used it more than once and it works well. Many export solutions don’t handle attachments but yours does. So kudos for that.
A few years ago I tried to write my own exporter using AppleScript but got stuck by a showstopping bug in Notes AppleScript implementation. Or so it seemed. I hope that bug is fixed now. I just moved on to use your app instead.
I used AppleScript for the first two years of this app as well (Can't believe it's been 7 years since I started it). But it's hard to handle complex programs, and handing off data to Swift or Objective-C from AppleScript. So I switched to [ScriptingBridge](https://developer.apple.com/documentation/scriptingbridge). It provides better Objective-C APIs to handle Apple Events.
This is so cool and way cooler than my tool!! I chose JSON because it was the easiest option. And also because it is the format I wanted to support first, but mostly because it was the easiest.
I consistently struggle with Notes and iMessage sync issues, and I suspect it’s due to being a customer for so many years and making my way through many devices over those years.
The UX of the Apple ecosystem is so nice but when it goes sideways you’re kind of out of luck, even if you have applecare.
I wish they had some sort of premium “fix my esoteric appleID account issues” service you could pay them for once a decade or so.
I agree- long standing issues that have popped up and are never fixed over the years:
1) My iPhone has never properly synced screen time with my other devices (presently my other devices: 2x iPads, MBP) sync fine together. This has persisted across multiple iPhones and major releases of iOS.
2) Apple News has a permanently saved “story” of a specific magazine issue’s table of contents. No idea how it got saved, but since it isn’t actually a story (it’s a ToC instead) the UI option to “unsave” it is greyed out / unavailable. I reported it to a CSR about four years ago, he took down bug reports, I even sent a screen capture demonstrating the bug… it still persists and Apple provides no mechanism to manually clear all saved story data. I can remove it locally from the device from disabling iCloud sync for Apple News -> selecting yes when prompted to remove local data, but it will still persist in iCloud and there is no way to wipe it there… it’s back as soon as I reenable sync.
I had a sync related issue with Messages.app on macOS recently. Escalated through apple support. The solution was to reinstall macOS from scratch. Not pretty but a small price to pay.
Now my lingering issue is with search: Calendar events and Reminders prior to the reinstall date do not appear in search results. This has been an issue for years on both macOS and iOS! Much Google searching was no help.
I wish that premium fix service included useful forums and…search.
Love this. I am an Apple Notes absolutist (see: https://pbs.twimg.com/media/FY1rZvNWIAAIRzs.jpg). I have tried Evernote, Workflowy, Dynalist, Notion over the years for marrying personal / professional worlds but always (always!!) come back to Notes.
I have dreamed of the ability to have a Notes web app with custom CSS for years and a few other features.
Good luck with this project I'll be watching closely
I'm curious — as someone who's had a hard time with Notes' maybe... lack of features and less-than-friendly formatting IMHO (no Markdown, unconventional keyboard shortcuts, layout formatting quirks...) — what about Apple Notes appeals to you more than other popular tools? I feel I may be missing something :)
Just works: fast to load, works offline, I don’t have to think about sync
Just notes: It’s not trying to be a programmable networked knowledge engine or a no code app builder or a toaster.
Not ugly: You can technically change fonts and crap but it uses clean system styles out of the box, so even as a font snob I don’t get distracted changing formatting instead of writing.
Free, or at least really cheap if you have a lot of stuff in iCloud
I don't understand what value this tool adds over the library it uses under the hood (which includes a command line tool for exporting your notes; I've been using it for years). What is it for?
Literally 0 added value here over the other library. In fact, my tool is a net value subtraction from the other library. The other library has way more features and works great! This was mostly a project for my own learning of how Apple Notes stores its data. I honestly didn't expect it do get any attention!
Oh neat. I’ve been using Notes extensively since the beginning of this year. Daily journaling. Personal TODO lists, grocery list (shared w my wife), and gym/workout tracking.
It works really well and has way more features than people realize. If you’re looking for like a “Notion-lite”, the answer might already be on your phone/MacBook.
As a windows user, while we don't have an OS built-in equivalent, the standard for such tools (in the form of Sublime and the like) is just to operate with temporary buffers that get written out as files on user request. Sublime will write internal data to some persistent storage somewhere and be able to survive sudden crashes and restarts but when you'd like to save the data out to a fixed filepath it's pretty trivial to do so. I do find it quite weird that Apple has managed to cultivate a system where storage in separate files isn't taken as the default.
I am excited to see more ways to export data out of Apple Notes. Long term it would be amazing to see a solution that allows more interoperability with other apps.
Depending on what you are looking to achieve there are a few ways to export Apple Notes to Markdown files, but it's only a one-way out solution.
I have been adopting Apple Notes more after trying to use a lot of other note methods/apps. Always fall back to simple text in Notes. And I seem to be hearing of more people doing the same.
I recently set up a shortcut that works on both my Mac and iPhone to create a new note each day with the date as the title. And then each time I trigger the shortcut that day, it enters the time, location and weather. Easy way to keep a running "diary."
Very nice reverse engineering work! I wrote a python library and CLI that does something similar but interacts with Notes via Scripting Bridge: https://github.com/RhetTbull/macnotesapp
But instead of pulling the data out of Notes, I took it from SMTP mail.
I don't know if it still works this way, but when I did it, if you used a non-iCloud email address to sync Notes, it saved the data as pain text in email messages that could be easily parsed.
You should absolutely just do this. 100% the better option. I wanted to learn more about how Apple Notes stores its data and see how I could interact with it.
Awesome initiative!
I wish Apple Notes would support Markdown.
This is my go to app for so much of my life, à Notion style editor would be an absolute game changer.
I'm the author of python library & command line tool, https://github.com/RhetTbull/macnotesapp, that will allow you to use Markdown to create (but not yet edit) notes from the Terminal. I use Vim + markdown to create notes all the time: `notes add -m -e` adds a new note using Markdown by opening the default editor.
On my system it is located at /Users/YOUR_USER/Library/Group Containers/group.com.apple.notes/NoteStore.sqlite. I have a relatively older mac osx version though and an ancient 2013 MBP. Could be that the file location has changed in the meantime.
I wish that one day, someone from the development team would write an article about it.
[0] https://devlog.notespub.com/2022/08/site-generator-for-apple...