- Do NOT migrate to new tools, even if they are really shiny and popular - this may end up being a big hurdle you'll regret down the line (looking at you, Notion, Todoist, Google Keep, the list goes on). Just stick to your tools for at least a couple of years before exploring alternatives.
- Try search instead of organisation. I used to have elaborate folder structures for everything... but have since given up, still keep some basic structure, but I find most things via search anyway.
- Help your non-technical friends and family. You may have discovered the best clouds, NAS solutions, software, methodology... but they still use post its for passwords and keep family recipes in that one notebook in the cupboard. Guide them, share your subscriptions with them, give them pointers.
- Use fewer services, even if they are not the best tools for the job. I find it much easier to reason about a handful of storage media (and their backups, pricing etc.) than having an app for every single activity in my life (e.g. I stopped using apps for recipes, I use my synchronised note taking app instead. Same for shopping lists, use my todo app. Etc.)
I've been taking notes for 2 years, zettelkasten-style, and 90% of them are just dumped in the same directory, without links or tags. If I'm looking for something and remember that I might have that in my notes, I just search for it.
That also implies that, when writing a note, I sometimes add a line with a few related keywords and synonyms.
I spent some time to build my own web-based markdown editor, and was using minisearch[0] to index and do fuzzy search on my notes. I've recently switched to Obsidian and plan to make a similar extension in the next weeks.
If you're on a Linux system and the notes are just text files a
grep -rn . -e 'search term'
is probably sufficient. Use
-rnw
If you're looking for whole words.
Maybe leave your friends and family alone - sure, it's fine to mention a product if the topic comes up but I find one of the things humans tend to want help with the least it's organization - everyone has their own system.
I agree. I rarely ever use new tools, but I switched to Notion a few years ago and even recommended it to many friends and colleagues.
Big mistake. Lessons learned. It's been slow and buggy all this time but I still kept using it. The final nail in the coffin was it started using 15-20% of a CPU core on IDLE, and has been doing so for 5+ months. Tried everything and gave up. Going back to simple text files.
Completely agree here. For better or worse (well, definitely worse), I've noticed that some part of my brain that loves list-making and re-organizing. This leads me to an impulse to be in a never-ending process of migrating from one tool to another, wind at my back with a new motivated epiphany about how to re-organize everything.
Of course, when I say it out loud, it's nonsense, but moment to moment it doesn't feel like that's what I'm doing.
I really love the wiki-style organization but I don't want to depend on a tool with features stuck to that tool. I want something universal, like text.
Right now I'm using Simplenote and its killer linked notes feature, but I fear perhaps I've got a bit of lock-in there too. I at least believe in the ability to easily backup and migrate out of Simplenote, I think it's kinda portable.
But I really think simple principles, like search don't sort, may cut through the unnecessary complexity that my brain loves to produce and administer.
If you want to stick with text, I can wholeheartedly recommend Obsidian. It's a bunch of Markdown files on disk and Obsidian does a great job of layering organization features on top of that. (I personally also pay for their Sync service, but you can sync to other devices using other cloud services — they're just Markdown files on disk, after all!)
I guess in truth I have a handful of "requirements" - text based, able to avoid lock-in, able to sync, and cross-platform availability. Looks like it checks all the boxes.
Your last point is a big one for me. Although I did not personally code the tools I use for 'PKM', they don't depend on any service provider remaining in existence in order to function. Not even my ISP.
> ... even if they are not the best tools for the job
A tiny bit of me - just a tiny bit - thinks: what's the point?
Or - maybe it's more like: what percentage of this activity is actually about organisation, and what percentage is about having a fun time using the organisational tools?
I'm not belittling it, at all. My knowledge organisation journey has gone through similar (not quite so well-thought out) paths: Evernote, docs, Keep, finally now on Obsidian with the knowledge that MD is just ...MD...
But - if I'm honest the "what's your PKM suite" conversation is such a well-oiled meme on HN that I can't help but wonder how much of this journey is actually about the journey, the fun of setting up a new set of tools, the enjoyment us nerds get out of coming up with new structures, new ways of approaching things, groovy metadata schemas, tagging, search....
Ultimately, does it make us quicker at finding or using or remembering stuff? I'm not sure. I know plenty of people who put literally no thought into any of this and can still lay their hands on that bill from 1986, or pluck a fact out of thin air.
As I say, this question shouldn't stop us from trying, but it's interesting to reflect on the little frisson of pleasure I know I get (and I suspect everyone else does, too) when they implement The Next New Thing and realise THIS IS IT.... for the next 6 months... :-)
I've been down the route of "save all the things" and I mostly veered away from it - The vast majority of that content is never revisited again, at least for me, and the comfort of having it is outweighed by the time/energy it takes to save.
I rely on my memory, and on my email search. I keep an occasional note in Bookstack, and some photos/videos in Seafile & Jellyfin.
Everything else gets deleted, at least once a year, when I wipe and install a fresh OS. My total storage requirements are under 100Gb (under 10 if we don't count photos/videos), and I can fit it all on a usb drive easily.
I don't really feel like I'm missing out.
No judgement for the data-hoarders. I've played that game and it's fun (and a great learning experience most of the time), but eventually I had kids and it just felt like a complete waste of time vs actually spending time with the people in my life.
There’s a lot about this I can relate to (mind mapping, 20+ years of notes on a wiki at https://taoofmac.com, hierarchical, date-based storage of photos, building a personal CRM, etc.) but I have to question how much of it all is actually _useful_ and not, well, just packrat instincts that add so much background noise it dilutes any ability to actually find things.
search technology, in particular, seems to have failed us. Searching the Internet and our personal computers peaked sometime in the mid-2010s and has become steadily worse (witness Spotlight and it’s Linux and Windows counterparts). Contextual and topic searching has become “premium”. Finding an e-mail in your native language about taxes in 2016 would be impossible if, well, it wasn’t in “Personal/Finance/2016” — because there is just no way I can tell Siri (for instance) to find it for me, even if it was written in English.
And that’s the _easy_ part. Photos are a problem, too. I have an offline copy of everything filed by date, but have tried iCloud, Google, OneDrive, etc. Tagging people in photos is a challenge, and having weird, creepy “memories” reminders from any of those services popping up unprompted (and often mis-matched) makes me think nobody cracked that yet.
I do believe that we can do better than plaintext notes, but (at least so far) Zettelkasten and Obsidian aren’t it, either (I don’t get why Obsidian doesn’t leverage standard front matter, for instance, or why folder names can’t really be used as namespaces).
I'm actually also halfway writing one! Two remarks:
- It's really interesting to see strategies, but the overarching themes are often mixed up in a subjective soup. All sentences begin with "I" and this is understandable, but it's harder to grasp this angle. More constructive, one could separate better known general topics like "personal media/memory keeping", "mind/thought mapping/notes", "data hoarding/sorting (for general media)".
- On the other hand I'd also love to read a research/analysis on such articles. What are the general ways on how users approach digital data management, how do people juggle between these topics above? I wonder if a holistic view on this is possible, without the subjective decisions and narrativizing elements. I feel there is a niche between best practices (strategy given a specific scenarios) and personal stories (how I distilled it to my life).
Great points, there's indeed a lot of subjectivity in this one. I decided to keep it that way because I cared as much about writing down my story (and diving back into my memories) as explaining a bit how I managed to keep my sanity (to some extent ).
I intend to write more articles on the subject, diving deeper into each topic.
I love this! I'm exactly like the author except a little less organized.
Some of my insights just to add to this:
1) I started curating existing collections to delete/cut down size. So I have a tv shows folder and I made a _cut folder and moved every show that I wasn't sure if I liked or if it was something I wanted to keep. This left the main folder with only my _absolute favorite_ shows.
2) I use sqlite for movie/tv/music/game/bookmarks databases and i try to combine data from different sources so like I combine imdb and boxofficemojo data in one sqlite db so that I know how many theaters every movie opened in and the rest of the info like actors/director/etc.
3) I make websites with lists and/or "collections" of the movies/things I like the most, and my personal stuff like code. That way I have to select and curate all of the items so that they are organized and accessible. Making a website is like publishing so can't publish drafts or half-finished stuff. Making a website for something really helps finishing the organization and collating all the disparate info.
4) Scraping info for all new movies/games/tv shows into the sqlite db's. Website for searching/browsing based on genre/whatever, and having the info locally (no third party).
5) Only publishing working code without any cruft to github. Makes it so I know every single commit in the commit history works and there is no need to organize everything since it was already organized and "fixed" when commiting. Also is essentially ALL code that I care to keep (I have a lot more "work in progress" code locally but that doesn't matter, only the published finished code matters), and of course local clone of all the repos.
6) I have a text file with all my command line one liners, tweaks and tricks for linux and/or website things but this is the one area where I am less organized. I don't externalize all the things I "know" as much, but I want to. I use my own memory for most things like that but would be nice to have a knowledge graph of some kind - looking into Obsidian.
Thanks for sharing your approach, very interesting! You went way farther with the movie/TV/music/game/bookmarks management, that's super cool.
At some point I went batshit crazy with automation and actually automated scraping, downloading of episodes, tracking of what I saw and did not. But over time I just let it all go because I didn't have the buffer to maintain all that anymore.
I omitted various things because I got tired; the article was way too long already
But I do know that saving everything is just piling up problems for your children. My Dad died 7 years ago, and left behind an astounding amount of data and programs on CD, VHS, hard drives etc. My two brothers, who were his executors (I was not doing well mentally at the time), looked at each other and said: "Shall we go through all this?", "No". And it went on a bonfire - probably breaking all sorts of local polution statutes.
Realistically, the amount of useful information you need to store can probably be put on to one USB stick, or less (is there anything less these days?).
There is a solution between these extremes, which is:
Store all the data you want, but make a document stating clearly what is worth keeping after you pass on, and how to access it. Then just update the doc once a year.
IANAL, but this is standard advice a lawyer will give you when drafting a will/estate plan.
Interesting take. I did not think about the burden I'd leave behind.
TBH this is easily solved by documenting what matters/is valuable and what isn't.
But yeah, good point. Truth be told, most of the stuff I've collected really only matters to me. Most bits and bytes can be destroyed as soon as I'll be gone :p
Frenchie here. Digital natives is a word we were always taught is en-US. Fun to read that you doesn't know it as a international user. Typical french startup bullshit ? This depict children born in 80s, 90s and supposedly comfortable with computer work and data as they were born amongst them.
As an en-US person, I can say confidently we do not use it in daily speech. I can intuit what it means, but I've never been taught it or been in a conversation where it was used in any way that stood out enough to create a memory.
The term "digital natives" has been around for many years.
I don't know the official definition of the term, but I've always thought it was someone born in 1995 or later, where they never lived in a world without online services like AOL, and later Google, Facebook, etc.... Where they never lived in a world that existed before the World Wide Web. Where they never lived in a world without being able to access data online through something at least as fast as a 56k baud modem. Where they never lived in a world without cellphones and portable home phones.
They may not have had all these things themselves, but they knew people who did.
And this future is also unevenly distributed, of course.
In Germany the term is used relatively often. At least in the circles I frequent.
Especially in the context of who the target audience is (in marketing, when thinking about UX and so on - the prototypical agency/consultants/client talk intermixed with a lot of BS and important sounding words).
This is HN, a safe space. We are all web maniacs. That's how we got here. Sure there are a few suits hanging around with ironic hoodies and smart looking eyewear but mostly just us nerds. You are among friends here, let your phreak flag fly (it is -p or --phreak BTW)
Nice reflection over personal data management. I can relate to several moments. I agree specially with one thing: that a good file naming convention is essential and is perhaps the most fundamental piece of data management in conjunction with a wisely chosen folder structure. ISO8601 is perhaps the most important standard in this regard, but I often used the compressed version YYYYMMDD (instead of YYYY-MM-DD), followed by _ and whatever descriptor is suitable.
But as others have already written I have overtime become less and less keen to keeping a lot of stuff. My family knows where the important documents are (say, for valuable inheritable stuff). Digital photos live in an external hard drive. Other things worth keeping for posterity like books, articles, letters, notes etc that may have intellectual, historical or emotional value for others can be kept in a less than 32 Gb external drive.
The rest is just ephemeral dust or electrons ... as perhaps we all are after all :-)
I used to be just like the author in terms of data collection and curation.
A traumatic experience took it all away from me, and changed me in many ways. I now view everything as ephemeral, and treat it accordingly. Mostly what I do is just write about things using plain text files, with some markup that allows recall of information as needed. Instead of snapping a bunch of vacation photos, I'll just write about it. If someone sends a postcard or letter, I can quote them in my journal, but there is no need to keep a scan or verbatim copy of the communication. Of course, requirements change based on circumstance, but in general, the written word is a great way to store information. It takes up virtually no space. I don't foresee ever requiring more than a 64GB thumb drive to back up the entirety of my personal data.
I was a long time user of one or more NAS boxes (first was a Synology DS101g+), and ran everything self hosted up until 3-5 years ago.
These days i just throw everything in the cloud, using Cryptomator[1] for privacy where applicapable, and everything else lives on a Mac Mini at home with a couple of USB drives attached, whose main purpose is backing up the cloud data. It also acts as a local server for media files, but i don't "hoard" as many files to confuse Plex enough that it can't figure it out.
For photos, i use whatever format Apple Photos use. I keep 300k photos in there, and it serves it up happily, and allows me to search fast enough by dates/locations and to some extent content.
While i'm "too old" to buy into Spotify/Apple Music/Whatever, and still purchase all music i listen to, i do buy it from iTunes, and i let iTunes handle the organization. I simply backup the data (only music, as movies have DRM, so no point in backing it up)
For mail/calendar i also just use the standard MacOS/iOS mail client.
All of the above "works". It gives me a searchable interface to whatever i want to find, without me having to bother too much about how it's stored. Of course i organize files in folders/directories by content/purpose, but beyond that i don't do much sorting.
The only place where i still struggle to find a good workflow is for note taking. While i don't write many notes, i do take notes every day, but working from multiple locations, with different computers, and not being able to use the same cloud solutions across home and work networks, i find it difficult to find a "one true source" approach to digital note taking.
I'm currently using a RocketBook[2] for notetaking, which is essentially an erasable/reusable notebook that supports easily scanning and archiving the page thanks to a unique QR code on every page, along with a "checkbox" that specifies where the note should be filed (email, OneNote, EverNote, etc). It also does OCR, and supports sending my sometimes illegible handwriting as a PDF along with the OCR attempt.
Over time I also reduced my time investment in all this. As years pass, we realize more and more clearly that our time is limited :)
Regarding note taking and your issue with having a single source of truth, you might be interested in the ideas I've shared on Reddit a few days ago [1]
My current method is simply having multiple RocketBooks "scattered" across the places where i'm likely to use one. One at my work desk, one at my home office desk, and one in my computer bag for taking notes "on the go".
I much prefer taking notes with pen and paper, and the RocketBook elegantly solves the digital filing for me.
Now if only it would integrate into Apple Notes, i would be happy :) Again, like with the other tools i use, i don't need "bells and whistles", and will happily settle for a built in solution that is "good enough" as long as i can search it with ease.
Thanks for the Cryptomator recommendation I've been looking for something that can encrypt cloud backups more easily than uploading encrypted 7z files. Not sure whether it would work for your use case since it doesn't support handwritten notes, but I've been using Notion as a cross-platform notes app and found it works well
It supports A LOT of different cloud providers, and does proper versioned, deduplicated backups.
Another great looking (but not yet stable!) option would be Kopia (https://kopia.io/) which is open source, and does much of the same stuff as Arq, and even supports deduplication across multiple backup sources. Duplicacy (https://duplicacy.com/) does as well, but i've not used it much.
> "Between 1997 and 2000, I continued burning tons of CDs."
> "Over time, disks became larger and larger. Prices also dropped. The limits and constraints I had before slowly vanished. I stored even more data."
> "And again, I collected the data. I kept the movies, the TV shows, the documentaries. Everything. It was tough for me to just "let go"."
How does one approach composting knowledge, things that no longer serve, or weren't theirs in the first place?
Saying this as someone who's in love with the latest 'second-brain' tools and such yet goes through some old bookmarks and find myself sitting if one really needs it. Similarly just went through digitizing a chunk family history and while there's some important bits, there was plenty to either let go of or not try to organize.
The author later in the article...
> "I feel like I've come to terms with the idea that it's time for me to let go of the past. I don't pressure myself, but I'm getting rid of more and more things. I do this slowly. Thoughtfully. Not so much because I fear losing something important, but rather because I have fond memories, and I realize that it takes me time to accept deleting some things, even if I know I'll never need those again. While discussing with my friend André he mentioned emotional attachment to things and gently letting go of those, as Marie Kondo recommends. I gave it some thought, and while it's true that I'm not attached to many actual things in my life, I'm actually attached to many digital ones."
Composting knowledge, I like this. I find that with papers, books, audio CDs, video game cartridges, etc., it is easier to filter, to know what you have, and to get rid of what you don't use as you go.
In some ways, the default of a physical medium is loss, and you have to put in some non-zero, repeat effort to retain it. In contrast, the default of digital media is retention, sometimes even in spite of efforts against (think trying to remove data from social media). Every Apple Note, iCloud Photo, browsing history, bookmark, etc. will continue to persist unless you make an effort to get rid of it. So, the challenge for me, is not so much how do I retain all this digital data, but how do I ONLY retain the bits that are relevant, and how do I cull those bits occasionally?
If the author is reading this, please remove the fixed menu at the top. Eats a ton of space on a small screen and makes reading unpleasant.
This is more about data management than knowledge management. I'm the opposite of the author. I have little interest in archiving all my data. Not that there's anything wrong with it if that's your thing.
The menu is not fixed for me, it scrolls with the rest. It’s just permanently “expanded” as a vertical stack, but perhaps that’s intentional. (iOS latest, iPhone 8)
>I had clear naming conventions for everything. For movies: <Engligh name> (YYYY) (EN|FR|JP|...) (<Quality>). For TV series: <English name>\Sxy_<EN|FR|JP|...>. Consistency was essential for me to be able to stay sane. It also made it much simpler to automate various operations.
This made me cringe. Why renaming a proper warez release ? Back in the day, the scene was very organised and got pretty strict schemes and rules regarding naming release, and today it still allows to know exactly what you got.
>And again, as you can imagine, harmonizing the names was tough. And there was more. Sound also needed to be normalized. Fortunately, there were tools to help with that.
The guy clearly didn't have access to proper releases ;)
Interesting article, but a shame that the UX removes the scroll bars (consumed via iOS).
You can’t get an overview of how much content there is, hence how much more reading you’re in for…
Personal Knowledge Management has been one of the more obnoxious buzzwords I've run across lately - comes across as an overestimation of the value of one's holdings. It's just a filing system by another name.
Thanks for sharing!
For me it's
/music/genre/artist/year-mm-dd album name/index song name.tta
/video/genre/(/series/)name (year).mkv
/photos/category/DDDDMMYY-event name/DDMMYYHHMMSS_index.jpg
> Application error: a client-side exception has occurred (see the browser console for more information).
...and an site that won't load unless I block JS. :D This completes my craving for knowledge for today.
On the one hand, I find myself aligned with many ideas exposed here. On the other hand I fight myself constantly to avoid keeping too much around, or to prevent the archiving process from becoming more important than the archived material itself.
Unlike others I ended up consolidating everything into a big dumb set of folders on my NAS just because it’s cognitively easier to keep track of than a smattering of online services.
But then again… what is worth keeping, really? How often do we go back to old photos? How often do we need a 10y old bank statement? How hard is it to redownload some film or music?
The value is not in how often one revisits the past, but on how important (and I mean from a personal point of view mostly) a billionth of that stored past might become in the future. If you can, why not?
True, a for a good part it is that "maybe" that makes me (and others) do it, but I can see how it's akin to choosing not to make a choice. There's a tangible cost to this however… which is manifested quite clearly in this very comment thread.
All I am saying — to myself mostly — is that to learn to be OK with skipping a few beats when it comes to archiving digital artefacts is just as important as the archival strategy.
I get your point, in fact, I don't put a lot of effort in getting things organized to any degree of "good". I "invest" in archiving (I almost never wipe a memory card, a floppy disk, or a hard disk, I just store it. I do copy CDs into hard disks though). If I had, let's say, two months just to organize the data, I would, and to me, it would more than compensate the time doing it. Every time that I had to look for something I had done 10, 15, 20 years ago I find, in the process of searching for that specific thing so much gold, so much joy, that it paid off. But of course, I do not obsess with categorization. I have two axis: the "technological age" (dictated by the kind of storage device) and the way I categorized it then within that device. I think I have been more or less consistent with how I name folders, where I put things, etc. over a long time.
Knowledge Management will always be idiosyncratic to the organization and individuals inside of it. I used to think differently early career, but twenty years in content systems has changed my mind fundamentally.
Over the decades, there have been so many attempts to augur "inherent" underlying "types" of natural language, and all of them are, at best, moribund. Outside of those that have gotten regulatory buy-in, of course, but those too shall pass as the enabling technology (and its users) age and depart..
I definitely agree with "letting go of the past". Notes and memories are important, but so is managing what you keep. Otherwise it becomse untenable.
Anyway, I made a show HN today, but it's kind of relevant (https://news.ycombinator.com/item?id=30917504). I made a notes extension for taking notes quickly and in the moment, before you decide to keep it or that you're done with it.
>Reduce the number of tools you use for the same purpose to 1 if you can
This was the focus of Q1 for me. I had 2-3 apps doing the same thing so I had to make a choice and choose the most suitable and stick to that.
It's funny that this advice goes beyond PKM. Even programming tools, pick and stick to one. You can always explore later. Go with your use case but I digress.
I don't know about you, but if I don't use knowledge from 5 years ago, it is unlikely if I will ever use the information even again in my life time. How many of you are in the boat?
I can't agree with a heavy emphasis on text search and having a minimal or even no organizational structure, as several folks here have suggested. Text search and an organizational structure each have their own advantages and disadvantages, and they complement each other. This because extremely clear to me when I worked as a patent examiner. If an examiner did only text searches (or only patent classification searches, the equivalent of an organizational structure), their search would not be considered complete because it almost certainly would miss relevant documents.
In my own files I have an organizational structure. It's not perfect, but I usually can quickly find what I'm looking for with it by simply browsing. Finding what I'm looking for quickly is what matters. And I can find things that would be quite difficult to find with a text search, like documents that use inconsistent terminology. Yes, I have to put those files in an appropriate location, but that's not a lot of work in my experience when "onboarding" a file. Even when I add a bunch of symlinks to the file in other locations, it's still worthwhile in my experience.
Sometimes, however, I can't remember where something was filed, and I turn to text search (usually via grep or pdfgrep, though I'll be trying Recoll in the near future). In that case it's not uncommon at all to be unable to find what I'm looking for, even for my own notes where I'm trying to be more careful about using consistent terminology! I frequently don't remember the terminology I used, just like I might not remember where something was filed.
Text search is convenient as it often doesn't require any additional effort like organizing files does, but it has limitations that a good organizational structure would try to compensate for. However, text search should have additional effort to apply SEO techniques to help you find a file again. I will often add keywords to my own notes to help me later.
My organizational structure is not arbitrary, for instance, putting files into folders for each letter of the alphabet. Even if automated, that's fairly pointless because anyone can jump to a particular letter anyway. I organizing things based on actual scenarios.
> ...I will often add keywords to my own notes to help me later...
I do this too! I learned it from my web dev days (back in early 2000s), where adding keywords helped search engines, etc. Half the time i just list out some synonyms in a "reference" area of any document/text file that i am drafting...the other 50% of the time i actually try and use synyonms inline, in the context of my writing...this helps for SEO sure, but also for my own writing it keeps things fresh, and (though small gain) helps me improve the extent of my writing vocabulary. The result being that it helps generally with searchability for my own material later on, even years later.