Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: How to organize personal knowledge?
403 points by _1tan 6 months ago | hide | past | web | favorite | 220 comments
In the last few years I have collected a large amount of ebooks, papers, articles, movies, pictures, etc. (all digital).

I am at a point where I would like to organize these items into a cohesive, searchable and maintainable system. I have looked at various methods (e.g. Niklas Luhmanns "Zettelkasten", Zotero, Org-Mode) and I am not satisfied yet. Any tips?

This is what I learned from designing data architectures:

If your data is variegated in format and form, don't look for a single tool/method solution to organize them. There is no single all-purpose organization method.

Don't start with figuring out a method to organize your information. You will end up overengineering your org method.

Instead, organize your information per "use-case". Start with a specific use-case/project (e.g. writing a blog or paper), and work backwards to figure out how to organize your data to meet the requirements of that project. Do a couple of iterations. After a couple of projects, and you will naturally discover your own data use patterns.

If you go with a super organization system on day 1, it will likely be too general and require too much effort (tagging, keywords, hierarchies, version controlled, branches, etc.) you will end up expending resources on metadata management on data that you may never ever need to retrieve and very soon you will abandon the effort.

My PKM system is very simple: a single unorganized Google Docs for quick thoughts and ideas (just bullet points), separate Google Docs files for specific projects, etc. and Dropbox for files. It's simple, searchable, and multi-device.

I also occasionally use some specialized tools like Jabref (BibTeX) for specific types of data like references, but I hardly ever write papers anymore, so these have fallen by the wayside.

I've tried wikis but due to their multipage nature, they segment knowledge too finely (often there are wiki pages hidden in deep in the link hierarchy that I forgot existed). Wikis don't fit the PKM use case that well, so these too have fallen by the wayside for me.

For me, PKMs need to in some way feel like a single broadsheet where I can easily see and touch my information without having to drill-down hierarchies and follow too many links.

p.s. I've heard good things about Evernote. It's a little too heavy for me, but many people seem to find it useful.

Don't start with figuring out a method to organize your information. You will end up overengineering your org method.

I think this is an important point. I like to start with a few simple rules:

- To retrieve information, I should know where to start: a Schelling point.[0] For me, this is the home page of my wiki. For wenc, it's a Google Doc.

- It shouldn't take me more than three clicks to get from my starting point to the information I'm looking for.

- Links/URLs will tie everything together. They are the edges in my knowledge graph. But as wenc notes, keep the graph shallow.

Then I need to be rigorous, reorganizing things when they don't work intuitively and adding new nodes when something I need has not yet been recorded. As wenc puts it, "discover your own data use patterns."

Wikis do work for me, provided it's organized around Schelling points. I've used and refined these principles in setting up wikis at my last 3 companies and it's worked pretty well for organizing a collective knowledge base as well.

[0] https://en.wikipedia.org/wiki/Focal_point_(game_theory)

What software do you use for your wiki?

I use TiddlyWiki (https://tiddlywiki.com). It's brilliantly simple to set up (doesn't require a database for example), has a small but nice plugin ecosystem with things like Markdown support, etc.

I'm not currently using a personal wiki, but both MediaWiki and Confluence are pretty easy to set up. Confluence/Jira licenses are cheap for self-hosted personal use. MediaWiki is oss and probably not going anywhere anytime soon.

I second Mediawiki. I prefer that for professional use:


Personally, I've been using WikkaWiki for years:


I would prefer something that supports markdown. Neither of these really do.

Dokuwiki is what I use. Easy to setup, easy to use. And with Dokuwiki on a stick there is a zero-installation, portable local option.

> It shouldn't take me more than three clicks to get from my starting point to the information I'm looking for.

I see this sort of criteria and really don't agree. There are many operations that take more than three clicks. Navigation a UI is a graph in itself, and we can handle way more than 3 nodes. In the same way that I can remember how to get to work, or get a book from one of my bookshelves.

Anyway, small point.

Trivial inconveniences.


There was a joke in one of Stephen Hawking's book that each equation included in the text cuts the number of readers by half. In a similar fashion, you can imagine that each extra step you need to get to the information will cut the number of occasions you do so by half. If you want the system to benefit you daily, in many areas, it has to be as simple and seamless as possible.

Fascinating observation, and could be a corrolary to Nudge theory [1], which won Richard Thaler his Nobel Prize this year, which says Opt-in rather than opt-out is more effective for changing behavior.

Underlying idea: friction disincentivizes.

[1] https://en.m.wikipedia.org/wiki/Nudge_theory

Yeah, sounds like the same thing.

I would not expect you could get a Nobel Prize for that, though. Maybe Scott Alexander should start submitting his articles wherever it is you need to submit them to get a Nobel in economics.

> In 2017, economist Richard Thaler was awarded the Nobel Memorial Prize in Economic Sciences for "his contributions to behavioral economics and his pioneering work in establishing that people are predictably irrational in ways that defy economic theory."

Proving that all the other economists are modeling things wrong is more impressive than you make it sound.

Having a simple system that works efficiently for each use-case is a good idea, just be careful that organizing your notes doesn't become a goal [0] itself. It is much better to just start using some system, and see where you end up, even if it turns out you made a bad choice for your initial system you can always adapt mid-way (e.g. tools like `pandoc` can convert between a variety of formats), focus on the content and linking, not on the tools themselves.

[0] http://blog.dilbert.com/2013/11/18/goals-vs-systems/

I think there actually is a way to organize everything (arbitrary knowledge) with a single tool/method, and am excited about it and want to make it work better for us all to organize knowledge individually and together. I have written about it extensively at http://onemodel.org (where the current take on the idea is available for free download, AGPL), though not all the features are there yet. There is some more info in a comment farther down on this page if you search for "onemodel". It is extremely efficient and effective (physically and mentally, once you learn it -- about everything needed is always on the screen): you can get around really fast.

I attest to the per use-case notion for organizing anything.

I'm fairly settled on a homebrew style of noguci's system including pen, folded paper in my back pocket, a few paper notebooks as well as markdown, google docs, simplenote (via notational velocity) and google keep.

Here's the post that inspired me towards this system - http://www.literatureandlatte.com/forum/viewtopic.php?p=1592...

However, my system is stunningly poor at filtering and modelling information into memory. I tend to frequently write, bookmark, capture things and leave it at that. That does not help. I have recently begun exploring concepts of spaced repetition to this effect but its too early to comment on this.

I have learnt to differentiate between archive and active areas. Screens and the internet while great at archival, just dont work well for active information that I want to recall at will.

Just wanted to point out that important difference between active and archive information. That link I posted above was from my archival system but remembering Noguci's system and that it applies in this case is from repetition over active information systems.

I endorse the idea that over-engineering is counterproductive. Focusing on principles over tools is good; tools come and go. My list of lessons learned (which has some overlap with yours) is:

1. Make sure your files and their organizational scheme can be transferred between tools. Your files need to be searchable, and at some point you will need portability. Text-based notes are easiest, but formats that have ubiquitous support (like docx or html) are ok too. Similarly, a collection of files is more reliable and flexible than a database tied to a subscription-based service.

2. Don't bother with elaborate tag or folder based taxonomies. Search is more efficient. I, at least, cannot accurately predict what taxonomy I'll need 5 years in the future, nor can I consistently tag every file correctly. There is some empirical support in favor of search over tagging [1].

3. Keep your notes organized; store your other files all over the place, wherever it is convenient. Use your notes to record context and thoughts relevant to a project/task/other activity, and use hyperlinks to connect those notes to the relevant files (wherever they are). Being able to embed or preview images in your notes is also very useful.

4. Organization is worthwhile for files that you have read/watched and thought about. The purpose of organization is to help you revisit and develop your thoughts on a topic. Organizing files you haven't read is not progress and does not improve your capabilities; you have to actually read and think about them to get any benefit.

5. Organization is also worthwhile for files that you know you will need for a specific project/task. Add a link to the relevant item in your notes for that project/task. Include a note why you thought it would be useful to future-you.

6. Files that you have not thought about / critiqued, and which do not fit a specific project/task, are not worth organizing. Dump them in an unorganized "to read later" list or just forget about them. The idea that you actually will get back to these files is mostly fictitious, and you can easily re-find them, or a superior equivalent, via search engines, browser history, or library logs.

7. Switch tools as needed based on circumstance. For example, my personal work notes are in org-mode, academic papers are in Zotero, personal work data is in a git + git-annex repo, and shared work notes and data are spread between Google Drive, Dropbox, and a private file server, depending on the requirements of the other people involved. I also have personal life notes in org-mode, Simplenote, and some ancient notes still to be migrated from Evernote. Hyperlinks tie files together as needed.

[1] https://people.ucsc.edu/~swhittak/papers/chi2011_refinding_e...

I agree that spending time on tagging/categorizing everything is not worthwhile, I tried doing that for my Firefox bookmarks/Zotero, and ended up abandoning it in favour of search. Having said that it is useful to have a limited number of temporary categories:

- things I want to (re)read next

- "best" / "mind-blowing" technologies, usually built around a simple, minimalistic, but powerful concept (e.g. reagents for composable lock-free data structures, parallel prefix sum, Futamura projections, etc)

For everything else full-text search in Zotero has served me well.

About portability between tools I would go with Pandoc, it supports conversion between a number of Wiki formats and markdown, which should suffice for most purposes. Definitely avoid tools that lock you in their proprietary formats.

>collected a large amount of ebooks, papers, articles, movies, pictures, etc. (all digital).

Since you mentioned those specific examples, I see a commonality and so I'm going to assume you're primarily concerned about organizing files that are not authored/generated by you. (Except possibly pictures/photos since you might be talking about jpgs from your personal iPhone or digital camera.)

This distinction is key because software like Org-Mode, Evernote, MS OneNote, etc is better for personally generated note taking -- like classroom lecture notes, or brainstorming, or personal todo lists. Those tools are not suitable for saving and managing a terabyte of movie files. However, they have one exception to the "being bad at managing collections of the world's data that I didn't write" -- they do have very good "web clipping" functionality. Instead of saving url bookmarks that might suffer digital rot, you can use the web clip tool. (Personally, I prefer to save important webpages as .mhtml archives. This doesn't work for javascript heavy sites but for the type of informative webpages I want to save for later reference, this usually isn't an issue.)

Likewise, the software that's good at cataloging a "library" of collected files with capabilities for custom tagging (e.g. Adobe Lightroom for photos, iTunes for music, Zotero for pdfs, etc) are the wrong tools for archiving personal notes.

For a disparate collection of digital files, the best method I've found is to leverage the native filesystem in Linux/Mac/Windows. Create a hierarchy of well-named folders to build a sensible taxonomy and put all your pdf, epub, mp4, jpg, mp3 in meaningful locations. There is no good universal organizing software for organizing all the file formats that has the longevity and transparency of the native file system.

> However, they have one exception to the "being bad at managing collections of the world's data that I didn't write" -- they do have very good "web clipping" functionality. Instead of saving url bookmarks that might suffer digital rot, you can use the web clip tool.

Let me warn you: I tried to use OneNote this way. The web clipper is really nice and I used it heavily to clip everything interesting on the web from my desktop and my phone until my OneNote file reached 8GB. Syncing just completely gave up and there is no way to export everything in a usable format.

Since then I use a separate Firefox 52.5.0 instance with the old ScrapBook plugin. I can send sites to it from a current Firefox instance and from mobile. It works ok and, in case this setup should stop working at some point in the future, I still have all sites sitting on my local drive.

Unfortunately the browser makers don't give a damn about saving and organizing Websites locally, although I'd believe that this would be far superior to Bookmarks. I love scraping my local library for useful stuff and I've been bitten by link rot far too often.

For personal notes and todo lists I prefer simple text files with a few categories. Lately I'm also giving Boostnote [1] a try, which simply looks a bit better and allows me to add checkboxes and proper headings/separators.

[1] https://boostnote.io/

Sometimes the pages can still be found on archive.org, archive.is, etc. but I usually find out too late that a page I bookmarked wasn't actually archived. I've just found this Firefox plugin [1] that automatically sends your bookmarks to archiving sites. (one at a time, so unfortunately this won't immediately help with existing bookmarks, unless you visit them all again).

[1] https://addons.mozilla.org/en-US/firefox/addon/archiveror/

I wrote a little script that I run as a daily cron job which fetches all URLs I bookmarked/saved in Pocket and submits it to archive.org.

I wish more people would do that as link rot is a very real thing, certainly for technical information often saved in gists or github repos that are often harder to find back.

While I agree leveraging the filesystem is the way to go in order to sort binary data (ebooks, papers, movies, etc), it's not true org-mode is not suitable to organize those.

Since it supports hyperlinks to a wide variety of data, and Emacs is so easy to script, you can definitely do that. For example, org-ref has a lot of functionality implemented to organize academic papers, including the corresponding PDFs.

In fact, a personal org-mode wiki could easily have one or more files devoted to personal pictures, books, papers and hyperlinks to those plus whatever metadata. There are some examples around if you google for that.

I do keep my personal wiki very close to a zettelkasten, but implemented in org-mode. Nothing special. Very simple. Small files that mimic cards, plus hyperlinks. I search using ripgrep, and I version control using git. A key feature is that I keep my task list and calendar inside my wiki too, but not version controlled.

How do you mimic the desk feature (lay out a few cars on the table) in org mode?

You can do this in many ways. With dired and several windows, or a view like org agenda that let's you explore several cards concurrently.

Apologies for the naive question, but how are the native filesystems for metadata storage, like say if you wanted to store Author for a video, or Location for a photo? Do they have preset available fields, or can they store an arbitrary number of key-value pairs?

Well, there are several tag formats, ie Id3v1, Id3v2, Vorbis comments, APE tags, which have have most needed metadata keys already part of the spec and many can add custom pairs.

Thanks! Just had a look, aren’t those all reliant on the file type? There’s no standardized metadata scheme built into filesystems?

> they do have very good "web clipping" functionality.

Are there services for this where you can link in your own backend?

Joplin supports nextcloud, dropbox, onedrive personal.

I'm not sure why noone in 150 comments recommended RDBMS yet, but it's great way to store information in general. I put a lot of personal things into RDBMS either imported from external sources, or added manually: invoices, double accounting, time tracking info/worklogs, bank account records, phone company records, passwords, email, irc/chat logs, things to download, things I bought in aliexpress incl. description and images, links I liked, scraped backups of my data from SaaS services I have to use...

It's replicated from SSD to HDD on the same computer immediately (I use PostgreSQL replication and run multiple servers locally), so if one of those fails I lose nothing.

If I ever want gui or cli tool it's easy to hack up electron app or PHP/node cli script in an hour or so. DB access is much easier than parsing text files. It's easy to extend (add columns/tables) without messing with some random txt format.

You can also much more easily answer questions about your personal data by just firing up a psql and writing a quick SQL query with a much better chance of finding an answer to wider range of questions without resorting to actual programming/scripting.

You have to write some schema, so that naturally brings some organization to your knowledge as you have to think about it.

I'd be interested in hearing more about the system you use.

Have you written anything about it before, I'm curious about more specific examples around what you're doing.

Do you automate most/some of what you put in?

I've thought about writing some scripts to export things from various online accounts/services/apis and importing them into a personal database mostly for backup purposes. It sounds like you're already doing that, and more.

I don't know where to start, so here are some random thoughts.

- Some of the data import is for archival purposes and mostly one-shot. I'm old enough to have seen many services where I put my data in, or where I had interesting conversations, go and delete everyhing in the process.

- Some of it is for ease of access and offline use. For example Jira can be a very slow system, where each action takes 10 seconds to complete. Some services have too bloated an UI to be useful on slow computers. You can easily lose access to some data/technical comments if permissions change, etc. If you have data locally, it's possible to avoid all those problems and present the data in any way you want.

Yes, I automate imports from external services. In the past I used PHP, now I mostly use Node or Electron, because it has a transparent support for binary data and JSON columns in postgresql, and if I want to parallelize HTTP requests it's easier to do in Node.

Import/sync scripts always have the same structure. First there's some way to gather the data from the external service, then I import it into DB with update/upsert/insert helpers, for example:

    upsert(db, 'table_name', {id: 123, name: 'qwe', ...}, ['id']);
I always try to find and use entity IDs from the external data source, which eases future syncing. Usually there's not much data so I fetch everything and update the database (keeping what was deleted by the remote data source). Often times the service supports queries that result in entities that changed since some date, which is ideal for incremental syncing. If not, the service usualy has at least a way to order entities by date, so I sync new entries until I start hitting entries that I already have in the database. It's very useful to use async generators for this in javascript, so that I can separate data fetching and data storage logic in a clean way.

One other method I use, with web services that are too messy or complicated, is to create a userscript for Violentmonkey that gathers data as I browse the website and posts it to the database via a small localhost http service. This is useful for JS heavy websites that don't have nice JSON APIs. The script just runs on the background (or can be triggered by a keyboard shortcut) and uses the current state of the DOM to get the data and send them to the database. This method of import is invisible to the service itself.

Most of the scripts are fairly simple (< 100 lines), because most services are built around ~2 interesting primary entities that are worth storing.

Because I use upsert most of the time, I can and I sometimes do add some extra columns for annotation purposes. For example I mentined Jira. I added columns for ordering issues and for marking them with whether I wait for some feedback or not, and then I have a simple Electron app for ordering and listing my issues that I need to work on where issues that wait for external feedback are not shown until the feedback is provided. It's uncluttered, and as fast as you can imagine, being backed by a localhost database.

In Node I use: https://github.com/request/request-promise-native and https://github.com/request/request and https://github.com/vitaly-t/pg-promise

PostgreSQL replication is well docummented elsewhere. I replicate the entire cluster, which doesn't require any maintenance when creating databases/tables, so it's absolutely painless after the initial setup. All you need to do is check the logs from time to time. All I do is that I run the backup database server on a different UNIX socket and disable the TCP/IP interface. It is very flexible, as you can also put the backup server on a different machine, you can have multiple backup servers, chain them and it just works even if you have intermittent connectivity between the machines.

Most of my system is not universal or systematic and is very specific to my use cases. But that will be true for anyone wishing to preserve some of their data in a world that is fairly hostile to data portability. In fact, lack of universality/configurability reduces the complexity by a lot and makes it all easily manageable for a single person with at most a few hours/data source of time investment.

I am curious about a longer more technical explanation of what you do, too.

Org-mode. Since it's a plaintext format it's easily searchable, and it's the most feature complete and ergonomic piece of software I've ever used for task tracking and knowledge management (including MediaWiki, TiddlyWiki, Google Docs, Evernote, plain txts, and more).

One major pain point: no totally seamless mobile sync that I'm aware of. I just write things down in Google Keep and transcribe them when I'm on my laptop next.

For iOS, the fairly new (~1 year old) Beorg is way better than other options I've tried, to the point of actually making Org viable for me, since I have such a dependence on mobile. https://itunes.apple.com/us/app/beorg/id1238649962

For Android systems there is Orgzly. http://www.orgzly.com/

For iOS another possible option is Editorial. There are some script addons convert orgmode to MD ↔ ORG. This is incredibly hacky though, and I'm unsure how it works.

I use beorg but it still needs a lot of work to be what I'm looking for honestly.

Not a perfect solution for everyone as you need a computer to be always on (eg. a server, a NAS, whatever) for sync, but I've been using Orgzly on Android with Syncthing to handle the sync between all my computers and my phone, and it's much better that everything I've tried before.

That's my setup too and it has worked so well that I sometimes feel like I have to double check that it's still working because it runs so smoothly.

Occasionally I have to consolidate multiple edits with ediff-files, but that's normally only after I have accidentally disabled wifi on the phone for longer periods of time.

Just to add to this, if you take a lot of mathematics notes, you can embed latex fragments in Org-mode. Making use of the yasnippet package, writing latex is actually quite efficient.

On top of this, if you read a fair bit of papers, check out org-ref [1]. Referencing and notetaking of papers becomes a breeze.

Finally, you can also export your notes to a bunch of different formats (e.g. html, pdf).

[1] https://github.com/jkitchin/org-ref

Also check out the Emacs packages interleave or org-noter (they have similar functionality): they give you a good way of taking notes on pdfs (linking the notes to positions within the pdf).

I use termux, along with dropbox and dropsync, to keep my org folder synced. It required me customizing a lot of evil-mode commands to use the basic android keyboard, but it's working out great now. Heavy leveraging of org-mode templates is key.

MobileOrg (+dropbox) works for me http://mobileorg.github.io/documentation/

also, comments in 'the software engineering notebook': https://news.ycombinator.com/item?id=15473702

This is what Vanevar Bush was trying to create with his memex [0], plus further what Douglas Engelbart was trying to build upon [1].

Gordon Brander's patterns [2] (on HN recently) seems quite close to these and what you're after.

[0]: https://en.m.wikipedia.org/wiki/Memex

[1]: http://www.dougengelbart.org/about/vision-highlights.html

[2]: http://gordonbrander.com/pattern/

I use Dropbox to store everything. There are no files that I care fore that are not managed by Dropbox (or iCloud, for my photos/videos). I use iA Writer to create lists and documents for everything in Dropbox folders. I also have a folder for PDFs/papers (also on Dropbox). There are also folders for my study nodes ( I study codebases ), notes on personal stuff, notes on Programming, on pretty much everything. I use Spotlight (on macOS) to instantly locate what I need among the 100s of such files.

I ‘ve tried all kinds of ideas before settling for this setup, and they all felt forced or just too much trouble for what they were to me. Text files, Dropbox, and Spotlight have been a perfect combination for my needs for years now.

I do exactly the same thing. Nothing beats text files, a deep folder structure and full text search (and ubiquitous access via Dropbox). I get a ton of value out of my system.

The thing that I find really makes things click is to figure out/set up good keyboard shortcuts and then practice them, so that you are able to save files to anywhere in your organiser hierarchy very very quickly. Say an average of five seconds per file. They say a 10x quantitative change is a qualitative change; once I had this down, I started using it a ton more and treating it kind of like an extension of my memory.

For Mac users some useful starting points for this technique are Default Folder X and ClipMenu.

I try to do this but having a plain text editor built into their site would make it much more powerful for me. There are multiple times a day where I find myself on a system that I don't have my Dropbox files synced to.

This, and I have, in the root-level folder, a folder for every spheres of my life. For example:

- Personal (health, finance, hobbies, spirituality, etc)

- Work (things for my company, invoices, PDFs for learning tech stuff, etc)

- Church (I volunteer for a couple of churches, so that's very misc)

The exact same structure exist for my Google Calendars (separated by spheres that encompass several projects/concerns).

Hmm I have a similar setup except I use gitlab private repo... but really there isn't that much of a need for a 'commit history' in notes... I think I'll try the dropbox approach.

How about a private git repo with SparkleShare?

I do the same with plain text files. The only problem I have is searching. I wish there was a Google for searching local files (i.e. concept search).

For macOS you can download Alfred, it can search all your filesystem for text inside files

There was and I shit you not it was from Google.

I miss it more than I miss Reader :(

I kinda went the non-organized method of organization: dump it all into a google drive and just count on search to find things for you.

A lot of people think of GDrive as pure storage but unsurprisingly Google search within drive is pretty good at finding stuff. Although when you know it's there and search fails, it's frustrating as hell.

I wonder if, when you put something new on, you should immediately do a search and then bookmark the search. Then when you can't find your target in Drive, you can search your search bookmarks. And when you can't find it there, you can search your search bookmarks search bookmarks.

Or you could write the locations of your targets on the backs of turtles.

For "ebooks, papers, articles" I use calibre as the database - it manages all the relevant meta-data and converts between formats. I then sync the calibre folder in google drive for search and cross device access.


For notes, I found the majority of my note-taking is research and reasoning to support decision making for either software or business projects. For that I built my own tool to capture rationale make it useful:


I went down a deep rabbit hole on this about a year ago. Nothing feels totally right but currently I use a mix of TiddlyWiki and Airtable. I haven't found a great solution for media outside of Dropbox but those two tools work well for text and data.

TiddlyWiki is rather inscrutable at first, but extremely customizable and very thoughtfully architected. What I like most is that it is doesn't rely on an internet connection, it's very customizable and non-linear. The basic idea of being able to interlink things and create new pages just by adding brackets around a word is something I miss anywhere I don't have it.

I use TiddlyWiki in an Electron wrapper made with Nativefier. With a bit of custom CSS the interface is very pleasant.

Can you say more about how you use Airtable? To me Airtable has a really appealing low/zero startup cost, but then I'm always tripped up by not being able to do arbitrary queries, joins, etc. to create new views.

I use Airtable for "quantified self" type things like logging workouts and health metrics, also tracking reviews of books and movies, planning travel, and anything that I would otherwise use spreadsheets for.

By the way, you can do simple joins in Airtable by using the "lookup" and "rollup" field types, but I agree that it is relatively limited in that respect.

I use Airtable for this stuff too. Unfortunately it doesn’t work offline (not even reads!) so I’m very close to reverting back to spreadsheets.

I'd recommend perkeep[0] (previously known as camlistore). It is built for exactly what you are asking for. It's written in go and it is very easy to contribute.

[0] https://github.com/perkeep/perkeep

How has your experience using it been like and how does it facilitate note-taking?

Personally i can recommend http://zim-wiki.org/

you can organize the "media" think in "attachment" and tag them or create page about them, and you can still find the attachment in your file manager

"Zim is a graphical text editor used to maintain a collection of wiki pages. Each page can contain links to other pages, simple formatting and images. Pages are stored in a folder structure, like in an outliner, and can have attachments. Creating a new page is as easy as linking to a nonexistent page. All data is stored in plain text files with wiki formatting. Various plugins provide additional functionality, like a task list manager, an equation editor, a tray icon, and support for version control. "

zim-wiki for me, absolutely. Text-based, offline, decent plugins, cross-platform, easy. What I love is that if zim is not available I can use Sublime Text to view and navigate as a near replacement as long as the SidebarEnhancements plugin is installed. I stand by the supremacy of zim-wiki although it does lack "pretty".

Do any of you sync your Zim/open it from mobile, specifically Android? I've been using Zim for a few days now and it works well, but would be much better if I can access the database while on the go...

"There is currently no Android application for Zim available." from the github page https://github.com/jaap-karssenberg/zim-wiki/wiki/Android-ap.... An Android app would be great. What I do is start up the zim server and look at my notes over LAN. A poor substitute.

I used zim for a while, then switched to Joplin. Huge fan, cli, modern and mobile interfaces. Choose your own storage. open sourcen

Thanks for the mention of Joplin. I just started looking at it and it looks pretty decent for an evernote replacement.

Some of these have been mentioned at various points:

https://zettelkasten.de/posts/three-layers-structure-zettelk... https://github.com/renerocksai/sublimeless_zk https://tiddlywiki.com/ http://journal.mcmorgan.org/view/incremental-paragraphs/ward... http://fedwiki.org/hack.platform.earth/fedwiki-for-notetakin... https://github.com/burtonator/polar-bookshelf

Perhaps a single system is not the answer, but a combination, e.g. for taking notes I'd pick fedwiki/sublimeless_zk/Zim/org-mode.

For organizing PDFs with full-text search Zotero is quite good, and you can link to your PDFs from your notes, but I wouldn't store my notes about PDFs inside Zotero (because then it'd be separated from my notes about non-PDF articles).

For keeping track of articles Firefox bookmarks (with Firefox Sync on Android acting as a 'read me later' bag).

DEVONthink - great for all things bookmarks, data, searchable PDFs etc - can also use it for org-mode files and open in your fav editor. Many sync options. Mobile too.

Seconded. I have databases for household scans of bills, etc.; one for computer science stuff; one for my work; and so on.

I used to dump every thing in GoogleDrive and search. I was enamored with keeping my notes in plaint text, so as to not tie these with any specific tools/app/formats as these go out of fashion every few years. However I realized I also have links/clippings from Web, which needs to be part of my searchable knowledge-base. And I want to be able to clip/bookmark those from my phone as well as Mac and Windows machines. Evernote works great for such use, and has tags(it does not keep notes in plain text format though, these are supposed to be exportable though). I would not mix video/movies with text based content though(links are fine).

I use http://papersapp.com/ for handling PDF papers from arxiv.org etc.

It seems they have merged with readcube who want a yearly subscription however I was still able to buy a life time license to papers app a few months ago. I guess support will slowly disappear but for now its working great, with PDF sync via dropbox onto my iPad or iPhone, the ability to draw and take notes and search as well as download new papers. For your own PDFs I guess this would still work pretty well since you can search and annotate, create smart groups etc.

Notion for everything: my work, personal, and wife+I joint workspace.

Notion's UI is so genius. It's clean, intuitive, flexible. It's like everything I love about Google Docs, Quip, and Workflowy but with Slack-like slash commands, mentions, and integrations.

Moved to Notion from Marxico+Evernote a couple of months ago, haven't looked back.

Are you using the free plan, or the personal paid plan?

Free for now, but will start paying when I run out of blocks (the usage isn't as high as I expected, only at ~500 out of 1000 right now, though I have imported 4-5 years worth of notes).

Also it's not a product or a service, I really like Gordon Brander's "Patterns" [1] on his website.

[1] http://gordonbrander.com/pattern/

Dynalist.io is the best note taking software I've seen in 12 years of using note-taking software. Features endlessly hierarchical lists, with tagging, links to other nodes, images, zooming to a single node, full-text search, Chrome web clipper, Dropbox backup.

- It's like Workflowy, but better... - I've tried Evernote, OneNote, Apple Notes, Google Keep... they all just end up becoming a massive mess of unorganized notes.

And don't forget inline LaTeX support. With this feature, listening technical conferences, lectures or noting mathematical ideas are really comfortable.

I’m super interested in this space! Agree with sibling that focus here is on knowledge management while question mentions media, which is probably even more interesting challenge. I recently dreamed up something that kinda resembled Airtable (which you should check out), albeit I was thinking more of datomic like system as opposed to tables (more flexible, but tables are nice and simple). I like the mention of wiki like system, I still find wikipedia to be my favorite knowledge source. I guess for me the ideal system has bunch of requirement you haven’t mentioned:

- mobile and desktop friendly (especially input) - full local backup, including any metadata (so that I can always use another software, not cloud-first-with-export) - can be purely single-person (plenty existing sharing platforms) - it’s gotta support arbitrary nesting (I hate evernote/one note three levels to a wisiwig page rigidness) - any others?

I also wonder whether there’s enough market for selling this software (Airtable is going after business use cases, the single-player use case makes it quite limited).

Forgot one: full tree history. No way to lose data, even if I do undo, delete a page, etc. Super important.

Another shameless plug: you might find https://johnnydecimal.com useful.

Forgive the design, I’m right this very weekend updating the whole thing. I’m busy writing a little web app which will replace the spreadsheet you’ll currently need to keep track of your numbers.

I had come across johnnydecimal at one point but had a hard time resonating with the idea of having numerical enumerations since I already use tools to help navigate folder hierarchies (e.g. alfred, quicksilver, etc.).

Do you still find value in the enumeration aspect? Is this something where there's more value in it within collaborative environments?

This can be difficult. I was looking for a system to organize all the personal knowledge beginning of the year. I tried Evernote and several of the alternatives such as OneNote, but none of them stuck with me well. My main issue was storing code along with the notes. Evernote does provide code formatting but not very friendly with the syntax highlighting. I also tried Quiver for storing code, but it is specialized towards code storage.

Currently, I am trying StackOverflow Team but only for myself. I am so used to question and answer format that it makes it easy for me to dump the information. Most of my searches on Google ends up on StackExchange families of the website that this solution has that familiarity. Plus, it has some versioning built into it. It lacks on some fronts such as limits on the size of images. I am still looking for a better solution.

> I also tried Quiver for storing code, but it is specialized towards code storage.

Can you expand on this? Was quiver not good enough at basic text?

Yes, it was but I wanted to access information with a web browser as well. I might give it a try again.

Personally I use markdown files synced with Dropbox and search with the silver searcher. But I think I better answer to your question is perkeep (formerly camlistore): https://github.com/perkeep/perkeep

Tiago Forte has a lot of thoughts on this topic and a partialy free/paid blog: https://www.fortelabs.co/

I'm not affiliated in any ways but his ideas a really well thought out and apparently thoroughly tested with clients.

I would highly recommend the idea of "personal knowledge management" (PKM). I started being much more organized on this front starting last year and it is a gamechanger.

Personally I store raw thoughts and ideas in Keep, then transfer them daily or weekly to OneNote. Every so often I review the notes.

I have notes from anything important I've ever learned, ranging from technology stacks to economics to capital markets to psychology or personal well-being. I now never worry about losing thoughts or not seeing "the big picture" - it's a picture (note really) that I've saved over years now. And ever so gradually, I just build on that knowledge base, which I see as an extension of my own brain.

Edit: and if you don't use the cloud, _make sure you back up every few weeks_. It takes 5 minutes and you will really regret losing part of your brain if you break your laptop.

Very practical approach. Reviewing the saved info is what makes this real knowledge.

I find that any kind of knowledge search system eventually gets saturated and requires increasingly more time to scroll through partial hits to find what was asked. Our brains work the fastest in associative mode, it'd be nice to have some kind of associative storage (other than tags) that is personally biased/customized.

Reviewing the kept info is one way to refersh these associations in our organic minds.

But it'd be cool to ask your 'MindAssistant' to retrieve you the whole context of your recent or old idea, complete with your thought pathways: 'Hey, Mind, please remind me what was I thinking about the shapes of the clouds in the sky', '-Here's your Mindmap, thought-journey, and most important insights, Sir. At the conclusion of these thoughts, you read the following articles via HN...'

Would help us to know more about the specific use case and goal you had in mind.

For now, generically speaking, I use OneNote for keeping track of my own generated ideas and pasting in things from other sources. For larger media I recommend having a network drive or using Dropbox.

I'm happy with OneNote.

For my more technical ideas where I'm brainstorming I have a nice leather journal (no lines) that I use for brainstorming. I tried using a Surface Studio as well with the pen and that works good too because you have an infinite canvas and can move things around. With paper / whiteboard brainstorming I frequently run out of room. Digital means I can just drag a bunch of stuff to make more room. Plus having the pen and choosing different colors makes it really freeform.

I loved OneNote until I tried to put some code snippets into it. Somehow over time they become unusable for cut and paste into *nix systems. Did you find a way to overcome this? I have no time to dos2unix everything.

OneNote even has a code format, but it still messes with capitalization. I gave up when MS announced mobile only and the desktop app is done.

No, I don't really use it for code. I use git repos / gist for that.

I use Evernote for ebooks, papers, articles and personal notes. Probably wouldn't work well for movies or large photo albums (I iCloud for stuff like that). I use their chrome extension, desktop app and mobile app and it works really well. It's really nice to be able to search everything I've ever written in notes or clipped on the web. It also has some support for OCR for paper notes, and does a decent job of indexing text in PDFs. I searched for an open source or self hosted solution for a long time, but ultimately broke down and chose a well supported commercial solution.

Unfortunately Evernote gradually becomes worse and worse, at least on Mac. The very basic functionality of editing notes is buggy and slow. Issues keep creeping into both simple text notes and web clippings. Users are complaining about this for years, the devs do nothing but change the looks instead and bolt on corporate features (apparently corporate users don't need basic functionality working well?).

They have the weird community forum where employees don't post anything except for announcements. And now they have an extra tier before actual support, where your question is 'answered' by some community member or someone like that, who apparently can't be bothered to understand the question.

The Android app is a slow mess eating all available memory right away and/or refusing to edit notes. Inability to use SD cards is just what I want when the notes take 6 GB. And the website now “doesn't support” mobile browsers.

Pretty much the only thing where Evernote doesn't have better competition is the web clipper.

The web clipper is definitely the main reason I still use evernote. I've tried a few alternatives, but haven't found anything nearly as good.

Other than the speed issues you mentioned, I'm also worried that evernote is eventually going to start selling my information (if it doesn't already) and I don't feel safe putting personal notes on it.

Evernote's search sucks(at least for a free user). can't(sometimes) even handle synonyms well. Sadly, there's no better alternative.

I haven't had issues with synonyms specifically, my praise for it comes more from it copying the full text of websites when I clip them and parsing the contents of hand written OCR scanned notes and being able to search over those.

are you a paying or free user?


I've just put this complete thread as a link into my Netsso, to study later. If I want to avoid future link rot, I tell Netsso to "save content" to my Dropbox/ GoogleDrive/OneDrive. In doing that, I could also click to encrypt each file separately, with a 40 character unique password,that only I know, and only decrypts on my local machine. Netsso remembers the passwords but only I know them !.It stores any type of document, any url. I load files from my computer to my Dropbox, and immediately link them to my Netsso, so I can access them on any device without formally opening the Dropbox. Online Note-making, with rich text, images included, even videos. Save any new urls with quick click/copy/paste. Add 400 character description to a link, to include in later search. Very sophisticated categorisation, sharing (even encrypted notes), drag/drop, inbuilt credentials for apps, etc (i.e password management included)...8000 links so far. Preview or Open on line, download, edit. Also I have a sharing version for Teams, with same functionality (plus encrypted Chat). Built for private,personal use - all my web knowledge available anywhere- but very easy to share links, or collections of links, if required. I can also Back Up complete collection to any computer in seconds, encrypted by my master password. Netsso seems to be a unique combination of bookmarker, online file creator and manager, internet file security enhancer and team collaborator.

I use WorkFlowy[1] as my starting place for all notes, info, and tech tips. It's quickly searchable and the mobile client is easy to use. I really like the structure of bulleted lists for just about all types of information.

It doesn't really cover your file storage needs, but I think that's a different beast than notes and information. When I need to store stuff like that I dump it on Google Drive which makes search/sync/browsing dead simple/cheap.

1 - https://workflowy.com/

I've been using Workflowy for several years and recommend it very highly.

The ability to expand/collapse, zoom in/out, and do everything via keyboard shortcuts makes it very pleasant to work with.

Judging by Workflowy's development pace, the entire team is spending the profit on opiates. No features were introduced in the past couple years, aside from the desktop apps—despite users begging for an API at least. Now Workflowy went the way of Evernote: changing the looks and adding bugs.

The outline paradigm is great, and Workflowy is a nice influx of fresh blood in that field, but I think I might actually kludge Org-mode into proper form with good mobile synchronization (and bonus privacy and customizability) before Workflowy makes any progress.

https://dynalist.io/compare shared above seems to have more features than workflowy

Being able to zoom into lists of lists is Workflowy's best feature

I believe Org-mode has that feature as well. I was and am very impressed by Workflowy but a) using plain text files b) better hyperlink handling have me migrating to Org-mode + Orgzly

problem is inline screenshots

Agree to wenc. If you try a hierarchical organization (you could use the same classification as in official libraries for example) then you won't find what you want easily. The best solution is NOT to sort too much, and use a (free) global tool to find what you search. If you're under windows, then "everything" of voidtools.com should help. (This program is realtime, it's amazing.)

if you're using any Unix flavor, activate "locate" and use it. Both do an index of the filenames. So naming correctly your files is important. The 8.3 schema is over now ;)

On a WinPC I also use TotalCommander (of ghisler.com) to make an advanced search (filename and/or content). Of course, find&grep under Unixes do that well!

I got more than 40Gb of ebooks, pdfs, videos, podcasts, etc. in 4 languages (French, English, German, Spanish) and the sole way to find out quickly is global searching.

And constant reorganizing. (With no tool to update, it's a breeze ;))

If you prefer a hierarchical way, then using aliases (symbolic links, on unixes, but since Win7 you can also create links under windoze) can be an alternative to have multiple entries of the same files in your "database" of filenames, making possible to have "NoSQL for dummies" in "RDBMS" as well as "BigData" and "books for beginners" categories.

P.S. I work on documentation systems since 1984. Full text indexing is probably the best solution.


For personal notes, check out Nvalt [0]. The search is blazing fast. I linked it with SimpleNote [1] so changes sync instantly between my laptop and phone. It also plays well with the Typora [2] markdown editor

[0] http://brettterpstra.com/projects/nvalt/

[1] https://simplenote.com/

[2] https://typora.io/

Devonthink is a great digital archive

I really think what we need for this is a new sort of file manager, and it should look more like iTunes or Kodi than Thunar or Dolphin.

I always enjoy reading this post about tag-based file organization, which feels like it would be a great foundation for this: https://www.nayuki.io/page/designing-better-file-organizatio...

I keep things simple. The following process has served me well for ~20 years:

For brain dumping text I put things into text files, named by YYYY-MM such as 2018-08.txt and they all end up in a notes/ folder.

Then I have a bunch of folders for things that are non-text, organized by whatever it is. I try to keep folder nesting at a minimum.

Searching text is super simple with grep and searching for specific files works well enough with file searching tools (and are also easy to browse manually).

I did try to find a good cohesive and cataloged system, but in the end I just ended up going with folders and descriptive filenames.

For instance, my music resides at "Music/Artist/Year - Album/Track# - Tracktitle.Filetype", with appropriate additions to the Tracktitle to signify bonus tracks, remixes, live tracks and featured artists.

I've adapted pretty much the same system for all of my other documents. It's a bit old-school, but it works.

I try whenever possible to push as much personal knowledge into Wikipedia and Stackoverflow, and then use bookmarks to keep track of that info in my browser. This ensures that others can take advantage of that info, forever.

EDIT: I'm still trying to find a good machine readable way to represent the metadata for all that information. Alas, there are only so many hours in a day. Will probably end up with tags and Pinboard.

Personally I use vimwiki. It stores everything in plaintext files that I can search, keep track of in git, and also comes with quite a few useful features for building articles.

The main downside, as someone else has mentioned here with org-mode, is the lack of a good mobile editor. This is something I’ve wanted to find time to fix for years but still haven’t had a chance.

How are you currently viewing/editing your wiki on mobile?

I sync to a server with vim and vimwiki installed and on my phone I just mosh[0] in. I don't generally do extensive editing on the go, though. Did the solution you have in mind include a specialized editor for touchscreens?

[0]: https://mosh.org/

I recently started using Markor for that purpose. While it wasn't built as a vimwiki client, it is good enough for my purposes. Some info at https://github.com/gsantner/markor/issues/317.

I generally control the organization of stuff by the way it comes in, and then have a secondary means to have media organized that comes in from a non-standard ingress.

Most information is stored in of 3 ways

  1. Synced to a NextCloud instance running on a VPS.
  2. Synced to a home NAS running Syncthing (though I may change this to IPFS at somepoint).
  3. Added to Wallabag (A read it later like service)
Notes, List & Project based Todos

These are put into Orgmode which syncs with Owncloud. I can then use the WebDav endpoint to pull the files to add or edit on my phone.


These get added to wallabag then are accessed via an RSS reader.

Personal Documentation

Added with Camscanner, OCRed, and added to NextCloud. At some point I will probably also store these on my NAS, so I can extracted the OCRed text as a text file so I can search them, but currently the collection is to small to justify this.


My NAS handles most of this, and organizes it using Emby. Syncthing is my secondary means for this, and Emby watches the directory to organize this.


Syncthing to add to the server. A headless calibre instance then organizes it.


Most pictures come my from my phone, nextcloud will automatically sync these. If the pictures are part of notes, then instead they go into Orgmode, which works with images as well.

There are a couple of other organizational thing that use Syncthing + Organize, but nothing of major note.

This is not A single cohesive system but several scoped systems with a standardize way to add content to organize (e.g. folders). Searching the content is generally done via plain text searches, or inside of the application that manages that context (see emby & calibri).


  1. https://github.com/nextcloud/server
  2. https://github.com/syncthing/syncthing
  3. https://github.com/tfeldmann/organize
  4. https://github.com/wallabag/wallabag
  5. https://www.camscanner.com/
  6. https://beorgapp.com/
  7. https://manual.calibre-ebook.com/server.html
  8. https://emby.media/

I use Personal Knowbase, a small Shareware era program. It has minor annoyances but does what it says on the tin. It's fast, minimal and allows narrowing down to related notes in a jiffy. The makers update it slowly though.


After trying different things over the years, I wrote this:

http://onemodel.org (yes I plan to move to https sometime)

You can think of it something like a personal wiki + emacs org-mode, using postgres, but with a much larger vision than today's features, including sharing (linking/copying) between instances, and computability of the info. The most current code is in github (AGPL). Comments/questions very welcome, preferably via the mailing list; be patient if my answers are slow. Announcements list is low-volume.

Edit: it can store files, but isn't especially smooth about it (yet). For personal notes of all kinds, it is the most efficient, effective, flexible thing I know of.

Edit: the FAQs link to a discussion comparing it with emacs org-mode and others.

Edit: it has built-in text search, and some finicky but very functional import/export.

Really enjoying this one: https://itunes.apple.com/us/app/mindscope-thought-organizer/...

Based on this concept http://www.mandal-art.com/

but original mandalart author had/has less effective implementation

Truisms from time: - Content > tool config - Simple is better - Have good search - ISO 8601 date format, line based entries

Old and current flames over 20 yrs: - Google Desktop - TheBrain - MindManager - Freemind - NotationalVelocity - TiddlyWiki - VimWiki on vim as Deft folder on emacs in evil-mode - Org-Mode in native and evil mode - log bash script - sqlite3 - Traveller notebook, pencil and paper

My tip is to break down your knowledge into very small chunks and store them somewhere easily accessible and searchable. I have been looking for a solution for this, and that led me to build a system for myself and open source it [0].

In short, I find that writing them down every day in bite-sized chunks is most helpful. This method probably will not help you store non-texts such as movies, pictures, but for other mediums, it is great.

After about two years, the main benefits I find are:

* Bite-sized notes are easier to index and search for later

* We can easily share those notes with others when in a technical discussion etc, saving a lot of time

* We can do spaced-repetition on those notes easily because they are concise. (I wrote a short Go program to do this for me)

[0] - https://github.com/dnote/cli

I started down this road several months ago, and decided to write my own system that can grow and change as I discover how I want to organize things. It's roughly modeled off of a traditional categorized card-catalog system, with the addition of zettelkasten-style numbered links. As a concrete example, here's a card that I added today:

  [[379]]  Sjö Milljarður Manneskjur
        |  Tölvuleikur - Forritun
        |  Inventory - Steam library
        |  Computer Science - Concepts - Multiprocessing
        |  ,,Seven Billion Humans''
        |  2018; re: concurrency
        |  Backreferences:
        |  [[380]]  Tomorrow Corporation
        |        >  2018: [[379]] Seven Billion Humans

I still use Together 2.6.8[0] for collecting and organizing just about everything. Multiple libraries, built-in previewing, editing, and Spotlight search, no proprietary database (uses Finder under the hood), etc.

There were some changes in version 3 and the subsequent rebranded Keep It[1] which kept me from upgrading, but it might fit your needs.

[0] https://web.archive.org/web/20130315081858/http://reinvented...

[1] https://reinventedsoftware.com/keepit/

I use OneNote currently. Dump (almost) everything in there. The UX leaves something to be desired - Evernote is so much better. Shame that Evernote kept losing my data.

And Google Drive for the "larger” and better organized stuff like photos, e-books, or invoices.

Try Onetastic, the calendar feature has saved me countless hours. Desktop app only though.

I don't look back at my notes. So, I don't think I should spend a lifetime organizing them. I write notes while learning and reflecting. Once finished with that process, 99% isn't used again.

I use meaningful titles and tags. That's enough.

Shameless plug: https://morethemes.baby/themes/log-lolla-pro/

I do organize my knowledge since years (http://metamn.io/beat, http://metamn.io/pulse, http://metamn.io/gust) and now I’ve collected these experiences into a Wordpress theme.

Still under development / test phase ... you are welcome to try it out.

Trees -> Typed nodes -> Graphs

MacInTouch More (1)

macOz Omnigraffle (2)

(1) https://en.wikipedia.org/wiki/MORE_(application)

(2) https://www.omnigroup.com/omnigraffle


TreeSheets: Free Form Data Organizer


Quetree – A Q&A site that is tree-based and hierarchical


Trilium Notes is a hierarchical note taking application (electron js)


Digital: Notes in Google Keep, Text files in Dropbox. Paper: Bullet Journal, Lists

Downloaded data: Folders PDF, Video, TV. Own Generated data: Photos in a YEAR/MONTH Labels Textual: In Dropbox, Excel, Text, Word

All data on master Hard Disk, two more copies.

Nodebook¹ is the tool that I've been using for the last couple months. IMO, it beats Zotero and Evernote, or any other tool I've previously used.

¹ http://nodebook.io

I use NValt [0] and plain text files for all my personal notes/info tracking system. I use 1Writer [1] to access/add to it from iOS, and on Mac I either use NValt itself or Deft [2] in emacs. I try to convert any visual information into a textual tip, otherwise, most images, PDFs and ebooks go to Dropbox while I'm actively reading or are in "pending", after being read they go to long term storage.

  [0] http://www.brettterpstra.com/projects/nvalt
  [1] http://1writerapp.com
  [2] https://github.com/jrblevin/deft

I always though the smallest federated wiki would be prime for this. http://fed.wiki.org/view/welcome-visitors

question is more about personal media collection. but title and most answers (so far) are about actual personal knowledge

I use Evernote (for notes) and Dropbox (for files). What helps me to stay on top of things is that I write everything in Evernote myself and then add the references at the bottom.

First, that helps me to rephrase knowledge in my own words (except for quotes, of course), which is good for retaining knowledge.

Second, I use tags and note links that help me to navigate through pieces of knowledge.

Third, I only store files in Dropbox, with a very well-maintained folder structure.

I don't clip stuff in Evernote and don't just copy/paste. I think that bloats the whole thing up and makes it unnavigable.

I'm surprised to not see any YAGNI voices in the comments. I set up Evernote with lots of collected papers I had read. However, I found I would rarely go back, and finding things within a collection of two hundred items was already not easier than just googling it. Perhaps with improved semantic search the situation may improve. I just think summarizing down to a small set of core ideas and storing them in a single small text file or document is better than keeping extensive originals or even per item notes.

Yeah. Summarize and Anki. Otherwise you are just collecting.

Shameless plug: miki (Makefile Wiki) https://github.com/a3n/miki

"sitemap.html" and "catalog.json" are the most relevant parts. The project itself assumes you already follow some organizing principle based on the filesystem, and documents it for you, with links.

It's based on Makefile, and existing utilities, so you should be able to use it as is, customize it, or as a start of something better.

I now use gitea to host code and markdown/notes and generic filesystem(or NAS) to host large size blob(movies,pdf,etc).

for a universal knowledge/media/file organizer the way might be a smart tag-based system, could not find one still.

used drupal, wordpress for CMS in the past before switched to gitea. Can't use static-site-generator as I don't want to put random personal notes on internet without even a login.

a simple, login-included, markdown driven cms with search/tagging will be really nice.

I use Zim but I can't say that I've done an exhaustive survey.


Why are you unsatisfied with the particular methods you listed?

The key to organisation is a WORKFLOW, followed by tools that fit your workflow.

I found and paid for https://www.buildingasecondbrain.com/ . It was immensely useful for me. The course provides a workflow specifically on managing all the information that feeds into our lives and shows you how to use Evernote to do this. Happy to answer any questions from my experience.

I use Pocket to grab interesting content as it swooshes by. :-). If it's worth keeping, I send it to Google Keep, then use Keep's color codes to organize material by rough use cases (code, product management, hobbies, behavior, ...)

I then periodically clump similar content into a single Keep note, usually by simply cut/pasting the relevant links. This happens once every 3-4 weeks.

Now if only Google's Keep team would publish an API...

I've been using the offline version of Draw.io for this. You can save diagrams as individual xml files. Unfortunately the data is base64 encoded by default so you have to manually convert them to a searchable format. I wrote a script to do that: https://github.com/JesseAldridge/draw.io-convert

For books, PDFs : Zotero (if book/pdf requires to be sync have a copy at iCloud) - Because I usually need to cite these things.

For notes: First condense the thought and put it nicely to Dynalist - Because it is all easily searchable, linked and think-then-write tree-structure. - Mobile app is a must to capture ideas, thoughts on mobile.

For todo: Things/Todoist etc. would work. - Because you need deadlines, reminders.

I wrote my own Evernote style system and used it for a while. Then I changed to my current system: Emacs and org-mode for notes, todos, tasks, etc. I also use Notes in Fastmail (and they show up in Apple Notes automatically). For PDFs of purchased books, Communication of the ACM, good articles converted to PDFs, etc. I use Google Drive because it is searchable.

By searchable you mean Google Drive can search _inside_ of documents like PDFs and images?

I don't think search works inside images but works well for PDFs. Just this morning I was trying to find a particular paper on LSTMs and NLP and I found what I needed very quickly.

I also follow many of the recommendations here and keep text files for my personal knowledge. I prefer to keep PDFs rather than HTML files. Currently I print webpages to PDF after rendering the page in (Firefox, Safari) Reader mode.

An idea on my back burner is finding a solution for auto-generating taxonomies using Python NLP. But I've been saying that for years. Soon.

I use qownnotes and Nextcloud on a NAS and back it up on external storage, as well as every desktop/laptop also has a local copy. It uses Mark Down. So far it seems to be working fine. Nextcloud is very flaky but better than owncloud.

I keep a lab paper notebook for everything I do and I write the date, time, and whatever I need. Then I type it in.

I'm working into this right now at https://lyfepedia.com, but it's not fully ready yet.

The idea is that you can publish/edit wiki pages, "twitter-like" messages, pics, videos, do whatsapp-like group messaging etc while maintaining the privacy of what you write.

You can organize ebooks with caliber software

Links: pinboard.in

Wiki: I host my own mediawiki instance.

Documents: I host my own upspin: upspin.io backed by ZFS.

Images: Combination of upspin (above) and iCloud Photos.

I also host my own MediaWiki instance, complete with VisualEditor and a mobile skin. Out of curiosity, why are you not loading documents and images into your wiki? For images, I don't use it as an image repository, just for images of things specific to personal knowledge.

Well in general I want documents to be available locally. If I were to upload them to the wiki, I'd had to 1) download them from somewhere, 2) upload them to the wiki, 3) when I need them, download them again for the wiki. In contrast, if I use upspin, I just download it in a directory served by upspin, so it's just step #1 above, and I always have them available.

Would you upload pdfs and things like that to your mediawiki? Bills, invoices, paychecks, for example.

I do upload pdfs including bills and invoices (not paychecks since I don't have a use case for that). My wiki is private so it's only available to me. Plus I have extensions that allow me to render the pdfs nicely and index all the text in pdfs for easier searching.

I have both a public and a private wiki, but most of my PDFs are 5000+ page instruction set architecture monsters, so I much rather prefer to view them offline in some decent viewer instead of viewing them in the browser.

I also have passports scans and other sensitive stuff. I don't trust that I can secure mediawiki well enough to put those in clear on the internet (in upspin they are encrypted with a local key and the storage is my own server, mediawiki is in the public cloud).

I've never actually heard of upspin before this thread. Their homepage doesn't give a clear view of what it actually does, but I'm looking over https://upspin.io/doc/overview.md now

This video is pretty good: https://www.youtube.com/watch?v=ENLWEfi0Tkg

What do you use for indexing/searching pdfs? I didn't think about that even being a feature for something like mediawiki, but I guess that would make sense.

One of the big features I like in Evernote is the ability to search pdfs and images. I have a lot of technical manuals that I save, along with receipts/etc that are easier to search for.

I have this problem, except in my case most of my documents are LaTeX. Tried Gitit but didn't like stick with it because only a tiny bit of MathTeX is supported and categorizing has to be done manually and you can easily end up losing pages that you forget to link to others.

I bought a $10 starter confluence license and using it since last 6 months. I am happy with what I have got so far. I have documented lots of things from my job related to personal life. Although I am also looking it to change to a different system.

This is a crossover between bookmarks and content curation. Also looking for a good solution.

I was thinking of building my own hacker news: own links, curation of content, tags, and even ranking.

Evernote (Premium): Saved Articles, PDF, Notes, Project Documents, Scans of my physical mail

I also export final documents there, eg Mindnode maps, Presentations etc. I write notes using iA Writer.

iCloud: Documents, Photos, Videos, Music (Production), Programming

Omnifocus: Reminders, Project Tasks

I have to recommend checking out Bookmark OS. It's like an online desktop and makes it super easy to keep bookmarks and notes organized https://bookmarkos.com

Perhaps you could be more specific with "not satisfied" and what kind of documents you need stored. I synch all research documents in mendeley, synch all links in firefox, synch all structured text using zim-wiki+owncloud.

VSCode and personal wiki extension of my own design, https://github.com/grigoryvp/vscode-language-xi

I build mindmaps using freemind/freeplane to keep track of my research; I literally have hundreds of mindmaps from 3Dprinting to ZFS. I also link mindmaps to other mindmaps (crude "topic mapping").

Take a look on https://www.thebrain.com/ It's better alternative to mindmaps

Can you elaborate on "better"?

I would say Evernote but I’m not sure you want to cram movies in there. Works for everything else though, even searches on its attempts to recognize handwriting. Won’t give you a transcript of that though.

I'm just impressed to see the name Niklas Luhmann in HN. Well played.

My pinboard,onenote, and evernote is just a dumpster ground.

I organize everything important in dynalist or file folders in dropbox

Whats importanf for me is I can take my notes from any solution and not rely on any one service

emacs org-mode. I make my own wikis for everything I want to remember.

I use personally hosted Confluence ($10 a year) to keep snips of notes and personal project related stuff. And also Google docs.

Yes I'd be interested in having a tool to keep videos organized.

With the exception of photos delete it all and don't worry about it. A cluttered digital life is no better than a cluttered physical one.

The answers here probably have the clues. But someone has to work hard to figure out the ONE answer that is minimal and satisfies all needs.

A combination of Pinboard (for links / images with URLS) and Notion (for personal notes, thoughts, and lists) works well for me!

Same here, pinboard and notion are a great combo. I use NewsBlur for feeds, but wish pinboard would integrate that.

The biggest disappointment with Notion is that there is no on-prem. I would've rammed it through at work by now if there were. I've searched high and low for a comparable open-source alternative, and the best I get is outline: https://getoutline.com, but it's got a long way left to go.

I recommend Notion as well.

Tiddlywiki is my favorite so far.


I just use common folder and use the file name as the tags, something familiar I use to remember what I'm saving.

Someone knows a good software to search a large amount of text pdfs locally? Something like mendeley & zotero?

For MacOS, foxtrot pro is fantastic: http://www.ctmdev.com/foxtrot/

It gives you just enough context preview around your search term to judge if the PDF is useful or not. Currently using with 10k+ research papers and very happy with it.

I personally am a long time fan and user of leap (ironic software) and yep!

Countless trello boards.

Trello is absolutely great for this


I have several more or less organized ones for specific topics and a catch all called "personal knowledge base"

with cards for a bunch of random things there is no more specific place for but not enough for it's own board

Focus on learning only a few things and get really good a them. That reduces the need to organise anything.

Has anyone found something that works with Markdown, code highlighting, and Latex?

There was a show HN this week with something called Madoko which sounded like exactly that.


I use pinboard.in mostly for a raw collection of links.

And I use Gingkoapp to organize my thoughts.

This screams like a startup idea you should pursue. I would buy the service.

evernote has served me best, with one notebook, and using search

it supports attachments and other stuff

eg when i research “how to do password less ssh”, i will create a note with that title for future use. it has worked so far.

For the mundane among us (like myself): trello works pretty good

Could OP outline why he is not satisfied? That might be helpful.

ElasticSearch can automatically index your documents.

EverNote is neat for what you're trying to do.

evernote if you want a "well supported" platform

devonthink if you want a personal knowledge database with good searching

Maybe Apache Tika?


I copy pasted everything from my blog here. If you want the actual links to things I use, just use the link since I can't dump it in hackernews without it being really unreadable.


I only put all the ways I organize my things via software, still a WIP (work in progress) for hardware things I use (pens, notebooks, etc)


## Notetaking / Productivity

Dynalist – Primary notetaking software

Airtable – Google spreadsheet meets microsoft access. A swiss army knife for business. Very useful for making a minimal viable prototype

Pinboard – Bookmarking. I consider this 3rd party chrome extension a must-have, I bind it to my CTRL+D key

Anki – Flashcard software. I use it to speed up learning math, programming languages, and computer science.


## Software Development

Chrome – My everyday browser

Slimjet – A simple chromium based browser that I use for webdevelopment occasionally. Because then I can have a seperate desktop icons

VS Code – My primary IDE

Github – Git + cloud storage for programming projects

Codepen – Isolated frontend environment. Used to A/B test components and explore cool things. Organize codesnippets here like I do in evernote.

Postman – API testing

LucidChart – Making flowchart diagrams and data models. Not free though, for that I prefer draw.io


## Design Tools – (macOS and windows)

Flux – Makes screen red at night so I can sleep easier

AffinityDesigner – I use all adobe products as well, but I prefer this over illustrator

AffinityPhoto – Same reason, I prefer over adobe photoshop

AdobeIndesign – Catalog & Brochure design

Figma – UX design tool, applesketch is macOS only

AutoCAD – Architectural drafting program, but not free

Fusion360 – 3D modelling program for DIY projects

GooglePhotos – Photo & video backup

Dropbox – Cloud storage for files/folders



Phraseexpress – Keyboard macro automation. Checkout my 14 phrase-express guide to see use cases

Autohotkey – When I need an advanced macro that phrase express doesn’t work on. It has a high learning curve and ugly syntax language though

DirectoryOpus – Native windows file explorer sucks, this is best file explorer in market. Has a learning curve. Checkout use cases here

7+ Taskbar Tweaker – Smaller icons on my windows desktop screen

Blank Spacer Exes – It gets hard to look at so many icons in windows, so this gives me eye relief by grouping icons together

Greenshot – The best windows quick image editor out there. First thing I install on friends/family PC normally, its that good. MacOS version is not good.

ShareX – ShareX is greenshot image editor that supports gif uploads / custom image-hosting options. Version 11.6 is the best. MacOS has no equivalent.


## MacOS specific – alot of programming apps

Iterm2 – A better terminal environment. I binded it to ⌃⌥[spacebar]

Oh-my-zsh – My preferred bash alternative for development

Penc – Gesture based window snapping/resizing

Spectacle – Keyboard based window snapping/resizing

Keyboard Maestro – Keyboard automation for macOS. Similar to phraseexpress but better

Alfred – Better alternative to macOS’s spotlight for quickly opening files / apps. I binded it to ⌥[spacebar]

Captured – An inferior version to shareX without gif support, but best thing I could find.

Dropzone 3 – For quickly handling common tasks like moving folders around

Cheatsheet – Because I’m a new mac user I need to quickly memorize commands

Unarchiver – So I can one click install things on macOS


## Chrome Extensions – Productivity

Silverbird – For quickly tweeting things on twitter

Tabsnooze – Set reminders to myself to pay bills at end of month, and to do incremental reading

Pagenotes – When I need to save some specific information on a website. E.G. terminal commands, regular expressions on regex101.com, etc.

TheGreatSuspender – Suspends unused tabs after 30 minutes. Because chrome is a memory hog

Tampermonkey – Injecting javascript on a page to enhance notetaking tools. Its like your own chrome extension. Read about how I use it here

Ankitab – Relatively unknown, but its the opposite of Momentum. I use it to review flashcards thoughout the day

DynalistCompanionClipper – I use this with my notetaking app dynalist, I dump all my ideas here as they happen.

Stylus – Add custom CSS to webpages. Since stylish was tracking user data and got banned

VideoSpeedController – So I can play youtube videos up to 16x speed with captions to speedread. Its how I consume information very quickly

Imagus – Hover over an image to see its full view. Useful for my notetaking app dynalist, as well

GoogleDictionary – Select a word to see its definition.

PowerThesaurus – Select a word to see synonym and antonyms. Thesaurus are my favorite business and coding tool, naming is important

ClipboardHistory2 – SelectAll, CopyAll – you now have a saved copy of whatever your typing on reddit / hackernews / wordpress

Lastpass – Chrome extension for managing passwords. Cross platform

AmazonAssistant – So I can dump all my favorited wishlist items anywhere to amazon, export it out later


## Chrome Extensions – WebDevelopment

Wappylzer / Builtwith – To check what websites are running these days, accurate 50% of time

Page Ruler – Measure things on screen quickly

Webdeveloper – Powerful swiss army knife for webdevelopment, but I never use it

ViewportResizer – For responsive webdevelopment, I use this daily. Better than chrome’s native tools. Checkout bookmark version here

Gifscrubber – I need to pause gifs especially if I want to make a codepen of it from dribbble, etc. PlaytheGif is a close equivalent.

RefinedGithub – Better looking github made by Sindresorhus

Octotree – Left side file explorer for github repos


## Android / General

ColorNotes – Every other noteapp I’ve used sucks. I write things here, copy+paste it to dynalist later

GoogleKeep – When I need to quickly sync an image or text from my phone to PC and nothing else works

Meetup – Use the specific link, login, and add URL to homepage. Meetup android app sucks. Shows only the 28 techmeetups I’m subbed to

SolidExplorer – File explorer

Should I answer? – Tells me if phonecaller is a scammer. Not that I pick up phonecalls I don’t recognize anyways, that’s what voicemails for.

Slack – Team communication

OfficeLens – For making PDF’s from documents using your phone camera.

waze – If I know a driving route is slow, I use waze

okgoogle – My favorite commands are: “Play spotify”, “Next Song”, “Navigate Home”, “Exit Navigation”, “Find Parking”

Spotify – Music

Musixmatch – Lyrics for spotify

GoogleCalendar – Calendar


## Content / News

Youtube – I only put an alert on people that post high quality content infrequently. E.G., SmarterEveryDay, MarkRober, TomScott

ProductHunt – Where I like to see all the new fun tools out there

Hackernews – Lots of smart and interesting people here. Also, where I find out about news in the tech environment

Reddit/subreddits – Copy this multihub, delete ones that aren’t relevant to you, add cool ones like r/welding, r/cableporn, r/engineeringPorn

Everyone else here has covered explicit file naming and data structure. That's totally key, but I'm trying to do what you're doing with Keybase[00] right now. Migrating away from Dropbox, Google Drive/Docs, etc.

A caveat is that files are limited to 250Gb currently and private repos are 100Gb personal and 100gb per team. I'm not sure on team creation limits and I don't think they are offering paid storage yet. Because of this, I'm also looking at IPFS[0] for cold storage and using other methods to catalog and index.

They added encrypted private git[1] registries and dropbox like file management[2]. That plus the mobile app, teams (groups), chat, and that it's primarily an identity provider I think it's a killer platform for file management and organization. It's still kind of immature in the chat and friends list area (not in a way that makes it unusable, just a little unpolished), but it's open source[3] and all of the functionality that is there works flawlessly for me so far.

Their recent edition of "bomb" messages[4], where you can send a private message with an expiration, I've found to be super useful. It has clients for all major operating systems and mobile platforms. Source is right there on github if you'd like to modify/enhance. You can even browse your git repos from your phone! They haven't added the ability to commit/push to git from mobile yet.

Now I don't have to worry that if I change a credit card and forget to update dropbox that I'll loose my stuff. Which happened this year. I didn't loose anything, but I realized I had not checked in on my over a decade worth of archives in a bit and hadn't paid them.

I took a look at Keybase a couple months ago and decided to make this move after hearing the Software Engineering Daily podcast with the found Max Krohn[5]. It's a pretty good walk through on their goals and current offerings.

Even with it's lack of commercial SaaS flair, I've actually found using it for chats, file sharing, git, and organization to be a fairly large productivity boost. Maybe because it's pretty no nonsense. I've moved half of my work dev chat to it from slack, and I don't miss Slack at all except for video chat/screen share. I especially don't miss it's the terrible file management / organization.

The next phase of my organization, I'm working on connecting it to a personal web portal where I can share and access control the content, but also curate my links, writing, and media I create in an HN/reddit style feed and use Kb to federate to other people's (friends) sites for content to and group that content by relationship and content category. So basically a smarter web ring (Everything old is new again!) with RBAC/circle of trust access control and collaborative versioned content editing.

I'm also exploring other similar tech in tandem or instead of Keybase like IPFS/Mastodon/WebTorrent. So far, for a lot of what I want though, I think Kb has it nicely tied up with a bow, though I doubt it'll scale to any interesting level of traffic without some elbow grease on the ops/storage side.

[00] https://keybase.io

[0] https://ipfs.io/

[1] https://keybase.io/blog/encrypted-git-for-everyone

[2] https://keybase.io/docs/kbfs

[3] https://github.com/keybase

[4] https://keybase.io/blog/keybase-exploding-messages

[5] https://softwareengineeringdaily.com/2017/10/24/keybase-with...

While keybase has been reliable so far github readme says "It is approaching release ready, though currently it is still in alpha. There may still be bugs, so please keep backups of any important data you store in KBFS. Currently our pre-built packages are available by invitation only."[0] so make sure to have a proper backup of your most important files.

[0] https://github.com/keybase/kbfs#status

Also one thing which bothers me is online only mode, you can't cache the data offline and in Europe those speeds are far from great (around 20Mbps for download).

I have a large collection (it's 1000+ articles, pdfs, photos, notes, etc) of mostly "research" for several fiction books I'm writing as a hobby and also random ideas and DIY inspiration, etc. and although nothing has solved it yet and my use case is slightly different, I keep seeing people mention Evernote and similar programs, and I doubt it will work for your use case. I have tried:

- Evernote - It's been a while, but at the time it had no highlighting. Formatting text was also a bit of a pain. Attaching large files was a problem because of free limits (I didn't care about sync but you need it for clipping). But the biggest problem was organization. The notebooks were not enough and it was really slow having everything in one account. Tags did not really help the organization problem either because what I really needed were custom fields. Search was slow and it was hard to be precise because there's no custom fields to be precise about. Additionally for any actual writing, there's no note history on the free plan.

- Most of the Evernote alternatives - Although I didn't move my collection to them, I did try every alternative I could find. If there's any in specific you're considering, I have probably tried it. The problem with most is they were either too simple, buggy, old/unmaintained, or just plain Evernote clones, which already wasn't working. Also many good ones that might have worked were subscription only because they were cloud solutions with sync. Then some sounded promising on paper (wiki-like alternatives), but they would just make me waste time linking and weren't really what I needed. Also many use only tree organization which just does not work for me. Or they don't support attachments well. TagSpaces looked really promising for an on-disk solution but didn't cope well with moved files when I tried it.

- Scrivener - This is what I ended up moving to. It's not the most intuitive piece of software. Development, especially for the Windows version is incredibly slow. It used to not have sync, now it sort of does, but it's not like Evernote (no clipping). I don't really care for sync though, also I like that it was a buy once sort of thing. Now it's still pretty slow with search (made worse by the fact it sort of freezes instead of showing some loader), but I can divide what used to be my main Evernote notebooks into completely separate projects. You can also have custom "views" (lists of notes) which helps reduce a lot of searching. You can also easily search just one specific field although I do kind of miss Evernote's query language. What really sold me though was you can have a completely custom organization scheme. You can add different types of labels/tags, you can change the note icon, you can color code notes, but best of all you can set custom fields (it's like a really fancy database really). You can link to other notes. You can have a custom note card summary for the cards view. There's several different view types (list, same tree-depth together, note, card/corkboard). It was designed for writers, so it's easy to write in and keep everything in the same format, etc (all copied articles are nicely formatted about the same). You can take as many history snapshots as you like. Best of all you can have huge attachments. I have videos, pdfs, etc (known image formats and pdfs, you can see inside the program, videos, and unknown files open externally). You can also set a custom theme and shortcuts which was also really important for me (Evernote still doesn't have a dark mode!). Also although the on disk project structure is not exactly easy to navigate, it exists (i.e. attachments just get copied to a folder in the project, all notes are plain .rtf files) so you can use Drive/Dropbox to sync to other computers or edit notes externally.

I'm working on my own custom solution now, but Scrivener has worked the best for me for notes, articles, and "clippings" if you have a lot of that. I would not used it though for ebooks, or movies. Ebooks I use Calibre because it also has the tools to edit them, though you could have ebooks in Scrivener if you want, and Calibre as the reader. Movies I have only a few, and just use a folder. For quick notes I use Google Keep or Simplenotes. Then every once in a while will clean them and extract the good ideas to Scrivener.

I use evernote.

Pinboard: https://pinboard.in/u:shawnpresser

Also, throw everything into your ~/Downloads folder and use ripgrep.

Its useless. Just let your brain do it and ull be able to find the information back when u needit.

Im not kidding. For any task you dont need more than 5 references. Dont keep useless volumes of stuff and move on.

Although one has to be careful that collecting these doesn't turn into a form of hoarding, it is useful to keep a list of useful/interesting stuff to search later. Especially for things you wouldn't have found just by using Google alone : e.g. after reading some HN posts there are a few "gems" that you haven't heard of before, those are worth bookmarking because they'll be hard to find later on. For things that are easy to find on Google anyway, or not relevant in a couple of years time, I agree that it might be beneficial to just delete it. At some point I realized I had 5000 bookmarks, went through the titles, thrown out most and was back to a 1000 (still a lot, but I intend to prune it more).

This isn't a task you should be doing. Simply put, an OS should keep record of where you got things from, and where it was sent/copied to. That info should stay with you when moving between computers. It should try to tag the files accordingly.

Disk indexing isn't enough to answer "where is that file", when you moved it for example to cold-storage.

Note that by this definition Windows is not an OS :) Linux came closest with the failed Nepomuk.

You need to master ONE method sufficiently and you may easily become satisfied.

Simplest method there is is the file system and files. That method is offered on each computer. Make your own hierarchy.

In general, I suggest 2 folders, one for individuals, one for groups.

In the folder of individuals, you are sorting by name or also by their ID number their files, files that arrived from individuals.

You may symlink the individual folder to a group file, thus knowing which individual belongs to which group, or individual may be symlinked to multiple groups.

Within groups, you can have further hierarchy, those can be groups of knowledge, for example "All about God" put in one folder with the hierarchy below, there you can place your research on Elohim, Summerians, or Annunaki, whatever you may want.

Other folder may be for your personal mind development, you place things underneath.

Some folders may be related to the humanity, groups, countries, and similar.

Sciences can be organized that way.

All you need is a computer, text editor, and file system, and knowing how to use the file system.

People placing everything on "Desktop" have yet a long learning period to go to understand why is there a file system.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact