I started a draft describing my 'process' , but it's pretty incomplete.. there are some bits scattered across exobrain too.
Also lately I started playing with Logseq  to get a more interactive representation of exobrain  (warning, it's pretty heavy, needs some optimization and I might have messed up the physics). Logseq could be a great gateway for people who want to get into org-mode but not ready to go 100% Emacs, highly recommend to try it it!
There are moments that I'm tempted to go down a similar organizational road because I'm curious about the possible lessons from the data aggregation, but I usually pass because the payoff didn't seem large enough.
So that makes me want to ask: Have you learned much from the data? or is it more about organization/keeping control of life?
The "quantified self" bit is what you could call learning, but ironically so far I've mostly failed to find any significant correlations in my personal data (which is also an interesting finding, in a way :) ). However I did learn a lot about nutrition/exercise/sleep while tinkering with data, so it was totally worth it so far.
I write about it a bit more here: https://beepb00p.xyz/tags.html#quantifiedself
Have you considered monetizing this in some way? As a person of limited technical skills, I would definitely pay for a set of tools that would help me aggregate my data.
- I don't really see a good way (not that I'm entrepreneurial in this aspect though), maybe except for providing some infrastructure etc
- dealing with others' data is a completely different kind of headache with legal and security responsibilities
- I don't really have a product or anything -- more of a vision which I want to communicate, a bunch of patterns, techniques and repositories. Maybe know some esoteric gotchas.
- It's a very niche thing to think about and 'want' in the first place (although I am trying to change it through my communication :) ). Most 'regular' people don't do backups, let alone this!
But it would be very cool if someone comes up with a business model to provide such service.
I am working with drawing graphs at the moment and evaluating what library to use, its coincidentally nice to see what you built but it also seems like it was thoughtfully laid out (less edge crossings etc). I am trying to do something more dynamic so it may not be as applicable for me.
I am looking at your code as well, thanks for providing that
There definitely are some weird things when you try to plot complicated things, fighting with weird placement, clusters etc. But not sure if it's me or Graphviz to blame for this. But I don't really know a better tool. If I knew how the diagram would look in hindsight I might have drawn in manually in inkscape or something, but when I started I didn't know what I would end up with, so needed to be an automatic tool :)
To minimize the manual work, I ended up with a mix of DSL in python and raw graphviz commands: https://github.com/karlicoss/myinfra/blob/fc6345c31c4e49b534...
Depending on the things you want to represent a better fit might be force layout, for example something like https://observablehq.com/@morvasaaty/d3-force-notes
Thanks for the code, taking at look at that.
If you need auto-layout, it's hard to do much better than graphviz. The creators of graphviz have some good papers on what it takes to do that right. They'll discourage you from wanting to re-invent that particular wheel, if you read them. Tricky bit here is, depending on your exact graph and constraints, it's not exactly fully solvable in the general case, if you want to have no overlap or no crossing lines under any circumstances whatsoever. Still, you'll struggle to do better, and there aren't actually a ton of implementations out there other than Graphviz that are anywhere near as good (at least, as of a couple years ago).
Templating or building programmatically? Great, it's just text. Generating on demand? Decent, it's fairly fast.
If you need pretty, and I mean really pretty, and especially pretty and interactive... well it's less enjoyable for that, but then nothing that is enjoyable for that purpose is much good at the other points above.
Graphviz seems to be great for getting you a mostly-useful auto layout majority of the time.
I have a requirement to be able to mix fixed-positioned nodes along with dynamically changing nodes, I havent used graphviz to know if it could do that. A library like cytoscape works ok or maybe d3 (I am still evaluating these)
Honestly coming into this, I thought graph drawing would be fairly easy and I could look up online for examples - but it seems to be quite opposite, its really complicated to get graphs that are both dynamically generated with the nodes positioned at the same place in case there are similar nodes
For exporting and owning the data the keyword is "data liberation"
Otherwise, the keywords are perhaps "personal knowledge management" and "memex". Have some links here, although they all have different components in scope https://beepb00p.xyz/exobrain/memex.html#cmmnts
I usually maintain related links at the end of articles, so perhaps you can find something there, for example https://beepb00p.xyz/sad-infra.html#links
Either way don't think I've seen anything better, so the best way would be to suggest them updates or help maintaining :)
Everything in Promnesia  is super useful, makes my internet lurking much ... efficient? This word has somewhat bad rap, but it is what it is -- doesn't mean I don't have fun, just makes it easier to keep track and discover new things.
Not sure about surprising data sources -- I'm trying to integrate them all as far as possible and make sure they work for me passively, so I don't have to think about specifics :)