On: The sad state of personal data and infrastructure

[simonw]

I've been working on a project along these lines recently. I've called it Dogsheep - the basic idea is to have scripts that export all manner of my personal data (from Google, Apple HealthKit, Twitter, LinkedIn etc) into SQLite database files, then use my Datasette web app to browse them and run interesting queries.

More about that here: https://dogsheep.github.io/

The tools I've built so far are under https://github.com/dogsheep

[gpchelkin → simonw]

I think that Perkeep https://perkeep.org/ is also worth mentioning here. One of their latest tools is the one for exporting Google Photos: https://github.com/perkeep/gphotos-cdp

[ocdtrekkie → simonw]

Whoa, this is super exciting to me. I've been looking for something that would get some of these types of data into a web-based interface that I could package for Sandstorm. This looks like it would potentially work for that fairly easily.

[rasz → simonw]

I went similar route after discovering Google Checkout refuses to return my YT comments. I wrote quick&dirty tempermonkey script logging submitted HTML forms (addEventListener "submit").

[TurkishPoptart → rasz]

Did it work the way a scraper does? Curious to know how you did this.

[rasz → TurkishPoptart]

I was only interested in stuff I posted on the internet for indexing/easier search, https://developer.mozilla.org/en-US/docs/Web/API/HTMLFormEle... and http://www.meekostuff.net/blog/Overriding-DOM-Methods/ covered form submissions, result goes to tempermonkey localstorage (GM_setValue).

[karlicoss → simonw]

Looks cool, I'll check it out! Might be good to join forces on scripts for data retrieval/parsing.

[austil → karlicoss]

Hey, I'm on this journey too ! https://github.com/austil/datapuller

[jakeywankenobi → simonw]

Nice. Any interesting use cases for the genome one?