
Show HN: Interface for all digital aspects of my life - karlicoss
https://beepb00p.xyz/hpi.html
======
simonw
My version of this is called Dogsheep, and involves building tools that suck
my data from various sources into SQLite databases so I can query it all using
Datasette.

[https://dogsheep.github.io/](https://dogsheep.github.io/)

I enjoyed reading this author's essay on why he doesn't think loading
everything into relational databases is the best approach:
[https://beepb00p.xyz/unnecessary-db.html](https://beepb00p.xyz/unnecessary-
db.html)

I think I have a workarounds for most of the issues he describes there. One
example: I mostly create my base table schema automatically based on the shape
of the incoming JSON, and automatically add new columns if that JSON has keys
I haven't seen before.

Very thought-provoking writing (and software) here.

~~~
mholt
Cool, that's similar to my project Timeliner:
[https://github.com/mholt/timeliner](https://github.com/mholt/timeliner)

It downloads your data from social media (or other online sites) into a single
sqlite database. It can even draw relations between various accounts/people. I
use it to back up my Google Photos.

~~~
anantse
Oh wow that's awesome. This sounds a lot like my project, in philosophy and
practice: [https://github.com/NickSto/life-
browser](https://github.com/NickSto/life-browser)

I've focused on Google Takeout exports so far. Only gotten Hangouts and Voice
working at this point.

I feel like all of us need to be working on a common codebase for all these
parsers. I just found out a week ago that the Hangouts format changed
significantly and totally broke my parser (which I actually adapted from
[https://bitbucket.org/dotcs/hangouts-log-
reader/](https://bitbucket.org/dotcs/hangouts-log-reader/))

~~~
karlicoss
Thanks, I'll check it out!

Yep, agree about the common codebase, that's why I'm trying to keep everything
modular and thinking a lot about it. I describe this bit of my
philosophy/design here [0].

I've only written parsers/exporters if I haven't found any (or any that
wouldn't be a complete pain to adapt) so far. Google Takeout processing is a
part of HPI, but I was going to extract it in a separate repository, so
perhaps we both could benefit from it.

[0]
[https://beepb00p.xyz/exports.html#bindings](https://beepb00p.xyz/exports.html#bindings)

~~~
anantse
Hmm it's interesting how diverse the data sources people are actually using,
but there's definitely overlap. I also have tried searching before I write
anything myself, and often I find something (both Hangouts and Voice were like
that). But it's funny how my use case is always significantly different, so I
end up having to put in plenty of work on top of what they've already done.

I suppose one thing that could be universally useful is common documentation
about what we've discovered in reverse-engineering the formats. I often have
extensive notes like this: [https://github.com/NickSto/life-
browser/blob/master/drivers/...](https://github.com/NickSto/life-
browser/blob/master/drivers/hangouts/hangouts.txt)

I remember finding a wiki about this a while back but I can't find it right
now..

------
nicbou
I have been toying with a similar idea for a very long time.

I want to have an all-encompassing personal history for the same reasons I
want good access and error logs. I want to see when something wrong happens,
and I want the means to diagnose the problem and fix the damage.

A few good examples:

\- Did I forget to write down any tax-deductible expenses?

\- What is my net worth? Is my current lifestyle sustainable?

\- Which parts of my city have I yet to explore?

\- Can I search all my conversation with John across 3 messaging apps, email,
SMS and phone?

I took a few jabs at these problems. Invariably, the services I consumed data
from shut down their API or hid my data better. In some cases, I wanted to use
a different service, but didn't feel like updating my scripts.

I'm currently looking into this again, but this time it's also a matter of
privacy. I want to own all that data I create. I'd rather integrate tools
under my control than services who actively try to hold my data hostage.

~~~
chrisma0
> Which parts of my city have I yet to explore?

> Can I search all my conversation with John across 3 messaging apps, email,
> SMS and phone?

These are totally pain points for me / questions I'd love to have some answers
for... Anyone tried to get the historic iOS location data to answer the first
question?

~~~
ajphdiv
There is a open source digital forensics project called APOLLO
([https://github.com/mac4n6/APOLLO](https://github.com/mac4n6/APOLLO)) -- that
could be used for personal use to get location information off your phone.
Here's a blog post by the author, with a little bit of explanation:

[https://www.mac4n6.com/blog/2019/8/21/i-know-where-you-
were-...](https://www.mac4n6.com/blog/2019/8/21/i-know-where-you-were-today-
yesterday-last-month-and-many-years-ago)

~~~
chrisma0
Oh, awesome, thanks for the links!

------
ozim
One day you wake up and realize that you spent your life maintaining your
system to improve your life instead of actually improving your life.

~~~
twicetwice
Only if you think of "maintaining your system to improve your life" as work. I
would think of it more as play. It's not time wasted if you enjoy it.

Hm. I actually don't agree with that sentiment as written. It's not time
wasted if you enjoy it and have something to show for it? That's not it
either.

Maybe this. I view the work we're discussing like a programmer's equivalent of
woodworking in your garage. Sure, it's not necessary. But it's creative,
fulfilling, and enjoyable. There's something immensely satisfying about using
something you made with your own two hands, even if it's not perfect. As long
as all these things remain true, I wouldn't call it time wasted.

To me your critism reads something like, "if you wanted to canoe, why did you
spend a year making your own? you could have bought one and spent that year
doing what you really wanted to do!" Well, I wanted do that, but I also
enjoyed making something to do that too. I feel my life is better for having
made it.

~~~
karlicoss
> It's not time wasted if you enjoy it and have something to show for it

Yep, agree! I really enjoy it, my only problem is that there are too few hours
in day :) I guess it depends on the goals though -- it's perfectly fine to
build something just for the sake of it, as long as you have fun. One of my
goals through is to stop/pause the active phases of building and spending more
time using it. Partly because of the lack of time, partly because it means
iterating and reflecting on whatever I've already built. So ultimately I agree
that it's important to improve your life instead of building something that
may improve it one day.

I mean, do I want my time back? Yes, sure, like with any learning in hindsight
it always feels that I spent more time than necessary om this.

But the thing is, I did improve my life while building this! I've learnt so
much: obvious things like building better and more resilient software to my
preferences in terms of tools and services. I've started the blog to share
this system and the related things, which made me appreciate writing, and I
feel like I'm getting better at it. The whole quantified self thing was super
useful: it motivated me to think and learn how my body works, to eat healthy,
etc. However stupid it is, self-tracking gives me extra motivation for the
regular exercise and trying to pish my limits.

Every new bit of data I'm adding is easier and smoother, so I can easily
imagine myself using this system for years (and fixing minor bits that break
once in a while, just like people fix things in their homes). Sharing it also
means I can potentially improve others' lives, which makes me regret the time
I spent building and researching much less.

> "if you wanted to canoe, why did you spend a year making your own"

I feel it's more like "you can rent/borrow a canoe, but you only have a spoon
instead of a paddle which you can't switch. Oh and it can also collapse
anytime. Why would you spend a year making your own?"

------
rl3
Fantastic job with this! Feels like I've stumbled upon a [vastly more
productive] clone of myself.

The best way I can describe this type of thinking is a very strong, overriding
compulsion to think and live in an augmented, external, hivemind-like
capacity. It's beautiful, but it certainly isn't normal.

It makes one wonder what the etiology of such thinking is. Were I to guess,
based on personal experience: hypomania, ADHD, obsessive tendencies,
indignation at tech platforms sucking so bad, technology fracturing our
fucking minds. Any combination thereof.

> _I 'm not willing to wait till some vaporwave project reinvents the whole
> computing model from scratch_

Sorry, turns out it's kind of hard! :)

~~~
metalliqaz
>Feels like I've stumbled upon a [vastly more productive] clone of myself.

yep, same

------
A4ET8a8uTh0
I am of two minds about it.

First off, there is something deeply satisfying about seeing your stuff
properly categorized and easily accessible when needed so I definitely
understand the impulse. And the result looks like it does exactly that. Kudos.

There is a part of me that dislikes this level of insight someone can have
into my life and habits ( yes, it is intended for owner use, but the data is
there, neatly organized for someone to access ).

It is a weird comment for me, because I absolutely see the benefit of this
project.

~~~
karlicoss
I can see your concern! I guess I've got several thoughts on it:

\- One is along the lines of post-privacy:

Lots of the stuff that I'm collecting one can find online anyway if they
deliberately search for it. Tweets/reddit comments/instagram photos, etc. Some
of it isn't public, let's say, Amazon purchases. But would I mind sharing it?
Maybe not, who cares what I buy? Some ML algorithm that would show me ads?
Whatever, I don't click on them anyway.

The more stuff I don't mind to be public in the first place, the less is the
chance someone can use it against me. But I understand it's ultimately
something very personal.

\- Second is that I agree that the security professionals would handle the
data much better than me.

I don't even mind Google keeping my data, if only it was easier for _me_ to
access it when I need it! So generally I would happily pay professionals for a
service to keep my data safe, being able to access it and integrate together,
and for the infrastructure around it.

Same way I'm using a bank to keep my money safe. But such infrastructure for
data simply doesn't exist at the moment, because there is no demand for it
from people.

And silos only benefit the companies that keep this data, so the change isn't
going to be come from them. E.g. if you can easily access your tweets through
a nice and fast local app, why would you even visit bloated twitter.com, which
is trying to 'entertain' and 'engage' you?

I'm hoping that with projects like this, I can inspire people to think
differently about their services and tools, and _demand_ (from professionals,
i.e. engineers/designers/product managers) for better means of using their own
data.

------
AJRF
Polar opposite - Does anyone enjoy digital amnesia? I feel like I might be a
little weird here, but every so often I'll just go in and delete all my notes,
reminders, apps, photos, etc and start again.

Wondering if anyone else does this?

I like doing it because it helps clear my head, even though the files or data
are digital they take up some space in my head. I do it every few months, just
have a fresh slate. I wonder if that says something about me - I'm not really
sure. It's probably just another form of procrastination.

I once read Stephen Wolframs post about how he has a key logger that backs up
every keystroke he types and its all indexed and he has a little front end
over his enormous amount of data and it made me feel anxious just even reading
it.

~~~
graphpapa
I do something halfway in between - i routinely get frustrated with everything
that’s not relevant to my present concerns, and just throw it into a folder
called ‘archive’ and then move THAT into the big central archive folder.

I never really look back through any of it but when I do it’s quite nostalgic.

~~~
AJRF
I should probably start doing this for the (very) few times over the years
I've regretted deleting something.

------
Communitivity
Kudos - this looks like a great project and is fully usable now.

I have a similar project, DL, that's unfinished. Mine revolves around using a
custom API in both Rust and REST to aggregate all my digital life events using
ActivityStreams 2.0 and extensions to that, in a manner that is decentralized
and ranked/categorized through machine learning. I am still working on it and
releasing it Open Source is one of this year's goals.

My motivation is that the amount of information I receive from
Twitter,Mastodon,Facebook,Reddit,HN,various Stack exchanges,blog postings,etc.
has gotten to the point where it's too easy to miss things.

Jeremie Miller, one of the creators of XMPP, had something similar revolving
around the Telehash protocol. As far as I can tell, that effort is
discontinued, or at least no longer Open Source.

~~~
21stio
Hey, we are building something similar. Please checkout
[https://metamate.io/blog/most_advanced_hackernews_api](https://metamate.io/blog/most_advanced_hackernews_api).
For now, we concentrate on reading public data from various social sources. We
just released a HackerNews service and going to add more over the next couple
of weeks. Here's a little application build on top of a MetaMate
[http://showcase.metamate.io/hackernews-
activity](http://showcase.metamate.io/hackernews-activity). Please let me know
what you think :)

------
Neph
This is awesome, I'm working on an personal, self hosted, IoT System from
scratch where I have several fetch agents in huggin to get data and store it
in "The Archive" (DataStorage Server)

Than I have different API's to that my IoT devices call to fetch the data from
The Archive to trigger events and generate dashboards.

This might be a great way to optimize it and save my some development time for
things that I don't have yet implemented. (Twitter archive, messages, etc)

Some examples of things already implemented:

\- Raspberry Pi Touchscreen on my desk with dashboards

 _- > Calendar_

 _- > Financial Dashboard (Net Worth, Stocks, Expenses)_

 _- > Reminders (ongoing to do list, train delays etc)_

 _- > Dashboard with my indoor air quality/temp and plant sensor_

\- Self made night light + alarm clock

 _- > Calculates time I need to go to bed to get 8 hours of sleep and
indicates it with a red light._

 _- > In the morning triggers alarm and reads highlight, light shows weather
forecast_

~~~
21stio
Sounds interested. Do you have a link to share?

------
amelius
If only we had well-maintained open source projects for scraping every data
type.

~~~
the_duke
Yes please.

I've been working on something quite similar to this project (in Rust), and I
had the same need.

I started tinkering with a library + DSL that describes data fetching and
transformation, where all data is transformed into schema.org types.

The DSL can then be either interpreted or client code generated for various
languages.

This would potentially make such a API collection library much more feasible.

Sadly such a DSL turns out to be more of a full programming language if all
requirements are covered.

------
alexmingoia
What’s the coolest thing or most useful thing you’ve been able to do as a
result of having all this information?

~~~
karlicoss
I list my most common usecases here [0], but if I had to name just one thing,
it would probably be promnesia [1]. It's an extension that I'm working on,
with the purpose to actually use all this information and integrate it in my
browser, like a remembrance agent.

[0]
[https://beepb00p.xyz/hpi.html#usecases](https://beepb00p.xyz/hpi.html#usecases)

[1]
[https://github.com/karlicoss/promnesia#demos](https://github.com/karlicoss/promnesia#demos)

------
edraferi
Very cool!

Intake [0] is another package that might help here. It organizes a set of data
sources into

(1) plugins that actually connect to the data source and map the data to
standard Python data structures like Data Frames

(2) catalogs that reference the plugins you want to use alongside project
specific metadata like usernames/passwords/source URIs

(3) convenience functions that persistence, concatenation, etc

(4) a GUI for browsing data sources

Looks like OP has already written a lot of data access logic... I might fork
them into Intake plugins for my personal use.

[0]
[https://github.com/intake/intake/blob/master/README.md](https://github.com/intake/intake/blob/master/README.md)

~~~
karlicoss
Wow, looks interesting, thanks for the link. Please share if you integrate it!

------
chrisma0
The linked post and video (including a demo) of Andrew Louis's 'Memex' project
is really neat as well!

Really cool interface there:
[https://hyfen.net/memex/](https://hyfen.net/memex/)

~~~
hyfen
Thanks! Had a nice chat with karlicoss earlier this week to compare notes.

------
vbsteven
Cool project. I've been toying with and thinking about similar ideas lately
but more focused on my productivity as a freelance software dev/ops person.

My approach is basically to view myself as a distributed system that receives
various inputs (jira tickets, PR comments, slack messages, emails) and
performs work that produces outputs (PRs, emails, PR reviews, deploys). I'm
trying to model workflows for all types of work I do, and write plugins that
process various inputs and outputs so tasks can enter and progress through
these workflows.

------
doelie_
Cool project. I have been on the lookout for something like this.

The idea of exocortex seems to be something that many people want, and almost
universially re-invent because the way in which structuring makes sense is
different for everyone. I like the approach of making it modular so people can
pick and choose parts and do their own integration.

What I learned from thinking about this problem is that this kind of
integration is really hard and almost impossible to tackle without an
evolutionary approach, and it made me go meta and focus on incremental
development of the integration layer first. My interpretation of the idea of
exocortex might be a bit different in that the focus is not so much about
information per se, but more about systems that interact with the physical
world. Here's a writeup in case anyone is interested in where it took me:
[https://github.com/zwizwa/erl_tools/blob/master/doc/exo.org](https://github.com/zwizwa/erl_tools/blob/master/doc/exo.org)

------
samanator
This has been on my development todo list for a year! What are your thoughts
about allowing others to contribute new modules?

I'm interested in a my.spotify and a my.film.traktv

~~~
karlicoss
I don't mind merging new modules at the moment, so would be happy if you try
it out and contribute! But it's also possible to use your module without
merging in the upstream HPI package [0]

In the long run, I'm not sure how sustainable it is, since there are many
different data sources and ways of representing them. This is also
particularly difficult when you don't use the service, e.g. as a maintaner I
wouldn't have any means of testing Traktv.

So far I've kept all HPI modules close, because a monolith is easier for
prototyping and refactoring. But my fear is the fate of oh-my-zsh, or
spacemacs, which are overwhelmed by the pull requests.

Ideally I think it should be a simple core, only containing the common utility
functions, extraction helpers, error handling, caching, logging, that sort of
thing; and make the rest third-party.

It's possible to achieve this in Python, thanks to the namespace packages. The
only problem I see is managing these small individual packages, and declaring
dependencies between them. This is possible with PIP and setuptools, but there
is certain overhead involved, I feel like this step is ought to be simpler,
especially to people who don't want to fully dive into Python.

[0]
[https://github.com/karlicoss/HPI/blob/master/doc/DEVELOPMENT...](https://github.com/karlicoss/HPI/blob/master/doc/DEVELOPMENT.org#modifyingadding-
modules)

~~~
simonw
Have you looked at Pluggy much?

It's the Python plugin system that was spun out of pytest. I'm really
impressed by it - it's a very clean design, and integrates great with Python
packaging.

I'm using it extensively for Datasette, which means myself or others can add
new features to the core software without needing to ask permission and in a
way which supports trying out crackpot new ideas without sullying the design
of the core software.

I've been thinking about usi it for my Dogsheep personal analytics suite too,
which is currently split up into a bunch of separate tools in separate repos
(since the only unifying interface is that they all spit out SQLite
databases).

~~~
karlicoss
Thanks, looks great, I'll check this out! So far I haven't even done proper
research of potential solutions, because I feel like it would be
overengineering at this stage, I'm still figuring out the core.

~~~
simonw
When I decided to add plugins to Datasette I asked around and pluggy was
pretty much the universal recommendation - with hindsight it's worked out
really well so I'm happy to pass on the recommendation.

------
srat17
Cool!

Offtopic: how did u create the blog? Did you use WordPress? If so, which
theme? If not, what UI library did you use?

I ask because I have a blog myself which looks like crap haha.

~~~
diffeomorphism
A right click, view source mentions hakyll
[https://jaspervdj.be/hakyll/](https://jaspervdj.be/hakyll/)

Probably in combination with org-mode
[http://wikemacs.org/wiki/Blogging](http://wikemacs.org/wiki/Blogging)

~~~
karlicoss
Yep, most sources are org-mode, there are few ipython notebooks too.

Hakyll in HTML sources is actually just an artifact left in my template, from
the times when I did use Hakyll. Should probably remove it.

I found it a bit overkill for my purposes and often overly restricting me, so
I've switched to a Python script to generate everything:
[https://github.com/karlicoss/beepb00p/blob/master/src/build....](https://github.com/karlicoss/beepb00p/blob/master/src/build.py)
It's a bit ad-hoc, but ended up the same length as my old Hakyll code, and
allows me to experiment much faster.

------
ybbond
I just noticed about "own your data" movement ever since I know about
IndieWeb.

Your work is a great example for that and I like how you display them

------
nico_h
Reminds me of the Feltron report[1], in the self-quantization realm. The
person did it for 10 years and stopped in 2014. The accent was put on the
visualization.

[http://feltron.com/FAR14.html](http://feltron.com/FAR14.html)

------
andrey_utkin
This is amazing.

I am putting as much as i can into my TODO+archive database (see
[https://github.com/andrey-utkin/taskdb/wiki/Live-
demo#workou...](https://github.com/andrey-utkin/taskdb/wiki/Live-demo#workout-
dashboard) ), and it is pretty neat already for analysis with querying and
visualization. But your stuff is orders of magnitude bigger. Possibly I will
set up HPI for myself some day.

~~~
karlicoss
Looks cool! HPI is more about accessing data, so it should be possible to
combine the projects. I was thinking of using Grafana too, and adding an
automatic influxdb integration (via
[https://github.com/karlicoss/cachew](https://github.com/karlicoss/cachew),
which I'm already using for sqlite), so let me know if you set up something!
:)

------
ajflores1604
This is incredible. I've been slowly mapping out a similar idea, and gaining
the programming chops to build it, so I thoroughly appreciate all the work
this has to have taken to make real. Heres an older post I made with thoughts
on this topic of centralizing data and what could be done with it:
[https://news.ycombinator.com/item?id=21673846](https://news.ycombinator.com/item?id=21673846)

------
jasode
_> HPI is a Python package (named my), a collection of modules for:

\- social networks: posts, comments, favorites

\- reading: e-books and pdfs

\- annotations: highlights and comments

\- todos and notes

\- health data: sleep, exercise, weight, heart rate, and other body metrics

\- location

\- photos & videos

\- browser history

\- instant messaging_

I've been studying the concept of "life management software" for decades so
thanks for sharing your project and I enjoyed reading your thought process.

The concepts I always think of that the idealized software would manage:

\+ timeline: [past] --> [present] --> [future]

\+ The Big 2: time & money

\+ reflection vs growth : mining the past data for patterns and metrics -- vs
-- managing and prioritizing a future wishlist to deliberately design a future
life

I've come to the conclusion that it's very hard to come up with a good schema
that unifies _all aspects of life_ that's important to me. For example,
looking at your list above, it is heavily populated by digital artifacts
(things that happened in the past). One of the exceptions in your list that is
"future" oriented would be _" todos"_ and possibly _" notes"_.

I'm also interested in life "situational awareness".

E.g. the other aspect of life data is _money_ which means financial
budgeting/planning. Another aspect of life data is _time_ so a digital
calendar of future events -- and goals -- is essential. Yes, I keep a list of
books I've read (the "quantified self") -- but also books I _plan to read in
the future_ (designing a future version of myself a.k.a personal "growth").

There's nothing wrong with your project. I'm just explaining my observations
after seeing various attempts at this over the decades. This includes 1980s
software like Borland Sidekick, 1990s PIMs (Personal Information Managers)
like Lotus Agenda, late 1990s PDA (Personal Digital Assistants) like Palm
Pilot, 2000s software like Evernote, and a hundreds of 2010s SaaS
"todo/calendar/notes" websites, or uber-geek tools like Emacs Org-mode.

None of the above really do what I want so for now, I just split my digital
life across various tools and files. My daily notes in "journal.txt". My
financial planning (and bank & credit card data downloads) in "budget.xls".
Saved webpages in ".mhtml" files. Planned programming projects in
"projects.xls". Etc.

~~~
karlicoss
Thanks!

Yep, totally agree it's hard to incorporate the future, especially when it's
in free-form like org-mode notes. Past data is much easier becuase it's at
least somewhat structured and easy for the computer.

I guess my take on this is that subjective metrics, like the one people use to
define 'success', 'growth', etc., are orders of magnitude harder to grasp than
objective metrics (e.g. hard data, like sleep or excercise logs). Yet, we
don't even have good means of incorporating the hard data in our lives, so I
chose to start with an easier problem.

Yeah you're right, my future planning is all in my org-mode notes, I reflect
on it with my brain, but so far not sure how I can use software (apart from
organizer software) to aid me with it. It would certainly be an interesting
area to explore.

------
adrienchauvet
I think you should check this project form Tim Berners Lee:
[https://solid.mit.edu/](https://solid.mit.edu/)

~~~
karlicoss
Hey, thanks. I'm listing it as once of the links, and I tried using it, but I
kind of struggled to see if it can integrate with the existing infrastructure.

------
pheme1
I wish to see some kind of encryption and token access for each module. A
quick glance of the setup.py, it seems to rely on appdirs to store user data
unencrypted?

~~~
karlicoss
appdirs is just a module to find the config directory in a portable way, the
data can be stored anywhere you point the config to, but you're right that
there is no encryption involved.

Note that it's meant to be running on your own computer, and it's using the
filesystem to access the data, no network interactions involved. Ultimately
it's about how you're protecting the data on your disk and whether you trust
youself with it. Of course, my code has to be trusted too, but it's at least
possible to run it in a sandbox/use Docker, etc.

The modules run untrusted Python code, so if you keep your token on the disk,
they can potentially steal it token too, unless you use some elaborate
authentication system.

------
wakkaflokka
Like a lot of other posters, I’ve been working on something like this off and
on but never near to completion. Thank you for taking it all the way!

------
op00to
I just need a simple way to make sure I am doing the things that are important
to me and make my life better.

------
chrisweekly
Fantastic! Thank you for this. Agreed on most of your points, and stoked to
give your scripts a try soon. :)

------
jackklika
> What it isn't?

> It's not vaporwave

I look forward to the day we replace vaporware with vaporwave, ala
segue>segway

------
random_kris
that looks awesome. I think this area is ready for a great new application. I
am not sure how it could be monetized or what it would even do.

------
GistNoesis
Is there a myDigitalLifeInAGlimpse function that would generate a full
synthetic report, ready to be shared on social networks ?

If you can add some social scoring, it would be great too. Maybe if you have a
way of certifying it too, it would be great indicator to put on resumes to
show how much better you are than those other candidates.

What about the next step ? Have you considered just putting screen recording
(+ body cam if you want to extend it to all life monitoring and not just
digital) and store all the footage. Then it's just some video indexing. You
can always post-process the data as new deep learning algorithms become
available.

This way you don't have to mess up with specific api for each service. You
handle the general case and you are done. (That's what smart people do
[https://writings.stephenwolfram.com/2019/02/seeking-the-
prod...](https://writings.stephenwolfram.com/2019/02/seeking-the-productive-
life-some-details-of-my-personal-infrastructure/) )

Have you thought about plugging an optimizer into it so we could make the most
of our lives ? I have got an Hedonic treadmill to climb.

/s

~~~
karlicoss
Hey, I've got a crude timeline of events (simply rendered as giant HTMLs), but
it's private at the moment. I'll probably share some of it in the future,
especially considering that some of the data is public anyway (e.g. online
comments or tweets), it makes sense.

Next step: yep, I throught about it a bit [0], agree that it would be great as
a service-agnostic way of accessing the data, the same way we can see it with
our eyes.

But figured I should start with strictly easier problems for now, ones that I
can solve at least to some extent. For now my goal is simply enjoying using
it, making the setup simpler (to make it accessible to more people), making
sure it's robust to the changes, flexible enough, etc. I want to inspire
people to own their data. In many ways, I'm advocating an approach to owning
it and working with it, rather than a specific set of tools, formats, etc. It
would be great to have more people experimenting in the same area and trying
to loosely integrate with existing efforts.

Optimizer: to some extents, yes! Not sure about 'general happiness', but I'd
happily let the computer dictate me what to eat, how to exercise and how to
sleep. Maintenance is probably one of the most annoying things in my life.

[0] [https://beepb00p.xyz/pkm-search.html#future](https://beepb00p.xyz/pkm-
search.html#future)

~~~
GistNoesis
I don't like your project. Data aggregation is powerful but very dangerous and
I feel that you are handling it carelessly. I also get a strong Borg vibe
here.

In the current state, people don't know how to handle their data correctly.
You are just making it easier for some people siphon other's people data even
more. And having their data played against themselves.

Most people are not quants. And becoming one is not something trivial. Having
people play unguided with their own data is like giving uranium to little
kids. They will have a great fun !!!

~~~
karlicoss
> making it easier for some people siphon other's people data even more

I guess you mean someone could easily steal your data, etc? I agree, it's a
tradeoff, would be interesting to explore how to make it more secure.

> Having people play unguided with their own data is like giving uranium to
> little kids

Well, handling Uranium is a bit different. Can you elaborate on some specific
examples?

I've seen people misusing statistics, and drawing misleading conclusions,
sure. But even more people are using anecdata and broscience, which I think is
worse.

~~~
GistNoesis
Not even stealing some data is needed. Just sharing publicly some data, may
leak information that can be used against you. Bots do build profiles and will
happily ingest any information you give.

If your top ten "most interesting Slate Star Codex" contains certain keywords,
they will flag you in their database.

>Can you elaborate on some specific examples?

-People having sport injury from those sport performance training app. Their knowing of their data made them push-themselves past their limits.

-Little kids getting bad marks, and assuming they are just bad and not persevering. Their knowing made thing worse.

-Hypochondriacs becoming ill from the stress of all the illnesses they think they have.

-People doing weight loss badly focusing too much on the scale.

-Musk tweets :)

-People censoring themselves from knowing that having an unpopular position will cost them karma points.

As soon as you make a system self reflective, you impact it in profound ways.
You generate some reinforcing feedback loops. That tend to either break the
stability resulting in chaos, or tend to lock it into a specific behavior from
which it's hard to escape.

~~~
karlicoss
Ah I see what you mean. Yeah, absolutely, an aggergate system would enable
such behaviours.

I guess it comes down to a personal level of paranoia. And to the old question
"Is ignorance bliss?", for which people had very different answers long before
the software.

