
The idea maze of personal logging (2016) - pcmonk
http://pcmonk.me/2016/10/13/logging-idea-maze.html
======
conradev
This is a very interesting read, because I took the opposite approach.

I started out with the premise that manual logging is very hard to do well. It
takes effort on my part (and thus hinges on my discipline), can be
inconsistent, and can change depending on what my important metrics are.

I automatically collect as much information as possible. I log every time I
change a window or tab and what I change it to (extending from native windows
on macOS all the way through to virtual windows in gmail.com). I log the same
from my iPhone. I throw all of my data in a Postgres data warehouse, and then
use a BI tool to normalize it and put some KPIs in a dashboard.

This system allows me to ask anything from "what articles did I read last
week?" to "what percentage of my time at work did I spend programming?" to
"how much did I sleep this week?" to "who do I spend the most time
messaging?".

It's also an exercise in understanding that metadata is incredibly revealing.

~~~
pcmonk
Sounds interesting! Sounds like you've hand-rolled this, have you published
anything about your system? I'm sure the details would be fascinating.

~~~
conradev
I haven't yet! I should really get around to doing that.

------
buro9
I treat Google Calendar as a time-series database.

I have 14 calendars logging approx 500 events per month, and I have a UI for
all of this... Google Calendar.

I also have API endpoints for all of this... Google Calendar. Which can be
triggered by IFTTT for automatic stuff (i.e. did I Instagram something or
Tweet?).

My email is linked to Gmail, so flights, itineraries, online shopping
delivery, restaurant bookings can all flow in.

So the only thing I need to do manually are the few things it cannot
automatically know, such as what film I just watched or what book I finished
reading.

I only log public things, that is... data that exists in someone else's
database somewhere. I do not log private things, i.e. Signal conversations or
anything 100% offline that didn't involve a public database.

For me it's a simple thing... this data exists, can I make store it myself in
a structure that I can do something interesting with it.

I have about a decade of this data.

Oh, and Google Calendar is really long in the tooth and is becoming both slow
to use, hard to search, and clunky.

~~~
johnchristopher
> Oh, and Google Calendar is really long in the tooth and is becoming both
> slow to use, hard to search, and clunky.

In your personnal case (because of the many events) or in general ?

I feel like calendar has been forgotten by Google (no material design for
instance) and could be on the list of next things they chop off.

~~~
buro9
In general.

I don't think they'll chop it because it's critical to the GSuite business
side of things.

But it's probably not receiving love as whilst Ai has seen things like Photos
revamped and the emergence of Google Assistance, the applicability of this to
the business stuff is more difficult for compliance reasons and politics, etc.
It's easier to focus on consumer when innovating.

The weirdest bit of all of the Google offerings, is the ever-diverging
difference between the capabilities of the consumer Gmail accounts and the
GSuite accounts. Google's tagline of "One Account, All of Google" is long dead
in reality, and with it there seems to be a very different focus, innovation
happens to the consumer and not to the business side, and never the twain
shall meet.

Consumer accounts cannot use Google Calendar's "Find a time", and business
accounts cannot fully use a list of products so long that it's boring listing
them.

Google Calendar is central to how I run my life... I even logged phonecalls
with links to the audio files uploaded to Google Drive, as well as scans of
post received (and categorised in Google Drive but linked from Google
Calendar).

But the usefulness is being reduced, silo'd, splintered from the general
usefulness of Android and the Google related family of products due to the
split of Google into consumer-focused vs business-focused with less overlap.

I've not bought a Pixel phone for this reason either. What's the point when so
much of it's selling point is software and that software doesn't work with
GSuite accounts.

~~~
salvadors
> I even logged … scans of post received (and categorised in Google Drive but
> linked from Google Calendar)

Oh, very nice idea. What's your workflow for doing this?

~~~
buro9
Imagine a 1990s batch flow system :)

I have a Fujitsu ScanSnap S1500, I scan through that... it auto names my post
based on timestamp and drops it into a folder.

I rename it how I want to file it within my "Files" folder in Google Drive but
leaving the datestamp intact, i.e. "2016-05-03T07:13:01 Finance - HMRC -
P60.pdf" and I move it into another folder.

A cron comes along, looks at everything in that folder, fires a little Go util
that uploads it to Google Drive (splitting " \- " into subfolders), and
finishes off by adding it as a calendar ever.

This would be a lot easier on Windows or Mac as I think you have Google Drive
clients that sync whereas on Linux this doesn't exist.

I thankfully don't receive a lot of post, but when I do it's only 30 seconds
to scan something and a few seconds to rename and move into another folder.

------
TACIXAT
I made my side project ([https://socialite.ooo](https://socialite.ooo)) to do
personal logging, but I had a slightly different problem. I would remember
small things about people but not their names. I was going to the dog park
with my girlfriend a lot and couldn't believe how often we'd remember
someone's dog's name, or the walk we met them on, but we wouldn't remember
them.

My solution was to correlate people to events, and events to locations. This
allows you to not only remember what you were doing, but pivot off of any
piece of information to better recall an event. Logging into a web app to
enter information is a bit of a slog. I think making a mobile app is really
crucial. That could even lead to some automation of the logging. For example,
"You just called [CONTACT], would you like to create an event for this? ", or
at the end of the day, "You texted [CONTACT] today, give a quick summary of
what you spoke about". From there you can make the analysis smarter and
smarter.

It's really strange, all these companies have so much data about us. We don't
have access to our own data in a decent format though. My call history is with
my telco, my location history is with Google (and probably many others), my
conversations are spread out among so many different apps. What happens when I
want to query the data about me though?

~~~
xkxx
> Socialite is a paid application. We made this choice in order to give you
> peace of mind. With no advertisers involved, you won't have to worry about
> who is getting their hands on your information. Pricing will be targeted at
> $10/year. Don't worry though, we won't charge you until you're making use of
> the application.

> Socialite is a paid application. We made this choice in order to give you
> peace of mind.

That sounds very funny. I would get much more peace of mind if you said that
your client app is open source and uses strong encryption before saving data
to your server. If you pay somebody, they must be trustworthy? That logic is
laughable.

~~~
XorNot
It it's free, you're the product, not the customer.

~~~
dublinben
If it's proprietary software, it uses you.

~~~
Diederich
Can you expand on that?

------
udkl
I remember using AutoHotKey [1] (windows only) to randomly prompt me what I
was up to every hour.

This was many years ago so I don't recall the details but my intention was to
understand where/how I spent all those hours in the day as a student.

While there was no formal syntax that I followed that would have helped to
analyze the logs programmatically, it did help me get a general understanding
of my schedule and identify things I should change.

Ref : [1] [https://autohotkey.com/](https://autohotkey.com/) [2]
[http://lifehacker.com/5582372/use-a-daily-log-to-keep-
yourse...](http://lifehacker.com/5582372/use-a-daily-log-to-keep-yourself-
focused-on-productivity)

~~~
obstinate
Beeminder's TagTime does this, but according as a poisson process averaging
once every N minutes.

------
inetsee
In college I had to keep a life log for about a week as part of a psychology
class. It was a completely manual process (i.e. pencil and paper; this was
quite a while ago), and I was supposed to log in as much detail as possible
what I was doing throughout the day. At the end we were supposed to create
categories from the log, and add up all the time spent in each activity.

The purpose was to find out how much time was _really_ spent on each activity,
rather than what we thought we did. It was an eye opening experience.

~~~
pvitz
I did something similar back in university after reading about Sen. Bob Graham
[0]. For a period of 6 months, I would write down every single action with
place and people involved in a small notebook. At the end of the day and the
week I would review it. It was really eye opening and allowed me to change
some bad habits.

[0] I can't remember which article it was, but this one describes his logging
activities: [http://quantifiedself.com/2009/05/politician-as-self-
tracker...](http://quantifiedself.com/2009/05/politician-as-self-tracker/)

------
anticafiamma
I've been using RescueTime
([http://www.rescuetime.com](http://www.rescuetime.com)) for around three
years now, which pulls data from all of my workstations as well as my phone,
but it lacks the ability to handle soft skill logging like social encounters
and dietary habits with the finesse that pcmonk is working towards. Right now,
I use it to track things like how much time I spend in IDEs, in StackOverflow,
Hacker News, etc.

For some previously unexamined reason, I am giddy about tools that enable me
to surveil my own productivity and procrastination metrics, but feel slightly
uneasy about trying to capture daily intangibles.

I'm not sure if I would want to know how much time I've spent with each of my
friends; the analytical side of me excises and optimizes, and I worry how that
might manifest in negative ways if I start to quantify friendships and phone
calls.

Anyone identify unexpected truffles and/or landmines in the exploration of
your personal logs?

~~~
wingerlang
DayOne could help you, it can add a little popup on your device asking "what
did you do today?" where you could enter little or lots of details of your
encounters which are not loggable.

------
hertzdog
My very first (and maybe not so useful comment): I really love your approach.
I was thinking something more automatic (I have some withings stuff) but you
convinced me to start small and manual. Really interesting. Thank you for
sharing.

------
DenisM
How about screen video capture? Rewind at 60x at the end of the day and take
notes in whatever structured form you want.

It would also help to have an app that reacts to inactivity, that is you
stepping away, and ask you what you did.

This works well if your day revolves around a desktop or maybe Android, but
not so well with iPhone or iPad.

~~~
pcmonk
Sounds like a fascinating idea, but I suspect a video will be unmanageably
large. However, a relatively low resolution time-lapse would probably be
sufficient.

~~~
Houshalter
You could sample say one screenshot every minute. It wouldn't take up too much
space.

~~~
paulryanrogers
Those files will add up very quickly. On Linux I had screenshots for every
click for months and the results were numerous enough to bring 'ls' to its
knees

~~~
yjftsjthsd-h
I think the idea was to analyze and cull at the end of every day

~~~
Houshalter
Well the goal was to make a video. The screenshots could be compressed into a
single video file. At one screenshot a minute, and 24 frames per second; 24
hours of screenshots would give you a one minute video.

------
tyingq
Really well thought out. Would also map well to an Alexa skill, like:

Alexa, tell plog I ate a burrito at 8am.

------
cbanek
Being able to log and analyze your data is great. It not only provides
insight, but also provides motivation ("I'm logging this to improve. Let's not
screw it up." etc).

I think my three big tools that have helped in this space are: my fitbit
(tracking exercising), habitica for recording daily tasks and todos, and
private journaling at the end of each day.

I've been terrible about routines and such, but all of this has really helped
get things back on track.

Overall, I think also logging has to feed back into both organization and
action, such as new goal setting. Which adds new things to log.

------
billwilliams
This is great. The structure and simplicity is bomb and the data will be easy
to mess around with while still being easy to clean up even if you mess around
with verb usage like "scarfed" instead of "ate".

------
M_Grey
There's some evidence that this kind of behavioral monitoring can be useful in
a number of areas, especially weight loss, and other gradual lifestyle shifts.
This seems like a particularly smart way to implement that.

------
hashemian
Most of the human-subject research done in public health, sociology, and
similar domains uses data collected in a similar way. They basically ask
people to take notes on specific behaviour or fill short surveys multiple
times a day (they call it ecological momentary assessment), then they analyze
it in the context of their research.

So yes, a tool like this has value in helping with better understanding
ourselves, but also can help population research as well.

------
kornakiewicz
GUI is not a problem, you can always tweak it or delegate it for little money.
The big problem is how to force yourself to do it regularly in long term, at
leat in my case.

I think that good idea is to integrate it with bot on your IM platform of
choice. Messenger bot, in my case, could ask randomly what I'm doing and track
it.

------
spdustin
I've used Timing.app on my Mac for a loooong time. Love it.

[http://timingapp.com/](http://timingapp.com/)

------
ibgib
ibGib's current web app used today by my girlfriend and myself for personal
logging while our shower is getting remodeled:
[https://www.ibgib.com/ibgib/Saturday%20II%5E818E0366E6B009CA...](https://www.ibgib.com/ibgib/Saturday%20II%5E818E0366E6B009CA0F71083DDD001EC99E5B0E708795B9F7FDBBF43C29225220)
_(NB: After a temporary session "login" at the bottom of the page, you will be
redirected...also, the links are long because the last part of it is a sha-256
hash that guarantees the integrity of the content of the ibGib data)._

I post this because I think personal logging is a perfect fit for ibGib's
engine (and it's partly why I've written it), and it may interest you as a
back end. It would allow you to basically have a graph database for a back
end, while also allowing you to create and evolve your semi-structured data
much like evolving an Event-Sourcing event in version control like Git (since
each and every datum is versioned and maintains a complete audit history). It
also is designed to be Big Data friendly, with the ibgib website's datastore
acting with _completely_ Open Data.

Also, I wanted to mention that I've actually used it recently in the exact use
case given by @TACIXAT where I was walking my dogs in my neighborhood and I
wanted to be sure I remembered the neighbors' names down the street when I met
them a month or two ago
([https://www.ibgib.com/ibgib/Walk%5EE3EE1755557357849C41C1511...](https://www.ibgib.com/ibgib/Walk%5EE3EE1755557357849C41C151117424D87EEB806D28216987B0F92F71ED1A2A39)
\- just doule-click the big yellow circle). I definitely like how the
socialite app looks, and it's great in that it helps you remember things _in
context_. This is also what is primary about ibGib, allowing for pictures,
text, links, etc., to be nested "inside" other "objects" (so each node acts as
both a file with content and a folder with relationships to other nodes).

So for your personal logging, you could create a semi-structured "type" with
your subject-verb-DO-etc. and then create instances of these. Then you can
relate them via any named relation to any other construct (called an ibGib).
So it's very much like a graph datastore that also has data hashing for all
"children" nodes, which makes it a merkle graph. There's all sorts of neat
properties, and like I said, I thought your use case would be a good fit for
the engine. (Btw, the structure also lends itself to auto-generating dynamic
and collaborative blogging from your data, e.g.
[https://www.ibgib.com/ibgib/WaffleGib%20Jr%5E59B6D3828D35209...](https://www.ibgib.com/ibgib/WaffleGib%20Jr%5E59B6D3828D352091B0251F2A017511C2A681BE300E179398F4D7CB8BC56B650A)
or
[https://www.ibgib.com/ibgib/Grillin%20burgers%5E9E0151A452C1...](https://www.ibgib.com/ibgib/Grillin%20burgers%5E9E0151A452C17F5814602D9A119865A60E12E53E8A9819FFB41DE33C0D05CDBB))

