Right now, things are fairly sparse. You can check the repo out here: https://github.com/jdjkelly/bodyimage
An open source Rails API to track "quantified self" data.
Right now, the hardware companies in this space are storing your data in closed systems. Let's change that.
Take the Withings WiFi scale or the FitBit, for example. Both are essential pieces of the quantified self hacker, yet both store your data in their respective, siloed databases. What happens when one of them goes bust? Or decides to pivot towards an ad-based business model?
This project includes a Rails + MySQL RESTful API to store your data.`
Contact in profile.
Tent is an evented data store for arbitrary JSON blobs which can be typed b the developer. It has the added benefit of operating as a decentralized system that lets you push and receive posts with other users via webhooks.
Tent can be used to create applications and systems from the quantified self to microblogging to cloud-backed file sync to collaborative real-time document editing.
All we need is a place for storage and we add services like apps.
With ipv6 around the corner everyone will get his personal ip attached anyway so we can build upon that.
Many of us have this idea but no one made a breakthrough so far but it will emerge. We have not figured it out quite yet we have the technologie but we
need to get the idea of how we glue everything together.
I will look forward to the day when i pay a certain amount of money per year to have my own node without being the product of some megacorp thats only interested in quarter performance.
Since you said you would pay for such a thing, can you elaborate how much and what you'd expect (in terms of functionality)?
The basis everyone would except are normal social network features sharing of "items" galleries friend system feeds and groups.
Maybe this part should be an open source solution like for example diaspora.
On top of that you can build an app/service network. Maybe something like app.net.
Important on all parts is that you own the data and you should be able to pull it at will from any part of the network.
Then other API calls can be built on top of individual people - College students in the US, citizens of India, etc. The possibilities are pretty endless.
Sorry, maybe I'm being a bit facetious. There's definitely a lot of cool stuff that could be done with a personal API though. But how do you get people on-board? You have to have equipment to track/update stats right? Are people willing to buy into this potential network?
Get your DNA in there too! https://api.23andme.com/
Is there an opportunity here to create a secure data store for your own structured data hosted securely somewhere else and that providers would pay YOU to access?
Of course you'd give free access to it for your doctor who needed to pull some info. But what if you had total control over your own API. This is a kind of like Facebook but I would never let facebook hold some of my most personal data. I would like to control authorization to my own data though to certain people.
Basically he's disrupting capitalization, which I've been thinking about a lot lately.
What I'm trying to say is that with a personal API platform, apps simply become a front end to standardized personal data. Switching costs would be reduced significantly, which would encourage a lot more innovation.
Here's a spec I've written out for my work on my own personal API. It's not done yet, and the code isn't open sourced yet, but hopefully it will inspire others to hack on the idea.
Once app developers adopt the Open mHealth approach, much of the work of getting data out of these silo'ed apps will be done. This is important because the real value of this data comes about when the streams are combined to be analyzed and visualized.
I am posting here because many of the issues outlined in this thread and in the original post are things we are actively trying to solve.
Is anyone interested in using my dashboard app?
* Secure access. It's great you decided to share everything publicly. I'm looking at fairly confidential data like blood tests, though.
* How do you express permissions? It's likely you'd give out sets of data, not per-url permissions. (I.e. my photos from date A to date B, in this general vicinity - your vacation pics)
* How can others discover what data sets & types you've shared?
* Related: A type description. Some events cover time ranges. Some are momentary samples. They all use different units. Some have location info attached, some don't. (I.e. my heart rate doesn't. Pictures do. Steps/day - geolocation is meaningless)
Either way, it's exciting to see people push ahead on this - many of these questions will probably be answered as we go.
Fitbit holds a lot of the data hostage, but it's at least something that anyone wearing a Fitbit can easily grant access to.
See https://wiki.fitbit.com/display/API/API-Get-Intraday-Time-Se... for more information. You can retrieve data in 1 min or 15 min increments.
So if this comprehensive personal dataset becomes the norm at some point in the future - what happens when this is under somebody else's control against your interests (whether an illegal attacker or a legal authority)? And what about more gray areas, such as marketing, insurance etc (which would likely encourage the creation of that kind of API in the first place)?
I've used it to back up and then delete my personal FB account, and continue to archive instagram, linkedin, twitter, blogs, etc. etc.
What's more, even though their interface is kinda weird, they store all kinds of granular data in a very nice SQL database which contains more then they represent in their reports, and can be be repurposed in many different ways...
Express it as json ld to make it more modern, and you have a distributed ecology of services that all talk the same sort of language, about you.
this is why i believe a smart scale is one of the best examples of good data tracking: you step on it whenever you think of it (not on some weird schedule; nor do you have to wear it or take it with you). it does the mundane stuff of capturing the data and sending it to the cloud, which you would otherwise have to write into some notebook.
if it's a system that's fully automated though, one might lose interest in collecting it in the first place: the QSers always like to say – when you add a little friction to a process, you end up paying more attention to why it is. so instead of just blinding capturing your weight, setting up alerts for goals and seeing trends fall and rise will get you to make the most of the data – instead of just collecting data for data's sake.
Namely, set up a commitment device that forces you to keep your datapoints on a path to some goal. That's what Beeminder does.
I had actually mocked up something similar with openapi.me, but never got to release it.
It included different level of permissions and data including addresses and payment information. The goal was to have a centralized repo of data to update only once and have it propagate to all the other services asking for the same data over and over (email, address, links, etc).
Friends/relationships, for checking general activity and check-ins.
Potential businesses, for seeing interests and the like. Or, if currently employed, the employer could randomly check to see how the employee has been sleeping (are they going to be tired/cranky today?) or to see how often they check-in at the office.
Aggregated with more data from people doing similar things and you can start to get a look at populations (what if everyone in SF was logging this data? What if a good chunk of those in the US were?).
Facebook also has a pretty cool Timeline UI to boot which will probably only get better as time passes.
you could probably do it today without the use of a private account
i believe in app settings, you can restrict the visibility of posts by any particular app. this way, your scale's posts could be seen by just you and not your friends.
I suppose API access to that would be a logical extension - this is very cool.
At CES, Qualcomm had a bunch of devices working with it through a plugin wall unit. They offer free data APIs for storage, etc..
Storing, doing simple metrics, making assumptions and returning the data is not a problem. Standardizing input methods is.
Faced with legacy systems communicating (or not communicating) via various channels, Middleware, ESBs, SOA, message queues, content-based routers, business process management, event processing, and data virtualization have all been created so that a development group can factor these systems into canonical representations of business events.
At that point, the input method ranges from the simple (existing web app to API) to complex (treating a batch-oriented system in a semi-synchronous manner or screen-scraping an ancient smart terminal as a queriable data source).
At that point, the real challenge isn't the input method but master data management and security/authority/privilege.
As an engineer, I crave data. As much as anything, this process would force me to organize that data into some archivable, more useful form.
if you're just getting started and you aim to keep it light, then i'd start with the withings scale. you leave it in your bathroom, set your goals, step on it when you feel like it and watch trends and alerts from time to time. i find my weight tells me more about me over time than many other metrics: it shows my bad weeks from my good and, in indirect ways, shows when i stress out and get bad sleep (and therefore, end up eating poorly and exercising inconsistently)
Also, a stalker would love this :P
There are already APIs built on top of you. There are the ones you know about, like Facebook and Twitter. There are the ones we know exist but have no direct access to, such as Amazon's backend or your police record. And there are the ones we may never know about, such as the NSA bot scanning this post right now and correlating it to my recent overseas cell phone calls.
I'm intrigued by the concept of aggregating this data. Imagine if the wealth of data collected about you all went to one place, and, most importantly, was controlled by you. Imagine if Company X had to pay you to know about your Amazon spending habits instead of paying Amazon. Imagine if scientists could perform long-term health studies in an instant because the data they need is already collected and just needs to be crunched.
I think the concept that's missing from your blog post is the idea of discoverability.
It's a thought provoking book.. and slightly worrisome to boot (about the state of things at the moment, with regard to the economics of "sharing data").
thanks for the book recommendation - i will check it out this weekend!
I'm keen to commercialize the work when it's ready but it's a research project precisely because it's a tough problem. Of course, one approach is to just make that personal info public but I don't think that will really benefit users in the long term (companies would certainly benefit, though)
Ignoring how JL looks and behaves (which is not really germane to the point), the commoditisation of everyone's data by a select few "Siren Servers" (to use Laniers term) without any payment to the originators themselves is - in the long term - not a viable strategy.
Regardless, I stand by my point that his book is thought-provoking and worrisome, though it does not give any concrete answers.. that of course is up to us as a tech "community" to fix (before we all merrily cut off our noses to spite our faces).
edit: corrected the username i was referring to, and spelling.
VRM stands for Vendor Relationship Management. (...)
Social software is very complicated because of issues like this. Something like a personal dropbox is straightforward - single customer, no community. You only have to worry about people hacking you, not what they do to each other. Anyone who gets into social software inevitably sets themselves up for making rules and judging disputes, and that often requires hard decisions.
Or compare email with social messaging; the rights and responsibilities are different and it's due to the software. Personal servers only support setting ground rules in a certain way (for example, the inability to revoke or prevent copies). Centralized services can support communities with different rules, for better or worse.
e.g. - you want to discover the 'right' kind of people to follow, as to have better or more entertaining or more useful or _____________ information. What is 'right' for you may not be 'right' for me, since our goals most likely aren't the same.
I love Naveen's idea (and more importantly his implemention) & will be playing with it a bit more this weekend. But, the value may not be apparent right away.
Aggregating and mining one type of data across a relationship may be more valuable. It may be that I have to track a similar set of data for myself and my significant other to make sense of the aggregates.
If you liked this article, you might like:
i) Egont, a web orchestration language: http://blog.databigbang.com/ideas-egont-a-web-orchestration-...
ii) Egont part II: http://blog.databigbang.com/egont-part-ii/
iii) Parsing s-expressions in C# using OMeta: http://blog.databigbang.com/parsing-s-expressions-in-c-using...
Because your stuff is defined as an s-expression and then optimized to a DAG.
Even more than Project VRM linked above, this is precisely the eventual goal of Tent: https://tent.io/
That said, I'm skeptical of these projects', or anyone's, ability to achieve this utopia. Why would Amazon or Google adopt such a system? Your shopping/searching history is valuable data that gives them a business advantage.
Also, while a personal data store that you control may be a geek's wet dream, for an average nontechnical person, it would likely be difficult to use, insecure and offer no clear benefits. And if it ends up just a toy for geeks, the lack of scale would make it even harder to do anything interesting with it or get any companies of significant size to play ball.
I do hope ideas like this surface in some form, but I'm skeptical of the personal data store ever really becoming a thing.
it would likely be difficult to use, insecure...
Who says? How can you assume this?
and offer no clear benefits
Some would consider the ability to keep your information private at will a benefit. Others would consider the ability to sell said information a benefit.
the lack of scale would make it even harder to do anything interesting
Lack of scale? An individual's data would be controlled by that individual, but the system would scale across all of humanity. I don't think you're thinking big enough -- I'm not talking about each person running his personal EC2.
As for why it would be difficult to use and insecure? I'm guessing most of the people you hang out with are probably geeks. For perspective, take a look at this video:
Imagine each one of those people in the video has a personal data store that contains their complete medical history, right alongside all their Google searches, Amazon purchases and tweets.
How secure of a password do you think they picked?
How hard would it be to trick them with a phishing attack?
How much do you think they even know how to do with it in the first place?
Even though some people are confused about the difference between browsers and search-engines doesn't mean they don't know how to use either. You'd probably have got the same response if you'd asked about the difference between the Web and Internet. Or any other complex system they don't have deal with day-to-day (internal combustion engine, electricity transmission, etc)
 edit: think of things like mass-assignment issue with rails and github or the recent facebook exploit that was posted. Centralized services are just as much of a honey-pot as a personal cloud might be.
Amazon could make these available through a simple RSS feed or export feature, but they don't. The only way I can get my hands on them is to scrape their website, which depends on some fragile scripting and is subject to their terms of service.
I really like this idea, but without some kind of positive government regulation, I don't see it happening. At least not in the sort of meaningful ways the OP envisions.
A simple example off the top of my head. Let's say there's a popular app that collects your pulse from your cellphone somehow, and that data is stored in your "Personal API" in some format. Some scientists write a routine that scans people's pulse patterns (with their permission, say) and warns them of possible risk factors. Later on hospitals start feeding their pulse data into the same personal API, but in a different format and using different units (bpm instead of bph, say), because they've never heard of the cellphone app.
I take "discoverability" as a general concept addressing how the scientists' routine might discover that the hospital data exists in a person's "Personal API", discover that the thing being measured by the hospital is the same as the thing being measured by the app, and discover how to map both formats into a common workable format and units.
There was that interesting article a while back about how Target was able to figure out a teenage girl was pregnant before her own father did:
It glosses this point:
Target can buy data about your ethnicity, job history, the magazines you read, if you’ve ever declared bankruptcy or got divorced, the year you bought (or lost) your house, where you went to college, what kinds of topics you talk about online, whether you prefer certain brands of coffee, paper towels, cereal or applesauce, your political leanings, reading habits, charitable giving and the number of cars you own. (In a statement, Target declined to identify what demographic information it collects or purchases.)
I'd be very curious, and a little scared, to see all this kind of information collected on me.
1. Excessive or erotic interest in
oneself and one's physical
2. Extreme selfishness, with a
grandiose view of one's own
talents and a craving for
So explain to me; what does quantified self have to do with narcissism?
The thing I like about personal data collection and cool new apis is what it could do for the individual, i.e. offer new insight, movitivation, encourage better behaviours.
But I hate the sharing part because it opens the door to over-sharing. Not just in terms of corporations having all kinds of data and using it in their interests, but also just simply the noise that gets added by people sharing information that frankly, has very little relevance.
I don't really care if someone just completed a 2 mile run and I certainly don't go to twitter to read automatically generated tweets about it.