
 Cost of storing all human audio-visual experiences - terrahawkes
http://lists.w3.org/Archives/Public/www-archive/2013Jul/0047.html
======
mistercow
I know it may be a little tangential to the original point, but I reckon you
could cut this down considerably if you had more accurate psychophysical
models than we do. Humans are probably not actually processing anywhere near
1.25 mbps of input, and as far as our actual experience, we're discarding the
vast majority of the information we even take in.

Consider, for example, that if you could perfectly predict the exact path of a
person's fovea as they watched an SD movie, you could aggressively compress
everything more than a dozen pixels away from that. A few pixels further, and
you wouldn't even need to store chroma information at all. Let's say you can
do that at 100 kbps, and the pixels on the fovea are 1 mbps. That would put
your total bitrate at about 105kbps for video. If you don't optimize audio at
all, that brings the cost down to about $170B/year.

And that's not even with new psychophysics; it's just better input. Sure,
playing it back would be hard, but _in principle_ , you could fool a brain
into thinking it was experiencing the data firsthand. I think it's reasonable
to think this is still a _massive_ upper bound. It's not unreasonable to think
that a lifetime of human sensory experience could be encoded on the order of
hundreds of gigabytes, if not less.

~~~
gknoy
If I were recording my life, I would not want to discard that, however -- I
don't want to record merely what I actually saw/noticed, but rather what I
would have seen or noticed if I had been looking elsewhere in my field of
view.

------
kalleboo
Charlie Brooker did an episode of Black Mirror[0] based on the premises that
everyone records every waking moment of their life.

[0]
[http://en.wikipedia.org/wiki/Black_Mirror_(TV_series)](http://en.wikipedia.org/wiki/Black_Mirror_\(TV_series\))

~~~
dhughes
I wanted to do something like that but it didn't work out very well. Mostly
"everything" meant on my computer (audio files are massive!) or phone and I
couldn't wear a video camera at work or scan every piece of paper I came into
contact with.

But still I find it interesting it's almost possible.

------
Udo
It's interesting to think about what these factors actually mean. The
perception of what's "good" quality will continue to shift hugely. Life
expectancy, hopefully, will go up. Hard drive prices actually haven't come
down again after the floods, so I'm guessing they'll stay artificially
inflated for a long time - especially considering storage density won't
increase in the coming years as we seem to hit a wall there as well.

So that's 60€ or $60 per TB for the foreseeable future. Let's say we're using
a well compressed H264 stream with a 500 MB per hour data rate for 16 waking
hours per day => 3 TB or 180 USD per year and person. Plus electricity and,
realistically speaking, bandwidth costs.

Still, it would be nice to have a searchable experience database like that.
Ideally I'd want that with a cybernetic eye ;)

~~~
joering2
> especially considering storage density won't increase in the coming years as
> we seem to hit a wall there as well

[http://www.southampton.ac.uk/mediacentre/news/2013/jul/13_13...](http://www.southampton.ac.uk/mediacentre/news/2013/jul/13_131.shtml?1)

~~~
Udo
There is any number of similar articles you could have linked to. However,
there always seems to be a huge disparity between sensationalistic storage
innovation announcements and what's actually ending up on the market.

Also, just snidely posting a link that a good number of people here already
know without taking the time to write an actual comment seems kind of lazy.

------
binarymax
It's funny, because in 1000 years, this is an anthropologists dream come true
(assuming the storage can be preserved that long by eventually copying to a
less volatile medium).

Is it worth sacrificing our privacy now, so that their perfect glimpse of us
in "the stupid ages" teaches them enough to learn from all the mistakes we
keep repeating?

~~~
Zimahl
Man, this brings up a lot of privacy questions. Especially around who gets to
look at what. I mean is this a personal archive that only unlocks after your
death? Is it open to anyone? Is it available to law enforcement?

This about this:

Your entire sex life is replayable. This includes being with someone or solo
action. Do you want you grandkids watching that?

No longer are we contemplating whether someone committed a crime, we would
just watch the replay. Would you even commit a crime if you knew it was being
recorded? You wouldn't need witnesses, just their feeds. Heck, you don't need
forensic evidence really - if you never find a murder weapon who cares? It's
on video! The innocent always have an air-tight alibi. It's on video!

Would not allowing law enforcement to look at your 'life feed' automatically
make you a suspect?

~~~
clarkmoody
Robin Williams played in _The Final Cut_ [1], which is a film that explores
the idea of making a post-death movie called a "Rememory," usually shown at
the deceased's memorial service. Williams plays the role of the editor of the
Rememory, and he faces the internal and external dilemmas associated with
uncovering all of a person's gritty life experiences.

[1]
[http://en.wikipedia.org/wiki/The_Final_Cut_(2004_film)](http://en.wikipedia.org/wiki/The_Final_Cut_\(2004_film\))

------
elboru
This post reminds me of a cool project, where the author is recording a single
second of video every day, for the rest of his life.

[http://www.ted.com/talks/cesar_kuriyama_one_second_every_day...](http://www.ted.com/talks/cesar_kuriyama_one_second_every_day.html)

------
sharemywin
assuming somekind of google glass attached to a giant world wide data storage.
Could it be compressed? (which got me to thinking about compression and the
larger the size of the data stored the more likely the chance of larger and
larger streams being duplicate. How many parts of the image do I have to store
of my desk. Also, could we limit it to interesting moments(how many times do I
have to record me eating cheerios in the morning.)

~~~
joshuarrrr
Presumably you could also use lossy compression across users, especially for
the boring parts. I imagine that, within the confines of my bowl, my milk and
cheerios look pretty similar to yours. Do the actual differences matter, as a
record of some particular breakfast experience? I suppose only if they float
into some surprising configuration.

~~~
peddamat
Dedupe for life experiences.

------
ChuckMcM
"If storage prices continue to develop as they have since the 1980s," bzzzt
and thanks for playing.

Remember this truism, "In real life, exponentials are s-curves."

That said, there are folks today who record 24x7 stereo 44khz as a 'life
diary.' It isn't that hard to store these days and compress down. One could
imagine adding video to the mix.

~~~
onebaddude
>bzzzt and thanks for playing.

There may be tens or hundreds or even thousands of years before that
exponential curve looks like an s-curve, making his point more relevant that
yours.

~~~
ChuckMcM
Well I've been pretty involved in storage for the last 10 years, 5 of them at
Network Appliance and then later another 4 at Google dealing with their
storage scale. And worked with both Seagate, Fujitsu, and to a lesser extent
Hitachi as they have worked to increase the density of what can be reliably
recalled. More interesting has been watching the struggle of thermal noise and
the ability to push past areal densities of 300 - 400 Gbits/inch^2.

There are certainly nano-scale technologies which seek to store information in
the 'spin' of electrons (I'd be hard pressed to see you get better than that)
but may become impractical if the network bandwidth gets to the point where
the size doesn't matter any more. Specifically, what matters to the end
consumer is that they can get what they stored into something which can use
that information. If it comes from disk across the room that is just as good
as a local on board disk if the bandwidth is the same.

The effect this reality is having on storage is that fatter (and increasingly
more fragile) drives, are becoming less useful to consumers than larger but
more reliable storage attached via a protocol (be it iSCSI, iCloud, S3, or
NFS).

So once people are unwilling to pay to carry it around, the ability to recoup
your investment in making it possible to carry around a device that can read
electron spin is less and less likely. And your s-curve will become clear,
when for the last 5 years the drives have all been at most 6 maybe 10TB and
the price of those doesn't seem to fall all that much.

I realize disk growth has been phenomenal "your whole life" as CPU performance
growth was for most of mine, but CPU performance growth has kinda sputtered
big time. We've been multi-coring for a relatively long time now. Storage
systems (especially random access read/write systems) are in the same boat.
But the glory days are over.

I am curious to see if storage will have its 'multi-core' moment. That will be
interesting if it happens.

~~~
mtrimpe
My bet is still on a break-through in write-once storage, which wouldn't just
be useful for life logging but could also work wonders with a storage system
like Datomic (with persistent/purely functional data structures.)

~~~
ChuckMcM
I read an interesting paper in Science about backing up data to DNA as a
future archive scenario. At the time I hadn't realized you could just store
DNA by drying it out and putting it on an index card, I had visions of cryo
vats of test tubes :-). Such a system would be a 'write once' sort of
solution.

~~~
mtrimpe
Indeed. That's exactly the kind of thing I'm talking about, although from what
I've heard access times for DNA are still a bit steep though ;)

------
thefool
Episode 3 of this tv show is very relevant:
[http://en.wikipedia.org/wiki/List_of_Black_Mirror_episodes](http://en.wikipedia.org/wiki/List_of_Black_Mirror_episodes)

------
_greim_
What about the cost of sharing all human audio-visual experiences? Because
simply storing them, while cool and/or terrifying in and of itself, will leave
a lot to be desired.

------
VikingCoder
I think a typical DVD is more like 4.5 Mbps...

~~~
Ogre
That's true, but the OP said "Typical DVD-quality SD bitrate". DVDs use MPEG-2
encoding, but there are much better codecs available today, such as H.264,
which can achieve equivalent quality at substantially lower bitrates.

Opinions vary on what the "best" bitrate for transcoding DVDs to H.264 is, and
it depends on what your content is too. In a quick search I found people
claiming as low as 500kbps(+128kpbs for audio)[1], which is half what the OP
uses, all the way up to 2500kbps[2], double his figure. All of which leads me
to believe that 1.25Mbps is a pretty reasonable napkin number.

[1][http://forums.macrumors.com/showthread.php?t=842536](http://forums.macrumors.com/showthread.php?t=842536)

[2][http://www.networkedmediatank.com/showthread.php?tid=6867](http://www.networkedmediatank.com/showthread.php?tid=6867)

------
consonants
Once whatever successor to Google Glass becomes ubiquitous, I can see this
becoming a reality.

This is an advertising research agency's wet dream come true, unfortunately.

~~~
cinquemb
I wonder, with personal income on the decline this past decade, will
advertisers spend less money on adverts or more on people who have more
disposable income from mined data?

~~~
consonants
Doubt it. If firms could analyze and deconstruct the purchasing decisions and
preferences from such 24/7 feeds, it will just allow them to more efficiently
funnel money from the consumer to the top. The lower income market is
intensely profitable, mostly due to the limitation of choice a disenfranchised
person has.

It's disconcerting that, literally knowing everything about you, puts
advertisers in a position to accurately target you in insidious ways.

Advertising is incredibly good at convincing people that they need [product]
as it is. Assuming the 'rational actor' theory ever treaded water, this sinks
it.

It does seem like the logical progression for Google and friends. What will be
interesting is how such invasive monitoring will be pitched as innocuous.
People seem to be a-okay with corporations mining their data, though.

~~~
cinquemb
> _The lower income market is intensely profitable, mostly due to the
> limitation of choice a disenfranchised person has._

From what I've seen, this has actually started to attract foreign owned
corporations in some industries like food (snacks in particular), I'm sure
that's not the only industry.

> _It does seem like the logical progression for Google and friends. What will
> be interesting is how such invasive monitoring will be pitched as innocuous.
> People seem to be a-okay with corporations mining their data, though._

Agreed. From someone trying to get into the data mining space, I feel like the
"positive" direction to go in is to basically admitting that there is no
putting the lid back onto this box (simply too valuable to many services and
provides utility for people can't be botherd to learn more outside of how to
press buttons) and trying to make data more valuable to people on an
individual level. (instead of just for sigint agencies and their outsourced
private equity owners portfolio companies like BAH).

