If this is real and can be commercialized it could lead to some pretty devastating fallout for the entertainment industry. You've got to know that there are a few obsessive archivists out there with pretty comprehensive collections of films spread across rooms of RAID arrays. When they can start backing those collections up for friends onto a single disc, ie a single disc that has more films than you could possibly watch in your lifetime the entertainment industry is going to pine for the days when torrents eve the big problem.
The same goes for rogue librarians, or Google Books employees dumping entire libraries onto discs and leaking them out.
> more films than you could possibly watch in your lifetime
I was curious if this statement was true...
So assuming, 4.4 GB her HD film & 2 hrs long each, you can fit roughly 232 films per TB or 232,000 films total (for 1000 TB), that's 464,000 hours of HD content or 53 years (!) back to back.
So, yes, that statement is true. Now the next logical question: Has the human race produced 464,000 hours of video content?
Let's assume there are around 100 cable channels showing content in HD video 24/7 (with no overlap, there are more but a lot of duplication). That's 2,400 hours of content a day, 876,000 hours a year or 101 years back-to-back worth of watchable content shown each year.
But realistically that will be at least 70% duplication (particularly looking at it over several years). So even if we just look at 24/7 news and shopping channels which always produce original content, you're still talking about easily 200,000+ hours/year.
...which comes out to 144kh/day. In other words, one of these discs would be enough to store ~3 days worth of contributions to YouTube. So a box of these discs, let's say 1000 of them - which isn't all that big (e.g. http://www.amazon.co.uk/Neo-Aluminium-Storage-1000-sleeves/d... ), could hold an archive of every single video that has ever been uploaded to YouTube.
Edit: got beaten to mentioning YouTube, but there's many other video sharing sites on the Internet, and of course there are probably countless hours of one of the types of video the Internet is well known for: porn.
The majority of that is junk anyways. But I suppose some people obsessed with archiving everything wouldn't filter anything. If I were to grab "everything" from YouTube I would at least limit it to subscriptions and only subscribe to channels that don't upload bogus junk.
Then the valuable commodity would become directories of worthwhile content. You would have people doing deep dives into the crap to find the locations of a few good pieces to share with their buddies or trade.
So, that would be 4,640 minutes or less than a week. Lower quality, and probably with (near) duplicates, but I guess it is safe to say we have that amount of video.
Another way to look at it: I expect we produce more content in unique wedding videos every year (at an hour each, that takes half a million weddings or a million people marrying. At an average of one marriage per life and a life expectancy of 75 year, that takes a population of 75 million people)
> So assuming, 4.4 GB her HD film & 2 hrs long each
People bitch constantly, but a typical feature film encoded by YIFY at 720p will clock in at somewhere between 650-900 megs, leaning towards the lower end of that range.
Not that it matters, the rest of your math makes it look ridiculous. Without too many tradeoffs, you could certainly put every American (Hollywood and indie both) film on such a disc.
> Now the next logical question: Has the human race produced 464,000 hours of video content?
Somewhere Google has a gauge that shows how many hours of videos they store... but I can't find it.
> So even if we just look at 24/7 news and shopping channels which always produce original content, you're still talking about easily 200,000+ hours/year.
Agreed. But I don't think that they have archival policies in place where everything is kept permanently. Even the news channels may have been in the habit of dumping everything but stuff deemed important, well into the 1990s.
>Let's assume there are around 100 cable channels showing content in HD video 24/7 (with no overlap, there are more but a lot of duplication). That's 2,400 hours of content a day, 876,000 hours a year or 101 years back-to-back worth of watchable content shown each year.
I'm thinking that 100 channels provide exactly 100 years worth of content per year instead of 101. Unless you rewind the good parts :-p
And there are of course vastly more than a 100 cable channels. BSkyB alone broadcasts more than 600 channels for the UK market (more than a thousand if you include radio). In 2010 Ofcom passed 1000 licensed UK channels.
The US is supposedly the only other country with similarly high number of channels, but I don't think it's unreasonable to assume that there are more than 10,000 channels worldwide.
Library genesis is the best thing that happened to research since Google (and a huge number of researchers use it, secretly). Being able to access any obscure technical book in the world in a few seconds is literally orders of magnitude more efficient than waiting a week for an interlibrary loan. I dread the day it inevitably gets shut down due to pressure from the copyright mafia. It's ridiculous that such a service isn't 1) legal, and 2) publicly funded (with due compensation to the authors).
Could you explain more exactly what "library genesis" is? Is it a collection of basically every book ever published or only modern things? Did someone scan everything, like Google books?
It's basically a huge collection of documents, mostly scientific with a FOSS codebase for serving. It comes with metadata. It is mirrored a lot thanks to bring completely open. It originated in Russia and has had quite the history. It is a great step into the direction how documents (especially scientific) should be available to anyone in bulk without restrictions.
gen.lib.rus.ec and bookfi.org both work for me, some time ago I investigated the origin of those collections and apparently they began as a set of 30+ dvds that have been distributed presumably primarily in russia for quite some time.
Especially in math and physics they are every poor graduate students dream. Even large libraries have only one copy of some of those books stuffed away in some underground vault.
And engineering and CS, both undergrad and postgrad. Fantastic website for when the library was out of stock of that book published 40 years ago that there's only 3 copies of in the country...
You are forgetting that the speed at which optical drives can be written is bound by the rotation speed. If you have a single laser of this type and mount it in a traditional drive as we know, it would take over a year to write a single disk :) Still lots to do :)
Really? Assuming that the rotation speed is the same for a normal disk and this denser format, wouldn't said laser be moving over 250,000 times as much data in the same time period?
At 1X speed, a DVD's data rate is about 11Mbit/s - so at 250,000 times the density, isn't it safe to assume a theoretical data rate of 2.75Tbit/s? Of course, pushing that much data at that rate is another problem...
If my assumptions are incorrect, please let me know - I'm just trying to understand.
You're forgetting the square-cube law. Or rather, in this case the linear-square law. Think of each bit as a square, with side length 1/n. Oversimplifying, but the basic principle is there.
The total amount of data the laser goes over in a second is proportional to n - because it's related to how long the laser takes to get from one edge of the square the the other edge. But the total amount of information stored goes up as n^2 - it's related to the area of the square.
Or: to put it another way, the number of tracks also goes up when data density increases.
52x speed has been the limit for a long time because the outside edge of the CD gets to speeds where it could damage itself. I guess it would need a cushion, like a nearly applied (car/bike) disc brake.
It's actually about optical discs. But "CD" is the vernacular for optical discs. Kind of like "hard drive." Modern laptops don't have hard drives--they have solid-state drives, but people still refer to their storage as a "hard drive."
Hmmm. Maybe you could use a prism/mirror array (like in digital projectors) to use a single laser to write in multiple places at once? (Or rather, in sequence but moving faster than the disc rotates?)
I'm wondering if the entertainment industry is the reason for which Hyper-CDROM[0] is not commercially available. If so, something is very wrong with the way society incentivises development and commercialization of new technologies.
Will this really be a larger "problem" for the entertainment industry than online file sharing? Is there many people out there that would want such a disc that already doesn't have fast speed Internet and can download any movie they want within minutes?
As of a couple of years ago, the entire content of The Pirate Bay[0] was under 100MB[1].
Think about that for a moment - it's the index for one of the largest collections of publicly available content in the history of humanity, and it fits in my wallet 80 times over[2].
[0] Not the P2P content that people download - just the main site itself, containing all of the magnet links, etc. TPB's 'content' is literally just an index (it's not even a tracker).
[1] I don't want to link to TPB from HN, but just search for "pirate bay archive". There's a large archive that contains all of the site data (db backups), but the magnet links are all you really need content-wise.
> the entire content of The Pirate Bay
> Not the P2P content that people download
I think you're confusing different definitions of 'content' here. Also, there are many more sites that have publicly available indexes of other links much greater than the pirate bay, google.com for example.
Why would you want that? The magnet links alone are rather useless. The whole site can be crawled within 2-3 days easily. There are ~1.2 million torrents indexed, lets just assume that a single magnet-link plus some metadata is 0,5-1kB... so that would be an total of ~1GB data.
> Why would you want that? The magnet links alone are rather useless. The whole site can be crawled within 2-3 days easily.
But the entire site archive is already available as a torrent indexed on TPB - why bother crawling? That's what I'm referring to - search "pirate bay archive" on TPB and you'll find it.
The magnet links are not useless, because they are able to query the rest of the metadata from a tracker and/or DHT.
I'm thinking about this from the perspective of 'minimal amount of information needed in order to reproduce TPB', not 'how much information does TPB index (which is obviously much larger).
(Remember that TPB no longer even operates its own tracker, so if TPB itself were taken offline, the magnet links would still work fine).
The idea to break the diffraction limit by using the this doughnut-shaped excitation spot comes from the STED microscopy (http://en.wikipedia.org/wiki/STED_microscopy) developed by Stefan Hell (Max Planck Institute for Biophysical Chemistry in Goettingen, Germany).
He actually showed that an additional intensity-dependent term can be added to Abbe's diffraction limit allowing sub 100um resolution with light. I think this idea is awesome and it sure won't be the last invention made possible by this trick.
> With the $18,000 fellowship, Gan will collaborate with industry and researchers around the world to work on new breakthroughs for data storage devices, and also see how his existing research can be used on a larger scale to rapidly improve the capacity of optics-based information technologies.
$18K seems like a drop in the bucket for powering this kind of research. Dr. Gan needs someone to introduce him to Kickstarter.
I guess this is because the technology only solves one of the problems (getting a very narrow writing beam/point) and not the others, like: making the disk so stable that it doesn't wobble for more than a few nanometers during rotation; keeping the laser beam focused on the disk; figuring what kind of rotational and seek speeds (=read/write speeds) you can handle with these limitations; making everything small and ready for market.
Nope, it's $18k, but it's not a wage. It is low (Australia doesn't have the money that US universities are awash in), but it's for things like international travel to conferences and the like.
True, its a pure travel scholarship, with all likelihood the student won't even get to touch the money. It will all handled by the university's finance department (it was in my case).
And $18k is huge for an Australian travel grant, they're usually just $2k to $5k, some examples:
I see where they mention writing the data, but does anyone see how/if they are reading the data?
From his other pubs [1], it looks like a two-photon process is involved as part of writing in deep-subdiffraction limit for lithography. Will be fun to try and reverse that.
The real breakthrough would be a way for regular CD/DVD drives to at least be able to read the new format. But I doubt that this will be practical for consumers. Most consumers dont really need a CD drive anymore, companies like Apple arent even including them in laptops any longer.
> people will still want to archive their downloads
I don't know how true this is. Even the tech-savvy folks I know rely primarily on Netflix/Spotify/etc, and in the rare instances they buy something, they trust iTunes/Amazon/etc to hold it for them. (I continue to think that is nuts.)
I'm using Plex to store our videos... sold our DVDs on craigslist a few months ago. Within 3 weeks, had another copy of every disc we got rid of.
Eventually I want to get a FreeNAS machine up and running, and any important files will be backed up to Google Drive (though, obviously not movies that can simply be downloaded again if desired).
A big problem with spinning disks is that storage grows inversely with the area of the dots, while reading and writing speed grow inversely with the diameter of the dots.
If you store 100 times as much data, all else being equal, time to write a full disk will be ten times as large.
Ways around that are increasing rotation speed (been there, done that. There is little to gain here without making the disk a lot stronger = heavier, if that is really possible at all) and using multiple heads (harder to do, but may eventually be the better solution, certainly if one can completely do away with head movements)
The parameter of interest is specific strength, which is the material's (ultimate tensile strength/density).
I forget what the plastic in DVDs is, but it could probably be reinforced with carbon fibers (chopped, nanotubes, etc.) to give a useful increase in rotation speed.
You are right and I am not an structural engineer, but I understand that pure strength isn't the only limiting factor for disk; they start wobbling before they tear apart (http://superuser.com/questions/554717/will-there-ever-be-fas...). That, I think, is more a matter of thickness than of pure tensile strength. Barring the construction of hollow disks, that means heavier ones.
Also, now I think a bit more about it: in a 50-speed CD player or a 16-speed DVD player, the edge of the disk moves at about half the speed of sound. Doubling rotation speed would push it over the sound barrier. I guess it would take quite a bit of research to make a CD/DVD/BlueRay player that will work fine at those speeds (a way around that is to make the disk smaller. That would be an option for a disk with the storage density discussed here, but it wouldn't win you more than a factor of four or so, at most)
A 50GB Blueray disk can be written in 12 Minutes with a home burner but a few seconds in mass production. Scaling that up this would be (1,000,000 / 50) = 20,000 times the space but take sqrt(20,000) * 12min = 28 hours to write at home.
However, assuming you can R/W at the same rate that's 10GB / second speed which might be tough for a home PC to pull off anytime soon.
PS: I suspect if this where ever out into production they would change the form factor to enable even faster R/W speeds and a significantly reduced capacity.
By the time this is ready for production, flash or disk will have already caught up. I've seen it time and time again - the old tech continues to improve exponentially while the new tech takes forever to turn into something practical.
I'm pretty sure "DVD" isn't any more accurate a term for this tech. The bits are way smaller than the pits on a CD or DVD. Also, the laser used for writing is purple, while for DVDs it's red, and CDs use infrared.
One hopes that the write speed of these disks will be proportionately faster than DVD write speed. A DVD writing at 16X transfers about 22MB/s. At that rate it would take 1.44 years to fill up this disk writing to it continuously. Even at 1GB/s (roughly the write speed of an SSD) it would take almost 12 days to fill up one of these disks.
I doubt it. There's a long, long way from the theoretical finding to the practical device.
Writing and reading speed has been mentioned. Another point is that as the dots become smaller, the mechanical parts need to be much more precise, moving the device from consumer parts to really expensive parts and assembly.
I would love a potential use case as a backup that is redundant dozens or hundreds of times over. I'm not sure the exact mechanism by which DVDs start to lose data over 10-20 years, but this tech might be a way to mitigate that risk (as well as scratches and other physical damage).
Awesome tech. I wonder how data will be written and read from such a disk? How long will it take? The transfer of 1000TB will be an interesting problem indeed. At current theoretical blu ray writing speed of 12x or 400Mbps [1] the time to read/write such a disk would be:
Could probably be written all at once using immersion lithography. You could threat them as big read only memory 'crystals'. Buy a set of 100 containing entire YT archive etc
> Every day, humans are producing more data than ever before - around 90% of the world’s data was generated in the past two years alone - and there will come a point when our data storage centres and the cloud can no longer keep up.
Is this true? Can anyone point to a some kind of study showing convergence of storage capacity and data production over time? I was under the impression that we've got far more storage than we'll need, at least in the near future.
Given that we can today fit tens of TB in a fistful of SD-cards (512GB per card), no I don't think it's true at all that we'd get to a point where our data storage can't hold up. Our ability to filter and identify gems outside of the most popular stuff is a far greater challenge.
This could be a DVD - it would be an easy enough thing to perform tests on, and they say they're using a red laser for the actual writing. If the dye can handle the very small writes, that is. (And I wouldn't trust Memorex for this ;) )
Error correction for these is going to be interesting, even a speck of dust could obscure many megabytes of data.
The researchers never used any discs. The 1000TB figure was for a surface area equivalent to a CD or DVD; they're the same physical size, and these descriptions give the reader an idea of what a 9nm feature size means in practical terms.
What they actually developed was a novel resin and a method for etching 9nm dots in it using a specific optical laser setup. If it were to be commercialized, it'd be a new type of disc, and CD/DVD lasers would not be able to read it.
Why downvoted ? Allegedly, blue-ray recordable discs are used by Amazon for Glacier backup. Surely they will be the ones very interesting of further development of this technology.
Wow this seems huge If it's practical to implement everywhere. You could keep every picture/song/movie/document/game from your whole life on a disk, and hey, make a backup too.
Oh great, another 1024-bit daisy-chained medium with a 20MB UeFI boot partition. We have watched this way too long and it was dragging us down...(cough)
Oops, accidentally implemented as a medium-hash CAMFS in disused packaging-and-optronics 45nm fab. Please write test, mind 5W limit if retaining cyano/pyridene dye in media.
What if I could hand you the totality of Netflix, iTunes, etc.? Not wait for it to download, but transfer large chunks of the Internet in seconds? and no longer have to hope that, say, Netflix won't lose vast sections of their library for contractual reasons (as it has done of late)?
Backups: permanent archiving of everything you have all on one disc.
It makes vast swaths of data yours, in your hand, under your control, in a ridiculously compact & cheap media. No more hoping that you can get X in time, or that it will be there some time hence.
I've seen profound transformations in computing when cheap fast storage increased by an order of magnitude of orders of magnitude. This will bring that about again, to similarly disruptive and amazing results.
Netflix would put the entire catalog in your home, encrypted, and the only data they would have to transfer would be for the control plane (billing, recommendations, etc) and encryption keys to auth you.
Totally side-steps last mile monopolies as well. Don't want to work with us? We'll drop an LTE chipset in as well for the admin stuff. Redbox In Your Home.
The encryption keys would be time limited (monthly basis, certain titles, etc). If you don't top up monthly, you key expires, the underlying data is re-encrypted with another key, etc.
You could have a master-key (or set of master keys) on a read-write medium, which you could encrypt/decrypt as many times as needed? You'd be vulnerable to someone scraping the master key from memory, but no more than someone just passing the video to a recording device.
Yeah that's true, but I would say the majority of people aren't ripping Netflix streams in case Netflix's content licences expire. If they are given a master disk with everything on it, anyone with that disk has the data on it in perpetuity (barring storage failure)
If you mail a single disk (nevermind a box full of discs) to your friend, and it arrives in 3 days, you have achieved about 31.6 Gbps. If you mail a USPS flat rate box with 100 discs, you have achieved 3160 Gbps.
You cannot buy a three thousand Gbps internet connection.
What other medium are you going to store the data on? What are you reading it onto?
You would read from that plastic disc just like a CD or DVD, i.e. as you consume the content. This makes the throughput of any link in this chain irrelevant. (I guess except the read speed of the plastic disc)
"the internet" is a more fragile medium than a disk, that's why. High-density, cheap, persistent data storage is critical to make sure that most of the cultural expressions of the last few generations don't vanish into a black hole sooner than later. It's our version of ancient Babylon's cuneiforms etched in wet clay. Just imagine how much crap Jason Scott could fit into a storage container with these. A lot of crap. A loooot of crap.
http://www.mdisc.com/ or similar. There are tons of options for archival quality backup media. If the data is very important make a few copies and leave them in airtight cases if you'd like in different geographical locations.
Sure you could use an entirely different storage medium for backups, that's not my point. I'm saying that if someone happens to drop the disc on a carpet, that could be enough to erase big portions of data. It's not about losing whats on the disc, it's about the fragility of the media itself. I don't know much about the tech being used, but I could see it bordering on volatile if they're embedding that much information, and volatile storage isn't very useful unless its extremely fast.
Who's going to store data on something that will erase portions of itself when dropped? Consumers won't, because they won't know what to do when their 50,000 movies suddenly start skipping ten minute chunks. Scientists won't use it because they want strong guarantees on retrievability, and won't want to bother with stringent protocols on handling the media when other options exist. Logging systems might have a use for it, if they can stand potentially losing big chunks of data.
Yeah, taking it into space or mining (WiMPs) would wreck plenty. You have to like the ECC you're on. you're on. you're on...
How often were you going to swap discs? When you got SESSION COUNT EXCEEDED, or UDF timeouts? Every time you give contact info...'and here is a capsule intelligence of things you may wish to contact me about in the next 15 years...but just call.'
Skip most of the spinning and size, put it in a fat SIM card, and have recovery pipelines (cleanroom et al) for failures. If you throw down for the 8TB (not really 8TiB) multiplatter +9 Fondleslab Of Regret maybe you can throw in the service premium to go visit the one retailer the future holds for us, do key exchanges, and have them print out a scan of peak versions of your besties, or let you write Abnormal_brain on the top with a marker?
Huh, didn't know that was the original name. I personally prefer the Hacker New moniker. I imagine 'Startup News' was just a working title for the project as it was only kept for ~7 months.
I reckon there was only so much you can read about startups on the web before it gets boring. (I certainly had that impression at the end of that phase.)
The same goes for rogue librarians, or Google Books employees dumping entire libraries onto discs and leaking them out.