
Facebook’s Blu-Ray Powered Cold Storage Data Center - 1SockChuck
http://datacenterfrontier.com/inside-facebooks-blu-ray-cold-storage-data-center/
======
bloat
I used to have a job as one of those robotic arms.

The datacenter were I worked had lots of shelves full of tapes, and the
computer would beep and display a tape number and a drive number. You'd have
to jump up, find the tape and put it in the drive.

During the day you were running about constantly trying to keep the thing
happy. On the night shift you could sit a watch movies and only get up once or
twice an hour.

~~~
chinathrow
One of the real robots dropped a tape within a swiss data center on December
24th 2010 and subsequent error states stopped all debit card transaction as a
result...

~~~
chkuendig
on the day everybody was scrambling to get their christmas shopping done.

i still remember the chaos (I also had to get some presents...)

edit: it was actually in 2000: [http://www.boerse-
express.com/mobile#/mobile/pages/96409](http://www.boerse-
express.com/mobile#/mobile/pages/96409) (german)

~~~
chinathrow
Thanks, too late to edit the post for this typo.

The ATMS were dead too.

------
jacquesm
I wonder how Facebook is able to implement the EU privacy directives with this
technology as their long term storage solution. If a customer requests
deletion of their data you're supposed to comply with that, even on back-ups.

~~~
junto
In all seriousness, it is more than likely a flag:

    
    
      User.IsDeleted = true
    

Or

    
    
      User.Status = Status.Deleted (or Status.Deactivated)
    

Facebook allow users to deactivate their accounts, which can then be
reactivated. Facebook users can also delete their accounts. Facebook say that
this is permanent:

    
    
      When you delete an account, it is permanently deleted from Facebook. 
      You should only delete your account if you are certain you never want 
      to reactivate it. 
    

Permanently deleted still doesn't mean deleted. It just means that you can't
recover it. Facebook have never publicly stated what their deletion process
is.

Is it logical or physical?

Since Facebook keep anonymous profiles, I wouldn't be surprised if deleted
accounts just get migrated to anonymous profiles which they can still profit
from. That's what I'd do if I was Facebook.

~~~
evgen
"Permanently deleted" means exactly what it says. Deactivated means that the
account is inactive but the data is still there. When you delete your account
the data is gone within 14 days, no options for recovery; it happens in bulk
and with a bit of a grace period for 'buyers remorse' so it is somewhere in
the tail end of that 14 day period. And yes, Facebook has publicly stated
their deletion process on multiple occasions and has explanations of how it
works in online help docs and FAQs that you just choose to ignore.

~~~
junto
FB isn't exactly good at trust when it comes to its users. The whole business
is about extracting value from its user base as an asset. Companies normally
don't like throwing away assets if they can help it.

Until I see evidence to the contrary, I will assume that FB doesn't delete
user data completely. That is just my preference based on the evidence of FB's
previous actions. Others can believe what they will. I don't trust the online
help docs or FAQs as far as I can throw them. FB has made a business out of
abusing user privacy.

    
    
      Zuck: Yeah so if you ever need info about anyone at Harvard
      Zuck: Just ask
      Zuck: I have over 4,000 emails, pictures, addresses, SNS
      [Redacted Friend's Name]: What? How'd you manage that one?
      Zuck: People just submitted it.
      Zuck: I don't know why.
      Zuck: They "trust me"
      Zuck: Dumb fucks 
    

Finally, as programmers, we all know how hard it is to remove elements such as
a user from a graph. A user is key element of FB. It is challenging to remove
a user, since they have a multitude of cross-related information, linked to
other users and other user content. Either you delete the user logically and
then hide their linked content, or you delete the user physically, but first
delete all of the linked content, or you replace the user with a 'deleted user
profile' and hide their linked content. Any other ways?

FB obviously hadn't done this back in 2010 [1] and maybe they have been trying
hard to fix this issue, but I'd welcome an official post from FB tech team
that states how they went about implementing this feature. Maybe a bit of
transparency would go a long way in helping the more paranoid of us to think
twice about FB.

[1] [http://www.zdnet.com/article/facebook-does-not-erase-user-
de...](http://www.zdnet.com/article/facebook-does-not-erase-user-deleted-
content/)

~~~
rimantas

      > Finally, as programmers, we all know how hard it is to
      > remove elements such as a user from a graph.
    

Yet blocked users do disappear from the graph.

------
hsshah
I am glad companies like Facebook is using Blue-Ray. Hopefully this will
trigger progress in increasing density of optical discs and/or decreasing cost
of high capacity discs and writers for consumer market.

Eager to archive my yearly ever increasing personal media (thanks to having a
kid) onto a high capacity disc and just store it in the safety deposit box.

~~~
bitJericho
I don't know what the life of blu-rays are but burned optical cds and dvds
have an estimated lifespan of only 10 years. Make sure you continue to keep up
your backups! How I do it is I update my backups every few years but I also
keep everything on functional and "hot" hard drives at all times (so I can see
it's all still there).

There's also [http://www.mdisc.com/what-is-mdisc/](http://www.mdisc.com/what-
is-mdisc/) but I've never used/seen it.

~~~
slantyyz
The main downside is that MDiscs have a max capacity is 25GB on the Blu-ray.

I have some myself, but I can't attest to the life span, since I've only had
my BDR for about a year now. How does a normal person prove the lifespan
claims of an MDiscs other than waiting?

I still find myself burning most backups to 50GB disks, and making redundant
copies of only my most important stuff to the 25GB MDiscs.

Where I live, a 3 pack of 25GB MDiscs runs around $20 vs $70 for a spindle of
25 RiData 50GB discs. That's ~6.66/25GB MDisc vs ~2.80/50GB BDR. Not a huge
premium for piece of mind, I guess.

~~~
hsshah
I see that they have 100GB BDXL MDisc available now. Have to invest into BDXL
writer but that's just one time investment.

------
samch
Just to help add some clarity, this isn't Blu-Ray in the common consumer
sense. I've seen some comments on here about the stability of the dyes and
capacity. This technology, as mentioned at the end of the article, is Optical
Archive storage [1]. The media are in the form of cartridges with 12 discs in
each cartridge, and Sony claims that the media is "50 year rated". Sony
currently manufactures cartridges with up to 1.5TB of storage[2]. They also
make a 7U, 10-cartridge library [3]. So, in a standard rack, you could fit a
maximum of 90TB of Optical Archive storage. That's admittedly not a lot given
the storage density of hard drives, so I'm guessing that Facebook's decision
to invest in this is partly driven by energy savings or other considerations.

[1]
[https://en.wikipedia.org/wiki/Archival_Disc](https://en.wikipedia.org/wiki/Archival_Disc)

[2] [http://pro.sony.com/bbsc/ssr/cat-recmedia/cat-
oda/](http://pro.sony.com/bbsc/ssr/cat-recmedia/cat-oda/)

[3] [http://pro.sony.com/bbsc/ssr/cat-datastorage/cat-
opticaldisc...](http://pro.sony.com/bbsc/ssr/cat-datastorage/cat-
opticaldiscarchive/)

------
grinich
A few months ago, Frank Frankovsky left Facebook to start a startup based on
this tech! He was previously "VP of Hardware Design & Supply Chain
Optimization" and has also been super involved in the Open Compute Project.

I'm pretty excited to see what they produce, and hope they can ship before
someone like Amazon tries to acquire them. This level of cold storage is going
to be critical as the world generates exponentially more data every year. He
also mentioned this is an open source venture, which is doubly awesome. :)

[https://www.facebook.com/frankovsky/posts/10203257723264639?...](https://www.facebook.com/frankovsky/posts/10203257723264639?stream_ref=10)

~~~
lucd
FTA: Last month that startup, Optical Archive, was acquired by Sony.
[http://www.datacenterknowledge.com/archives/2015/05/27/sony-...](http://www.datacenterknowledge.com/archives/2015/05/27/sony-
acquires-facebook-born-startup-repurposes-blu-rays-cold-storage/)

~~~
grinich
...sigh :(

------
cc438
So how does this work with the way Facebook delivers content? I understand the
advantages of disc storage for a service like Amazon Glacier as the customer
expectations are set for significant time and bandwidth constraints but
Facebook is supposed to deliver personal data as fast as it can be served up.
They've created a "customer" expectation of instant access that can't be
provided by an offline disc.

It makes for a good solution as a 2nd or 3rd level backup but you'd need some
form of RAID to prevent users from experience a lengthy delay every single
time a disk fails. That's a lot of money to invest in a novel solution that
creates a lengthy interruption in service should it ever be needed. Maybe the
power savings are worth it when weighed against the rarity of such use and
expected level of revenue lost to user frustration but that's an equation made
fragile by so many variables.

I could also see it used in conjunction with some amazing use-prediction
algorithm to store data that will most likely never be requested. If Facebook
truly keeps everything then they have millions of photos, videos, hidden
posts, and other digital detritus tied up in dead, inactive, or orphaned
repositories. I know half the groups I joined back in high school are now
impossible to find but the data they contained is still important for the
purpose of linking people and interests together. I also have several friends
who have passed away, making their account a kind of digital mausoleum. Their
private data will never be accessed again but it must remain in FB's
datacenter until someone is given access to the account to delete it. The
number of Facebook accounts tied to the deceased is only increasing so these
racks of Blu Rays might be considered the first digital cemetery. I don't know
how many resources are allocated to this kind of data but this is where cold
storage makes the most sense.

------
mark_l_watson
I really like how open Facebook is about their infrastructure, hardware
designs, etc.

I would have thought that they would be trying something with higher capacity,
but off the shelf == lower costs.

Interesting that FB needs to handle 900 million photo uploads a day. With
higher resolution cameras on smartphones, their storage requirements will keep
increasing. My Note 4 takes 4K video and very high resolution photos - FB,
Google, etc. can store lower res versions only, but the pressure will be there
for higher resolution support, especially with 4K TVs, and other high
resolution devices.

------
giancarlostoro
I wonder how projects like Archive.org could benefit from something like this.
I know it has it's up and downs, but if you have enough backups it shouldn't
be too bad. I'm sure there's tons of untouched content that could easily be
archived (an archive inside the archive), but other sites might as well, such
as wikipedia. Of course they would need a bit of funding for this somewhat new
endeavour. Maybe facebook can 'donate' the implementation to such sites.

------
Fuxy
Can I have one of those Blu-Ray storage units searching all my discs would be
way more convenient if i didn't have to do it manually :)

------
sschueller
Isn't the life span of a burned disc quite bad?

~~~
kalleboo
Isn't the life span of a magnetic disc quite bad?

~~~
slantyyz
I've gone through over a half dozen Seagate 2.5 drives (spindle and hybrid)
and one 3.5 external in the past five years, and I'm lucky if they last 2
years. This includes drives that are mostly offline.

I've switched to WD, but I haven't had them long enough to say that they're
only better. The WD Green in my Time Capsule has lasted a few years though.

All these drive failures led me to buy a BD-R burner, which has worked out
fairly well, although I would say it's better to buy burning software than
using the free software (on Windows at least) if you want to avoid making a >=
$3 coaster.

I find the 50GB capacity to be fairly easy to work with, since I'm much less
likely to have to span files/folders across discs than the 25GB ones.

------
therealmarv
I wonder how long it takes to put data from that storage online. What happens
e.g. if I want to see my 5 old images fullscreen?

~~~
phire
I think they still keep a copy online. But they only have to keep one copy as
the cold storage is used as a backup instead.

------
y0ghur7_xxx
All our chat history is there

~~~
CodingGuy
And our cat images! :)

