Of all of Facebook's behavior online, the advertising, the tracking, the privacy snafus.. this is probably the one that bugs me the greatest.
How this was not made a priority zero, drop-everything-and-fix-this-crap-yesterday internally is beyond my comprehension. I hate to think of the condition of their internal systems if they can't even get files deleted from their CDN in a timely manner.
3 years is not a timely manner. A month and a half (their target, FTA) is not a timely manner.
You generally don't delete files from a CDN. They just get evicted when they are the least recently used, which may never happen if the URL is still in circulation.
Facebook may have set near-infinite Expires/Cache-control headers on their static content, since it never gets overwritten. In that case, the CDN never goes back to the origin to check if the file is still there. The CDN they are using probably does have the ability to explicitly remove files (e.g., to respond to DMCA take-down notices), but probably doesn't have the ability to do this at Facebook picture deletion scale.
In other words, Facebook probably can't do much about it unless they get the CDN provider to make a big investment, build their own CDN, or switch to another provider that does support it (usually a costly decision).
No CDN caches content for that long, even if the origin tells them to. (Services like Akamai's NetStorage excluded, which Facebook most likely does not use.) It's quite obvious that Facebook is actually keeping the images on their origin servers long after they have been deleted.
I have this same issue with my site, but in case anyone is looking for a solution---I have a tentative one: NetDNA (or MaxCDN if you're not looking at their enterprise options) has a XML-RPC API that lets you purge individual files. So you can delete something on your server, then push the purge to them.
So while things have been bad in the past--they're getting better. Having everyone try to use s3 as a CDN has really forced the industry to evolve.
Doesn't Facebook run their own CDN, though? (At least that's what the URL's would suggest).
And in the case of major CDN providers like Level 3, removing assets is dead simple. They provide an API for you to pass assets to expire immediately. Or barring that, having a person spend 5 seconds on the keyboard to log into the control panel and paste in the URL to kill.
It is possible - the author of the Ars article had his pictures erased after writing the article.
In other words, they are now valued at $100 billion, but providing the most basic privacy functionality imaginable, after being given years to do so, is just too expensive?
>How this was not made a priority zero, drop-everything-and-fix-this-crap-yesterday internally is beyond my comprehension.
Its simple. Minority concerns like this don't upset enough of their users to impact their bottom line in any meaningful way so they don't do anything about it.
Class action lawsuit anyone? That'll bump it up the priority list.
The more this becomes public the more likely it is that stories will surface about how "my life was destroyed because FB didn't actually delete the photo and now I need to be compensated" and there will be a whole bunch of lawyers waiting in the wings to take up their case. They may wait until the IPO has been completed...!
#1) If it was easy they would have done it. Pushing photo deletions up to the CDN is hard without flushing the entire CDN cache. And Facebook has 90 billion photos, so flushing the cache is not an option I guess. (Wrap your head around 90 billion photos. The scale of the problem is huge.)
#2) Anyone with access to the link to the CDN had access to the original photo. So this really doesn't protect someone from viewing your photos you wanted to delete - since if they wanted it before they could have had it.
So ultimately this should not bug you the greatest of all the bad things they have done. The ultimate blame belongs to the person who uploaded the photo to Facebook and then later wanted it deleted.
Yes, but the CDN is not going to cache 90 billion photos indefinitely. Clearly photos being on their for years after deletion, presumably being accessed very rarely, are going to be prime candidates for flushing. In all likelihood these photos are still on the origin server.
Do people really think a company whose engineers wrote a PHP-to-C++ compiler because PHP is slow as balls are incapable of solving this engineering problem?
This is either advantageous for them (somehow) or the cost to make it happen it isn't worth it.
The cost to make it happen is developer time (money). The benefit to making it happen is... appeasing a few geeks? They haven't lost many users because of this, and they probably never will; I doubt their value from doing this will ever justify the expense.
Not to imply they shouldn't do it, merely that they have little motivation to do so, aside from it being a good thing they should do.
There are privacy laws somewhere that cover the case of "you published a photo, and sometime later changed your mind and decided you didn't want it publicly available any more."?
I'm not only not-at-all surprised at this, but I'm somewhat amazed that anybody even thinks it's a thing worth commenting about. You upload a photo to a website and make it public (or allow the defaults to make it public) - it's now out of your control. Changing your mind later is _exactly_ the situation the phrase "close the stable door after the horse has bolted" was coined for - and the used of stables and horses in that idiom shows at a minimum how old the type of problem is.
I love hating on Facebook as much as the next privacy-concerned-geek, but I find it hard to feel any sympathy for people who are calling this a problem - any more than I'd feel sympathy for people who want to unscramble their eggs.
Privacy laws in most Western countries outside the US (at least all of Western Europe) are based on the principle that you own your data.
The rest simply follows from that. If you tell Facebook to remove your data, they are, within reason, obliged to honor that request.
Now that photo may still be out there if it has been copied (of course when then come into the realm of copyright and such), but Facebook should no longer have it in their dataset, let alone be publishing it.
The privacy angle isn't about the photo being public, it's about Facebook having it in their database along with all the other stuff they have on you.
This is why American companies like Google and Facebook tend to run afoul of European privacy laws: even if the data they collect comes purely from public sources, doesn't mean they have the right to collect, and especially collate it. At least not in the European concept of privacy.
The purpose specification principle of the OECD Privacy Guidelines (which all substantial privacy laws are based on - EU, NZ, HK, etc.* ) says this:
"Finally, when data no longer serve a purpose, and if it is practicable, it may be necessary to have them destroyed (erased) or given an anonymous form. The reason is that control over data may be lost when data are no longer of interest; this may lead to risks of theft, unauthorised copying or the like."
Keeping an image URL online three years after it was requested to be deleted almost certainly counts as keeping personal data for longer than is necessary. When you upload a photo, it is usually for the purpose of sharing (as the case may be). When you decide to stop sharing, the data no longer serves that purpose and hence should be removed as soon as practicable.
* The US and Australia notably have not implemented the OECD guidelines.
There are privacy laws somewhere that cover the case of "you published a photo, and sometime later changed your mind and decided you didn't want it publicly available any more."?
Not yet, but the upcoming European Union new Data Protection Directive will give the public the right to forget " http://europa.eu/rapid/pressReleasesAction.do?reference=IP/1... ). This may offend your idea of basing things off quaint proverbs, but we do that all the time with laws.
However, this will soon be explicitly a legal requirement that Facebook (and other companies) will have to abide by.
You are not considering the case when a private picture is shared by some other person. Should allowing anyone to take a picture a themselves be a permission to do whatever he wants with that picture?
> Do people really think a company whose engineers wrote a PHP-to-C++ compiler because PHP is slow as balls are incapable of solving this engineering problem?
Please don't make this about PHP and technical competence :-( This is clearly not a technical problem (deleting a file is pretty easy in PHP to, y'know?).
I don't think he was, he was pointing out the engineering prowess of Facebook. Essentially, if they can write something as complex as HipHop, they surely could figure out how to properly delete photos.
I did below. PHP is a joke of a language/platform (and I say this as someone who has six years of professional PHP experience), but that wasn't my point.
The point was: PHP is dog slow, Facebook solved that particular problem with a gargantuan engineering feat, deleting photos is trivial by comparison, therefore they either do not give a crap or it's somehow beneficial to them to keep them around forever.
I don't see how it makes much difference. Removing the link from the site means that nobody will discover the photo. People who discovered the photo and saved the link could equally well have saved the file, and in fact I think saving the file is more common.
I doubt that. If the image is being removed, say because it contains an embarrassing picture that was accidentally posted, you can bet someone copied the link and shared it or posted it somewhere. Also, image lookup services like tineye could also grab a hold of them unless they are gone for good.
Don't get me wrong, I still believe that posting ANY picture on Facebook you don't want the whole world to see is just plain stupid.
Let's remember that it's not only the people in embarrassing pictures who upload them. Sometimes rude friends or colleagues take pictures of us without even telling us, and upload them.
Even if you don't save it explicitly, I find my browser cache is full of facebook photos when I go to clear it, so can retrieve it through there after the fact as well.
There are lots of non-technical users who don't know about saving files (seriously!). Removing the original link would cut off some people from access to the file. It's like computer security. You can never be "100% secure", instead you start taking steps that each will reduce your exposure.
Not only that, there are potentially legal differences between publishing a link to an embarassing photo and actually sending the photo itself. After all, it's common on this site to draw the distinction between linking to content and the content itself.
With the photo in their possession, they may crawl it sometime in the future and extract (via computer vision, facial recognition tech, whatever) further information about the person, their friends in the photo, etc. if they hadn't done it already. They may mine the metadata in the image itself for camera GPS position, etc.
There is still very much value in these photos, and a great chunk of it can be unnervingly personal.
Because no one ever shares links online ... email, twitter, delic.io.us, pinboard and pretty much every other method of communicating excepted, of course.
As a point of reference, Amazon CloudFront charges $0.005 per file to invalidate, after the first 1000/month.
No additional charge for the first 1,000 files that you request for invalidation each month. $0.005 per file listed in your invalidation requests thereafter.
I'm currently an undergrad in Computer Engineering. As an engineer we are taught to look ahead when designing software and to do it with the "good of society" in mind.
I find it VERY hard to believe that Facebook did not plan ahead and implement an efficient method of photo deletion. If they truly did not, I seriously call upon their skills and insight in creating software for the public.
Come on Facebook ... it's deleting a damn image! This should be top priority in the modern age of personal privacy.
Honestly, I'm not so sure. Looking at the APIs exposed by Facebook you'd be hard pressed to find anything that looks like its engineered. It seems to be a hodge podge of features thrown together over years with the stuff that works best/well enough sticking around long enough to be seen by the public. I think its a safe bet that many of the things not exposed are quite messy/nonsensical.
They could very well have never thought of deleting a photo; it doesn't seem that there is much interest in information removal there, and I could easily see if being a very ugly manual process that no one wants to spend time doing and as such it never gets done.
Note that the delete operation merely marks the data deleted - they don't actually purge or overwrite it. Perhaps there's some kind of bug in this deleted-marking system? Kind of weird they haven't been able to fix it yet though.
As an engineer we are taught to look ahead when designing software and to do it with the "good of society" in mind.
Keep in mind, that when the photo features of Facebook were added, they probably didn't do it while thinking about half a billion users or whatever gazillion terabytes of photos they have to operate with. Heck, they may have even started out with simply storing photos on disk in their servers where deletion was easy as opposed to on a CDN.
I've designed a couple of small scale websites before and I've always thought about how they would react if the user base would expand significantly. It's always something in the back of my mind when programming.
Even with a small user base; if a user flags a photo for deletion -- it should get deleted. Period.
1. Delete it from the public facing servers that we control.
2. Put a request in the queue so the backups, and thumbnails will be deleted.
3. Pray that the CDN will purge the picture eventually.
#3 is a big deal... we don't control the CDNs cache options, and can't push a 'delete' upstream so the picture might be gone 2 seconds after the deletion is completed on our end, or it might take 2 weeks.
I'd thought Facebook would have been able to engineer or negotiate better options than we do, but apparently not.
There's large scale, there's really large scale and then there's Facebook scale. They operate at a scale that almost no one does, and in fact, if you're thinking of how to operate at that volume when you start your new project, then I'd say you're doing it wrong (I know you didn't actually say that - just pointing out that no one should be thinking of how to operate at that scale when designing their basic architecture).
I don't really know how their photo system works - maybe it is a trivial thing to purge files from CDNs on demand - but I'm willing to give the benefit of the doubt and say never attribute to malice, what could be explained by earth-population-level scaling problems.
Also unrelated: Facebook has shown many times that they can't really be trusted on privacy issues - I never upload anything there under the illusion that I can freely delete it and it's really gone. But I don't personally believe this thing is a privacy issue they're intentionally balking on
Thanks to those of you who give Facebook the benefit of doubt. I don't work on this particular system, so I'm not qualified to discuss any specifics, nor do I speak for the company. I can say this though: People @ Facebook do care about these types of problems and work to solve them. I'm not some new grad; I've worked with large systems for a while. The scale is hard for me to comprehend. The systems I do work with illustrate very well that nothing is as simple as you think it might be and most of the obvious solutions get tossed out the window because they won't scale.
Maybe I mis-represented my argument. I didn't mean that I code small scale projects with large scale back ends just in case a user base boom happens. I meant that I try to code as using a "module" approach as much as possible. When (and if) the opportunity presents itself, the small scale project will be able to integrate a large scale solution much more easily.
You're making the same mistake that many awful managers make; namely "If the problem seems like it should be easy, it must be easy".
You aren't familiar with Facebook's infrastructure and comparing working on small websites to working on something that scales to roughly 1/12 of the entire population of the planet is laughably naive.
Saying you use a "modul[ar]" approach is nothing but handwaving. You think that facebook doesn't use modules? Or doesn't have a service oriented architecture to some degree? They friggin' developed their own RPC transport!
I'm currently an undergrad in Computer Engineering. As an engineer we are taught to look ahead when designing software and to do it with the "good of society" in mind.
Don't worry a few years in Industry and you'll learn what things are really like.
i.e. There is no end of shite software that wasn't developed. Yes you can make lots of money with hodgepodge software. Welcome to Industry :)
This kind of cache (mapping a url to a constant image) seems to have little to do with that kind of cache (mapping a memory address to likely changing memory contents.)
Can you clarify why you think the problems are similar?
If you start from the premise that there are only two hard CS problems then all hard CS problems must be a special case of one or both of those two. So cache in an on-chip memory sense is conflated with disk storage is conflated with distributed disk storage. This isn't necessarily bad, Smarty for example caches rendered files to disk storage and calls it a cache, but it does show off the problem of naming things. (That and every variable named "data".) Anyway, I imagine what the GP was getting at is that if your data distribution isn't particularly deterministic enough (e.g. swapping hard drives in and out when they fail) you have to deal with validation that a particular data changing command (store, delete, whatever) actually propagated to a sufficient portion (in some cases all) of the servers and that the introduction of new or changing or rogue servers doesn't affect that. The more apt term for this is consistency. Related is the CAP theorem which says for any distributed system, you can only pick any 2 of consistency, availability, or partition tolerance, though it's more interesting to talk about atomic operations, where transactions are/where they might be desired, and whether things get faster or slower with more data.
They're similar in that a deletion consists of a change to the image. This means that the image is now changing, so it's the same basic problem again.
The two scenarios have entirely different constraints, this is undeniable. It's still an instance of the same basic problem: verifyably deleting the image means invalidating all caches of that image.
In social software you, the operator of the service, never delete anything. Everything is a datapoint - valuable now and potentially further valuable in the future.
The contract a user makes with most social sites is that the social site can monetize the user's data and that the user gives them a non-exclusive royalty free license to use that data.
Well, when the user then ask them to 'delete' the photo, they are saying that they no longer have any interest in that photo sticking around. That doesn't mean that Facebook (or whoever) shares the same view that said piece of media is now useless.
They have no moral or legal reason to delete their copy of the media just because the user no longer needs it.
"Don't hate the player, hate the game" - and if you don't like it, don't upload your media to social sites like Facebook.
Social contracts and morals are not defined solely by corporations and their interests. We have a right and responsible to declare our opinions to converge on a mutually acceptable standard.
I'd like to point out that I believe what you're saying is technically incorrect - at least as it applies to Facebook. It is my understanding that when you ask Facebook to "delete" your content, that is (or at least should be) exactly what happens. From Facebook's "Statement of Rights and Responsibilities"[1]:
1. For content that is covered by intellectual property rights, like photos and videos (IP content), you specifically give us the following permission, subject to your privacy and application settings: you grant us a non-exclusive, transferable, sub-licensable, royalty-free, worldwide license to use any IP content that you post on or in connection with Facebook (IP License). This IP License ends when you delete your IP content or your account unless your content has been shared with others, and they have not deleted it.
2. When you delete IP content, it is deleted in a manner similar to emptying the recycle bin on a computer. However, you understand that removed content may persist in backup copies for a reasonable period of time (but will not be available to others).
From their Data Use Policy[2]:
When you delete an account, it is permanently deleted from Facebook. It typically takes about one month to delete an account, but some information may remain in backup copies and logs for up to 90 days. You should only delete your account if you are sure you never want to reactivate it.
And, finally, from a Facebook employee here on HN[3]:
Maybe that was true in the past, but today when you delete your data it is gone. Trust me, I wrote it myself. The law enforcement guidelines that have been circulating recently corroborate this.
I believe the above quotes demonstrate that Facebook, at least, do have a legal reason to delete their copy of your media when you no longer want it hosted there -- and that they actually do so. Maybe that will change in the future, but I hope not.
Personally, I think it is dishonest to use the term "delete" when you simply mean "remove this reference to", and I appreciate Facebook distinguishing between account deletion and deactivation. To most computer users, delete implies that the thing is gone and can't come back. Using that term to mean something else is deceitful. As such, I believe sites that allow users to "delete" their content do have a moral obligation to actually go through with that and delete it entirely, not just make it invisible to the user.
It is my view that once a service becomes culturally entrenched, it becomes accountable/responsible in many ways to its users. This attitude of "well, if you don't like it don't use it" is an interesting one (and it's something I adhere to for the time being), but I wonder how long it can last. Recent SOPA outrage demonstrates that people generally don't have this sort of attitude towards governments. The differences between the two are clear, of course, but there are similarities too. I find it interesting that people[4] will condemn, for instance, full-body scans at an airport as being an invasion of privacy but then have this cavalier attitude towards corporations like Facebook - hey, you agreed to the terms and conditions! At what stage does that agreement become entirely meaningless?
The response headers indicates photos are set to expire in exactly 14 days. Does Akamai not honor cache-related headers from the origin URL? If it does, couldn't Facebook just generate URLs that includes an rolling expiration date so that when the CDN pulls from the origin URL again, Facebook may send a file-not-found placeholder image?
The article doesn't make it clear, are these direct-link photos still subject to the permissions set on them? Would setting a photo's permission to "Only Me" prior to deleting it work around Facebook's failure? If they aren't subject to the photo's permissions, that's a giant security hole, or am I misunderstanding?
I am not a lawyer, but this seems to be a solution. Did I miss something?
1. Find a friend.
2. Write "I, Mr. Me, hereby transfer the exclusive right to publish the attached image online to Mrs. Friend, for the consideration of $1." Sign it and have your friend sign it and pay you. (At least in the US, consideration is required!)
3. Hand your friend a filled-out DMCA takedown request. Have them sign it and send it to Facebook. (If you're not in the US, figure out the local equivalent and do that.)
It is possible that you violate Facebook's ToS in (2), but Facebook doesn't look too closely at DMCA requests - and losing your account might be worth it anyway. Facebook is obviously not going to fight a takedown request on your behalf, so this almost certainly does get the photo removed.
It is ridiculous that one would need to do this, of course.
If you're not in the US, figure out the local equivalent and do that
I don't think any other countries have anything like the DMCA with it's safe harbour and take-down requests.
I'm also pretty sure that once you upload a photo to facebook, you give them copyright on the image, and hence you are unlikely to be able to claim copyright infringment,
Honest question: aren't ACTA and the like all about cross-border copyright enforcement? I can't say I'm an expert, but...
Note that it's your friend claiming copyright infringement, with an exclusive license in hand. Legally, the question is whether you can actually give an exclusive license after uploading a photo to Facebook (the ToS has been challenged and changed on similar issues), but that is not between your friend and Facebook.
Google "facebook dmca": Facebook will happily take pretty much any page down (e.g. http://www.techdirt.com/articles/20101007/01244411320/facebo...). Do you really think that Facebook is going to risk a court case where it has to prove (on its own dime!) that you actually did have copyright on the image when you uploaded it, risking its DMCA safe harbor status?
I'm also not an expert, but we've had cross border copyright enforcement for over 100 years with the Bern Convention. The UK will recognise and enforce USA copyright that happens in the UK.
Facebook has to take down DMCA requests to stay with the DMCA and the safe harbour rules. However that's a US law. I don't know if non-USA people can invoke it.
I'm also pretty sure that once you upload a photo to facebook, you give them copyright on the image
I don't think that's likely - there are plenty of photographers on Facebook, and I find it hard to believe they'd upload pictures to Facebook if it meant losing their copyright.
I don't mean to sound snarky, but: that's because they hire people who know how to merge linked lists or reverse words in a string or (insert interview problem du-jour) ; they don't hire people who can really solve real-world problems.
I may sound old when I say this, but my impression of interviewing there was that they're just a bunch of kids who like working on 'cool' or new things, and don't really care much about the real-world impact.
No, this is not sour grapes; I was recruited for my skillset, but was really disappointed to be tested on bullshit problems; I pointed out specific issues with their site that needed fixing and how they could be fixed, but it didn't really matter to them.
Surprised that Ars didn't jump to the most obvious conclusion: a legal compliance issue.
Presumably, five or six years ago, someone from the DOJ or FBI told Facebook it would be really handy if they stored user images indefinitely, and Facebook, not being a company that insists on seeing warrants or subpoenas, said "Sure, why not."
Can somebody remind me why we use facebook again? The argument that it's somehow more convenient than other communication channels is utter bull shit. It's just herd behaviour.
I use Facebook because it's the only way to communicate with lots of people who are important to me and my life.
Social networks are the definiton of a network effect. They are boring and simple technologically, but are phenomenally valuable when lots of people I care about use them aswell.
You cannot beat facebook with technical solutions, only social solutions.
How this was not made a priority zero, drop-everything-and-fix-this-crap-yesterday internally is beyond my comprehension. I hate to think of the condition of their internal systems if they can't even get files deleted from their CDN in a timely manner.
3 years is not a timely manner. A month and a half (their target, FTA) is not a timely manner.
A few seconds to a minute would be great, thanks.