
Facebook's e-mail debacle: One 'bug' fix, but rollback impossible - wyclif
http://news.cnet.com/8301-1023_3-57465433-93/facebooks-e-mail-debacle-one-bug-fix-but-rollback-impossible/?part=rss&subj=news&tag=title
======
ChuckMcM
Ouch. Talk about not thinking it through. Relating back to a common topic
here, one of the things that distinguishes between senior engineers and not
senior engineers in my experience is the ability to visualize both secondary
and tertiary consequences of a particular change to the system. They stop and
say "Ok, we change this, that means ..." and then walk forward past the change
in their mind exploring the ramifications. In order to do that they have to
have a broad understanding of the system involved from both a technical and
operational perspective. They come back with "Oh if we do that then we'll get
this effect and that would be bad, how can we prevent that from happening?"
and if the answer was "We can't." then they spike the feature.

One of the most frustrating things for me as an engineer at Sun early on was
bouncing some whiz bang idea off Bill Joy only to get one of his one line
zinger 'but that breaks X' type replies. Always annoying but often right as he
would take what was proposed, extrapolate two, three, or a half dozen steps
and then point out the now 'obvious' flaw.

~~~
codelust
This.

The problem is not limited to engineering. I have often had to push back with
product guys responding to feature requests from users to address edge cases.

The trouble with doing so in social networks is that the potential for the law
of untended consequences to strike is much greater in it as it is nearly
impossible to test for every permutation and combination available.

~~~
rhizome
If, as described elsewhere, the overwriting is due to using a specific and
common API, the only unintended consequence is user reaction. No need to test
for every possibility there.

------
pixelcort
Quoting:

> Today Facebook also admitted that its API for contact sync on phone and
> device apps was set -- on which devices and systems we're not told -- to
> take an individual's most recently added e-mail address and overwrite their
> correct contact e-mail in everyone's address books with the new e-mail.

and

>> Since Facebook changed the email addresses on contacts, I've actually lost
every single one of my email addresses including those for work.

From this article it sounds like actual data loss occurred on some address
books, where non-facebook.com email addresses were permanently deleted from
contacts. For those people who don't have a backup of their address books, it
might now be impossible to recover those lost email addresses.

Isn't this more worrisome than any issues around how Facebook handles incoming
messages?

~~~
TeMPOraL
It is. Everything else seemed to me to be whining about how someone who saw
you on Facebook will maybe not be able to contact you because your default
mail is a Facebook one (but why would anyone contact you by mail if Facebook
messages are better? They both a) show notifications on FB and b) send
notification e-mails to your real address).

But this, this is real, and worrisome, and it has done real damage, and it
will be damn hard if not impossible to reverse.

~~~
s_henry_paulson
So Facebook messages are better because instead of getting your e-mail, you
now have to jump through hoops to read it?

Got it.

~~~
TeMPOraL
Facebook messages are better when the person who wants to contact you _found
you on Facebook_. They are better because they show in two places: your
normal, original (not the Facebook one) mailbox AND in Facebook itself. So it
is at least as good as sending an e-mail, and likely better.

~~~
s_henry_paulson
If someone went looking for you on facebook, chances are they know how to
simply click and message you.

If they're going the extra mile to copy and paste your e-mail address, chances
are they want you to e-mail you, and not send you something through facebook.

Aside from the simple fact that not everyone has the same notification
settings, it's already been shown that facebook has been losing people's
e-mails. No notifications, no nothing, just gone.

[http://www.rluxemburg.com/2012/06/29/bad-facebook-email-
scre...](http://www.rluxemburg.com/2012/06/29/bad-facebook-email-screwup/)

In no possible way is this better.

------
51Cards
Sometimes it seems to me that FB doesn't have an impact assessment process.
This is a very important data change you're making to 900-ish million records
on a system that is inter-woven with and syncing to... everything. I
understand bugs happening, but really, the cascade effect of this email change
had to have been foreseen by someone... no? Bueller? Bueller?

I really don't mean to trash FB as a lot of good people work there. However...
when I read about something like this, and then I see a company like Google
figuring out a time smear function to seamlessly account for a leap second in
their servers to avoid cascading system changes... it's hard to feel the two
companies are functioning on the same level. (I know, a mis-balanced and
irrelevant comparison, but it's just for a 'gut feeling' sense)

~~~
rhizome
They knew what they were doing. Not having a complete rollback was
intentional.

~~~
eli
I believe Facebook is capable of doing sneaky things to advance their business
goals, but Hanlon's razor [1] comes to mind here.

[1] <http://www.jargon.net/jargonfile/h/HanlonsRazor.html>

~~~
rhizome
I know what Hanlon's razor is, and I can guarantee you FB Engineering knows
what a rollback is and how to know whether you need one or not. I'm not sure
where you're getting the idea that they're that stupid.

~~~
keithpeter
Would applying such a 'rollback' involve replacing facebook.com email
addresses with the original external domain addresses in the contacts list on
the devices affected by the write-over issue?

If so: Facebook have taken private data and placed it in a central location

If not: it isn't a rollback that I understand

Disclaimer: I'm not a programmer

~~~
rhizome
Yes, a (hypothetical) "complete rollback" would involve replacing removed
data. Sure, data privacy would be a concern, but it's beside the point that
rolling this back would necessitate un-overwriting.

------
reneherse
More good reasons to never:

a) Associate your phone number with your FB account.

b) Download the FB app.

The more you can keep Facebook sandboxed on your devices (and in your life)
the better.

~~~
Ogre
c) Associate your real email address with your FB account

Facebook gets an email address I don't use anywhere else. No one has me listed
in their contacts under the only email address facebook knows for me (well, I
presume. It's not hard to guess the address I use for facebook if you know my
real address, but there's no good reason anyone but facebook would use it).

I've done that on general principle for more than a decade any time I have to
give my email address to anyone other than an individual. This instance is the
first time in a while where it's actually felt worthwhile. Originally I did it
more for spam tracking. GMail's excellent spam filter fixed that problem for
me years ago. But I never stopped doing it.

~~~
reneherse
Indeed, I do the same.

And I'll also add: d) Never associate your real name with your FB account.

I use my first name plus a family name that matches my middle initial. It's
generally recognizable and accepted by friends.

When I first set up my account this way (items A through D) it did seem a
little "tinfoilly", and this last measure still does to some extent. But since
Facebook doesn't actually let you delete your account, I'd rather err on the
side of prudence. It will be nice to at least approximate a hole in their
databases if their policies ever get too onerous.

~~~
edoloughlin
I'll see your tinfoil and raise you. I've also poisoned their automatic facial
recognition by tagging random faces as my own.

------
j_s
Can anyone point to a Facebook representative saying that rollback is
impossible? (I don't care at all what the CNET author says.) It seems like
they should just put everything back to the email address they had before it
was auto-updated to @ facebook.com. Then their apps would put them back to
what they were on people's phones.

Lastly, if they were truly going to fix the problem, they would pump through
all the emails they got a chance to spy on, this time to the correct address,
maybe with an attachment or sending the email as an attachment with their
official apology.

Can't rollback the delay and that impacts a lot, but email is theoretically
unreliable anyway. Hurry up guys and gals... show me someone at Facebook
actually cares about the product (their users) they are selling to their
customers (their advertisers).

~~~
51Cards
A couple issues pop to mind. My understanding was that this update took a
week(+?) to roll out and may still be in progress. It could take them awhile
to even start to roll back. Then if you do roll back what do you do about all
the people that have changed their settings in the meantime? They have updated
their email addresses, perhaps with different ones, so do those get over-
written with the old data in all cases? I'm sure it could be figured out but
how timely it would be is questionable... not to mention that FB may still be
trying to salvage the bump in use for @Facebook.com they were aiming for in
the beginning.

~~~
j_s
I agree it's not a simple fix. Since they have everything stored, they could
act as though the address had never been changed. (Primary email should be
most recent non-auto-change. Email should be sent to value of primary email
address [without auto-change] as configured when email arrived.)

This mistake/bug/zuck-style enhancement has had a colossal negative impact on
many people. If Facebook tries to salvage anything from it then it's even more
clearly time to walk away. I am not cheerleading for Facebook at all on this,
I just want people's contacts back to normal and their emails delivered to
where they want them. I think we agree this is possible, in fairly direct
contradiction to the title of this article.

------
dmethvin
"Move fast and break things" should still require that you think through what
you're doing. The problem seems to be that the complexity of Facebook's system
is beyond the understanding of anyone who reviews these changes.

~~~
tobtoh
I'm pretty sure that FB knew exactly what they were doing. They did think
things through and decided the the short term PR loss was worth the benefits
of gaining access to the email of 90% of their user base.

~~~
alan_cx
Exactly. Users are stupid. Not because they cant read minds and find hidden
features, but because they take it.

Its the same thing with things like internet censorship. Every one bleats on
and on, and they just do it any way.

------
melvinmt
Let's say 1% of all their users is fed up because of this and 0.1% actually
closes their account. This leaves Zuckerberg with 98.9% control over e-mail
traffic from 900+ million users. Well played, Zuck, well played.

~~~
entropy_
Couldn't this be construed as an abuse of a monopoly(antitrust)? They have a
monopoly on social networking(there are alternatives, but they are very very
small in comparison) and they are leveraging that to expand massively in
another sphere, namely as email providers.

------
grey-area
The problem here is not one of competence or failure to think through
implications. Facebook know exactly what they are doing, and you can be sure
the potential implications of these changes were debated at length within the
company at a high level. They were taken precisely because of their
implications for non-technical users, not in spite of them - FB has replaced
users contact details because they want Facebook to be the only way most
people can contact others online.

Facebook have always wanted to replace the internet entirely with a system
they control. Hence the walled garden approach with incessant prompts to
signup to view content, the use of address book import to spread virally, the
use of beacon to track and announce users activities, the use of like button
js to track activity, the launch of Facebook apps and single sign-on to
attempt to corral web developers within their ecosystem and make them
dependent, the launch of a competitor to email/IM etc etc.

This latest move with email is simply the latest in a long line of moves
consistently attempting to steer users into spending all of their online time
within the confines of Facebook, it is entirely consistent with their past
behaviour and we should expect similar moves in future.

So the arguments about how they could have done this differently, or whether
Facebook has made some mistakes here are largely irrelevant - they will always
push to own your data, email, photos, and digital life, because it's in their
DNA, this is not a mistake, it's a pattern of behaviour - they can and will
control your digital life if you choose to hand it over to them. Consider the
quote from FB on the way their FB email works:

 _Regarding the "email loss" this may actually just be confusion around the
Messages Inbox: By default, messages from friends or friends of friends go
into your Inbox. Everything else goes to your Other folder._

All messages which do not come from FB contacts go into a folder which is
effectively a junk folder - they want you to use their email for everything
(hence the recent changes), they want your address book to contain nothing but
FB contacts, they want you to forget email even existed, and they want you to
encourage your friends to join FB so that you actually receive their messages.
That won't work for all their users of course, but their hope is that they
keep enough of the user base on board that they can dominate the internet and
replace email with FB and the web with a network of sites where they control
login and user info and all roads lead to FB.

So replacing email with FB mail wherever they can, hiding other contacts and
trashing user address books is not a mistake, it's part of a broader strategy,
one which is not in the best interests of their users, but which they hope
most will go along with anyway.

~~~
statictype
I deleted the Facebook app from my phone a long time ago and I only use it
from an incognito browser window. So that's how much I trust their reach. But
I find it a little incredulous that Facebook, in their quest to control all
their users's mail would deliberately overwrite email addresses in users'
contact lists. It seems like the kind of dumb plan for world domination that
only a caricature of an evil villain would execute.

~~~
j_s
How many times must the same type of thing occur before Hanlon's razor falls
to Occam's? :)

------
uptown
I used to have an Android device, and let the thing sync my Facebook contacts
into my phone. What I wound up with was a mix of people I'm actually friends
with, and also a big collection on "Facebook friends" ... those people I'll
never ever call, but knew from back-in-the-day and wanted to connect with on
Facebook. I don't need them in my phone today because the extend of my
correspondence with them will be what I consider "two-way read-only". They can
look at my FB stuff. I can look at their FB stuff, but that's pretty much the
extent of our relationship.

I'm on an iPhone now, and know that iOS6 will give users the same option ...
to link your device w/ Facebook ... sucking in all your "FB friends". I won't
be enabling that feature, and while the software apparently indicates you can
disable this part of the connectivity, I'm reluctant to even link it for fear
that it'll immediately hose my contacts in a way that requires me to manually
clean out that kid I went to middle-school with many years ago.

------
AznHisoka
There were so many obvious problems with associating a @facebook with
everyone's account. As soon as I heard the news, I thought.. hmm.. someone
could easily scrape 1 million facebook usernames, and be able to send 1
million emails to all of them using their facebook email address.

And make it all uber-personal because they know their full name AND all their
friend's names AND what they liked AND their hometown AND the college they
went to, etc, etc,. That's a spammer's wet dream.

~~~
entropy_
The @facebook was always there and could always have been used to reach a
person's FB messages via email. Also, most of that other information would
probably be marked as only available to friends or friends of friends. So you
wouldn't be able to scrape it.

Don't get me wrong, this is a _horrible_ move by facebook, but spam isn't the
reason why.

------
antidoh
"Apple might be able to salvage the relationship with Facebook, but it's
pretty clear that Facebook's relationship with users has hit the rocks."

I very much doubt that. Statistically no one will leave FBook over this.

~~~
pdonis
>>> Statistically no one will leave FBook over this.

If that's really true, it's very unsettling. If this doesn't trip a
significant number of users' "not worth it" detectors, what would?

~~~
blhack
Um. Hi. I use facebook, this pissed me off.

Then I changed my email address back to what I want it to be, and I have
ceased caring. I will continue using facebook to coordinate bar meetups with
my friends.

~~~
glesica
I think you kind of hit the nail on the head with your last sentence. You will
continue to use the service for fairly meaningless tasks. It seems Facebook
wants its users to use their service for more than that. So while this debacle
may not make many people delete their accounts, Facebook may find its apparent
goal more difficult to achieve going forward.

~~~
SwellJoe
I hope this is true.

I've always had a very healthy mistrust of facebook, based on their past
actions, and I never let facebook sync up with anything else I use (and I
don't use services that only authenticate via facebook). They simply can't be
trusted with anything important; not because they are incompetent, but because
they have no moral compass that prevents them from screwing users over.

This is just one more reason for me to continue to silo facebook off into its
own world, where it knows nothing about the rest of my life, particularly the
really important stuff.

~~~
five_star
I agree. Facebook seems to have little concern about customer satisfaction.
That is why I use Facebook minimally.

~~~
antidoh
FBook cares very much about customer satisfaction. Those are the advertisers.

You're the product. You're inventory.

------
andrewfelix
I've been using a great work around to the numerous Facebook issues, including
the lack of meaningful interaction with Friends and Family; I use a
combination of email, telephone and physical interaction to maintain and form
social connections.

------
crazygringo
So this whole debacle has taught me that I have an "Other" folder, which I
never saw before, and it turns out I have 20+ messages from people from the
past few months which I didn't know about!

I'm trying to find a way to get Facebook to send me an e-mail notification
when I receive a message that goes into "Other", the same way it sends me an
e-mail notification when I receive a message that goes into "Inbox".

But I can't find any option for this anywhere. Does anyone know? Otherwise,
how am I ever supposed to know I have a message from someone on Facebook who
I'm not already friends with?!

~~~
crazygringo
Wow, this is infuriating. According to _Slate_ :

 _So how can you make sure you find your lost laptop, appear on Israeli TV,
and respond immediately to job offers? Unfortunately, the Facebook rep
informed me, you can’t change your settings to get email notifications for
your Other messages, the way you can for your main messages. Your best bet:
Make checking the Other tab part of your daily Facebook routine._

So since I never log into Facebook, but use it as a place to receive messages,
which I find out about via e-mail... I am doomed to never find out about
possibly important messages from people who are not already my Facebook
friends.

This is possibly the worst design decision I've ever come across on the
Internet. Facebook has created a black hole for messages. Even when you log
into Facebook, it doesn't show a number next to "Other", so you can't even
find out if you have messages there without navigating to the "Other" page.

I can't wrap my mind around how insanely stupid this is. WTF, Facebook?

~~~
Tyr42
Time to use selenium [1] to automate it and send you emails?

[1] : <http://seleniumhq.org/>

------
bschlinker
"Facebook is indeed intercepting messages -- the e-mail is clearly passing
through Facebook's servers, but this seems to be where it stays, as neither
sender nor receiver are getting a copy."

Is this really appropriate use of the word "intercepting"? Typically
intercepting carries a negative meaning -- like "the FBI is intercepting my
postal mail and opening it before it reaches me". However, with their use of
the word, every email I send is "intercepted" by multiple servers before it
reaches its destination....

~~~
j_s
Facebook is intercepting messages by replacing the destination address (which
email clients often do not readily display [opting instead for full names,
even photos, etc.]), and now claims to have done so inadvertently. Multiple
servers relay messages before they reach their intended destination, yes, but
that is by design.

To continue stretching your analogy, I could claim that the post office
x-ray/bomb scanners are 'intercepting' mail sent to me when in fact it is an
understood part of the process to anyone who cares enough to research how
things work. Someone moving my mailbox [edit: filling out a 'change of
address'] and photocopying everything sent there is pretty clearly a
reasonable use of the word 'intercepting'.

------
acchow
I can't stop my @facebook.com address from appearing on my timeline. The
option shows up but clicking on it does nothing. Similarly, I can't make my
@gmail.com address show up on my timeline. The mouse listener is also missing.

------
reidmain
"Since Facebook changed the email addresses on contacts, I've actually lost
every single one of my email addresses including those for work." -Complaint
from iOS 6 user

My suggestion, don't use the very first beta release of a operating system on
anything important. Digital darwinism at work there.

~~~
s_henry_paulson
What about all the Android and Blackberry users who had their addresses
replaced?

What's the lesson there?

~~~
reidmain
Oh that is Facebook screwing up. I'm not trying to pardon that. I just hate
people who complain about stuff when they take a well defined risk and it
backfires on them.

------
SoftwareMaven
Given FB is already having a hard time generating revenue from mobile clients,
doing things that make mobile users afraid to use their service seems like a
really bad idea.

~~~
wutbrodo
Well, given that they can't monetize mobile usage and they can monetize
desktop usage (if somewhat poorly), technically driving users back to desktop
usage from mobile would be good for them.

------
ricardobeat
I'm still confused. Did the address book trashing affect only iOS 6 beta
users?

~~~
s_henry_paulson
It affected anyone on Android and Blackberry using contact sync, IOS6, and
Windows Phone 8

<http://www.bbc.com/news/technology-18687044>

------
GBKS
Now what happens if other sites and apps do the same? Will our address books
change every day and contacts fill up with dozens of unused email addresses?
Many of Facebook's growth techniques have been imitated by other startups so
going into the territory of manipulating data that has always been off-limits
is a big step. If another social site uses those techniques against to
manipulate data on Facebook profiles, will Facebook change it's mind?

------
leeoniya
i uninstalled FB app about 8 months ago, never used sync on it anyways since i
use FB more like a glorified twitter. been using mobile site since then (which
is quite good), posting photos by sending an email to a special status
address. so much more privacy. if they ever get rid of the mobile site to
force me to install the app, byebye facebook on mobile.

------
danso
I like FB so I won't be doing one of those "hey everyone, look at me I'm
quitting FB" acts. But I highly recommend that you don't put FB on your
phone...it's amazing how even a five second glance to check for new
notifications will ruin your train of thought. And also, not having it mess up
your phone account is nice

------
febeling
There is the product life-cycle which has the phases: introduction, growth,
maturity, and decline.

The point is that facebook needs to understand that "Move fast, and break
things" is an appropriate approach in the earlier phases, but it is quite
incompatible with later phases, when people start to rely on a service.

------
dm8
Why don't FB openly admit their mistake on their engineering blogs? At least
they can salvage already messy situation

~~~
Devilboy
They don't see it as a mistake.

~~~
rhizome
Or a mess.

------
dsirijus
Of course, they do thoroughly test their new ad options, limiting them to
region first, then measuring impact.

------
mariuolo
Why anyone would delegate handling of their address book to Facebook is beyond
me.

------
luminaobscura
this is why i have never sync'ed to fb or hotmail.

------
gitarr
If lawyers are reading this, a question:

Our governments are treating our emails more and more like real mail (aka.
letters).

Is there not a law prohibiting third party people or companies to mess
with(fake) my mail addresses?

Fact is, I do not want a facebook email address. And I have given no consent
for my email address to be changed.

~~~
rhizome
It's more like someone sending a change-of-address card to the post office to
have your mail delivered to their Mailboxes Etc. store.

~~~
tfb
That's actually a really good analogy. And the GP is right. There should
definitely be some laws to prohibit these kinds of things that Facebook is
attempting (doing), now that electronic mail/communication is becoming
standard.

~~~
rhizome
As a matter of fact, the (US) Postal Inspector considers this to be Identity
Theft.

~~~
tfb
Interesting. I wonder why no legal action has been taken against Facebook for
this recent fiasco... or has there?

