
Facebook's Realtime Updates API has been down for 12 days - jobeirne
https://developers.facebook.com/bugs/738043236218053/
======
bri3d
As noted in the comments, this is in stark contrast to Facebook's new SLA
policy announced at F8, with the patch seemingly slated for release in
Facebook's standard iteration rather than fast-tracked to "within 48 hours."

[http://thenextweb.com/facebook/2014/04/30/facebook-
announces...](http://thenextweb.com/facebook/2014/04/30/facebook-announces-
two-year-stability-guarantee-core-apis-sla-fix-major-bugs-within-48-hours/)

It seems as though they're already playing the "it's not a major bug" triage
game that most companies find themselves trapped in upon agreeing on an SLA.

Obviously depending on Facebook's APIs for your well-being has never been a
good move, but this API seems particularly egregious to break as it's mostly
used by branding/marketing managers, the very people who give Facebook their
revenue stream.

~~~
zenogais
You would be surprised. I worked on a marketing product for a major agency.
This product heavily utilized the Facebook Ads API, and I can tell you that it
is extremely fraught with problems. Often times we were left to work around
issues in the API as fast-tracking a fix was near impossible even being a PMD.

~~~
CaveTech
Ads API was and still is full of bugs and undocumented features.

~~~
hagbardgroup
Move at whatever speed and let it stay broken and confusing.

------
sicross
I'm the Product Manager on Facebook's API team.

On April 24th, it was first reported to us that the Realtime API was not
sending updates when people comment or post on Pages. As of now, a fix is live
and in production.

This was not an easy issue to diagnose the root cause of, and thus to
determine how many people, developers and apps were affected. But while it
took some time for us to diagnose, as soon as we did, we merged and pushed a
fix within a few hours.

At f8, we announced an SLA of 48-hours for the highest priority bugs. We
announced this because we believe stability matters. It matters to us because
it matters to the people who build on our platform.

In this specific case, the bug with the Realtime API didn't prevent apps from
accessing data about likes, posts and comments on pages - those were always
queryable via the API on demand. That, along with our understanding that this
issue affected only apps subscribed to updates on page content, meant we
didn't, on this occasion, classify the bug as within the 48-hour SLA. However,
we're constantly listening to our community, and may revisit how we classify
and prioritize issues like this going forward.

Every bug matters. The very fact that a developer has reported an issue to us
means it's affecting someone. It's on us to respond quickly, gauge how many
users and apps are affected, and get to work fixing it.

We apologize for the disruption this issue has caused the affected apps, and
are doing everything we can to reduce the chance of this happening again.

~~~
kungfooguru
> But while it took some time for us to diagnose, as soon as we did, we merged
> and pushed a fix within a few hours.

The comment in the thread says on May 1, "We have a fix for this issue and it
will be pushed on or before our next weekly push next Tuesday"

But it didn't go out until May 5.

~~~
sicross
We initially thought we had a fix, but upon further investigation, we
discovered a deeper issue. I agree, we could have been better at updating the
bug with this information.

In the meantime, the severity of the issue became more obvious, which is why
we accelerated the fix and deployed it today (May 5th) ahead of the regular
weekly push scheduled for tomorrow (May 6th).

------
sheetjs
Apparently you must have a facebook account to read this. Does the link point
to a status page or a discussion?

~~~
bri3d
It links to a bug in Facebook's developer-facing/public bug reporting tool.
Lots of angry comments about various products (mostly marketing related)
depending on the API for updates, suggested workarounds (rewrite to query/pull
rather than accept pushes), and an indication from Facebook that the fix will
go out with this week's standard Facebook API iteration (they're released on
Thursdays).

This is mostly of interest because at the recent F8 conference Facebook
announced a "48 hour SLA" for "major bugs" and it appears this bug wasn't
"major" enough to be fast-tracked.

------
striking
The tone of the comments is actually really striking. It almost sounds like
people are pleading with Facebook to get their act together.

~~~
masterleep
New to FB platform work, eh?

~~~
mjn
_" Like the catcall of 'whore' or 'crook,' the Facebook Platform passes
judgement before you even signed up for it. ... If you were innocent, why did
you start using the Facebook Platform in the first place? ... Maybe it's me,
maybe it's me, Facebook devs whisper quietly, alone, every Tuesday, before
heaving the deep, lumbering sighs of resignation beyond sorrow."_

[http://www.bogost.com/blog/oauth_of_fealty.shtml](http://www.bogost.com/blog/oauth_of_fealty.shtml)

------
yeukhon
I am having trouble understanding what the problem was. I don't even think he
explain anything...

 _In this specific case, the bug with the Realtime API didn 't prevent apps
from accessing data about likes, posts and comments on pages - those were
always query-able via the API on demand. That, along with our understanding
that this issue affected only apps subscribed to updates on page content,
meant we didn't, on this occasion, classify the bug as within the 48-hour SLA.
We're constantly listening to our community, and may revisit how we classify
and prioritize issues like this going forward._

I'd like to hear a detail postmortem.

------
mattyohe
Of course it is. “Move fast and break things”

~~~
psychometry
Don't forget the third and fourth tenets of Facebook API design: "Don't
document anything, but if you must, do it poorly."

~~~
Curmudgel
"Don't document anything, but if you must, do it poorly" is also sometimes
called the "Unix Philosophy".

------
sergiotapia
I -knew- something was off! My timeline hasn't been updating on facebook.com
for a long time now. I have to refresh the page for it to load new items. This
is a bummer, 12 days is a long time, especially for a company the size of
Facebook.

~~~
rohandhruva
This bug has nothing to do with timeline freshness: the bug affected
developers and not users. If you're having issues with the timeline, please
report an issue here:
[https://www.facebook.com/help/181495968648557](https://www.facebook.com/help/181495968648557)

~~~
rlu
That link is for reporting abusive content, not reporting bugs.

------
lurkinggrue
Nice, At least it's down in realtime.

------
alttab
On April 24th, we got one ticket out of a thousand like we do any other day.
We fixed this bug like we do all bugs.

Shits fucked here, ok? We had no idea if you were some asshole who can't read
a tutorial, or if you have some phantom dating app with 1,000s of users. When
we finally got around to it, we fixed it quickly because we are boss.

When we lay down our new rules, we said we would get to the important things
sometime this week. We want to demonstrate we have a process for our madness.
You continually bitch about it so we know how important you feel it is for us
to move slower and stabilize things.

You act like the API wasn't responding with data, or that it was all data all
the time. It was some obscure content edge case that doesn't provide us any
real direct revenue. How would you have expected us to react in a situation
like this?

We know bugs are important. We don't open them on the weekends for fun anymore
either. We get to decide whats important and how, and then fix it when we want
to or can. We fixed the bug, and are going to find someone to blame about it
so you can all stop worrying.

~~~
kamobit
Can't tell if this is a serious reply, but it seems to go against the
conclusion of F8 where Mark said, "My goal for our culture over the next 10
years is to build a culture of loving the people that we serve that is as
strong if not stronger than our culture of hacking at Facebook." [source: end
of the article here [http://goo.gl/iwWlck](http://goo.gl/iwWlck)]

