

Yahoo’s fantasy football debacle – and why it was being compared to Obamacare - circa
http://www.washingtonpost.com/news/morning-mix/wp/2014/08/25/yahoos-fantasy-football-debacle-and-why-it-was-being-compared-with-obamacare-last-night/

======
kareemm
I was one of the lead devs on ESPN's "League Manager" Fantasy Football engine
rewrite back in 2002.

The Yahoo outage reminds me of that experience. League Manager had no clear
project or product management leadership. After four months of 60h weeks, most
working 7 days, we ended up in a late-August crapstorm when we launched an
incredibly buggy product. Things stabilized in about week three of the NFL
season and I took the next three weeks off to recover, telling myself that the
Christmas bonus would be worth sacrificing my time, energy, health, and
relationships.

My key lessons:

0\. My bonus was $1500. Not only was it not worth it, it was a slap in the
face.

1\. No death march projects like that ever again.

2\. The opportunity cost to work on a project like that wasn't worth it, given
the tiny rewards. An instructive lesson that helped push me towards
entrepreneurship - working hard isn't a problem, but it is when the rewards
are controlled by someone else.

3\. In retrospect, I'm incredibly thankful that social media didn't exist. The
closest we had were user forums. A quick perusal of said forums ensured a
crappy day due to the vitriol and hate thrown towards the dev team, who were
busting their humps to fix the product. I remember one commenter who
sympathized with the dev team -- whenever I was down in the dumps, I would
read his comment to trudge on. It motivated me for weeks.

For anybody on the Yahoo Fantasy Football dev team, it gets better. Take some
time, reflect, and do your best to integrate the lessons you've learned here
as you decide what to work on down the road.

------
dchuk
I got majorly screwed in my draft yesterday because of this. What was strange
was that it only affected about half of the guys in our league. That still
means half of us auto-drafted though.

Yahoo just emailed out their solution to everyone:

"Dear Yahoo Fantasy Football Users,

Yesterday evening we encountered an issue with a portion of our Fantasy
Football drafts and one of those affected was yours.

We're so sorry for the inconvenience. To fix your league, the Commissioner
needs to use the reset draft tool and reschedule your draft date and time.
Please note, the new draft time must be at least two hours out from your
original draft.

If you need assistance, please contact customer care

Thanks and again we apologize,

Yahoo Fantasy Football team"

Doesn't really solve anything for anyone. I've been trying to get my league
commissioner to re-draft or at least open up a vote for a re-draft but he has
deemed that unfair to everyone else. There goes my season.

~~~
bmelton
If it's a pay league, I'd demand a refund. It _might_ be unfair to make
everyone that was unaffected redraft, but it is _decidedly_ unfair to have a
draft in which members of the league weren't able to draft through no fault of
their own.

If it's not a pay league, then it might not be the end of the world. You'll
never be able to get the same sense of pride if you do well though. :-\

Maybe you could propose a vote from the other members, as a first step, and
hope that they aren't as short-sighted as the commissioner?

~~~
dchuk
It is a pay league, but it's a little tricky because the commish is one of my
best friends/former roommates. Since it's fantasy football, I've taken to just
talking a whole bunch of shit until action is taken. It's looking like we'll
re-draft sometime this week I think, fingers crossed!

~~~
jbigelow76
Commish is pushing back against a re-draft? Safe to say he wasn't hit with the
auto draft bug then right?

~~~
dchuk
Bingo

------
yummybear
Oh god.

I'm a developer for a fantasy soccer, football etc. site and this is my
ongoing nightmare. Fantasy sport provides some interesting challenges as a
developer, but the fact that you are operating with deadlines and windows
outside of your control and server loads that vary by a factor of 5-10 in
minutes is... challenging.

If your site stumbles at the wrong moment (and we have on several occasions)
you are basically screwed. You can't call FIFA or NFL and ask them to delay
the event for a few hours while you put out a fire, or finish developing that
new feature.

I can feel my stress level rising even as I'm writing this...

~~~
peterwwillis
Relax. You are not the person responsible for the site being up. That's Ops,
QA, and support's job. All you need to do is provide a working product ahead
of time to the other teams to load test it and figure out stopgaps for any
holes found.

You can of course expect to be working at launch time, crushing random bugs
and pushing fixes to production. But those should be minor issues. If everyone
else has done their job, you will never have a Yahoo-esque outage, and you
certainly shouldn't be held responsible.

For the downvoters who don't find this comment helpful, I worked for four
years in CBS's fantasy sports operations team, handling hundreds of thousands
of hits per second (dynamic content) and tens of gigabits of sustained traffic
on launch days, and we never blamed the developers for an outage. (We did,
however, have them on hand to look at problems)

~~~
yummybear
Well we're a small shop, so I am Ops, QA and support ;)

~~~
peterwwillis
Well then, i'm sorry to hear that! But still, try not to get stressed if you
have problems doing the job of 10 different people. My suggestion is to focus
on making the application robust against errors and invalid data, and
determining the performance limits of various parts of your application stack,
then making sure you allocate headroom for the expected traffic _well before_
the event - none of this automatic scaling business.

