

Ask HN: Oprah's effect and scaling - gommm

One of my customer's website is going to be mentioned on Oprah's Winfrey show in a few month. Anyone here has had any similar experience? What kind of traffic should I expect?<p>The website in question is a SAAS with no free account developed in Rails. What should I do to make sure it scales?<p>Since users of the website don't need to interact between each other, I was thinking of having one load balancer running nginx proxying to different nodes with each node having it's own database and application server.<p>I would hash the email address to chose the node to proxy to and then store that information in a cookie. So nginx would either look for a cookie, or an email address in the post data and proxy to the correct node accordingly. Otherwise, it would redirect the user to the login page. Does this approach make any sense?<p>I'm currently using rackspace cloud for this which gives me flexibility in terms of adding and removing instances but does mean that I have less raw performance than with a dedicated server...<p>Has anyone got any stories of this kind of exposure to share?
======
patio11
Are you familiar with how middle-aged women actually use the Internet?

Step #1: Make sure you dominate rankings on "the Googles" for every
conceivable way someone with half an ear on the television could mangle your
name or category. She isn't going to product.com, she is googling for
[product], [product.com], [products], [praduct], [pro ducks], [what was that
thing on Oprah this morning?], etc.

Step #2: Expect less traffic than you think, because while Oprah's audience is
ginormous they are in TV watching mode rather than browsing the Internet mode,
and large portions are going to fall out of the funnel in a) trying to go to
your website and b) doing so successfully. Oprah's peak audience is somewhere
around 9 million people. If you're expecting 500k visitors to your website, I
think you're being overly optimistic (absent Oprah literally doing a call to
action on your behalf). So it isn't all that much worse than a Slashdotting,
and you can expect the people to come in over a wider interval than "right the
second the URL appears on the screen". Any reasonable website can chew through
500,000 visits spread over a 48 hour interval if it survives the initial bump.

Step #3: Scale the obvious bottlenecks and get ready to do some firefighting,
though you probably won't need to. If the home page makes 10 SQL queries right
now, a) fix and b) replace with static HTML for the day (or get a process in
place where you can do that quickly if the server starts to chug). I'd put a
big Welcome Oprah Viewers banner at the top of it with a sales pitch
leveraging that social proof for all you are worth and giving them "special"
advice for using the site, by the way.

Step #4: Have a backup plan. If it seems highly unlikely that you're going to
survive, switch all or most users to a static site which explains that "due to
overwhelming popularity" the service isn't available at the moment but you'll
email them the second it comes back up if they just drop their email address
into this MailChimp signup form. Do so. Give them an option from hearing from
you in a less transient fashion, too. (I would strongly consider "Check if
they're cookied for your site, if so let them in, if not give them the static
pseudo-site" but that doesn't scale to the moon like "hello new client, here
is index.html, bye" does.)

~~~
maukdaddy
_I'd put a big Welcome Oprah Viewers banner at the top of it with a sales
pitch leveraging that social proof for all you are worth and giving them
"special" advice for using the site, by the way._

^^ This. You are in a unique position of being able to PLAN for a huge spike
in traffic. While planning for the technology impacts are important, do NOT
forget the business aspects. You already have a damn good idea of the
demographics of more than 100k visitors that might be visiting that day, so
get started now.

Do you want to offer a special "discount" for Oprah visitors that day? Does
the signup process need to be simplified for less tech-savvy visitors? I
definitely agree with Patrick that you want to have a targeted set of
instructions or otherwise customized copy for the visitors that will come that
day. Above all else, PLAY UP THE OPRAH CONNECTION! A banner, special icon, her
picture, etc. Visitors should be smacked in the face early with some
indication that they have found the site mentioned on Oprah.

You might also want to find any Oprah-related message boards and create
accounts now. Then on the day the site is mentioned in Oprah, go to those
message boards and post a clickable link. Not only will this help the visitors
get to the correct site, but they can use it to share the correct link with
OTHERS.

------
fredleblanc
Oddly enough, I worked on a project that had a whole show dedicated to a site
that I built. There were banner ads on Oprah's website and a bumper in the
show with the URL on it.

My boss and I sat watching and waiting and we saw virtually no effect from the
show whatsoever. (If you were just watching top on the servers without knowing
Oprah was talking about it, you would have assumed it was any other day.)

We spent time theorizing why this was. We came up with the following:

\- The site that we built has a relatively long URL, so maybe it was hard to
remember.

\- The bumper that the URL appear at was towards the end of the show, at which
point we figured most stay at home moms/dads/people that watch Oprah were
probably starting to get dinner ready.

\- I'm not sure that people that watch Oprah are the same people that eagerly
sit in front of the TV with an iPad/laptop/etc.

Of course, your mileage may vary. If you're using Rackspace, I'd say you're in
good hands. It may be worth mentioning to them when it's going to happen and
seeing what they can do in case of sudden load, but for us it certainly wasn't
the huge gush we were expecting/hoping for.

~~~
bradleyland
I had the same experience. We had a customer whose cookies were mentioned on
Oprah and then Rachel Ray at a later date. We saw a bump in traffic, but the
customer didn't warn us until a few days before air, so our recommendation was
to leave the site alone rather than risk introducing changes that could
produce bugs. We put a little time in to tweaking the site's error pages to be
more friendly, but when the air dates came, nothing spectacular happened. We
actually saw a little bit more traffic from Rachel Ray than Oprah, probably
because cookies are a more relevant topic for her show.

------
stevenbedrick
In a previous life, I was half of the engineering team at a call center. On
more than one occasion, one or another of our customers (always a different
one) would call us and say that their product/service/organization/whatever
was going to be "mentioned" on Oprah "next week", and could we please set up
some extra phone-answering capacity for them in order to handle the inevitable
spike in demand.

At first, we would try and accommodate their requests- bring in an extra
operator or two, do a special answering script, etc. Besides being good
customer service, we figured that increased usage on the part of our customer
translated into more money for us, so it was worth spending some time on our
end helping them get ready for a spike.

After a little while, however, we eventually stopped taking "Oprah-induced
capacity panics" so seriously. Why was this? Because we never- not once- saw
any serious spike in calls from any of our customers' Oprah "mentions".

I'd suggest planning for some extra capacity, just in case, but don't go doing
too much special-purpose engineering work for this one customer. Unless, of
course, their "mention" on Oprah is really more of a "entire episode dedicated
to their product" kind of a deal- that might be a different kettle of fish
entirely.

~~~
gommm
It's more toward the side of the entire episode dedicated to the product than
to the casual mention...

~~~
stevenbedrick
Hey, well, in that case, your mileage might definitely vary from what we
usually saw. I suppose it's always a good idea to have plans for this sort of
spike, Oprah or no...

------
jhherren
First off, congrats! I was in charge of the website when Oprah featured The
Secret a few times on her show. Here's a few pointers from my experience.

1\. Remember the show airs in Chicago in the morning, then more than once
across time zones, so this will be an all day treat. We had a lot of buzz
(Oprah's own forums went down under load for our show), and our URL was
displayed on the screen. Regardless, you can expect a lot of traffic, mostly
during the show and a few days afterward.

2\. Your db will likely be your bottleneck. Make sure you don't have any
unoptimized queries, or better yet, cache as much as you can so you're only
hitting the db on writes for all the orders you'll be getting.

3\. If possible, offload static content to a cdn. Akamai saved us here,
because the site was not optimized size-wise.

4\. What ultimately caused us issues was slow clients. I would definitely try
to have a caching proxy up front. The snowball effect would cause us to tap
out our 100Mb connection at times. Since you're using cloud hosting hopefully
that's not a hard limitation for you.

5\. It's _really_ fun watching your bandwidth. I used a console tool called
bmon. You'll be able to tell when commercials happen during the show.

6\. +1 for a backup plan. Have a static page ready that you can switch over
to. I'd highly suggest having an alternate payment processor, even if you have
to go with Paypal. Since you're SAAS, you can't rely on retail or Amazon, so
you may not get visitors to come back if your site fails. Make sure you can
capture those customers the first time.

7\. If you don't plan on having a free account, offer something so you can
build your customer list and capture contact info. A whitepaper, newsletter,
etc.

Have fun!

~~~
gommm
Thanks a lot for the answers!

1\. Can you tell me approximately how many unique visitors you had during the
shows? I'm kind of curious...

3\. We're using cloudfront... So you'd advise on using Akamai instead? or
would cloudfront be ok?

6\. For payments, we're using Authorize.net (with the CIM feature to capture
credit card information)... Hadn't thought that they could be a bottle neck
but I'll definitely at least check up with them...

7\. Makes sense, I'll suggest that...

It'll be a fun few months preparing... I've been meaning to play around with
Varnish, now's my chance :-)

~~~
jhherren
1\. Can't give you any real numbers, but dude, it's Oprah.

3\. Any cdn should be ok. You just want to offload those files to somebody
else. Remember that browsers will only download a few resources from a single
domain concurrently, so even if you have a subdomain for statics you'll get a
faster page load. Hint: use yslow for more frontend optimization tips.

6\. Authorize.net can handle it--we used them. But if your shopping cart
can't, it's always nice to have a Paypal 'buy it now' button ready to go. Not
real elegant, but it's a backup plan.

~~~
gommm
Thanks!

3\. I'm a big fan of yslow too... It's really great for that :-)

------
ethank
I have the lucky position of having to do scaling for not only Oprah, but
american idol, the superbowl, x factor, inaugurations and more.

we have had artists on Oprah many times, probably a dozen in the last five
years. It has nearly no effect. The largest bump we see is during the central
timezone broadcast. The east and west code feeds have negligible effect unless
you ask for it by doing an "expensive" call to action on the site. I define
that to mean asking every user to do a db or server intensive task, like
signing up for a site, logging in, commenting, etc. We generally avoid db
writes during load events.

I would recommend a good cdn though and offloading as much as possible to
them.

Incidentally the biggest impacts for us have been xfactor, american idol,
glee, and the biggest was Tom petty on the superbowl. The only time I have
ever watched it willingly.

------
davidw
I think part of the tricky part in this is that, most likely, it has to scale
up, but not at all gradually, and then scale back down too, in terms of costs,
as the traffic dries up again. Good luck:-)

~~~
w1ntermute
Amazon EC2 might be a good option.

------
petervandijck
A good approach could be to put all the static stuff (the sales pages, pricing
info, everything that doesn't require backend) on 1 domain, and put that on S3
and Cloudfront. That definitely won't go down, and it will be fast and take
the brunt of the load.

Then put the interaction stuff (the forms etc.) on a subdomain, and put that
on heroku or whatever with plenty of servers.

------
apalmblad
While I've never been on Oprah, I have had my startup featured (~8 min.
segment) on a show on the CBC (Canada's national broadcaster.)

Hardly a blip in load.

We had a few more emails than usual offering "advice," though.

That said, it sounds like a good excuse to take a good look at your caching,
landing pages, and conversion funnel. And, if you are worried about needing to
scale fast, consider something like chef for automating infrastructure setup,
so that even if you don't need it for Oprah, it's work that you'll use when
you do need to scale.

------
orta
heroku?

~~~
DanielRibeiro
Heroku is great. However, if you use some other system besides rails iself
(like custom processes, things other than DJ for job scheduling or calling
process that require root access from you app) you may find it will be hard to
migrate. Otherwise it is a piece of cake.

~~~
gommm
I hadn't thought about using them, I've had bad experiences before with
engineyard's appcloud (mostly during their beta and because I needed to do
something a bit more complicated than what they normally support out of the
box, forcing me to play with custom chef recipes that started failing when
they changed their base system)

But, yeah I think I should definitely check them out

------
iterationx
Inadequate phone support is possible failure point.

------
earle
maybe jhh will chime in here as he went through this exact situation @ the
secret

