
What's up with HN? - kogir
I'll provide more details in a full writeup later.<p>We suffered a DDOS. The volume of traffic was sufficient to keep us from handling it in Arc like we always have before. Simply accepting and dropping all requests not from our office required 45% CPU utilization.<p>Now nginx is helping with some of the work. Ironically the transition was planned for today anyway, except it was meant to happen at night with no downtime. So it goes.<p>I'm fixing things as I find they're broken.  Please let me know if I've missed anything.<p>Edit: Yes, I know about and will fix all the SSL resources. Like yours, my Chrome window was also a portal to the '90s for a bit.<p>Edit Again: Your SSL resources should now be happy. Let me know if I missed any.
======
dpweb
Hi, it seems all the major sites that have been hit recently have been using
the "DDOS" term as a catch all, which really doesn't provide any insight for
those of us who are trying to protect our own sites and understand wtf seems
to be happening lately with all these high-profile attacks.

A site could, because of its own deficiencies in handling normal traffic, call
any outage a "DDOS attack". Not saying this is the case with HN, but see what
I mean.

Could you at least specify, is this a massive scrape, which would indicate an
attempt to pirate or steal information, or a SYN flood type attack (not a ton
of GETs) which would indicate an attempt to not steal information but disable
the site.

I believe some more insight into what is happening with all these major site
attacks will help us to protect our own sites better. Thx,

~~~
xb95
General practice in the industry is not to discuss the details of attacks
publicly.

As much as it sucks, the rule of thumb is that you need every advantage you
can get when it comes to being attacked. You gain little by talking about
these kinds of things publicly and stand to lose much (by giving away how
you're mitigating the problem, for example, possibly leading the attackers to
adjust their attack). It's just generally safer not to.

If you are someone who runs a web site, once you hit a certain size where you
have to worry about DDoS attacks you will certainly have the kind of industry
connections where you can talk about the issue privately and get help and/or
help someone. Below a certain size you just don't generally have to worry
about it -- and if you do get attacked, the response will mostly be done by
your provider as there's not usually a lot you can do if you're just a few
servers.

~~~
chimi
This sounds a lot like Security through Obscurity to me. Why does it work with
DDOS, but not source code?

~~~
lmkg
Security through Obscurity is actually a valid tactic, in most arenas. It
can't be relied upon _in isolation_ , which is what many people tried to do.
If you already have a robust defense system, it adds an additional layer.

Additionally, there are different trade-offs for DDOS vs source code. Source
code you leave behind obscurity, in order to get a well-tested and well-vetted
implementation. In DDOS, you're using ops, not code. All your responses are
custom-crafted anyways, so there is no well-tested implementation for you to
gain. The benefits of transparency are much smaller, and the benefit is the
same.

~~~
Hello71
Just in case anyone else was confused: The _cost_ is the same.

------
sillysaurus
Bug report for kogir: all items with an id less than 5million are 404'ing.

E.g. this loads: <https://news.ycombinator.com/item?id=5000000>

But this 404s: <https://news.ycombinator.com/item?id=4999999>

~~~
nwh
Looks like anything above ID 8000000 returns a 404 too.

~~~
kogir
Together, the two of you can probably guess the regex responsible :)

~~~
nitrogen
Does this mean that old content will be permanently inaccessible? One of my
(rarely exercised) pastimes used to be typing a few random five- or six- digit
IDs into the address bar, then following the chain up to the top-level
article. I found some really interesting stuff that way.

------
DanielBMarkham
I guess the rule is that you know you're making it as a startup when somebody
sues you. You know you're making it as a website when they DDOS you.

Geesh.

~~~
borlak
A DDOS of HN is quite strange indeed. There should be some kind of ransom
involved, but seeing as HN doesn't make money (I assume) from these forums,
there is nothing to ransom.

~~~
manojlds
For the people doing it, the gain can be about trying out their skills, not
being bored, and getting the bragging rights of having DDoSed "Hacker" News.

~~~
gingerlime
So if they then brag about it and it get posted on HN, should it get upvoted?
(not a rhetorical or cynical question, I'm really curious)

~~~
pc86
I'd upvote it if it was interested and they talked about how they did it (and
it didn't involve LOIC).

Which is to say the blog post about it probably wouldn't be worth reading :)

------
rwg
For what it's worth, <http://ycombinator.com/images/grayarrow.gif> is being
referenced by pages on <https://news.ycombinator.com/>, leading to mixed (SSL
vs. not-SSL) content fun in browsers.

Also, if SSL is now a permanent thing for HN, it would be a nice bonus to see
"add_header Strict-Transport-Security max-age=31536000;" in the nginx
configuration block for the https server...

~~~
xPaw
I think arrow should be replaced with raw css, or converted to base64, since
it's very small image, it won't be big as base64.

~~~
stock_toaster
Or at the very least set a long future expires set on it....

I doubt the arrow is going to change much, and if it does, just use a new
reference!

------
signed0
It would be really nice if there was an official Twitter account with status
updates for things like this. Might reduce the amount of refreshing.

~~~
CoachRufus87
Or we could step away from the keyboard for some fresh air :-)

~~~
sideproject
not possible.

~~~
LolWolf
Fresh air? What's that?

~~~
PhearTheCeal
I think it's a new Javascript framework. I hope it is REST-compliant.

~~~
LolWolf
Oh, yeah! Well, that makes two of us.

------
joonix
Thanks for spending your time on a site that's essentially one big favor to us
all.

~~~
hnriot
It's also (hn) an effective way to get quality beta testing of new startups.

------
heartbreak
For what it's worth, HN is loading extremely quickly now. Is that the nginx
transition? Usually it takes a few seconds to load, say, my user profile. Not
anymore. Nice work!

------
iatecake
Chrome is still throwing a small fit about insecure elements. Change the
favicon url to a protocol-relative one.

------
6thSigma
It's surprising that there still isn't a sure-fire defense against a DDOS
attack.

~~~
nwh
That's a bit like asking for a truck-proof bicycle.

~~~
cft
Not quite. There's one sure protection from DDOS actually: scale. For example,
Facebook or Google are practically immune from DDOS, since you cannot really
overwhelm an infrastructure of that size, even with 100,000s of bots. If you
have 10,000s of servers, IP anycast to multiple datacenters, multiple 10Gbps
uplinks, you are immune from most DDOSs- they are simply handled like normal
traffic fluctuations.

~~~
nwh
To extend the metaphor; Google and Facebook have 1000000 bicycles, and you
can't take them all down with a single truck.

~~~
martinced
Your second metaphor ain't correct. Google and FB's normal way of executing
doesn't consist in sending regular trafic to a bicycle (server) until it
crashes and then re-dispatching that legit trafic to non-crashed servers.

Sure a server does crash once in a while but it's not because they purposefuly
did overload it.

The DDoS attempt trafic is dispatched left and right just like regular trafic
and doesn't affect the normal behavior.

It's more like the single truck can hit any of the 1000000 flying bycicles.

: )

~~~
andrewem
Are you saying that Google and Facebook respond to DDoS attacks by just having
enough capacity to serve all the attacking requests, all the way to rendering
the pages as though they were requests from legit users? And if so, do you
have first-hand knowledge to back that up?

Consider the fact that each of those companies has many services, which can
vary widely in usage and capacity.

------
kostya-kow
>The volume of traffic was sufficient to keep us from handling it in Arc like
we always have before.

>Now nginx is helping with some of the work. Ironically the transition was
planned for today anyway, except it was meant to happen at night with no
downtime. So it goes.

Does this mean that HN no longer uses Arc, but nginx? Or are you using
nginx+Arc now?

~~~
rgbrenner
Arc is a programming language. Nginx is a web server.

------
jakozaur
Why not use CloudFlare? It is probably the easiest way to deal with that kind
of situation.

------
rcfox
So is the redirecting to HTTPS thing going to permanent then?

I guess I'll have to figure out how hack some Chrome extensions that haven't
been updated for 3 years since the creators seem to have hard-coded the HTTP
URLs.

------
Confusion
Well, it's lightning fast now. I don't believe I've ever seen it this fast.

------
gnosis
Is HN having SSL problems? Firefox is complaining that HN's SSL cert has not
been verified. A screenshot can be seen here: [1]

It used to be fine up to today.[2]

[1] - <http://img1.imagilive.com/0313/hn-cert-130311.png>

[2] - Just FYI, for me, news.ycombinator.com resolves to: 184.172.10.74

~~~
NathanKP
I had this problem the other day with a site I administrate, in Firefox only.
It turns out that Chrome and other browsers have robust and complete
certificate chains built in which allow them to trust certificates that
Firefox over zealously assumes are not verified if you don't explicitly define
a certificate chain file to link your certificate issuer to a root certificate
that Firefox does trust.

So long story short: Firefox demands a chain file for some certificate issues
while other browsers just trust the certificate.

~~~
schrodinger
I bet what actually happened was that your chain was set up incorrectly, but
was cached in Chrome so it appeared ok. If you've already hit another site
that uses the same root certificate as your chain, then your site would appear
fine. If you haven't (i.e. testing your site in a browser you don't use much),
than you'd see an SSL warning. So it might not have been anything different
about Firefox, just that Chrome trusted the chain because it had already
verified the root for another site with the same chained SSL certificate
issuer.

~~~
NathanKP
That actually could be. I use Chrome primarily and rarely use Firefox so
Chrome probably trusted my Network Solutions SSL certificate because it had
already encountered and cached a chain file linking Network Solutions back to
a trusted root certificate, but Firefox had not cached that chain file yet.

~~~
rwg
At ${PREVIOUS_JOB}, I despised that certificate caching behavior. It did
nothing but wallpaper over configuration errors that shouldn't be wallpapered
over...

Me: "Your site at <https://blah.otherdept.myemployer.edu/> is causing visitors
to see SSL errors because your web server isn't sending the certificate chain.
It probably got messed up when an updated certificate was installed the other
day."

Other sysadmin: "It works fine for me. Try clearing your browser's cache."

Me: "No, really, it's not that. Here's the openssl s_client output showing
that your server is only sending its own certificate and not any intermediate
certificates."

Other sysadmin: "I just tried from another computer in the office, and the
site's working fine. You should call the university's helpdesk since the
problem is obviously on your end."

Me: _profanity_

~~~
nwh
Funnily enough, I had the same conversation just last week.

------
twr_salt
SSL on HN no longer supports RC4 ciphers, see BEAST Attack and the full report
on

[https://www.ssllabs.com/ssltest/analyze.html?d=news.ycombina...](https://www.ssllabs.com/ssltest/analyze.html?d=news.ycombinator.com)

------
mikegioia
while you're in there, can you please increase the default font size on this
site!?

~~~
youngerdryas
Try ihackernews.com for mobile HN browsing.

------
hakaaaaak
It would be nice next time to direct to a static status page with info on what
is going on, if possible, and keep that site up all the time, e.g.:
status.news.ycombinator.com.

------
btipling
Why do people bother with ddossing, it just always goes away anyway. I can't
imagine that it would be anything other than kids since it achieves no aims in
particular.

~~~
taf2
Not true - it can be used effectively to distract from a primary more targeted
attack...

------
bsimpson
The //news.ycombinator.com comments links in the RSS feed aren't working from
Pulse for Android. Can we put the https scheme in the RSS feed permanently?

------
tolmasky
I have a suspicion that average karma isn't updating.

------
samspenc
Can you let us know if this was done by a "large state actor"? Any reason why
you think this could have happened? Any specific target?

------
martin_
I can't submit, it just says "Please try again" - been like it for a few hours
now

------
nasalgoat
What about the RSS feed? All the links point to file://.

~~~
nasalgoat
Aha, it's a problem with NetNewsWire not liking <https://> URLs. That's a
problem.

------
pmtarantino
Who did it?

~~~
spinlockmusic
Since attacks are distributed amongst several infected zombie machines with no
motive or intent, it can be difficult to pin-point the true source of the
attack.

~~~
skeletonjelly
We really need to standardise a protocol for this so at least victims know who
is taking them down

/half-s

~~~
xk_id
To the IETF!

~~~
yskchu
Don't forget to tell them to turn on the Evil Bit:

<http://www.ietf.org/rfc/rfc3514.txt>

------
jimmychu0807
Thanks

------
probablyshit
Interesting info, happy the site survived.

------
hallpined
Thanks for the update. But how is that ironic? Do you mean coincidentally?

Sorry, I don't mean to be a pedantic jackass, but still I said it.

~~~
paranoiacblack
I'd say that depends on what type of irony you're considering:
[http://en.wikipedia.org/wiki/Irony#Irony_of_fate_.28cosmic_i...](http://en.wikipedia.org/wiki/Irony#Irony_of_fate_.28cosmic_irony.29)

I imagine you're referring to situational irony, but there are a few more
literary ways to use it. Ignoring that, it really isn't hard to see how it is
ironic that they planned to do today anyways, but instead of everything going
smoothly, there was a severe unseen outage that they weren't prepared for.

~~~
edmccard
When the whole trend of "irony has only one meaning" start? Every dictionary
I've looked at includes something like "incongruity between the actual result
of a sequence of events and the normal or expected result" (I've got an old
Webster's New World from the 40's that gives the example "it was an irony of
fate that the fireboat burned and sank, which is exactly the kind of thing
that a certain type of person claims is "coincidence, not irony").

Is it all down to that one episode of Futurama?

