
OVH servers exploited through shellshock - spindritf
http://status.ovh.net/?do=details&id=8120
======
conradk
That's pretty crazy, though with the hack publicly disclosed, it was bound to
be exploited. What I don't understand is why this report incident comes so
late. The hack has been public for a large amount of time. Could anyone
explain?

~~~
atmosx
I manage 2 VPSs. I'm responsible for security patches and everything. If shit
hits the fan, it's because of me not because of my VPS provider :-) That's
what VPS is all about.

~~~
freshflowers
True. However, since most clients (especially those that don't maintain their
servers well) use standard images provided by the hosting provider, I don't
understand why those images don't come with automatic security updates
configured by default.

I know that many of us that run systems for which availability criteria mean
they want full manual control over all updates, but those are the people that
either use custom images or know how to turn those automatic updates off.

Security updates are finally pushed hard on desktop users for obvious reasons,
with everyone and their mother running cheap VPS's these days the same logic
should apply to virtual servers.

~~~
kijin
Automatic updates for server software can be tricky.

Ubuntu, for example, can be configured to run apt-get update && apt-get
upgrade automatically every day (using unattended-upgrades), but AFAIK the
default configuration only checks for updates, notifies the admin, and waits
for approval before actually changing anything.

Even though Ubuntu LTS, Debian Stable, CentOS, etc. are supposed to be stable
distributions, it is not unheard of for routine updates to break something
mission-critical. For example, the package maintainer might have made some
changes to the default configuration. (C'mon, do you really need to make minor
changes to nginx.conf all the time?) A daemon might go down and fail to come
up again after an update. (apt is particularly annoying in this regard, since
it forcibly restarts daemons during an upgrade, and not always in a sensible
order. Result: HTTP 502 Bad Gateway.) So unless a human is present to spot any
issues immediately after an update, you might be left with a broken system at
an inconvenient hour. Which is why the documentation doesn't recommend fully
automated updates.

Unfortunately, the set of VPS owners who don't apply updates regularly
probably has a large intersection with the set of VPS owners who won't be able
to troubleshoot a broken update anyway, so maybe this concern need not
apply...

~~~
robszumski
We're working on the auto-update model for servers at CoreOS. If interested,
you can read more about our update philosophy here: [https://coreos.com/using-
coreos/updates/](https://coreos.com/using-coreos/updates/). We think that this
is one of the best ways to secure the backend internet, similar to how auto-
updating browsers moved the front-end of the internet forward.

(CoreOS employee)

------
Tepix
I have had a bad experience with the OVH abuse department. I sent them email
twice (once about spam, once about network attacks) and never received
anything, not even an automated email delivery confirmation.

~~~
balladeer
OVH/Kimsufi has non-existent support. They claim to offer support (yes, every
kind - from listing your passport verification request to tax exemption
request) via a publicly accessible forum. Even that support is non existent.
All you get there is few useless answers from few uselessly enthusiastic
(sometimes just smartass) fellow customers.

------
ck2
Their network was so saturated, we were only getting 1KB/sec inbound
yesterday. Fun times.

Their VAC protects external inbound but does nothing in their intranet which
runs at full tilt 1gbps for most dedicated servers.

~~~
SG-
They've been having capacity issues for almost 2 weeks now at peak, you can
see it at [http://weathermap.ovh.net/usa](http://weathermap.ovh.net/usa)
usually. Capacity inside their network between Montreal and Newark was at 100%
until an upgrade was done last week (edit: the upgrade isn't finished yet, but
so far it's helped):

[http://status.ovh.com/?do=details&id=8038](http://status.ovh.com/?do=details&id=8038)

Tata transit in Montreal has been saturating at peak too.

I'm hoping all this was just internal botnet/DDoS that was ramping up and the
issues will go away as they clean things up.

------
cpsaltis
It is sad that so many servers are still vulnerable on an issue that has been
reported through all major mainstream news networks. We've witnessed several
attacks many days after Shellshock was fixed, and chasing down the botnet
scripts we saw hundreds of servers compromised.

The story and scripts were published on a blog post in case anyone wants to
check out a standard botnet attack:

[http://blog.mist.io/post/100582053116/anatomy-of-a-
shellshoc...](http://blog.mist.io/post/100582053116/anatomy-of-a-shellshock-
botnet)

------
napsterbr
I feel it's my clue to ask a question I've been wondering lately: how good is
OVH's _dedicate_ server service? (Regarding network, uptime, system outages
etc).

I used one of their dedicate servers for six months and never had any problem
with it, however I plan to buy (rent) dozens of big-data servers for my next
project, and while my experience so far have been good, I'd like to hear from
someone else.

Despite of that, I had several downtime issues with VPSs hosted at OVH. I
guess the difference is you can't oversell dedicated servers.

~~~
yc1010
I had a few expensive top end storage servers (36 drives), they worked fine
for years, only 1-2 drive failures, no power failures or major network outages
I can remember.

Last year i switched to collocation instead, but still have 1 server with
them, things have improved alot lately, now the servers come with ipmi and new
ovh control panel is nice.

Problem with OVH are noobs buying their servers trying to save every buck only
to realise that the support will not fix anything but hardware issues (loose
cable, broken drive etc). Obviously alot of people do not understand that
unmanaged means unmanaged

------
phreeza
Completely forgot I had a server running there, I think it is actually a
dedicated server, not VPS. It was vulnerable but seems to be uncompromised.
Fixed it now.

~~~
djm_
I'm sure you're well aware but that server should not be trusted now for
anything sensitive (or anything at all really) until it's been re-imaged.

~~~
werid
Just having a vulnerable bash isn't enough to break in. You also need
something that uses it. A CGI script available via webserver, or some other
service (I hear OpenVPN had issues...)

------
sspiff
I run a server on OVH, but I figured as I don't run any dynamic web app on it,
I was unaffected by shellshock - the only way in is through the web server
(which basically hosts a bunch of static files), or SSH.

From what I understood, shellshock was caused by poor sanitation of bash
variable extension, and so only posed a threat to people who used bash from
their public-facing interfaces, hosted bash CGI scripts, ...

Was I wrong to assume that I was safe?

~~~
SCHiM
Yes. I've seen attempts to exploit bash via http headers (the headers them
selves). Like so:

GET / HTTP/1.1\r\n Host: () { :;}; echo vulnerable\r\n User-Agent: () () {
:;}; echo vulnerable\r\n\r\n

So even static servers could have been vulnerable. I'm not sure if your server
was vulnerable or not, but I think you were definitely too soon with assuming
you were safe.

~~~
tokenizerrr
This exploit only works if bash is called at all during the handling of the
request. This is typically only the case with CGI scripts, or when the
requested application does a system() call or directly calls out to bash

~~~
SCHiM
Though it should certainly not happen, it's not uncommon for various
applications to 'shell-out' certain tasks. The point I was trying to make is
that you're not necessarily secure just because you don't use CGI or only host
static content, and that exploitation via the web server could certainly have
been possible.

------
api
The title is misleading -- it was _customer_ servers on OVH that were
exploited en masse and then proceeded to flood the network with DDOS and bot
traffic, not OVH itself (apparently).

This has likely happened to loads of VPS and dedicated server providers. One
consequence of hosting such a service is that you typically end up with a lot
of poorly administered crack houses on your network.

------
Lanzaa
It sucks that so many servers are still vulnerable to the shellshock issue. I
am impressed with OVH's openness in regards to reporting this issue. I am
amused and impressed by their final comment/conclusion; "Obviously it is time
to go to 2x100G on the private network between Europe and Canada."

------
jgreen10
This seems like the tip of the iceberg. Shellshock didn't generate nearly as
much attention as heartbleed, but is far more devastating and easier to
exploit. How many home routers have been taken over by now?

~~~
vertex-four
Home routers run busybox, not bash.

------
teepo
That's a pretty serious limitation of their automation if they are unable to
just whack the ACLs on the private network to the affected customer nodes.

------
concern2
Does anybody know if any of the hubic.com services have been affected by this?

------
segmondy
VPS providers should take a little responsibility to protect their customers.
I'm not talking about full on sys admin or managed services, but even a scan
and report or taking off badly infected hosts offline till the owner upgrades
it.

This will save them much time in the future, save their own some time, and
save other sites (that will be attached through the compromised hosts some
time).

Being a good citizen of the Internet boils down to taking some level of
responsibility. I wont put grandma on the internet without malware bytes or
avg, etc on her computer. So why should I give her a VPS and tell her she's on
her own?

~~~
M41K0
OVH is not a VPS Providers. OVH is a global service provider.

This hack is on Dedicated Server. OVH do not have the right and the access to
the system of thoses servers.

By the way, all the clients have been informed Two weeks ago by a global
mailling because of Shellshock. The responsability is all on the customer.

~~~
spacefight
"The responsability is all on the customer."

That changes quickly if links get saturated suddenly and other customers are
impacted.

~~~
mahouse
Then cut off the network connectivity of the compromised hosts because they
have breached the contract.

