
How we discovered a database leak in one of the biggest Swiss hosting providers - redsec
https://security.infoteam.ch/en/blog/posts/how-we-discovered-a-database-leak-in-one-of-the-biggest-swiss-hosting-provider.html
======
redsec
A little update on the service Security Guardian after the publication of this
post.

Thanks to Hacker News and its incredible community, there have been a massive
number of new users. We are working on adding more resources to the
infrastructure to make the scans quicker. For now, it is possible that some of
you have to wait some hours before receiving the first results.

Thanks for trying our new product, we hope to improve it with your feedbacks.

~~~
doorbellguy
> Then, they blocked the IP address of our scanner so we could not scan their
> server anymore.

Long time lurker, made an account just to ask this: What are your comments on
this unprofessional reaction from the Swiss?

~~~
iforgotpassword
They should release the name of the hoster. Not so much for internet points
but to warn the customers. Who knows what information could have leaked from
all those DBs, and I doubt that the hosted sent out a report on that incident.

~~~
doorbellguy
100% agreed.

------
bipson
For whoever was wondering who this provider is: according to whois-nslookup-
mxtoolbox_arin_lookup, the server hosting infoteam.ch is provided by metanet
(metanet.ch)

Not trying to ruin their business, but they should consider handling issues
like this one properly.

~~~
redsec
Sorry, but to avoid problem for now, we prefer to keep the provider anonymous,
however we can assure you that it is not metanet.ch.

~~~
piquadrat
You should probably mention that at the top of your blog post. bipson is
probably not the only one coming to this (apparently) wrong conclusion based
on your current hoster.

------
raducu
The moral of the story should have been -- change your hosting provider the
minute they commit such a blunder.

~~~
bringtheaction
Indeed. Like they say, fool me once, shame on you; fool me twice, shame on me.

However how do you go about picking a new provider? How do you know that
anyone else is any better?

~~~
twhiston
nine.ch is better (disclaimer - I work for them ;) )

------
Mashimo
Can't test the product they try to promote because emails with a `+` in them
are not valid.

~~~
codetrotter
Email validation regexes are so annoying. Everyone ought to just use .+@.+ as
their validation regex and not be more strict than that.

Beyond that just queue and try to deliver the email. Tell the user than an
email should arrive shortly and that if it doesn’t they should check their
spam folder and that they should check that they gave the correct email
address. When you say this you repeat the email address that the user gave you
(escaped for XSS of course).

I think some people “validate” against a strict pattern to keep their users
from mistyping, but really there are so many ways to make a typo and still
match those regexes that IMO it’s pointless to use a complicated regex and 80%
of the times those regexes end up rejecting actually valid (though unusual)
email addresses.

I think for a lot of developers the reason they do this is that they’ve
learned that they should validate data and so they decide to validate email
and to do so they either copy-paste some random-ass regex off the internet or
they write their own broken regexes.

All your regex should do is to ensure that there is an @ in the address and
that there is something before and something after. This keeps people from
mistakenly entering say for example their phone number because they didn’t
read what the field was for.

To prevent people from making your machine send your emails where it should
not, such as to root@localhost of your server or elsewhere on your local
network (don’t know why anyone would and also it wouldn’t be a _big_ issue,
just a tiny bit annoying), is a server configuration concern. Specifically, a
concern of configuration of the email server software and of your firewalls.

User presses sign up -> Send then to registration form, they fill in their
details which you validate lightly client side, they submit -> You validate
lightly server-side and either send them back to the form or on to the next
step -> You tell them “Thank you, your registration is now complete. An email
should arrive in your inbox shortly. If it does not, please check your spam
folder and also control that you entered your email address correctly. The
email address you gave us was somebody@example.com.”

~~~
theandrewbailey
Sorry, but .+@.+ isn't going to cut it if you want to confidently accept
deliverable email addresses. Regex valid, but not email valid:

codetrotter@example

code@trotter@example.com

code trotter@example.com

codetrotter@example..com

codetrotter@example.com.

.codetrotter@example.com

My company runs a website that has elderly people signing up for newsletters.
The client is paranoid about not getting every last drop of possible data, and
raises hell for every email address that isn't deliverable. There are LOTS of
ways to easily ruin a simple email address.

~~~
Xylakant
There is a other email that the regexp lets pass and that’s still not valid:

codetrotter@example.com

It conforms to the expectated format and could be a valid email, but it’s
actually not because no such user exists.

An email might also exist, but not accept mail from you.

The given email address might exist, by could belong to another user.

There’s a million things that can go wrong and you’ll have a very hard time
catching them.

The only way to identify if an email is valid and accepts mail is to actually
send an email there.

You can, as a help for the user identify odd looking email addresses and flag
them in the UI (“this looks unusual, are you sure”), but generally speaking,
chances are high that any strict validation will reject real world addresses
while not catching all errors.

~~~
theandrewbailey
Good points. Why do regex tests at all if emails could fail in any number of
ways?

If you're going to test for at least 3 characters with a @ in the middle, you
probably should implement some other simple rules to have a snowball's chance
on the internet:

only one @

no spaces

has TLD (guarantee at least one period after @, and something else after, no
consecutive periods)

can't begin or end with a period

~~~
Xylakant
Your “has TLD” test is already wrong: localpart@tld (example@de) is an odd,
but valid address.

------
nordras
Their vulnerability scanner is basically an on-demand DOS attack. Tried it on
my site and almost brought it down

~~~
TheRealPomax
Good. That also tells you that your server probably needs some DOS mitigation.
Because if their service almost takes it down, so can a trivial nmap sweep.

~~~
nordras
Yes, my site does - but that's not the point - it's a text field on a website
anyone can use to hit any site without effort.

It's also supposed to be a helpful service, which generally implies it's not
gonna behave in a way some would consider to be malicious, but it has no rate
limits. If I saw that hit my logs i'd consider it to be an attack, not a
friendly vulnerability scan.

------
willvarfar
This is a nice anecdote to promote their new Guardian service.

------
TheRealPomax
For the longest time a "clean" MySQL install would set up an no-password
superuser for presumably dev convenience. I don't know if they changed that
(it's been a while since I last installed MySQL) but if not, this could simply
be a security hole by design, with the maintainers simply not paying attention
to their install script flags.

~~~
strictnein
I thought it was no password, but only available via localhost?

~~~
TheRealPomax
All that requires to be exploited "on" localhost is some PHP script
interpreting unsanitized user uploads (uploading a php script that has an
image file extension's a pretty famous example) on any of a thousand customer
sites. You don't want that MySQL user to exist, ever =/

------
kennydude
Sounds like they may have had a deploy script which ran again a week later or
something like that :/

(Also overriding scrolling is not cool)

~~~
sdoering
I second the scrolling part. Between the massive fixed header and the
overridden scrolling the page is unusable to me. I am able to get rid of the
first with a little bookmarklet - the second one I haven't figured out yet.

------
peterwwillis
"If you use this network security appliance, you can stop all traffic that
doesn't match the profile of your normal traffic from leaving your network."

"That's expensive, and complicated! We'll just do regular audits and be fine."

[some time later]

"Someone exfiltrated all our data using mysqldump!"

= /

------
patte
currently infoteam.ch seems to be hosted on METANET (metanet.ch). Is there
anyone who can deny or confirm that this is the provider they don't want to
mention?

source: nslookup infoteam.ch; whois 80.74.143.113

~~~
ar0
Just in case anyone is reading here first and because this could really hurt
the business: The infoteam.ch people have stated above[1] that the provider
was _not_ Metanet.

(I do think, however, that it would be important to know.)

[1]:
[https://news.ycombinator.com/item?id=16529273](https://news.ycombinator.com/item?id=16529273)

------
stareatgoats
> "Hopefully, we had ‘only’ read access and could not write or delete
> anything"

Sounds a lot like feigned ignorance about the nature of the root user. Not
entirely sure if it would help them in a court of law. They should probably
anonymized the whole thing better to be completely on the safe side (not a
lawyer though).

~~~
dredmorbius
That reads more like an attempt to make a light-hearted quip suggesting the
exposure was limited. Say, more like: "OK, so we could directly access the
database, but _surely_ nobody would provide a full read/write administrative
account publicly accessible _and_ passwordless.... Oh."

Given a few other hints at English not being the first tongue (or possibly
even the fourth, CH having four official co-equal languages) of the writer(s)
in question, that could just be awkwardness from a non-native speaker. Their
English is vastly superior to my German, French, Italian, or Romanche.

------
parliament32
I'm getting a "Something went wrong. Please retry in a moment." error when
trying to submit a domain to Security Guardian (tried different domains and
email addresses). I assume you're being hugged to death.

------
unixhero
Moral of the story. For my sake:

Check the access logs.

Regularly.

~~~
peterwwillis
Why, are you bored? They can just access the db from an internal host that
already has connections and exfiltrate from there. That's part of why people
retrieve data using holes in web apps - it looks like a spike in normal
traffic.

~~~
unixhero
I'm sharing my takeaway after reading this. Access logs are fun to some people
;)

------
vectorEQ
:s i can't even get my mysql to get me to be allowed to login root without
password >.< that takes a special kind of negligence.... and really, how long
was it there before they developed a new product and tested it on themselves?
:/ seems logical, especially for a security service provider that with the
lack of such product still this would be noticed?

that besides pitching their own product for an issue any similar natured scan
would pick up i'd say it smells like marketing department at work more than
chinese hackers or shitty service provider.... >.>

i doubt they would have left a passwordless root on their mysql, or didnt they
check the initial setup they were given by the provider before taking it in
use?

~~~
barkingcat
If you read the story you would see that the database host is a shared host
(as in hundreds of other clients of the Webhost have accounts on it) and that
the error is likely the result of a persistent hack. As in there is a
vulnerability where someone can get access to the server and create a
passwordless root account, so that they can siphon the data out.

Once that account is deleted, a new passwordless root account is created by
the attacker in order to continue access.

------
orf
tl;dr, portscanned a server, found an open MySQL port with a weak password.

~~~
LeonM
Not just weak, but no password at all, for the root user.

And the problem returned a week after the 'fixed' it.

------
sneak
Important to remember that these “one of the biggest $x in $y” where $y is a
country with a population under 10 million means that the statement
encompasses many entities which are just a half-dozen people in a small office
somewhere.

I know nothing about the particular hosting provider in question.

------
smoyer
Security Guardian is not a he/him. It may be a translation issue ... or maybe
you've achieved human-level AI and it's become self-aware? In any case, I find
it interesting that your first response is that the tool might have a bug ...
and the link also on HN at this moment is about the Apollo 13 mission control
engineers thinking their telemetry might be at fault. This is an excellent
first response and it's important to provide a way to distinguish between the
two.

~~~
dredmorbius
Given the number of failures I've seen _as a direct result of security and /or
availability systems_, the instinct to first question your own readings _is_ a
valid one. It's a good check particularly to make sure that your tests _are_
valid, and that you aren't going off half-cocked with an alert (alarm fatigue
is a Real Thing, and attention is a scarce resource).

I've also seen, in general, far more false alerts than real ones. (Yes,
Nagios, I _am_ in fact looking at you.)

The gender-casing is a quirk of the article, and would reflect at least three
of the languages official within Switzerland (German, French, and Italian are
all gendered languages, I'm fairly certain Romanche is as well). It's fair to
note that, though perhaps not _quite_ to the extent you did.

------
rurban
They probably use a homebuilt admin panel and sw mgmt, and an update brought
back in the old root vuln. They don't use cPanel or Plesk. Or the Chinese
hacked it again.

Interestingly they -
[https://security.infoteam.ch/](https://security.infoteam.ch/) \- offer the
very same security service, automatic security audits for their customers.
Which explains their angry response the 2nd time.

~~~
bipson
Maybe I got you wrong, but infoteam.ch _are_ the guys who wrote the blog-post,
so your link is to _their_ product they are describing, and not one by the
provider? How does that again explain the angry response?

AFAIK there is no mention or link to the provider, I guess since they don't
want to make enemies just yet (and lose hosting while doing so). But I am
willing to speculate that they are looking for another provider just now :)

~~~
rurban
I got it wrong, not you. Thanks for the heads-up.

They didn't mention who leaked the data. I got confused. The symptom they
described looks like an automatically updated system with a leak. One of our
swiss customers, a big hosting provider, uses such a system and would fit.

