
Efnet klines all Hetzner netblocks - alternize
http://forum.efnet.org/viewtopic.php?t=5866
======
efdee
"One of our EFnet operators located a server that a client was using, the
server was most likely hacked and/or used for illegal activities." -- so they
contacted the ISP and the ISP sent their request to whoever owns the box.

That sounds entirely reasonable to me. Hetzner has no idea whether or not the
box owner is the suspected hacker (even EFnet assumed the server was hacked)
in the same way that EFnet has no idea about whether an Hetzner employee was
the suspected hacker. So three possible outcomes: (a) the box was hacked, so
sending said info to the box owner was not a bad idea, (b) the box wasn't
hacked, it was the box owner himself doing bad things, in which case Hetzner
just gave the box owner personal information about his victim, or (c) the box
wasn't hacked, and it's a Hetzner employee doing bad things, in which case
EFnet just gave the employee personal information about the victim.

So, exactly why are they blaming Hetzner?

~~~
laumars
The issue is that Hetzner passed on personal information. It's a breach of
trust.

~~~
efdee
You're missing my point though.

The operator passed on personal information without knowing if a Hetzner
employee might be the bad guy. Hetzner passed on personal information without
knowing the server owner might be the bad guy.

If the operator had contacted the server owner directly to let him know his
box was hacked, the result would have been the same.

Why did the operator have the expectation that an uplink of the server
wouldn't just forward the mail to the server owner?

~~~
laumars
Because Hetzner is an award winning business who sells professional hosting
solutions. You'd have a point if the company in question was some dodgy garage
operation, but Hetzner isn't.

If EFNet are going to report the issue then there has to be a trust worthy
contact at some point along the chain of network ownership, and Hetzner
_should_ have been that point. Quite frankly, it's irresponsible for a company
of that size to behave in that way.

~~~
gknoy
What would Efnet lose by redacting the name of the operator themselves before
doing it? They still show timestamped logs, and [operator redacted]@ef.net
would be enough to note that an operator reported the issue.

If the log included chat between the malicious user and the operator, the
malicious user __already knows__ the operator's nick, so there's no additional
information. If the log doesn't include that, then what reason is there to
include the nickname of the operator?

Rather than expecting Someone Else to sanitize our information (which fields
do you feel are critical to redact? real name? phone number? IP? zodiac sign?
Made-up-nickname?), it seems like it's a good idea to proactively redact the
information ourself before making that abuse report.

~~~
nknighthb
The operator is the one that sent the email. Who are you suggesting should
have redacted his name before transmitting it? Magic fairies in the SMTP
servers?

------
andor
_Unfortunately, the past few days many EFnet servers (and more are following)
have had to ban an entire ISP, which has not happened in over a decade, if not
longer. Naturally, something extreme must happen for this to be even
considered._ [... and the extreme thing is ...] _the email also contained
sensitive information about who this operator was, including nicknames_

This is ridiculous. Why would an IRC op want to keep his/her nickname from
becoming public? On IRC, even IP addresses are public. Hetzners reaction seems
entirely reasonable to me, especially if the server in question was "mots
likely hacked".

~~~
nknighthb
> _Why would an IRC op want to keep his/her nickname from becoming public?_

That isn't what they said.

> _On IRC, even IP addresses are public._

This may still be true on efnet. It has ceased to be true on many IRC
networks. And even where it is, VPNs, shell accounts, tor, and similar methods
are readily available to make the IP address worthless.

> _Hetzners reaction seems entirely reasonable to me_

Please tell us where you work so we can all be certain never to report any
sort of abuse to your employer non-anonymously.

~~~
DangerousPie
> That isn't what they said.

How not? The forum seems to be down now but the announcement heavily implied
that all that was "leaked" was the nick name of an operator, which they
claimed could somehow be used to deduce the real name and address.

~~~
nknighthb
The problem is not the public knowledge of a nick, but the information that
the person using that nick made the complaint. It paints a target on a person
whose nick is very often publicly associated with a real identity.

------
raphman
I have reported spammers/phishers to Hetzner in the past and experienced the
same: Hetzner's default policy is to forward a complaint to the server's owner
- which also kind of surprised and annoyed me. On the phone, support staff
told me that Hetzner sees itself just as an uninvolved messenger between both
parties. Apparently, their support ticket system automatically forwards all
complaints to the server's owner without any way to opt out. The support
person offered two alternatives: send anonymous complaints through a freemail
service, or send the complaint to the personal address of a support team
member, so that they can manually enter it into the system. Yes, this is
pretty annoying.

~~~
oellegaard
As a Hetzner customer I think this is more than fair. If someone reports abuse
from my server, hetzner should forward me the report. They do not have access
to the server, so why in the world would they do anything but forwarding it to
the person or company that does in fact have access?

Protip: Don't put your credit card number in the abuse report. Just put the
logs or whatever is required to prove that there was in fact abuse from that
server.

~~~
danudey
The problem is that if you have someone using Hetzner's network for spamming
or other illegal activities, you'd expect the ISP to want to keep their
network clean of these sorts of things and investigate. Instead, they just say
'whatever, not our problem'.

On top of that, though, forwarding the complaint to the server owner just
makes a mess of things, because now the server owner can target you and make
your life hell. They can send you threats, DDoS your mail system, flood your
inbox, etc., all the while knowing that Hetzner has their back.

It's like if you saw someone assaulting someone in an alley and called the
police, and the police took the information you gave them and gave it to the
person who assaulted them. Now they know who you are, and they can come after
you too. Now your only real solution is to just not report it when someone
does bad things, because you know it's not going to get fixed and it's just
likely to cause problems for you down the road.

~~~
mschuster91
Well, Hetzner does not state it in their RIPE record
([https://apps.db.ripe.net/search/query.html?searchtext=5.9.12...](https://apps.db.ripe.net/search/query.html?searchtext=5.9.120.209&flags=&sources=RIPE_NCC&grssources=&inverse=&types=#resultsAnchor))
that the email is NOT handled by Hetzner staff. In contrast, Manitu (another
German hoster) explicitly states this:
[https://apps.db.ripe.net/search/query.html?searchtext=217.11...](https://apps.db.ripe.net/search/query.html?searchtext=217.11.51.9&searchSubmit=search#resultsAnchor#resultsAnchor#resultsAnchor)

------
codesuela
> Unfortunately, according to trusted sources (ex-employees) of Hetzner.de,
> this is policy and not an exception. They have realized they can save money
> (by limiting attacks) by redirecting the attacks back at the person
> reporting them. That way, the hacker/cracker/kiddie using their services
> will not cancel their contract with Hetzner, and in return Hetzner will
> remain protected.

I don't think that this has to do with Hetzner needing criminal business but
rather with Hetzner not wanting to shut down an entire server if a part of it
has been breached. Forwarding that complaint in it's entirety is definitely
not best practice however making such allegations is neither.

~~~
nmcfarl
That was roughly my take away. Hetzner looks really bad - but the explanation
that they are incompetent and lazy sounds more likely than the criminal
explanation.

And EFnet looks wronged. But also looks bad for spreading allegations against
Hetzner of profiteering from criminals without much more substantial backing.

Sticking to the facts of the situation would have produced a more powerful,
and damning condemnation.

~~~
rhizome
This isn't court, I think it's reasonable to forgive humanity entering into
the issue. Besides, this appears to be a citizenship issue, so banishment also
comes as no surprise.

------
h2s
That announcement should have clarified whether the communication they sent to
Hetzner explicitly requested confidentiality. If not, the incident is as much
Efnets's fuckup as Hetzner's. Why did it even need to contain such sensitive
personal information about the sysop in the first place?

~~~
devicenull
Even if their initial email requested it, wouldn't you assume that a large
provider like Hetzner has at least partially automated their abuse handling? I
wouldn't count on a human actually reading it before it gets forwarded to the
customer.

------
subsystem
"This has worked very well due to our personal involvement with a lot of said
organizations. If we find an abuser on IRC, we try to not only ban him or her,
but also to contact the provider so that the problem is handled at the right
end, often with the involvement of law enforcement, as was the case with Kevin
Mitnick, t0rn and a lot of other well publicized hackers/crackers."

I'm not sure that is something to be proud of.

~~~
laumars
Of course it is. If someone cost you time and money and severely harassed
users of a service you're offering by illegally attacking your servers, then I
bet you'd be all for reporting them as well.

The problem with script kiddies is that they're too lazy to create anything of
their own and thus cannot emphasis with how hard it is to set up and maintain
a free public service. So they don't have any second thought about destroying
other peoples hard work. Plus the pseudo-anonymity of being online and the
ability to not look your victims in the eye make such crimes easy for even
some of the more ethical kids to shrug off.

The only way to have an open internet where everyone is free to create and
contribute is to rat out those who seek to destroy it. People like the
aforementioned might enjoy all the perks of an open internet, but their
actions have the opposite effect.

So I'm not only in favour of reporting abusers, but have actually reported
attacks on my own servers to their respective ISPs.

------
trotsky
God forbid we should mildly redesign the irc semantics so that splits don't
allow you to take over channels or collide clients off.

I mean it's only been a big enough problem for 20 years now that you can't
even host an ircd on most standard hosting contracts.

~~~
xpose2000
Efnet has had chanfix for quite awhile now, which makes taking over channels
from a split much harder, if not impossible.

~~~
trotsky
Really? Hmm. It has been 5+ years for me, my bad. But what's the point of the
ddos's then? I see netflow graphs of huge attacks on major efnet servers on a
regular basis.

~~~
xpose2000
I'm honestly not sure why people still DDOS servers these days. Maybe just fun
and games, it is EFnet after all.

Not too long ago there was an ircd exploit that killed servers and split 100%
of efnet, which had never happened before. Not to mention its down 50% of its
user base from a few years ago. Tough times for efnet.

------
DangerousPie
This seems like a ridiculous overreaction to me. Not only do I not see much
wrong with an ISP forwarding abuse e-mails to the admin of the hacked server
(who is probably a victim too), but I also find a bit hard to believe that the
nickname of an operator is enough to "derive a home address".

~~~
blibble
you'd be surprised

as an operator of a large IRC network I am forced to keep my real name off my
employer's websites, websites like github and my home address out of the
public record (director's addresses and such like)

it's a major nuisance

~~~
ulope
Here is an idea: use different nicknames for different things then you wont
have to live in a self prescribed virtual prison

------
mschuster91
Uhhh... that's bad. Real bad, for both sides.

I can totally understand Hetzner for just forwarding abuse complaints to the
client (for root servers, the hoster usually has no "emergency ssh keys"), so
the faster the original owner of the server can boot out the hacker, the
better. At least it's better than disconnecting the customer from the internet
entirely, especially as "haxx0ring" a server is damn easy these days, given
the numbers of aged Wordpress installs alone. Also, a server owner who knows
his server can support me as a hacking victim better than the hoster support
who often knows nothing about configuration details, OS, disk encryption etc.
on the server.

But I also understand EFNet, that their emails got blindly forwarded is bad,
too...

~~~
claudius
> I can totally understand Hetzner for just forwarding abuse complaints to the
> client (for root servers, the hoster usually has no "emergency ssh keys"),
> so the faster the original owner of the server can boot out the hacker, the
> better.

Removing confidential information from such an email should take ~2min, I
think that’s time well-spent, especially given that email per se is not a
real-time medium.

~~~
mschuster91
I guess their automated forward system regex-greps the IP out of the complaint
email and then forwards the mail to the applicable customer(s), so this system
actually _is_ realtime.

Also, it places the burden of dealing with spam on the customer, so that the
hoster doesn't have to do spam-filtering.

------
weinzierl

        "[...] someone at Hetzner chose to forward
        this complaint to the actual abuser him/herself. [...] 
        Unfortunately, according to trusted sources 
        (ex-employees) of Hetzner.de, this is policy and not an 
        exception."
    

I really hope this is not true.

In 2011 there started to appear pornography when I searched for my name in
image search. The reason was that a stackoverflow.com scraper showed my
answers (which I post under my real name) but my profile picture replaced with
porn. I have no idea why he replaced the profile pictures in the first place,
but anyway.

The scraper site was hosted at Hetzner. I phoned Hetzner. They told me to
write to abuse@hetzner.de, which I did (in German). I received an auto-reply
but nothing else happened. I tried to reach their phone support (09831 610061)
several times during their business hours (Mo-Fr 7:30-18:00 Uhr) but no one
picked up. I wrote several other mails to Hetzner but never received another
reply.

After about seven months the scraper went down but I have no idea if it had
something to do with my complaints because I never received any feedback from
Hetzner.

I have know Idea who was behind the site. They could have found out about me
anyway because I posted this publicly on meta.stackoverflow [1], but still:
The thought that they might have learned about my identity through Hetzner is
discomforting.

[1] [http://meta.stackoverflow.com/questions/81872/copycat-
site-c...](http://meta.stackoverflow.com/questions/81872/copycat-site-causes-
disturbing-images-to-turn-up-in-image-search)

------
itry
Can somebody tell me, what "kline" means? I never heard that word and dont
find a dictionary entry.

~~~
mikeash
I was just musing that the only word in this headline recognizable to someone
outside this small subset of the tech world is "all", and yet it makes perfect
sense if you know all the words.

~~~
mturmon
Happened to me. Incomprehension is the reason I clicked the link.

------
ciupicri
The article doesn't seem to mention if the sensitive information from the
email was marked as such.

~~~
TazeTSchnitzel
Why would it need to be? Surely Hetzner and the like would be familiar with
how sensitive it could be.

~~~
DangerousPie
If you consider nicknames sensitive you are gonna have to mention that,
because a lot of people wouldn't.

~~~
TazeTSchnitzel
Hetzner, as a hosting provider, would likely be well aware of the workings of
IRC networks, so I doubt that.

------
eliasmacpherson
Doesn't explain why efnet themselves are not at fault for failing to redact
the details if they are so concerned. What if the attacker was Hetzner staff?

~~~
ahnberg
Should we perhaps have sent the request through anonymized pidgeon? When you
send an e-mail to someone you can find out quite a lot of information from
headers, from a nick or a given name.

When contacting ISPs some people attach contact information so that the ISP
can request more information, maybe even make a call to verify identity and
discuss the matter to help resolve it.

This doesn't mean that the information is intended for the abuser himself. I'd
expect the ISP to send anonymized information/questions to the owner of said
service, and not just forward "everything" (especially without discussion with
the reporting party).

Reporting DDoS and abusers risks putting a huge target on yourself, your
company, your servers, etc.

That is what the issue is about.

~~~
eliasmacpherson
I specified redacting the details, i.e. the irc nickname.

If the information within the headers is that sensitive, the headers should
probably not be forwarded from efnet to hetzner. At its most basic their
servers are attacking efnets and so should be considered hostile.

By all means pass on contact information to resolve amicably - but don't leave
sensitive information.

There's no reason to pass on irc nicks or sensitive email addresses - what use
are they to hetzner? I have the same expectation of efnet as you have for the
ISP.

If the issue is that efnet is being identified as the reporter to the rogue
attacker, then that is not what is stated in the efnet admin's text.

~~~
ahnberg
EFnet didn't pass anything on. EFnet reported abuse, and the EFnet reporting
staff members information was the information that was leaked/forwarded to the
abuser. Not a third party to whom EFnet acted proxy.

You seem to misunderstand the situation.

------
devicenull
Are there any actual standards for how abuse reports should be handled by
service providers? What Hetzner is doing here appears to be pretty reasonable
to me (it matches what companies like Level 3 and nLayer do).

I can understand why you might not want sensitive email forwarded to the
abuser, but why would you send that information in your initial complaint? For
all you know, the person you're reporting to is the abuser.

------
cientifico
This posts makes me trust more in Hetzner as a provider than the other way
around.

I know that if there is any problem, they will forward the problem to me as
soon as possible, and they are not going to take the law into their own.

------
johndoeee
I got an abuse mail from Hetzner once (they mistyped my ip).

The original complaint was something autodetected by their own system, mailed
to themselves. The original complaint was attached, header and all. Something
about malware on an IP similar to mine.

Also, when will this be in effect, the server i tested from had no problems
connecting.

~~~
arianvanp
I had the exact same email last week .-.

------
TheSwordsman
Full text:

Posted on behalf of Silence, EFnet admin:

Dear users,

EFnet has always been a network that promotes freedom of speech. One of the
core pillars of a free virtual society is trust. Trust not only amongst
ourselves internally, but an undying trust in the companies that allow their
users to connect to our wonderful network. We have survived over two decades,
in a world that is increasingly image- and video-based. IRC can offer neither
of those. IRC is based on ideas. Ideas that are exchanged in text. With text,
as opposed to images and videos, one has to be put extra effort into the
subliminal, the meaning, the message. This has been our catcher in the rye,
and we intend to protect this content-based communication form, for as long as
it is appreciated by the hundreds of thousands who every day turn to IRC for
philosophical debates, dating and just about anything you can think of (I’m
sure a lot of the things in that last category does not belong here in this
text, but you get the picture!).

We rely solely at the goodwill of others, as is the case with most things
worth saving. There is no money to be made. We all do this for free. Sure,
some companies might have benefited from a small level of advertisement,
attracting customers to their products. But all in all, it has mostly been an
uphill battle against enormous attacks, sometimes exceeding 75Gbps of DDoS.
This has made it impossible for all but the largest organizations to host a
server on our network, or any other large virtual society. We are Don Quijote
and the weather mills are often winning.

One of our key strategies is to preserve a close relationship with the major
Internet- and Hosting Service Providers, as those are the networks that our
users connect through. This has worked very well due to our personal
involvement with a lot of said organizations. If we find an abuser on IRC, we
try to not only ban him or her, but also to contact the provider so that the
problem is handled at the right end, often with the involvement of law
enforcement, as was the case with Kevin Mitnick, t0rn and a lot of other well
publicized hackers/crackers.

Unfortunately, the past few days many EFnet servers (and more are following)
have had to ban an entire ISP, which has not happened in over a decade, if not
longer. Naturally, something extreme must happen for this to be even
considered. Almost always can we find a solution through the use of good old
fashioned communication. Alas, not in this case. Well, here is the story (to
the best of my knowledge):

One of our EFnet operators located a server that a client was using, the
server was most likely hacked and/or used for illegal activities. As IRC is
often a playground for these people to use, before moving on to more serious
targets (where they can make money through extortion), we take this extremely
seriously. Because of the serious nature of this, our operator sent an email
to Hetzner.de, a German hosting provider, to help them lower the abuse of
their servers, as well as ours. This is usually a fruitful symbiotic
relationships, where both parties stand to gain.

However, the big difference between this case and all the other thousands of
cases we have handled in the past, is that someone at Hetzner chose to forward
this complaint to the actual abuser him/herself. This might seem fair enough,
as anyone accused should be granted the right to defend him- or herself.
However, the email also contained sensitive information about who this
operator was, including nicknames (from which names can be derived, and thus,
also, home addresses). We know what an impact this can have on your social,
not to mention your professional life. We have seen people lose their jobs,
after constant attacks and we have also seen companies lose money that is hard
to fathom, considering this is still just a simple chat for friends. This is a
fundamental breach of that mutual trust that has allowed us to accept clients
from Hetzner to use our network - free of charge, just like we do with anyone
else wanting to connect.

This a give and take network, where mutual trust is vital for our survival. We
are maintained by the community, and we exist solely for the community.
Hetzner.de has broken one of the most fundamental aspects of any report of
criminal activity or suspicion thereof; source protection.

I expect us to get attacked now, which will result in a lot of work for the
company kind enough to donate money and time to continue to provide us with
servers, in an era where almost everything else would be more profitable. But
this is an ideological problem, more than a financial one. We have been
attacked before, and we will again. We are prepared. But these preparations
rely on the fact that we know who the enemy is. Hetzner.de has made that
impossible.

As a result of this, we have decided to ban all Hetzner IP ranges (both IPv4
and IPv6) from our servers. It seems other networks are following, and I know
QuakeNet has published a similar statement. We simply do not want anything to
do with a company that values money over source protection and integrity. Some
may argue that this was a one time mistake, and that we should not jump to
conclusions so fast. Could this have been a mistake? Sure. Does it matter,
given the consequences this could have had for this operator’s personal life
and health? No. We do not appreciate cowards that would rather see someone
else hurt, than take their responsibility.

Unfortunately, according to trusted sources (ex-employees) of Hetzner.de, this
is policy and not an exception. They have realized they can save money (by
limiting attacks) by redirecting the attacks back at the person reporting
them. That way, the hacker/cracker/kiddie using their services will not cancel
their contract with Hetzner, and in return Hetzner will remain protected. Left
are those of us that work for free, and who will continue to do so, for as
long as there are honest, reliable companies out there, willing to go the
extra mile to protect the freedom of the Internet, and, above all, freedom of
press and source protection.

Questions on this matter must be directed to Hetzner.de, as our involvement in
this situation is over. This has been their decision based on questionable
methods. It is unfortunate for them that they got caught, but it is good for
the sake of the free Internet.

Sincerely yours,

Johan Boger, on behalf of EFnet and anyone else believing in integrity, source
protection and a free Internet.

~~~
aspensmonster
> chose to forward this complaint

> ...

> the email also contained sensitive information about who this operator was

> ...

> This is a fundamental breach of that mutual trust

> ...

>Hetzner.de has broken one of the most fundamental aspects of any report of
criminal activity or suspicion thereof; source protection.

Excuse me? How is this in any way Hetzner's problem? The only "fundamental
breach" of trust I can see here is the one that occurred when efnet gave
"sensitive information" on its own operators to a third party at all. It
sounds to me like efnet is the one in need of a lesson on "source protection."
This reads more like a "we screwed up bad and are about to divert blame as
hard as we can" letter than anything else. But I'm more than ready to listen
to any further context that can be given to this.

~~~
dsl
For abuse reports to be taken seriously you need to provide timestamped logs
and legitimate real world contact details, in exchange there is the
expectation that reports are not forwarded in an unsanitizied fashion to end
users. (I worked an abuse desk for a few years)

~~~
cientifico
The problem is that if you have a problem and want to solve that problem, the
first thing you do is talking with the person.

Instead of trying to fix this like normal persons, you want others to work for
you by free.

Who gives you the role of a judge in this case, to even consider that the
actions from that ip where illegal.

If they were illegal why don't sue? If you ask a lawer the first thing he will
tell you will be to talk with the other parties before starting legal actions.

~~~
dsl
I really hate to use this analogy, but consider abuse desks the police of the
Internet for a second.

Your local ops team notices a lot of SSH break-in attempts coming from
Sprint's network (for example). They forward the logs on to the abuse team,
who reach out to the abuse team at Sprint. In the same way that a cross
jurisdiction investigation would be handled by two police departments working
together, the two abuse teams will work together to investigate and remediate
the problem.

Now imagine if the Seattle Police Department contacted the Chicago PD about an
ongoing investigation of a jewelry heist, and the Chicago PD says "we are too
busy to deal with this" and forwarded it on to the thieves so they could
respond directly and help the Seattle PD with their investigation... Now the
investigating officer is compromised and at risk. Sure, he was a known as a
cop to the bad guys before, but now his enforcement actions have made him a
direct target.

This is a system that has worked for 20+ years. We have our own mailing lists,
meetings and conferences, heck even a private direct dial phone system.

~~~
ciupicri
Except that the communication between police departments is clearly marked or
known (because of various regulations/laws) to be confidential. Here it's not
clear (as I've also mentioned in a previous comment [1]).

[1] <https://news.ycombinator.com/item?id=5518161>

------
mschuster91
I do see why EFnet makes a drama out of this situation... check out the RIPE
entry for one of their IPs:
[https://apps.db.ripe.net/search/query.html?searchtext=5.9.12...](https://apps.db.ripe.net/search/query.html?searchtext=5.9.120.209&flags=&sources=RIPE_NCC&grssources=&inverse=&types=#resultsAnchor)

They do not mention that these emails get redirected to customer!

------
gesman
Hetzner offer the cheapest dedicated servers with pretty hefty resources.
Hence the invitation for abuse.

------
JonnieCache
Nice to see efnet retaining some of its old magic :)

------
ttrreeww
This sounds like whining on the part of efnet...

