Hacker News new | past | comments | ask | show | jobs | submit login
Sinkholed (susam.in)
616 points by swiftsecurity on Dec 4, 2019 | hide | past | favorite | 130 comments

Hi, I am the author of this post. I had posted another link about this story a few days ago when this story was still unfolding.[1][2]

This blog post summarizes the timeline and the events that occurred to resolve the domain transfer issue. Like I have mentioned in this blog post, multiple parties such as Namecheap Support, the Shadowsecurity Foundation, and NIXI helped me in resolving this issue. Thanks to all of them and special thanks to Namecheap CEO, Richard Kirkendall, for looking into this issue on priority as soon as he became aware of it.

I wish the domain name management, in particular, and the Internet, in general, had a higher degree or decentralization, so that it were technically impossible for something like this to happen but I think there are many factors at play that are currently preventing from such an Internet to become mainstream at this time.

[1]: https://news.ycombinator.com/item?id=21671579

[2]: https://twitter.com/susam/status/1200678538254393345

My guess is since .in is one of the cheapest (if not THE cheapest) ccTLD the malwares/botnets are exploiting them. I received an email couple of years back from a top domain registrar based in India that the govt. requires actual Registrant contact address and not that of any privacy protection service (to protect from WHOIS); I don't remember whether it was only for .in or for any TLD's registered from India.

Anyways I've since then stopped buying .in and sticked with .com. I'm glad OP got his domain back, but I'm quite sure it wouldn't have been simple (or even possible) if he had chosen some registrar based in India (due to their incompetence and general nature to not contend with higher authorities) instead of Namecheap; especially since Namecheap's .in domain costs more than that of any registrar based in India.

This was a really good write up, thank you. And congratulations on getting your domain back.

This is something that can absolutely be decentralized. Problems aside namecoin shows that it can be done. The entire domain name system needs to be overhauled, central certificate authorities need to be avoided.

The internet in general is infrastructure, and protocols that are increasingly controlled by governments and organizations in bed with governments.

That also has its downsides though. For starters, it would no longer be possible to take down domains used for controlling botnets.

That's a similar nonsense as is the broad-scale surveilence to prevent terrorism. Botnets can already build a decentralized store of IP addresses, bypassing public DNS completely. Centralization makes some people, organizations or state just too much powerful.

Intelligent botnet authors will switch to the decentralized options once they are widely available and stable. While I think there's limited benefit to decentralization for any use case I care about, I don't think centralization is going to stop this trend in the medium term.

The GNU Name System https://gnunet.org/en/use.html already has a distributed DNS-like system built out.

This does not look like something my grandma would use.

I don't think she has to. As long as your ISPs run GNS themselves, and offer you DNS via DHCP, then you can just forego the ISP DNS server and drop in your own local GNS. Mount .com and the other TLDs you want from direct from ISP off their pubkey, and overwrite the TLDs you don't care much for.

This is great news!

Have you consulted a lawyer?

It seems that the Shadowsecurity Foundation did act recklessly. But you'd need to prove monetary damages.

But perhaps they'd settle to avoid the hassle.

Edit: This is an admission of guilt:

> He explained in his email that my domain name was sinkholed accidentally as part of their Avalanche operation.

Thank you for this comment. I have not consulted a lawyer. I have not suffered any monetary loss due to this yet. I use this domain name only to run a small personal blog (the one linked to in this post) and an Exim4 MTA. The fact that the MTA became unreachable via the domain name did mean that some emails sent to it must have bounced back. The primary loss I suffered was in terms of time.

In fact, I appreciate the efforts of the Shadowserver Foundation in depriving Avalanche malware families of their command and control infrastructure by sinkholing the domain names generated by the malware domain generation algorithms. I understand false-positives like this can happen. It comes with the territory. It just sucks that the domain name I was using happened to be a false positive. In this case, it ended well because the Shadowserver Foundation (along with Namecheap) acted quickly on my issue and asked NIXI to have the domain transfer undone.

Having said that, I do believe that you make a very good point. Actions like this should be taken with a lot more care. What if it were not a personal blog but a small business? Depending on the nature of the business, an inadvertent domain transfer like this could have affected the business seriously. The Shadowsecurity Foundation did say that they are improving their processes.

I am not very concerned about this particular foundation when I talk about this. But I am concerned about the systemic issue in the domain name management system that allowed a mistake like this to occur. There could be some type of peer review before a domain transfer like this is executed. I don't think there is a general and popular solution for this problem in the near future. I am hoping that the recent work that is going on in consensus protocols based on cryptography might pave the way to a more decentralized network and more decentralized name management that also become popular and mainstream.

Yes, I get that the Shadowserver Foundation does good work. And that they acted quickly, after being pointed to your tweet.

However, if your tweet hadn't gotten traction, and if Namecheap hadn't been proactive, you'd likely have never gotten the domain back. I mean, you had the Namecheap CEO on the case! And for a business losing a domain like that, it'd probably be fatal.

I get that many think that Americans are hugely too litigious. But there is the argument that there ought to be compensation for damages.

You say that "[t]he primary loss I suffered was in terms of time". But arguably your time is worth something. Such as your customary billing rate, times three.

Edit: Or just send them an invoice. At perhaps 50% over your customary billing rate, given that it was a rush job.

> I get that many think that Americans are hugely too litigious. But there is the argument that there ought to be compensation for damages.

That point of view is rather unfortunate. Why does it have to be about damages? GP even ends his comment on a very postitive note about things that would help. Not every mistake needs to be punished. It was a false positive, and it was heartening to see that all the parties acted fast enough. Why not just move on instead of outraging over hypothetical concerns?

EDIT: I have no idea how to quote parent comments here.

I was kind heartened by how quick they reacted. However it is quite true that Americans are very litigious; however, austria, germany, israel are worse than us, and England isn't far behind.

I don’t think I hear anybody from those countries talk about sueing nearly as much as any US citizen.

What is your source for that?

> Why does it have to be about damages?

If someone attacks you, and causes physical or even mental damage, they typically face criminal and/or civil penalties.

That's common, throughout the world.

> Not every mistake needs to be punished.

Maybe so. But the people responsible for the damage could proactively offer compensation. That'd arguably be the honorable and compassionate thing to do.

> It was a false positive, and it was heartening to see that all the parties acted fast enough.

It's clear that they acted recklessly. So it's not just a "false positive". And what's "fast enough"? A few days after a totally implausible attack isn't that fast. It should have been fixed within the day. After at most a few emails.

>But arguably your time is worth something. Such as your customary billing rate, times three.

Wouldn't the legal fees and time spent litigating exceed the winnings?

In a lawsuit, the fees can be added to the damages.

Generally no, they can't. In the US. Other countries have different rules.


Yes, that's a good point.

But maybe some attorney might do it pro bono.

You know, for the exposure.

Or just on contingency, if it's an open-and-shut case.

> I get that many think that Americans are hugely too litigious

No, really they aren't. There are a lot of lawyers in America, but most never set foot in a courtroom. They mostly just do "important" paperwork and give advice on following rules.

I know a lawyer who last month wanted to sue someone in federal court, only to discover that Joe Random lawyer is not allowed to file lawsuits in federal court; he had no idea that there was such a thing as a federal trial bar and that membership requires significant experience in federal court under the supervision of a member. He spent a week trying to find anyone that would be willing to file his lawsuit but none would. So he hired a law firm to sue in state court - that is much easier I guess. Anyway, the point is that he is a good lawyer with years of experience doing the lawyer thing, but no experience with litigation. That's normal.

Average Americans are not litigious, because the overwhelming majority of Americans would not actually be able to afford a lawyer if it came to it. Lawyers are expensive.

The scariest part is that it looks like this got resolved quickly only because your tweet got noticed and retweeted. I wonder how long it would have taken otherwise.

Props to you by reacting suitably: worried, but calm and measured and not jumping on some Twitter outrage bandwagon.

> The fact that the MTA became unreachable via the domain name did mean that some emails sent to it must have bounced back.

MTAs should queue undeliverable addresses for more than four days, so you may not miss much unless someone’s DNS resolver had a poor caching strategy or Shadowserver used a long TTL. At least the name still resolved (else the mail would have been dropped immediately). Some mass mail senders may go to less effort to get their messages through.

The fact that people responded so “quickly” helped here. I put “quickly” in quotes as it was quick enough to possibly cause no message to be lost — I’m sure it was super stressful and didn’t feel quick at all!

Glad you got your name back.

> so you may not miss much unless someone’s DNS resolver had a poor caching strategy

Microsoft Exchange used to (hope it doesn't still?) cache MX records until the service restarted. :( There's a long tail of braindead software in the DNS space

Yes, sue them, and incentivize a worse, slower (because lawyers involved) and less transparent resolution of errors.

Thank you for this. We are definitely seeing the declining use of centralized domain-resolution. It has advantages only when it is not _itself_ being gamed, and increasingly companies and governmental orgs have found ways to do just that. At the very least every domain should have dual central + decentral resolutions, and browsers should give you options when the resolutions conflict.

The server is up. However, it is possible that at your end "susam.in" is still resolving to a sinkhole address.

I have created a mirror of the blog on GitHub Pages for you. Visit https://susam.github.io/blog/sinkholed/ to read the mirrored blog post.

As someone that deals with sinkholed malware domains everyday, I have to day I slightly disagree with the logic and approach behind it.

The basic premise is that unwitting hosts are compromised by malware,this malware is talking to a domain and in order to protect the infected users and curtail the further spreading of the malware the domain is sinkholed.

First, a random authority, regardless of legal relevance has no standing to "protect" infected hosts without explicit consent of the owners. If the infected host is causing harm to other internet hosts then it needs to be taken offline by it's network owner (e.g.: isp or datacenter operator that owns the IP AS number).

Second, in case of malware spreading (e.g.: wannacry) and DGA domains: if the domain is not registered, instead of a sinkhole, an administrative restriction on registering that domain should be placed. If the domain is registered, you want the IP infrasructure to be taken down. IP blocks an reputation damage can be very harmful. IP subnet owners are much more responsive and where that is not the case, a null route can be blaced to "sinkhole" the IP -- null routes are advertised using predefined BGP communities, this means it will be unreachable only by networks that accept that community (e.g.: FBI sinkholes an IP, american networks accept the community and block the IP while other countries might not). You have to understand why there are so many malware domains for C2 and why DGAs exist, it is more costly to become in control of an IP address than it is a domain. If you block the IP as soon as C2 is detected on it,the attacker will just change the A record to point to another IP, but they have a much more limited set of IPs and costly IP infrastructure so they'll be running out of them fast. You can use DGA and dynamic domains such as noip.org (MS famously sinkholed them taking down millions of legit hosts!) But you can't as readily come up with IP addresses. I like this approach because the IP owner is always in a position to force remediation of the C2 server or infected host, they can ban the user or work with them to remediate the infection after confirming, they can request removal from the sinkhole. Most malware operators have no more than a handful active C2 IP addresses but from experience, I see them use dozens of domains,sometimes from different malware campaigns pointing to the few few IP addresses in their control.

I am sure this has challenges but it is a cleaner way of doing it and focuses remediation on the C2. If this approach was taken, OP's IP would have been accidentally sinkholed, her webhost would contact her about it, she would show proof that the server has not hosted malicious content and work with them to lift the sinkhole. Meanwhile,the site can be moved to a different host (if it takes too long) and IP address, since the domain is not being sinkholed it would just work. Malware researchers and law enforcement can see if infected hosts continue to communicate with the new IP or if the new IP responds to C2 initial traffic to decide if it should continue to be sinkholed (costing OP hosting money if it was an attacker, it might cost them money and access to compromised hosts).

Interesting. I noticed that the blog post mentions the Nymaim malware family. I read about Susam's case when it hit Twitter the other day and might have even followed a link to his URL. Then a few days later got an email from my ISP Virgin Media claiming they'd detected Nymain on my home network.

I run macOS only and as far as I can tell Nymaim is Windows only. Still, I ran an malware scan on my Macbooks and nothing popped up, so I'm pretty sure nothing infected my devices.

Still, I wonder if I ended up hitting the sinkhole, Virgin was somehow notified and this triggered their email? Or maybe it's just a complete coincidence.

Edit: Sure looks like Virgin works with Shadowserver: https://www.ukfast.co.uk/it-security-news/virgin-media-to-in...

I have been at the receiving end of German authorities reporting an uninfected server of mine to the hosting company as Avalanche-infected based on Shadowserver information. It was unpleasant, particularly because it happened a day before a family holiday, so my spouse was annoyed when instead of participating in preparations, I was researching what had happened and explaining my innocence.

Although my server wasn't infected, it had connected to a Shadowserver sinkhole.

While it's good that there are folks who work to sinkhole botnets, the next step of accusing others of being infected based of what the sinkhole sees needs more care. I'm disappointed that, evidently, my expression of these concerns to the German authorities three years ago hasn't lead to a substantial change at the Shadowserver end.

As can be seen from your case (and mine), you can get blamed even if the software at your end of the connection wasn't the botnet software. Considering how a basic premise of the Web is that it's safe to dereference a URL and everyone runs software that does so (Web browsers!), it's a bad idea that Shadowserver doesn't require a narrower indicator of compromise.

There'd be less chance of folks weaponising this system against bystanders by framing them as botnet-infected if the Shadowserver Foundation sinkhole required the other end of the connection to exhibit more specific hallmarks of the botnet software.

That is the purpose of sinkholes. That's why you don't just change the DNS record to (or similar) - you want to log the traffic that you're seeing so that you know who is infected and can help them.

I'm unaware of this particular international cooperation arrangement but it's great to see.

Yup. Looks like the system is working pretty well. Plus I'm pretty happy that I've got an explanation for the email I got!

I would say it should be concerning to you that your ISP is actually watching what DNS addresses you resolve. I'd either suggest using secure DNS or using your own DNS server or some DNS server not owned by your ISP.

This kind of thing makes picking a personal email address a tricky decision.

Do I go with a @gmail.com or other corporate address? Then I risk losing my email if my account is suspended.

Do I go with a domain I own? Then I risk losing it if something like this happens.

Either way is serious because email is effectively a master key into all my accounts.

I'm honestly not sure what's best.

> Do I go with a domain I own?

This one, it’s this one.

Losing your domain tends to require human action: from someone forgot to pay the renewal to someone messed up and sinkholed it because they thought it was a C2 server.

But because humans are in the system there tend to be layers of processes that try to prevent you from getting to this state and can get your world back to normal if you do.

Gmail offers nothing like this. When the ML algorithms decide you are too many sigma in the “abnormal” category, you are done. And there is no one to talk to who can fix your problem.

Although if your using your own email address, be prepared for emails you send to end up in junk.

This is an unfortunate risk with running your own. Instead, pay someone - my Fastmail subscription is absolutely worth every penny.

If it ever becomes a problem I flip my MX record elsewhere.

That's if you use your own _server_. You don't have to point your DNS records to a server you own.

Don't put all your eggs in one basket.

Maintaining a domain you control, with a couple of privacy-aware (Protonmail, Fastmail) fallback / out-of-band email contacts, and possibly a few online presences (Twitter, Mastodon, Reddit, etc.) which you can use for signalling, helps.

Knowing actual postal or phone contact information for key partners is also useful. You can use these to communicate in an emergency as well, and spread word.

Operating as a pseudonymous nym online for a decade and experiencing several lockouts / shutdowns over the period ... has been interesting.

Do both, and add both email addresses to every account that you can.

Huh, it seems obvious but I hadn't thought of this.

I think that's because I was coming at this from the perspective of trying to prevent getting hacked, but really I'm less worried about that than I am about losing access.

Option 3: Contract with a mail forwarding service like pobox.com. That way, you can move your mailbox service with the click of a button. Since they are not involved in delivering your mail, the chance that you'll get your account suspended for anything short of doing real crime is essentially zero.

Looking forward to hear from Shadowserver on a few points...

• What led to the false positive.

• What actions were taken to notify the domain owner about the actions being taken against them.

• Why there was not a comment put into the Whois entry — or in some other obvious place — saying what had been done to the domain.

I want to know why law enforcement allows a private organization to seize private property based on some algorithm.

I have heard bad things about shadowserver in the past. Now I wonder how much other collateral damage they have done over the years.

Domain names are not property, and this is not under the pervue of law enforcement. The country registrar (NIXI) is working together with someone to prevent abuse of their systems. When you purchase a DNS entry, you agree to this sort of thing as part of the ToS.

The domain got transfered to The Verden Public Prosecutor's Office.

Evidently law enforcement was involved in some capacity.

Right; this is essentially the same as a mail server operator relying on a DNSBL.


This is what I was wondering as well, it seemed bizarre to me and I expected there to be much more outrage here about it. I wonder if it's a bit like local parking regulations, which tend to be written in ways that allow towing companies to abuse you as much as possible.

I'd rather like to know, what gives shadowserver the authority to initiate such domain takedowns/sinkholing in an act that's pretty much vigilantism?

The decision is made by the registrar, so that's who should be answering these questions.

As far as I can tell the chain of events was:

1. shadowserver had an oopsie (i.e. added a false entry to their list)

2. that list was taken by the prosecutor's office to without further checks and by the authority of the prosecutor's office (i.e. state attorneys) of Niedersachsen in Germany and contacted the registrar in India.

3. The registrar transferred the domain

Between each of these steps, there are so many things that went wrong, on so many levels.

Between 1 and 2: Why is the word of some rando org taken as gospel? Without being independently investigated by the office, which sole raison d'être is (independent) investigation of the claims brought to it.

Between 2 and 3: A German prosecutor's office has no authority outside the EU; and even if it did, it could not tell anyone anything to take enforcing actions without a proper court order, even if it's an international request.

Also in the grand scheme of things, considering this was all done to "take down a botnet", this is a path to hell paved with good intentions. How about creating incentive that software and hardware manufacturers get their shit together and secure their shit?

Shadowserver runs the sinkhole, its up to the registrar/dns company (in this case namecheap) to point it at the sinkhole. The Feds provide a list of "identified" domains to all the major providers for the avalanche take down. Since this is an on going thing, someone at namecheap blindly ran the list without any verification.

I don't find this surprising at all.

This can happen in any scenario where a special shortcut has been added to get around a standard process (where standard process usually involves some human review and judgement).

I imagine that in most cases, the shortcuts were created simply to speed up a process where some (perceived) harm is significant, and a rapid change would alleviate this harm. This would allow some enforcement agency to rapidly take down a child porn site, for example.

Such shortcuts can also be designed to prevent any other party from knowing what is going on - why the change is being made. This would be the case where some high level governmental agency (FBI, for example) wants to change something to prevent a situation they deem as bad or to perhaps to enable better awareness of a process/communication by inserting themselves into the middle of it.

And finally, in this case, an automated or human error resulted in this person's domain name being included on a "bad list", and the shortcut swept their domain along with the trash.

This particular situation doesn't bother me as much as the "Google/Apple/FB suddenly closed all my accounts" scenario (which is often triggered by some opaque artificial wishful-intelligence system).

Regardless of the situation, it's not ideal that our best course of action in recovering something wrongfully taken is by complaining on public forums. It's a shame that we have to hope for the attention and generosity of someone with more power to champion our case to right the wrongs.

So I say, bring the humans back into the process! :)

This is not really ok. There must be a clear contact point for the affected people (not only namecheap).

Also it was very bad on the transparency front. If they are taking down a domain, the operation is not secret anymore, so they can tell why. No telling you is bullshit.

That being a German operation, I would expect much better on the democratic handling of it. And it being an international operation, India should have complained that it was badly done too. What would happen if namecheap didn't care?

I think there is a clear point of contact, and it is your registrar. This is part of the reason a hierarchy of registrars exist, rather than each TLD just being one organization maintaining its own registry service: so that the people with ultimate authority (the TLD, in this case) can have personal relationships with representatives of “constituencies” of domains (the registrars), rather than necessarily-impersonal relationships with a flat collective of millions of individual accounts (the domain owners).

By analogy: Google has necessarily-impersonal relationships with millions of Gmail users; but rather more personal relationships with far fewer GSuite and Google Cloud organization owners. If you were an employee of a company that uses either of those, and your service was breaking, you’d ask your GSuite organization-owner (i.e. the person Google has a personal relationship with) to contact them for you.

I did not say it was ok. I think it's a bad system, and I think it's getting worse - company by company, and month by month.

But when it comes to not telling someone, that's usually because there's some high level legal/law enforcement situation. In those cases, there is sometimes a reasonable argument for not alerting someone. The problem is that if there's no human oversight, innocent people sometimes get caught in the net.

> This particular situation doesn't bother me as much as the "Google/Apple/FB suddenly closed all my accounts" scenario

This situation bothers me so much more. Google/Apple/FB are private corporations. If they continue to adopt user hostile practices it is possible for some other company to out compete them in the more or less free market of internet services.

TLDs are government controlled entities administering scare resources. I certainly _feel_ like I have much more ownership over my domain name than I do my Google or Facebook account.

Here's the problem: A big corp is never going to alienate 10% of its user base at large, enough to motivate a competitor. But they are highly incentivized to alienate 1% -- millions of people, as a cost-cutting measure. Or they might even alienate 20-40% of potential users, as long as those users are not wealthy or powerful enough to "matter" (see: chattel slavery).

This is the history of society, of marginalization of the disabled or other outgroups. It's hard work to overcome that, often through force (Civil War) or the threat thereof (Americans with Disabilities Act).

I hope to see someone come up with a bright solution to this in the near future and post it here.

One of the most terrifying thoughts is that of losing domains to someone who points them at alternative mail servers thus gaining access to everything that uses email for verification.

It is really unbelievable that a legitimate domain can be transferred so easily without any verification or due process. Isn't there an EPP-code-based domain transfer process to prevent exactly things like this?

As mentioned in this blog post, the domain transfer was done as part of an international operation against the Avalanche botnet. As such, it was a legal action as opposed to an administrative action. Further, the action was taken at registry level as opposed to registrar level (which is one level lower than the registry). Therefore, no EPP code was necessary and the "clientTransferProhibited" domain transfer record was ignored.

It seems like there should be a notification after the fact so legitimate domain owners have a chance to reach someone who is in the know.

It looks like here the whole domain transfer was bypassed on the pretext of a possible botnet participation. The OP is absolutely right. Domain transfer needs to be much more robust than this.

The FBI and analogous TLAs do this all the time. And generally, there's no recourse.

In many cases, sites have resorted to distributing their IP addresses.

Personally I’d rather deal with the FBI accidentally seizing a domain than some foreign entity.

The National Internet Exchange of India shouldn't be a foreign entity for holders of .in domains.

> Shadowserver Foundation

The Foundation didn't seize the domain, it just suggested to the NIXI that they should do so.

“Why did you seize our domain?”

“Shadowserver said it was bad”

“But it’s not bad”

“Take it up with them. I hope you speak German.”

You'd be in luck, the Shadowserver Foundation is essentially a proxy for the FBI.

They control various seized assets seemingly unrelated to their publicly stated mission, like libertyreserve.com. Shadowserver used to host the seizure page for liberty reserve too.

That’s a pretty big claim. Source?

The Liberty Reserve connection arouses suspicion, no?

.in is a ccTLD, so there are no requirements on its transfer process whatsoever. There are plenty of ccTLDs out there that don't use EPP at all, and simply maintain everything in a hand-edited database or provide a web console (but not programmatic interface). I doubt that .in is one of these ccTLDs that doesn't support EPP, because the big ones tend to support it for obvious scaling reasons, but there's still no protections being offered because the Indian government can always just do what it wants, like provide an API to an anti-botnet company in Germany that can instantly yank any domain name it wants to.

A registry is little more than a DB at heart. In EPP you'd of course send a request with password, it'd go pending Transfer, need to be acked, etc. But nothing stops an employee from just changing things manually, outside of company policy and restrictions. Due to the large amount of immediate domain takedown requests(usually some danger or highly illegal activity involved), there's always a process in place. It sounds like this fault lies on the Gov't agency making the request, for not doing its due diligence prior.

Automated legal actions and takedowns like this introduce a lot of risk of collateral damage, but I wonder what the alternatives are?

The investigators would likely argue that notifying domain holders would reduce the chance that they can take down a botnet's infrastructure successfully, which seems likely.

Could there be some maximum time after which the 'rule set' for the auto-takedown code needs to be made open source / public? It must presumably be implemented as software and/or configuration files.

That would at least allow for inspection, confirmation and disputes about how it's implemented, and if this was 30 days or so, it shouldn't risk the takedown effort.

While top-tier network engineers are developing takedowns like this, presumably they'll do a good job of minimizing false positives - but as this case shows, it's not always going to be perfect - and I worry that if it becomes more common, we'll see sloppier implementations.

That could lead to connectivity and access issues for more users (again in an international context). It's great that the situation was resolved in this case but I imagine not all users would be able to raise a complaint at a similar level of technical detail and respectful tone and for it to receive the same amount of attention.

Maybe that's untrue - maybe injustices really do get amplified by social media and relying on companies to notice this 'works'. It doesn't sit particularly well with me as a remediation process though, and I'm not sure it scales.

How is one supposed to get this resolved if the CEO OF NAMECHEAP doesn't see your tweet to get involved?

Is your domain just gone at that point?

Is it really that easy to lose a domain name...by someone doing an 'oopsie'?

The idea would be that the support ticket with namecheap should be enough. Though I doubt it would be, for the average person.

Doubt is an understatement. lol

> I also wondered if a domain name under a country code top-level domain (ccTLD) like .in is more susceptible to this kind of sinkholing than a domain name under a generic top-level domain (gTLD) like .com. I asked Benedict if it is worth migrating my website from .in to .com. He replied that in his personal opinion, NIXI runs an excellent, clean registry, and are very responsive in resolving issues when they arise.

I'm not sure that's the correct conclusion to come to from this experience. Yes, the registrant happened to get lucky in this case, in that they had significant enough reach on Twitter and HN to get the right people to pay attention and get eyes on resolving the issue. But that easily could not have been the case (and might still be the case in the future), and with a ccTLD, you have no recourse.

I think the correct lesson here is to go with a gTLD, because if worse comes to worst you will always have recourse through ICANN if necessary (since the gTLD operator is contracted with them). On a ccTLD it's not always gonna work out. Heck, the registrant was already ignored by the ccTLD operator in this case anyway; it's frankly kind of lucky that they had the CEO of their well-known registrar go to bat for them. That's not the kind of intervention you should be regularly relying on to keep and maintain your domain name!

Would the experience be any better in gTLD? The registry of .com gTLD is VeriSign Global Registry Services. Would ICANN handle a domain-related dispute themselves or would they redirect us to Verisign? Is Verisign any better than NIXI?

Someone less technical would likely have no idea what happened to their domain. An individual relying on their web presence for income could be massively impacted by something like this. There really does not seem to be a clear way for someone to a) know what the problem is, and b) get it resolved quickly.

Every domain has a technical contact, it's part of the WHOIS schema. A non-technical website owner hires someone to handle technicalities, just as a non-mechanical car owner hires someone to handle their cars mechanics.

Sure, if you don't pay attention to the care of your domain, it can break in incomprehensible ways, just as if you don't pay attention to the care of your car, it can break in incomprehensible ways.

In theory, sure. But according to most of my domain WHOIS records, the technical contact is somebody named asdflkj_34890f@privacyprotection.com.

> My website was missing. In fact, the domain name resolved to an IPv4 address I was unfamiliar with.

Do you guys know this stuff? If my domain started resolving to a new IP address, that would be just as unfamiliar to me, as the current address.

Should I ping my domain and write the results down?

Normally, if you did any setup before pointing the domain to the host, or if you connect via ssh, then you at least have seen the ip and probably have it recorded in files like ~/.ssh/known_hosts.

But generally, the better your setup is documented, the better you can detect or diagnose when something goes awry.

I might not recognize the address. But `whois` will tell me whose network it's on. Anything other than my hosting provider would be a red flag. If it's the right company but the site still looks wrong, I'd try to SSH, which would fail because known hosts (see adjacent comment) and also authentication would fail. Then I'd reach for the VM console inside my hosting provider's site, and see the correct/expected address there, and hopefully realize what's up. You could also check your DNS provider, assuming that hasn't been compromised.

It can be a good tradeoff to know where/how to find information rather than actually knowing the information.

If you have a small infrastructure, knowing your IPs is useful.

For a larger one, the netblock(s), ASNs, or hosting. provider / region.

I tend to know recognize the networks my servers are on, because I only have one or two.

As a small business owner, this terrifies me. Since the TTL for NS records is 48 hours, a domain takeover like this could easily bankrupt a lot of SaaS companies.

What options are there to prevent this? Would a registrar such as MarkMonitor provide at least some notice or protection?

Namecheap rocks, been using them for years.

Really wish they'd pick another name: it's really hard to convince clients to take them seriously and let me choose them. If I had a dollar for every time someone insisted on GoDaddy... :-(

Ah, GoDaddy: a name that is much easier to take seriously.

What I find interesting here is the interplay and mix between private, public, and governmental concerns.

In the physical space, in the states you're free to walk out into the public park, put on a hat saying something atrocious like "I hate cats!" and peacefully petition your fellow citizens to destroy all cats or something silly.

When we moved to printed distribution, there was still a clear bit of guidance; as long as you weren't directing people towards violence, you were good. Most newspapers were locally owned and would even be happy to print your letter to the editor about cats.

But now? We've got a foundation doing something like a regex match on domain names, we've got a criminal element hijacking computers, we've got various government-condoned organizations for managing tlds, we've got registrars. All of these are different types of organizations working in different countries and established for vastly different reasons.

I am reminded of two things. First, Thomas Paine made the point that it was better to live under a dictator than a complex system that hurt you. Under a dictator, you had a guy to point to when things went wrong. It was them! They are responsible for this awful thing! Under a complex system? There's nobody. Bad things just happen, and when you try to ask about it, each party can explain to you that they were working for good reasons to the best of their ability. There was a problem, but no reasonable way to discuss, diagnose, or propose fixes to it.

The second thing was a story from the 80s about a U.S. official, Raymond Donovan. He served fairly well in public office but was accused of some serious crimes. He was destroyed in the media. Then they found out he was innocent. He asked a famous question "Which office do I go to to get my reputation back?"

I'm extremely happy this was resolved, but good grief, if I didn't know anything about the net, and my domain had just been set up instead of being in my control for 12 years, which office would I go to to get my domain back?

Either we own things or we don't. If every bit of our participation online is owned by somebody else, this should be a lot bigger deal than it is currently.

> First, Thomas Paine made the point that it was better to live under a dictator than a complex system that hurt you.

Interesting; can you point to the quote/source on that? (I'm also reminded of the Jobs quote: "Conspiracy is optimistic! You can shoot the bastards!")

I'm not sure one is "better", per se, but problems with complex distributed systems tend to be painfully intractable. Worse, I think human nature is to map such problems to narrowly-scoped villain narratives which feel tractable; and so not only does the root complexity go unaddressed, but great efforts are undertaken to thwart the alleged villains, sometimes making the real problems worse.

Blaming the dictator doesn't solve the problem. Sure, you could try to overthrow a dictator, but you can also try to fix a system, by talking to any of the people involved in it (or if there are no people involved, try to modify the system yourself, since there's no one to stop you).

This is the Fallacy of Gray. Just because neither option is perfect, doesn’t mean that one option isn’t better. It’s clearly easier for a bloc of concerned citizens to solve problems in dictator-land than in bureaucracy-land: in dictator-land, you just have to remove one (probably very unpopular) guy, while in bureaucracy-land, you have to... um...

In dictator-land, you first have to remove the dictator. You still have to actually solve the problem after removing the dictator.

It's easy to say "the dictator is bad" but historically getting rid of dictators without a specific and concrete plan for what to do afterwards has not turned out so well.

In doctor-land, you are trying to solve a particular problem with a particular patient, the assumption being that once that problem is solved, the patient will remain self-regulating.

In political-theory-land, you assume all actors are bad, are at least will become bad at some point in the future. The question then becomes *how can we organize our political structures so that if we can't prevent dying, can we at least provide guidance to the next government that follows?"

It naturally follows that the real goal is not solving problems: it is providing problems and solutions that people can accept. If nothing else, providing problems and solutions in terms that future generations can reason about.

Don't worry. It's still _definitely_ a good idea to centralize all Internet trust in a DNS-based PKI.

Wow! "Accidentally?" That's inexcusable. He should be compensated, and the people who made this mistake should not be doing this sort of work anymore.

> He explained in his email that my domain name was sinkholed accidentally as part of their Avalanche operation.

Would it hurt the people doing domain takeovers to at least try sending an email to the registrant with contact info and a case number?

I feel there is an important point between the lines here: a central authority is well positioned to stop bad actors by working outside the definition of “expected behavior” of the system.

P2P systems do not get this by design. For load bearing infrastructure like DNS, escape hatches to stop bad actors putting humanity’s hive mind at risk is a handy tool that has been used effectively.

There are the philosophical questions of “how do we define a bad actor” and “who gets to decide what’s acting in bad faith?” But many botnets don’t suffer from that ambiguity, a well deployed/utilized botnet can take down load bearing internet infrastructure.

Being able to stop future threats without requiring migrating a P2P network _seems_ like a feature, and reminds me a bit of common law. In law, we know there will be cases we can’t anticipate upfront and leave the courts room to interpret. P2P seems to still be in a state where we have to identify and protect against every form of “bad actor” upfront in our game theory in order for our network to be stable moving forward.

A couple of years ago we lost our domain [1] due to a registrar (that we were not a customer of) erroneously issuing a suspension. The amount of honor system involved in the whole process, particularly in ccTLDs without as much oversight, was really surprising.

[1]: https://medium.com/thisiscala/the-duct-tape-holding-the-inte...

That's worth resubmitting to HN.

This is so bad. This process needs transparency and notifications! It shouldn't be possible to sinkhole a website without the owner being aware.

So, did a court just order a domain to be taken down based on erroneous information from a private entity?

I don't see any mention of a court; the TLD manager (NIXI) probably took the initiative based on said erroneous information.

NIXI is subject to the jusrisdiction of CERT-IN (Indian Computer Emergency Response Team - https://cert-in.org.in/), which in turn was one of the participants of the Avalanche takedown program.

A legal request originating from Germany would have been approved by CERT-IN and NIXI would have had to comply.


AFAICT, CERT-Bund (the German CERT) believes what Shadowserver tells them. When CERT-Bund makes a request onward ,it looks more official, but I don't believe any more actual vetting happens.

Who said it was a legal request? I got the sense that Shadowserver has some monitoring software running, which triggered a warning for NIXI who sinkholed the domain.

most likely interagency request. NIXI as an Indian govt operator is subject to RTI queries (Right to Information) and court action.

However NIXI is subject to CERT-IN authority which is clearly part of the anti-botnet collective.

With all the domain buying out and weirdness, it is my fervent hope people seriously begin reconsidering what it means to have a domain name and to implement other structures to serve out names as opposed to the now, obviously bought and sold, DNS.

The server is up. However, it is possible that at your end "susam.in" is still resolving to a sinkhole address. I have created a mirror of the blog on GitHub Pages for you. Visit https://susam.github.io/blog/sinkholed/ to read the mirrored blog post.

> for some perceived violation

This is the fundamental problem with this sort of crowdsourced censorship - there's no accountability on the reporters and no repercussions for bad faith.

I still don't get it, what exactly was flagged that caused the domain to get noticed by... Who exactly? Did it have anything to do with the owner logging into his server? What is the sequence of events in bullet point format? I know the author tried to be clear but I'm confused as to what actually happened.

A particular malware contacted its command and control servers via procedurally generated domain names (to make it difficult to shut down just a single domain that controlled it).

Malware researchers reversed a sample of the malware and started blackholing domains that matched the pattern to get ahead of the malware by preventing it from communicating with the domain du jour.

It just so happens that the authors domain pattern-matched domains that would be contacted by the malware.

Thank you. So it was just a coincidence that this person logged into his server at the same time as the domain was taken down? It was simply a suspicious looking domain name?

tldr; Collateral damage from law enforcement taking down a botnet, resolved reasonably fast.

It wasn't law enforcement taking the domain name, it also was only solved fast because of intervention from the Namecheap CEO.

I didn't say it was law enforcement directly taking down the domain name, I said that it was collateral damage from law enforcement operation against botnet.

As the original operation was run by interpol, europol etc i think this was a fairly accurate description, especially in context of a tldr; which gives some constitutional rights to simplify things :)

Wow, so a German entity can just go around and confiscate internet domain names, without a warrant? That's really dangerous.

This is horrifying.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact