Hacker News new | past | comments | ask | show | jobs | submit login
Cisco Nexus 9000 Switches Allow SSH As Root (nist.gov)
221 points by sky_nox 7 days ago | hide | past | web | favorite | 106 comments





If you have the Cisco 9000 Series, patch them now! This SSH backdoor allows an unauthenticated, remote attacker to login as root.

This is exactly why I only buy belkin routers. I can't even connect to it.

Ha! Thanks for making me laugh out loud.

Without wanting to start a political flame war it would be great if there was consistency to how we in the tech community and the media treat these types of vulnerabilities. When Huawei have these sorts of bugs they are reported as backdoors. Bugs happen in software be nice if put the nationalism aside and reported it consistently as bugs or vulnerabilities

Nexus 9000, running ACI, not normal NX-OS, as opposed to the ASR 9000 series which are common internet routers.

Cisco model numbers are fun.


god I still have nightmares about cisco sales rep trying to push ACI as the "solution to all problems".

Also you should have ACLs in place and VLAN segmentation (assuming their use as pure layer 2 devices) so that only certain authorized sections of the network are even able to reach things like the management ssh and SNMP daemons.

Or turn that all that shit off and go full Out of Band management - ok there are some trade-offs here.

'unauthenticated' should have been in the title of this post

Given that it is a $30k+ piece of kit I suspect not too many people here have them.

  > Cisco Nexus 9000 Switches Allow SSH As Root 
Cisco Nexus 9000 Switches [have a vulnerability that Allows an attacker to] SSH As Root [over IPV6 using a default key-pair]

If a compamy as big as Cisco can screw this kind of thing so badly, the future of IoT looks bleak.

You know what they say: the S in IoT stands for security.

Can't wait for marketing people to catch up to this and call devices "SIoT" without changing anything about the product.

Perhaps "Security Hardened Internet Things" would be more appropriate?


Cisco hasn't had a good reputation for awhile IMO.

Like any other network solutions vendor?

I heard https://nvd.nist.gov/vuln/detail/CVE-2019-1804 is Cisco's ninth backdoor so far this year. Not ninth security problem total, ninth backdoor. The ninth security problem Cisco shipped intentionally.

Meanwhile, the router that serves my office is from a company that's had fewer than nine security problems in the past ten years. Two, I think, but I confess I don't really keep count (ditto the nine above). The precise number doesn't matter, because

1. If you want to be cynic you can point out that 9>0 and 2>0 and really they all suck.

2. And if you don't want to be cynic, then Cisco's recent record is in a league of its own. Steals the show. Makes other people's CVE count look like rounding errors.


> I heard https://nvd.nist.gov/vuln/detail/CVE-2019-1804 is Cisco's ninth backdoor so far this year.

Also the 9th they have fixed.

> Not ninth security problem total, ninth backdoor. The ninth security problem Cisco shipped intentionally.

How can you be sure it was intentional?

> Meanwhile, the router that serves my office is from a company that's had fewer than nine security problems in the past ten years.

How can you be sure? Did you audit all the source code yourself? Did you compile the source code yourself and are you running only binaries you compiled? Are you sure you can trust the compiler you used?

Or, are you assuming that, because there isn't a CVE, there isn't a vulnerability or security problem?

Fewer CVEs doesn't necessarily mean more secure, it may just mean less validation/testing etc.

But sure, it could mean more secure, it's just not a guarantee.


Someone at Cisco intentionally created a keypair and intentionally put it in the image build process. They may or may not have intended to put it in production builds, but they clearly intended to set it up in some form, when they could have just ... not. If you take the easy but risky approach, you have certainly intentionally put yourself at risk.

I've worked for a company that built OS images for distribution to customers. Putting my SSH key in development image builds would have been convenient, but there was too much of a risk of exactly this problem; instead we just made it easy enough to download an SSH key on a development build (and start up an sshd) once you've booted it and have physical access to a terminal.

Also, a practical concern with disclosed vulnerabilities is that non-nation-state attackers (which are most of the attackers most people care about) are very unlikely to find and exploit a vulnerability that neither has a public CVE issued now nor will have one issued for years. So even if the alternative vendor has difficult-to-discover vulnerabilities, there is, in a very real sense, reduced exposure from those vulnerabilities compared to things that are disclosed and fixed. And especially if Cisco's disclosed-and-fixed vulnerabilities originate from outside vulnerability reports, there's a definite correlation between whether a vulnerability can be found by someone who would report it and whether a vulnerability can be found by someone who would exploit it.


Backdoors aren't bugs like most others. Buffer overflows happen because someone mistypes or forgets a length check, etc.

Backdoors are unusual: They happen because someone writes code of the form addAccount("s3kr3e", "s3kr3t"), and that's code that's written. You can typo and accidentally omit a bounds check, but you can't typo and accidentally end up with a valid SSH key pair and code that installs it.

It's possible to ship that SSH key pair and the code to customers that as a bug, e.g. if someone writes that code on purpose, intending to add and deploy s3kr3t/s3kr3t but not intending to ever have that code on the branch that's deployed to regular customers, and then someone else mismerged. In that case serving it to customer X is due to a bug, it should only have gone to customer Y or test environment Z. What I'm saying is that shipping those backdoors at all must have been intentional.

(Personally I think shipping backdoors to test environments is fine. Including test environments at customers. Risky.)


> Backdoors aren't bugs like most others. Buffer overflows happen because someone mistypes or forgets a length check, etc.

> Backdoors are unusual: They happen because someone writes code of the form addAccount("s3kr3e", "s3kr3t"), and that's code that's written. You can typo and accidentally omit a bounds check, but you can't typo and accidentally end up with a valid SSH key pair and code that installs it.

Not sure if you're intentionally trolling but a backdoor is simply some code which bypasses security that a particular person knows about. It does not have to be obvious. The ones that have plausible deniability are the better ones as that is considered a feature. That way the company can say "oops, we made a mistake".

To say that a backdoor must be obvious is absolute nonsense particularly for closed sourced binaries where disassembly and using simple tools like https://en.wikipedia.org/wiki/Strings_(Unix) would reveal the presence of such backdoor.

> (Personally I think shipping backdoors to test environments is fine. Including test environments at customers. Risky.)

It depends. Unless that "test build" specifically has an option that disables all "test related backdoors" then the answer is no. You cannot risk having something slip through to a production build.

As the previous poster said:

> I've worked for a company that built OS images for distribution to customers. Putting my SSH key in development image builds would have been convenient, but there was too much of a risk of exactly this problem; instead we just made it easy enough to download an SSH key on a development build (and start up an sshd) once you've booted it and have physical access to a terminal.

Another very common solution would be a template where at build time the keys are generated/imported by whatever build system is being used.

That way if something unintended happens or is "forgotten about", the build simply won't have a key at all and therefore will not work, rather than having a set of keys that are the same across all production builds.


Can you tell us your vendor?

We are moving offices, and it's time to change equipment, Been reading about but still haven't gotten a good list.

The only thing i found is great micro tick for wifi routing/AP's


Mikrotik is okay for small routers, and so is Ubiquiti. If you get Mikrotiks, look for ones with angly metal boxes, not curvy plastic ones. And avoid the GUI stuff, use the CLI. If you're looking for larger routers I'd look at Juniper first.

All of those will give you hardware that does the job and stays up, and provide uncomplicated upgrades for many years.


Palo Alto seems to have a good reputation but that could be security through obscurity as no one can actually afford to use it.

Palo Alto (like every vendor) has had similar vulnerabilities in the past with their web management. Typically management of a switch/firewall isn't exposed to directly to the Internet.

An increased use of SSH keys for credential guessing is also found in our SSH honeypot: https://pmcao.github.io/caudit/

Heh - people are harvesting compromised secret keys then? Because "guessing" secret keys shouldn't be viable?

What product segment is this, what kind of organizations are likely have them?

Also, link to Cisco's own advisory: https://tools.cisco.com/security/center/content/CiscoSecurit...


It's a switch used in enterprise data centers. It's pretty critical because servers plug directly into it.

This is a pretty egregiously editorialized title; what we know is that there's apparently an SSH keypair authorized on these devices, for which the private key is available on the device. That's a terrible, ugly vulnerability, but it's as likely due to stupidity as to malice.

The right title is something like: CVS-2019-1804: Cisco Nexus 9000 Switches Allow SSH As Root.


It allows anyone who knows the default SSH key pair to login as root. How is that not a backdoor?

Backdoor definition: "A backdoor is a method, often secret, of bypassing normal authentication in a computer system."


People sometimes read "backdoor" as something intentionally left by an insider for later use by themselves or others.

Well Cisco wrote the code so it has to be in some way intentional but it doesn't necessarily mean it was done maliciously though. It could a private developer key used for testing accidentally got pushed out in production code or some poorly thought out management "feature". Regardless it is an epically dumb mistake for a company like Cisco to make on an enterprise product.

> Well Cisco wrote the code so it has to be in some way intentional but it doesn't necessarily mean it was done maliciously though. It could a private developer key used for testing accidentally got pushed out in production code or some poorly thought out management "feature". Regardless it is an epically dumb mistake for a company like Cisco to make on an enterprise product.

That someone might not be the company, it might be a developer.

It's entirely true that the company says it's not a backdoor, the developer says it's a mistake, but he/she was approached from an external organization.

Unless you can provide either way it's impossible to classify it as a backdoor or not.


> People sometimes read "backdoor" as something intentionally left by an insider for later use by themselves or others.

And considering you can never know if someone else knows about it, that means you can never know if it was a backdoor.


And what do you think this was? Virtually all router/networking devices have some kind of "hardcoded account" (read:backdoor) and this is only slowly changing. I believe the EU is going to ban the practice soon.

You’re joking right?

It’s “allow ssh as root with a publicly available ssh key”. Your version is making it sound mundane.


Sad state of affairs when terrible, ugly vulnerabilities are mundane.

If it was a genuine "backdoor" why would you want use a publicly available key?

Being the only keyholder reduces plausable deniability, so maybe.

The private key is on the shipped devices, from my reading.

Agreed.

I'm hypothesizing that you might do this, even with keys intended to be used as a back door, by shipping it on devices, you vastly increase the number of potential suspects for any backdoor abuse.

Continuing this train of thought - a hardcoded password is classic example of a backdoor, and just as "public" as including a private key.


This isn't a backdoor but it is a major vulnerability.

If mundanity is your concern, add an exclamation point to it.

Accuracy helps.

It is impossible to know the motivation of the person who put this here but these constructs have no place in firmware for critical devices and Cisco should have known that for a long time already. Either they truly are idiots or this is malicious.

More likely they are just imperfect humans.

That's a bit vague; imperfect humans are both malicious and incompetent, often at the same time.

And often neither.

Then from whence came their imperfection?

Just stop. Is it really hard to understand that normal people can make mistakes without being malicious or incompetent? Imperfect and incompetent are not synonyms.

And my response to the OP was to push back on the idea that only "idiots" could make mistakes. To me that is an absurdly reductionist view of human nature.


You tell me stop and then proceed to ask a question? How incredibly rude. I suggest you rethink your philosophy of discourse, and words (hint: most qualities such as incompetence and maliciousness exist on a continuum).

Huh? You were labeling people as incompetent and supporting the OP's idea they were idiots. I was asking for empathy and understanding that people can make mistakes. And when I tell you to stop, I'm the rude one?

Yeah, telling somebody to cease communication and then asking them a question is absolutely rude. I think we're all incompetent to varying degrees in different domains, I don't think it's rude to express that. From my perspective I don't feel I was rude at all in this exchange until my last message with the snarky "hint" part, but I was okay with that since you had essentially just told me to shut up.

You are misinterpreting my very terse "Just stop". If I were to expand it: "Just stop trying to convince me that people who make 'mistakes' can only be incompetent, malicious, or idiots"

That is not the same thing as "Just stop communicating" or "shut up". I shouldn't have been so terse. Without the verbal cues you made a different assumption about what I was trying to say.


In this context I don't really see a difference between "stop making your point about the subject we've been communicating about" and "stop communicating in general." Am I supposed to talk about the weather, or engage in a lengthy meta-discussion about talking about the subject we've been talking about?

I never used the term idiots, that's you putting someone else's words in my mouth.

It's hard for me to imagine how a person can make a mistake in a given domain without being at least bit incompetent in it, hence my point about competence/incompetence existing on a continuum.

Edit: ...and if the person were malicious, it wouldn't be a mistake to begin with.


So people extremely knowledgeable in a domain don't make mistakes in that domain? Or are they just "one notch too low" on the continuum- enough so that they make a mistake? I just don't think that holds.

Mistakes aren't always made due to incompetency and extremely competent people still can make mistakes.

I think most of the "seven factors that lead to stupidity" could still affect someone very competent: https://fs.blog/2019/01/how-not-to-be-stupid/


Well, I do apologize for my sloppy writing that made you think I was telling you to shut up. That wasn't my intent. I was just trying to say that your argument wasn't persuasive to me.

We'll just have to agree to disagree regarding human nature.


Ok, we changed to that. Thanks!

Submitted title was "Backdoor Found in Cisco Routers CVE-2019-1804".


I agree , this is not a backdoor in the more commonly understood sense of the term , post-Snowden.

At the same time, your suggested "right title" doesn't seem too accurate either (imo).

When I see the term backdoor, I think of something done intentionally.

Maybe something like: Embeded unsecured credentials , allow attacker remote root ssh access

( it's actually much harder than I initially thought , to phrase)


The mods asked me to email comments like this to the hn@yc.c address in the footer (Contact link), and have been responsive (not necessarily agreed, but they do reply!) when I've done so. I emailed them a link to your comment as the edit request with an attempt of my own:

> CVS-2019-1804: Cisco Nexus 9000 remote root exploit via SSH-over-IPv6

(Yes, it's a backdoor, I ran out of time sorry)


Clarification - Anyone can email, not just me

CVS-2019-1804: Cisco Nexus 9000 Switches Allow SSH As Root via IPv6 only.

Which makes it even more likely to be explained by stupidity as to malice.


It's still a bit editorialized since this only affects ACI mode, not the default (and far more common) NX-OS mode.

And plausible deniability is the #1 rule when being malicious. If you know enough to use an asymmetric key instead of a password, but not enough to think it's a good idea to leave the private key there, you're in a weird cross-section of expertise.

A broken script that turns a development build into a production/customer build could also be at fault, forgetting to delete the default key pair.

Equally as plausible which is exactly where you want to be if you're malicious.

I'll spin this around though: What would a high-quality plausibly deniable backdoor look like to you?


> Allow SSH As Root

That sounds like they just erroneously left AllowRootLogins yes in the ssd_config, which would not be a critical vulnerability.


What is a backdoor if not this?

A backdoor to me suggests an intentional loophole through a level of security. A bug that does the same is severe, but isn't intentional.

At least that's my reading.


But any competently inserted intentional backdoor is going to be indistinguishable from a mistake.

If Cisco had some SecretFBIChinaBackdoor() function somewhere the backlash would be way way worse (or at least an unknown). Whereas at this point it's abundantly clear that serious "non intentional" security vulnerabilities in networking hardware basically go ignored by the market.


>But any competently inserted intentional backdoor is going to be indistinguishable from a mistake.

Maybe, but without proof it's still just speculation. Real bugs do occur often, and sometimes in sensitive areas.

I understand wanting to be vigilant. In both assuming malice and assuming human error though, you're still forced to make an assumption.


You're not forced to make an assumption, you can just be honest and say you don't know. There's too many comments in this thread effectively saying "it looks unintentional, so it's unintentional".

>You're not forced to make an assumption, you can just be honest and say you don't know.

Indeed. That's what I was trying to say.


Yes, that is the most commonly used to describe an “undocumented” access credential, regardless of why. This has been used for credentials that were added and forgotten, credentials that were added to permit unauthorized access later, and credentials that were added to permit authorized repairs more readily. These credentials were included in an “undocumented” (or “unpublished” might be more precise) manner and can be used to bypass security, so “backdoor” is correct.

This of course says nothing about whether its inclusion was due to intent, incompetence, and/or malice. (If the private key includes “Comment: hack the planet” then yeah it’s malice :)


Seems like a not very secure front door to me.

Is there an explanation of why it's ipv6 only?

ACI control traffic is probably IPv6-only. They may be taking advantage of link-local addressing.

I don't own nor have I read the manual of one of these, and there's not much in the way of details on that page, but isn't this more like "use the factory-supplied default key to get in for the first time, then change it to your own"?

People are calling it a backdoor because presumably it was not documented anywhere.

can someone help me understand this better.. Did Cisco leave a user public key in the switch and the private key has leaked ? To exploit this vulnerability attacker has to get hold of that private key ?

The keypair is essentially some default known value.

You shouldn't be able to use this to connect at all, but apparently works over IPv6.

So you'd have to have the private key, as well as knowing the IPv6 address of the device you're connecting to, and that device would have to have a route to the internet or a location you could connect to it from.


Any idea why it works for v6 but not v4 ? SSH authentication itself is agnostic to the IP version, no ?

This is a nasty one! Sloppy in hindsight.

There is one bright side to otherwise disgraceful incidents: All the customers running older versions are now forced to upgrade to the latest versions. The burden of supporting really old versions suddenly vanishes.

Box vendors should really stop selling unmanaged boxes/solutions. In reality, customers end up buying service contracts anyway along with boxes. Instead, sell usage/service/connectivity and manage the hardware. A critical patch like this one could then be applied before a PSIRT is released. Frequent upgrades(security patches or feature/bug fix patches) are now commonplace. The user experience would be so much better if the solution were managed by the vendor (cloud managed).


Most places (especially where they have enough money to be buying Cisco Nexus 9k kit) will want some sort of change management, not the vendor to be making arbitrary changes to their critical infrastructure.

Also, given the number and severity of these sort of vulnerabilities in recent times, do you want to give the same companies remote access to your infrastructure as well? :)


> Box vendors should really stop selling unmanaged boxes/solutions

Users should no longer be allowed to own their own hardware? That'll be popular with both the hacker crowd and the high-security people.

What of devices that are never intended to be connected to the wider internet?


Cisco bought meraki that provides a cloud managed solution, but it's only office equipment.

I worked with Cisco a lot in the past. I am so happy we have more and more open source alternative to replace all those network solution vendors.

I surely can't be the only one who sees open down to the hardware replacements as the only solution to this type of problems.

As a former Cisco employee, I can tell you why companies never want to open source their security-sensitive products:

Pros of open sourcing a product:

- fewer total number of vulnerabilities

Cons of open sourcing a product:

- more publicly-known vulnerabilities

- less effort required to find new vulnerabilities

The product might be more objectively secure, with more bug reports and more fixes.

But it will be less practically secure. There will be more known vulnerabilities, and many customers can't upgrade, leaving more total vulnerable customers. And worse, now anyone on the internet can try and find new vulnerabilities for $0, while before they'd need to buy a $1,000+ piece of hardware to even get a shot at the compiled code.

The real defense against this problem is security auditing. Security engineers try to hack the device while asking a bunch of questions about SSH connections and private keys. This is the technique most companies employ, often combined with bug bounties.


Auditor: "Question 1. Did you manage to add backdoor keys to your production build?"

Auditor: "Big surprise. We once again recommend that you use a build system. Question 2. ..."


Really what you're saying is that if you open source your product the degree of shittyness will be then obvious to everyone, and that security by obscurity keeps the managers happy because they think there is less work.

For most enterprise IT products there is no "open down to the hardware" replacement and there never will be because there isn't a business model to create it.

The Facebook open compute network gear is, and is equivalent to the Cisco Nexus series

Dude, Cisco, you had one job.

And they completed it successfully. They literally had a new backdoor every month for years now. No company is that incompetent unless ordered to be so.

Are this devices normally left with accessible 22 port in the wild?

A Nexus 9K is an expensive piece of kit, and is not a trivial switch to deploy what with VPC and other configurations being commonplace, so just powering it on will not deliver a workable product. I suspect most if not all deployments follow best practice and have a management VLAN with access lists control limiting the source address of the connecting client, and blocking access to port 22 from other networks.

* Edit * Plus the Nexus the backdoor is only relevant if the switch in using ACI, and not standalone NX-OS mode. ACI training is a 5 day course for advanced engineers. https://www.cisco.com/c/en/us/training-events/training-certi...


Western governments: Huawei needs to be banned from our collective infrastructure because backdoors

Also western governments: Cisco will remedy their errors


HackerNews: Huueerrggg Huawei can't even write secure code

Cisco: Hold my beer


The Huawei stuff is pretty bad, but the comments read like they've been copied pasted each time here. It's the exact same talking points.

Because the anti Huawei talk is obviously not in people's interest. Just look how zero politicians worldwide lament that Cisco should be banned from anything related to internet infrastructure despite showing again and again that they are unwilling to stop implementing backdoors. Cisco makes it obvious that backdoors are A OK as long as it's our backdoors. No one acting against Huawei actually cares about peoples security.

Is it xenophobia masquerading as national security, or national security masquerading as xenophobia?

Neither... it's nothing but hegemony with the pretext of both.


Huawei you all feel about that now?

This is a true backdoor, not some silly telnet left on.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: