
Netgear Signed TLS Cert Private Key Disclosure - Maxious
https://gist.github.com/nstarke/a611a19aab433555e91c656fe1f030a9
======
CiPHPerCoder
Some commenters are decrying that this post fails to meet the bar for
"responsible disclosure".

Please stop using that phrase. "Responsible disclosure". It's wrong and
_harmful_ , and the person who coined it agrees with me:
[https://adamcaudill.com/2015/11/19/responsible-disclosure-
is...](https://adamcaudill.com/2015/11/19/responsible-disclosure-is-wrong/)

You want "coordinated disclosure" instead.

Netgear doesn't do coordinated disclosure. They do non-disclosure. In the
absence of a coordinated effort, full disclosure is the responsible thing to
do.

~~~
ryanlol
It should be noted that it’s essentially _never_ the people who find bugs
arguing for “responsible disclosure”.

It sure is easy to tell others what to do with their work product when you
have zero stake in the game.

There exists a really easy solution to the purported problem of full
disclosure, vendors could just offer significant enough financial compensation
for non-disclosure.

~~~
JshWright
I think the millions of people with Netgear equipment deployed have some
"stake in the game".

How does non-disclosure benefit them?

~~~
throw0101a
> _How does non-disclosure benefit them?_

And how does dumping the vulnerability without a fix help Netgear owners?
They're all flapping in the wind right now.

IMHO, the best scenario is coordinated disclosure. Full disclosure may be
necessary to force a vendor to do something, but let us not pretend it is a
good thing.

~~~
StavrosK
> And how does dumping the vulnerability without a fix help Netgear owners?

It lets us know to buy another brand immediately and never buy Netgear again.

> IMHO, the best scenario is coordinated disclosure

The original post said that Netgear doesn't do coordinated disclosure, and
subsequent posts were arguing whether non-disclosure or full disclosure were
better. Nobody was disagreeing with what you said.

~~~
swiley
Netgear making worthless routers isn’t really news.

------
bottled_poe
Netgear has reliably demonstrated to me over more than 10 years that they are
incapable of delivering secure products of an acceptable quality. I actively
avoid this brand like the plague.

~~~
shock
I have posted this in another thread, but I am posting it again because more
people need to be aware of Netgear's practices:

I recently bought a couple of Netgear Managed Switches (for Business)⁰ and in
their datasheet they list "Local-only management" as a feature. Only after
they arrived we discovered that you only get limited functionality in the
Local-only management mode, you have to register the switches to your Netgear
Cloud account to get access to the full functionality.

Reading up on it, this was achieved only after a community outcry because in
the prior firmware versions the switch would have to connect to the Netgear
Cloud on every bootup.

Needless to say I would not have bought the swiches if I had knew I needed to
register them to Netgear Cloud to have access to the full functionality
specified in the data sheet. If I had bought them as a consumer, not as a
business, I would have returned them immediately.

Netgear are now on our purchasing blacklist.

⓪ - the switches are Netgear GS-108Tv3

~~~
mdszy
>Netgear Managed Switches (for Business)⁰

>⓪ - the switches are Netgear GS-108Tv3

are we not allowed to use any sane method of doing footnotes? Who does this?
Why would you even use footnotes in internet comments other than to
specifically screw over anyone who uses a screen reader?

~~~
bmn__
> are we not allowed to use any sane method of doing footnotes?

There is no markup on HN for footnotes.
[https://news.ycombinator.com/formatdoc](https://news.ycombinator.com/formatdoc)

> Who does this?

Blame the people running the site, this is just a small inconvenience compared
with other lack of features.

> screw over anyone who uses a screen reader?

As GP's footnote is plain text – not markup – there is no problem with screen
readers (or any user-agent, really). Curious why you would think otherwise.

~~~
mdszy
Because it interrupts the flow completely when they could have just written
the model number along inside of the already-used parens like a normal human
being. With a screen reader, it'll get to the superscript 0, read that out,
and then won't be able to read the actual footnote content until the very end.
Can't jump around like our eyes can. "⓪" isn't even read by a screen reader
(at least not narrator on windows 10). Footnotes in internet comments are
useless and unnecessary anyway.

------
beardog
I discovered this same thing in 2017. My bugcrowd report was shot down as dupe

[https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-
rout...](https://www.chaoswebs.net/blog/tls-key-reuse-on-popular-router-
models.html)

~~~
ploxiln
The difference is that these newly discovered certs with private keys were
signed by real Certificate Authorities trusted by browsers by default.

~~~
tinus_hn
So their security got worse. It’s a miracle people even bother attempting to
do responsible disclosure to fools like this.

~~~
josteink
> so their security got worse

Only because current browsers makes it increasingly hard to run medium
security, user-trusted networking.

------
politelemon

        Tuesday, January 14th 2020 - Initial Discovery
    
        Tuesday, January 14 2020 - Tweet sent attempting to establish communications with Netgear  
    
        Wednesday, January 15 2020 - Reached out to Bugcrowd to attempt to establish communications.  
    
        Thursday, January 16 - Bugcrowd responds, but we are unable to establish a communications channel outside of the Netgear bug bounty programs.  
    
        Friday, Jaunary 17th - Conversation with bugcrowd proves inconclusive  
    
        Sunday, January 19th - Feeling we have exhausted our disclosure avenues, we decide to publish
    
    
    

I didn't understand, can someone explain this to me - I see the author tried
to talk to Netgear, then it's BUgcrowd the next day... it's not clear to me
what the relation between Netgear and Bugcrowd is.

~~~
Maxious
Bugcrowd run bug bounty programs like
[https://bugcrowd.com/netgear](https://bugcrowd.com/netgear) on behalf of
companies

They explained earlier that if they entered the bug bounty program with
Netgear, they would be agreeing to not disclose: "By submitting the security
bug, you affirm that you have not disclosed and agree that you will not
disclose the security bug to anyone other than NETGEAR."

So it seems they were trying to get Bugcrowd as a intermediary to contact
Netgear on their behalf to "establish a communications channel outside of the
Netgear bug bounty programs"

~~~
politelemon
Thanks for the explanation!

------
gene91
Netgear picked a terrible and clearly unacceptable approach. I I read through
this discussion, trying to understand if there exists a good solution.

Using plain http isn't a good solution because it involves telling the user to
ignore the "not secure" warnings by browsers. This seems like the best
available solution.

Without network connectivity, the traffic can not be tunneled through a remote
server with proper cert. Nor can the box request a cert in realtime at the
time of setup.

Pre-provisioned cert won't work if the box is purchased more than 27 months
after manufacturing (the current max validity period). For home network
equipment, I suppose this isn't unusual.

I see TLS-SRP mentioned in comments. But the commenter hinted at its limited
adoption in browsers.

~~~
jniedrauer
When I can control the client configuration, I just set up my own trusted root
CA and then have the distributed systems automatically submit a CSR on
initialization. This doesn't work when you have to support end user's web
browsers though (unless you're somehow lucky enough to own a public CA and
don't care about ICANN rules). It really does seem like there should be an
encryption-only option for TLS. Maybe anything that resolves by IP address and
not name should display the green padlock even if the SAN doesn't match the
name?

~~~
gene91
Encryption-only TLS doesn't protect MITM. It prevents sniffing.

At a higher level, TLS provides integrity and privacy when the network linked
is not trustworthy. As a result, I'm not sure what your proposal achieves.

Similarly, resolution by IP address does not provide privacy or integrity
unless you trust the network link.

~~~
jniedrauer
The green padlock indicates two things about the server to the user: "This is
really my name" and "no one can snoop on our conversation." It doesn't
indicate anything about the trustworthiness of the server. The first one is
meaningless in the context of an IP address. If a user is going to get pished
by an IP address, then they could just as well get phished by
123.sketchydomain.com. The important part when communicating directly with a
device by IP address, is that the payload is encrypted end to end.

> Similarly, resolution by IP address does not provide privacy or integrity
> unless you trust the network link.

That's a fair point. It's certainly no worse than our current situation where
you have to trust a self signed certificate on a case-by-case basis. The
router could forge a self signed certificate even more easily than it could
forge a certificate signed by a trusted CA.

~~~
nybble41
> The important part when communicating directly with a device by IP address,
> is that the payload is encrypted end to end.

Which you can't be certain of unless you can verify where the "end" actually
is. The IP address is no help here; the ARP cache can't be trusted. Packets
can be redirected to any host on the local network regardless of the official
IP address assignments. If you aren't authenticating the other end of the link
then you can't be sure no one is snooping. Active MitM attacks on local
networks are trivial. You don't even need to compromise the router—any peer on
the network will do.

> The router could forge a self signed certificate even more easily than it
> could forge a certificate signed by a trusted CA.

Sure. As a form of TLS without authentication, one-time-use self-signed
certificates would be worse than useless, offering only a false sense of
security. The options here are "trust on first use" (betting against a
malicious actor being present during initial setup) or offering some way to
manually verify the fingerprint of the device's private key. Both approaches
require authentication, not just encryption. For the second, more secure
option, the cheapest way would be to permanently provision each device with a
unique private key and print the key's fingerprint on the device's exterior.
For more flexibility, store the key in a small removable HSM (similar to a SIM
card) and print the fingerprint on the HSM itself and/or the accompanying
documentation.

------
mschuster91
The HTTPS cert is used for the router's login page - apparently putting an IP
address on the box backside label confuses too many people. It makes sense to
put it behind HTTPS because the browser _will_ whine "this page is
insecure"... but how is a router vendor supposed to include the neccessary
certificate that won't get leaked?

The only thing I can imagine here is a dedicated HSM chip... but that's
overkill for a 10$ router?

~~~
vbezhenar
1\. Just let it be HTTP. Stupid browsers are stupid, but at least they don't
prevent this page from working yet.

2\. Router coordinates with company server to get its own hostname like
n-123123123.netgear.com (which points to 192.168.1.1 or whatever), generates
private key and company server issues certificate for that key. HTTP requests
to 192.168.1.1 return HTTP 302 to this address. It requires Netgear to operate
CA or make some extended agreement with existing CA to let them issue
certificates. I know that Plex does something similar, so it should be
possible.

3\. Company server creates hostname which points to a public router IP address
and router uses letsencrypt to get a certificate for that hostname. But that
requires public IP and some providers are using NAT nowadays, so it won't work
universally.

4\. To configure a device you're talking with company server, rather than your
device and your device is talking with that server as well. It requires
working Internet, so not a complete solution as well.

I don't believe that HSM chip would work. You would just extract that chip and
use it as a private key and that's about it. You won't be able to extract
private key bits, but you don't need them. May be it might work with very
secure device like iPhone, where everything is signed, but yeah, $10 device
and someone will still hack it, causing bad situation.

~~~
berti
All but one make for an undesirable UX during initial setup before WAN is up,
which is probably the only time most users login to their router.

~~~
vbezhenar
Yeah, that's why I think that movement to HTTPS everywhere must at least
exclude private IP addresses. While it's expected to have HTTPS on public
resources, what happens in my local network is my business and should not be
considered insecure.

~~~
ajsnigrutin
The problem is, that someone setting up a public wifi (in a restaurant for
example), will be vulnerable to sniffing attacks (if they don't know what
they're doing).

~~~
franga2000
So get a cert before setting up wifi. This is not a hard problem. 1\. Connect
to AP with random key from the box 2\. Open admin panel over HTTP 3\. Follow
instructions to get Internet access 4\. As soon as it has a connection, the
router obtains a cert and redirects you to HTTPS 5\. Now you can make all your
poor security decisions

~~~
Terretta
Have you _met_ ‘normals’?

Everything is a hard problem if it’s any steps at all.

~~~
franga2000
I meant that "how to make router setup secure without making it more
difficult" is not a hard problem. My system is no more difficult than the
currently most common ones and about as secure as it gets.

Although, one could argue that setting up a router really shouldn't be left to
people who don't understand how they work...

------
regecks
6 days is nowhere near a justifiable timeframe for full disclosure.

Even if you disagree with that, you should have first reported Key Compromises
to Entrust and Comodo before publicly posting the private keys. They are bound
by BRs and their own CPS to revoke certificates such as this one - and they
would have done so promptly.

This is not what you should do as a security researcher - delete the gist
until the CAs have a chance to revoke it via OCSP.

~~~
mjevans
I think in this case it's to force browser vendors (who have the most
exploitable endpoints) and companies like Apple and Microsoft at the OS level,
to blacklist the offending certs.

Though the CAs in question should probably have an automated challenge-
response system to which a timed signed reply of a given message of their
choice causes a revocation of the key that signed the message. (As sufficient
proof of "this is vuln, kill it now, ask questions after".)

~~~
buraksarica
The OS level actions won't be fast. So this disclosure is closer to an
irresponsible attitude. At least CAs should be informed.

~~~
philprx
I tend to differ:

Netgear security is paid to manage security. They failed by not responding to
these legitimate communications requests.

The researcher are not paid. They did what could be done to _really_ fix the
problem. Of course you can always do better as a researcher, always, but
consider time available and paid vs. pro bono time. Also consider all the
people who probably found this before and may have sold it on black market,
you’re attacking the wrong people.

------
rodnim
I don't envy Netgear. They sell home routers to people who hardly know the
difference between HTTP and HTTPS. In this regard, I respect them for NOT
making those people submit sensitive information by ignoring browser security
warnings and/or accepting self signed certificates. I.e, not teaching bad
habits.

On the other hand, making private keys publicly available is obviously far
from ideal.

Damned if you do, damned if you don't. Again, I don't envy Netgear...

------
actionowl
HTTP(S) might just be the wrong protocol for initial setup of a router!

Maybe instead generate a password for each router and print it on the box
along with the device's unique SSH Key fingerprint.

Technical users can then SSH in and bootstrap the system (also generate/upload
their own TLS certs if they want to use a browser and connect over HTTPS,
etc).

Non-technical users get an app and they can scan a QR code on the box which
has the SSH user, password, and fingerprint for verification.

The App connects to the router over SSH to do router setup.

If an App is involved you wouldn't technically NEED to use SSH with the App
but it was the first thing that came to mind.

~~~
xg15
> _Non-technical users get an app ..._

So now I need an already-working smartphone to set up my internet connection.

~~~
actionowl
That's a good point actually. You'd need the internet to install the app too.

------
tgsovlerkhgsel
The cert for www.routerlogin.net (Serial
c1:a1:00:64:07:61:2c:07:00:00:00:00:50:f1:09:6a) isn't revoked yet:
[https://crt.sh/?id=1955992027&opt=ocsp](https://crt.sh/?id=1955992027&opt=ocsp)

The reporters should have asked the CAs to revoke the certificate. If the CA
doesn't do that within 24 hours, it's considered a violation of the rules that
the CA needs to follow to stay in browsers.

~~~
dlgeek
Anyone with access to the private key can submit a problem report to the CA,
they're obligated to revoke if the key is exposed to a non-subscriber.

I have submitted a report - they are obligated to revoke within 24 hours.

~~~
cjbprime
Looks like they've failed to perform the revocation in time -- other users
reported to Entrust >24h ago.

~~~
dlgeek
They revoked it 7 minutes shy of the 24 hour deadline after my report.

If other users reported it and they failed to act, that's a BR violation and
they're required to provide an incident reports to the root programs via
Mozilla's mechanisms.

If they don't, and you have evidence of those reports, you should provide them
to the root programs via the mozilla.dev.security.policy mailing list/group.
There's already a thread about this issue, though not claiming that EnTrust
was notified.

(Based on timestamps, it's also possible they revoked in response to that
thread).

~~~
cjbprime
Thanks. Here is the timestamp showing that it took more than 32 hours from
reporting to revocation.

[https://twitter.com/FiloSottile/status/1219147543667453953?s...](https://twitter.com/FiloSottile/status/1219147543667453953?s=19)

------
fulafel
Can anyone make out what the funjsq.com is about?

~~~
dylz
Chinese gaming VPN service or something, bypasses China IP blocks to allow
them to play on NA/EU servers.

~~~
bflesch
I'm wondering why such a VPN service is included in the netgear firmware
image? Wouldn't this resonate negatively with Chinese authorities?

~~~
lenova
Unless it was mandated/deployed by a state actor, looking to MITM/monitor VPN
users within the country?

------
mkeedlinger
This reminds me of something I have been looking for for a long time. If you
know any reasonable way to fix it please let me know!

This "any CA can authenticate any domain" system is ridiculous. I manage my
own CA, which is fine for my own self-hosted sites, but the issue is that it
doesn't protect me from a cert made valid by some other CA.

Is there any way I can whitelist my own CA such that IT ALONE will work for my
domains? (note: this is a me-only thing. Don't need anyone to be able to
validate these domains).

PS, I've asked this question elsewhere [0] before. I did not find an adequate
answer.

[0]: [https://security.stackexchange.com/questions/211401/can-
you-...](https://security.stackexchange.com/questions/211401/can-you-
whitelist-a-single-ca-for-a-domain)

~~~
zzzcpan
See DNS CAA
[https://en.wikipedia.org/wiki/DNS_Certification_Authority_Au...](https://en.wikipedia.org/wiki/DNS_Certification_Authority_Authorization)

~~~
mkeedlinger
Thanks for the reply!

This is very interesting, I will surely take a look. Only issue I see is that
it seems to use DNS, and that's not exactly secure either.

In my mind I am thinking of something that would be more theoretically secure
from any remote attack (closer to a form of key-pinning maybe?)

------
zyztem
This is a world after QUANTUMINSERT for you. Somehow everything should be
encrypted but key management are still unsolved

------
josteink
I guess I run a different firmware for my Netgear ReadyNAS 312.

This is easy to inspect because the FW is Debian-based, and enabling inbound
SSH is easy.

Checking on mine, it has a cert only for the "nas.local"-domain, and the cert
is stored in /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-
cert-snakeoil.key respectively.

Have to admit I love the naming of those files :)

Edit: Obviously it is a different firmware. Mine is a NAS, this was a router.
And “snakeoil” seems to be default debianism.

------
xg15
> _These certificates are trusted by browsers on all platforms, but will
> surely be added to revocation lists shortly._

So what about all the routers that were already sold and the customers that
bought them?

Are we ok with leaving all devices that are already in operation or that are
currently being sold unconfigurable for non-technical users, just to protect
against a highly hypothetical scenario of abusing routerlogin.net?

------
privateSFacct
What’s a good netgear alternative for switches from 5 port to 48 port - too
late for us now but for next buying wave

------
gene91
Is this cert revoked yet?

Sure, Entrust is required to revoke it. It is bound by BRs. If it refuses,
Entrust will get itself blacklisted by browsers. On the other hand, having the
cert revoked will cost Netgear dearly. As a result, I wonder if Entrust might
be dragging its feet on the revocation.

~~~
isclever
mini-app.funjsq.com is revoked
([https://decoder.link/result/418b8d20793d3f4daa4153752e45e78b...](https://decoder.link/result/418b8d20793d3f4daa4153752e45e78b87e8e293)),
can't check routerlogin.net/com as they didn't paste the public cert.

~~~
tgsovlerkhgsel
routerlogin is not revoked:
[https://crt.sh/?id=1955992027&opt=ocsp](https://crt.sh/?id=1955992027&opt=ocsp)

------
bilekas
Name and Shame.

Responsible disclosure ? How about reposonsible security pracitces first.

------
tapper82
This is why I use OpenWrt. Shout out to the dude sticking up for screen reader
users to! Thanks friend. Shit like footnotes and captchirs are the bane of my
life!

------
josteink
To all the people shitting on Netgear and security in this thread, just how
_do_ you propose one deliver a secure network appliance to end customers which
they can deploy on their network? And which is user-accessible to common users
in modern browsers rejecting everything not touched by a proper CA?

Really. Please educate the world with your ingenious insight. I’ll be waiting.

The unavoidable truth is: You _have_ to stuff the key in there somehow.
There’s no way around that.

Either browsers have to start accepting “local” certs more easily, or end-user
network deployed appliances can’t have ssl.

Take a pick.

~~~
oarsinsync
This was answered on the Gist in a comment[0] linking to a tweet[1]:

Things to do instead of shipping TLS certs and static private keys to
consumer-grade routers you sell by the thousand:

Generate a unique keypair per device.

Use this keypair to communicate upstream, in a similar fashion to CloudFlare's
Keyless SSL.

This keypair you generate on the device would need to be preloaded at the
factory, unique per device, and the public key would need to be stored in a
database.

When a TLS handshake comes in, you verify the entire request is signed by that
keypair before signing it.

Delegated online signing for a TLS handshake isn't even in the top 10
engineering challenges for cryptography in the 2020s.

Oh, and, if you don't have connectivity to upstream and your need users to
connect to configure something?

Fallback to HTTP reachable via port 80 on the private IP for the server.

[0]
[https://gist.github.com/nstarke/a611a19aab433555e91c656fe1f0...](https://gist.github.com/nstarke/a611a19aab433555e91c656fe1f030a9#gistcomment-3145485)

[1]
[https://twitter.com/CiPHPerCoder/status/1219228544548720640](https://twitter.com/CiPHPerCoder/status/1219228544548720640)

~~~
lima
An attacker can just dump the unique key, impersonate the device and use that
for a MitM attack. There's just no way to do this securely without completely
locking down the devices using hardware key management (which would be
unreasonably expensive for a cheap router, plus bad for people who want to
flash their own firmware).

Same level of security, and a lot of extra complexity and cost.

~~~
oarsinsync
Yep, and that compromises that one device in the field. Not every device
globally, today and tomorrow.

~~~
lima
Only if each device has a unique subdomain.

------
zonidjan
This is intentional on Netgear's part, and does not in any way degrade
security compared to the alternative (an untrusted cert or no HTTPS). It is
neither a bug nor a security issue.

