Please stop using that phrase. "Responsible disclosure". It's wrong and harmful, and the person who coined it agrees with me: https://adamcaudill.com/2015/11/19/responsible-disclosure-is...
You want "coordinated disclosure" instead.
Netgear doesn't do coordinated disclosure. They do non-disclosure. In the absence of a coordinated effort, full disclosure is the responsible thing to do.
Exposing something already public to speed up the resolution and make sure the impact is kept to a minimum considering the circumstances.
I like how Google Project Zero does it: give time, but don't extend indefinitely.
It sure is easy to tell others what to do with their work product when you have zero stake in the game.
There exists a really easy solution to the purported problem of full disclosure, vendors could just offer significant enough financial compensation for non-disclosure.
1. Netgear buys the bug.
2. They keep shipping firmware with a private key.
3. Someone with malicious intent finds it.
4. Someone with malicious intent MITMs a Netgear network and does something bad.
5. Bad actor sells bug online.
6. Millions of Netgear devices continue operating with compromised certs.
Instead, here's what happened/happens now...
1. Researcher tries to coordinate with the vendor, realizes NDAs are involved and the vendor intends to keep shipping their private key.
2. Researcher dumps details of private key on GH instead.
3. Mozilla and Google stop trusting the cert.
4. Now Netgear no longer gets to decide if they want to fix the problem. Browser vendors have fixed it for them.
5. Millions of otherwise insecure Netgear devices are protected by the browser instead.
6. Netgear rightfully gets bad press and hopefully the shaming makes them rethink their positions on NDAs in their bug-bounty program.
7. Future bug hunters submit bugs straight to a receptive Netgear, who has learned a valuable lesson about being a lazy hack.
There aren’t many people out there hunting bugs just to be nice.
Is this his best work? Nah, this is amateur hour on the part of Netgear. But am I glad it was him who found it? Definitely.
Keep in mind, there really wasn't anything _to_ this vulnerability other than copying and pasting from one website (and a firmware zip) to another. They just added a little pretty formating and saved you the trouble of extracting the firmware.
I certainly don’t think that there’s anything wrong with dropping bugs to pad your résumé. To the contrary, I think it’s fundamentally unreasonable to expect that more than a couple of people would do this work for purely selfless reasons.
How does non-disclosure benefit them?
And how does dumping the vulnerability without a fix help Netgear owners? They're all flapping in the wind right now.
IMHO, the best scenario is coordinated disclosure. Full disclosure may be necessary to force a vendor to do something, but let us not pretend it is a good thing.
It lets us know to buy another brand immediately and never buy Netgear again.
> IMHO, the best scenario is coordinated disclosure
The original post said that Netgear doesn't do coordinated disclosure, and subsequent posts were arguing whether non-disclosure or full disclosure were better. Nobody was disagreeing with what you said.
I don't think "benefiting the end users" is even present in that equation.
And besides, there’s nothing stopping the millions of end users from offering their own bribes in order to get shit fixed. $10 from each would go very far.
Personally, I haven't liked most Netgear hardware I've tried... but since the FCC cracked down on open firmware for consumer gear, options are really limited with most vendors.
Ah yes, that certainly sufficiently compensates independent security research.
Your complaints aren’t worth shit unless you’re willing to pay up.
I could give some leeway to a grey hat who finds the data but doesn't understand what it is and posts it. "Hey, what's this blob of data?". But this poster clearly understands it's a private key for a certificate in the Web PKI.
You can disclose the fact that you know a private key that isn't yours without revealing the key by various methods, but one that's very easy for non-experts is to create a bogus CSR. Just tell OpenSSL (or a tool that's actually good) that you have this private key and you want to request a certificate for it. Give bogus details, for example in the Common Name of the proposed certificate you can explain this is a Netgear key you found.
You can now publish the CSR, it is inherently proof that you've got Netgear's private key but it does not contain that key and so black hats will need to do their own work, for which you are certainly not responsible, to get that key from a Netgear box if they want to.
It would also make sense to tell the issuing CA, you should send them that CSR, although it's not terribly harmful to send them the actual private key and I guess if you're worried the CA's representatives don't "get it" this is a very blunt way to make your point. If the issuing CA doens't respond, tell m.d.s.policy both that you found this key and that the CA did not respond, and if necessary that can be escalated until the CA is distrusted (by Mozilla, and in my experience eventually everybody for reasons we'll not think too hard about here).
In the case of Let's Encrypt the process is, like everything else, fully automated. Call the API (by running your client software of choice) and prove you know the private key for a certificate they issued and want it revoked, its status changes to revoked and the next batch of OCSP signatures will show revoked for that certificate.
It wasn't like they had to hack a router, get root, and exfil the keys. All of this data was already made public, by Netgear! They just put together a document with prettier forms for everyone else to see.
Hell, they probably could've given you a shell one liner to grab it from Netgear and extract the keys yourself. :-)
Edit: and the Comodo-issued cert is already revoked. I'm too lazy to pull the other cert but I'd bet that it's revoked too.
I recently bought a couple of Netgear Managed Switches (for Business)⁰ and in their datasheet they list "Local-only management" as a feature. Only after they arrived we discovered that you only get limited functionality in the Local-only management mode, you have to register the switches to your Netgear Cloud account to get access to the full functionality.
Reading up on it, this was achieved only after a community outcry because in the prior firmware versions the switch would have to connect to the Netgear Cloud on every bootup.
Needless to say I would not have bought the swiches if I had knew I needed to register them to Netgear Cloud to have access to the full functionality specified in the data sheet. If I had bought them as a consumer, not as a business, I would have returned them immediately.
Netgear are now on our purchasing blacklist.
⓪ - the switches are Netgear GS-108Tv3
>⓪ - the switches are Netgear GS-108Tv3
are we not allowed to use any sane method of doing footnotes? Who does this? Why would you even use footnotes in internet comments other than to specifically screw over anyone who uses a screen reader?
There is no markup on HN for footnotes. https://news.ycombinator.com/formatdoc
> Who does this?
Blame the people running the site, this is just a small inconvenience compared with other lack of features.
> screw over anyone who uses a screen reader?
As GP's footnote is plain text – not markup – there is no problem with screen readers (or any user-agent, really). Curious why you would think otherwise.
I've got an R7800 running as the main router in my home, however, it's flashed with OpenWRT as their own software is a complete mess.
Only because current browsers makes it increasingly hard to run medium security, user-trusted networking.
Tuesday, January 14th 2020 - Initial Discovery
Tuesday, January 14 2020 - Tweet sent attempting to establish communications with Netgear
Wednesday, January 15 2020 - Reached out to Bugcrowd to attempt to establish communications.
Thursday, January 16 - Bugcrowd responds, but we are unable to establish a communications channel outside of the Netgear bug bounty programs.
Friday, Jaunary 17th - Conversation with bugcrowd proves inconclusive
Sunday, January 19th - Feeling we have exhausted our disclosure avenues, we decide to publish
They explained earlier that if they entered the bug bounty program with Netgear, they would be agreeing to not disclose: "By submitting the security bug, you affirm that you have not disclosed and agree that you will not disclose the security bug to anyone other than NETGEAR."
So it seems they were trying to get Bugcrowd as a intermediary to contact Netgear on their behalf to "establish a communications channel outside of the Netgear bug bounty programs"
Using plain http isn't a good solution because it involves telling the user to ignore the "not secure" warnings by browsers. This seems like the best available solution.
Without network connectivity, the traffic can not be tunneled through a remote server with proper cert. Nor can the box request a cert in realtime at the time of setup.
Pre-provisioned cert won't work if the box is purchased more than 27 months after manufacturing (the current max validity period). For home network equipment, I suppose this isn't unusual.
I see TLS-SRP mentioned in comments. But the commenter hinted at its limited adoption in browsers.
I suspect at done point the router manufacturers will give up and just force setup through a phone app (assuming you could download it over a cellular connection). That makes me sad, though.
At a higher level, TLS provides integrity and privacy when the network linked is not trustworthy. As a result, I'm not sure what your proposal achieves.
Similarly, resolution by IP address does not provide privacy or integrity unless you trust the network link.
> Similarly, resolution by IP address does not provide privacy or integrity unless you trust the network link.
That's a fair point. It's certainly no worse than our current situation where you have to trust a self signed certificate on a case-by-case basis. The router could forge a self signed certificate even more easily than it could forge a certificate signed by a trusted CA.
Which you can't be certain of unless you can verify where the "end" actually is. The IP address is no help here; the ARP cache can't be trusted. Packets can be redirected to any host on the local network regardless of the official IP address assignments. If you aren't authenticating the other end of the link then you can't be sure no one is snooping. Active MitM attacks on local networks are trivial. You don't even need to compromise the router—any peer on the network will do.
> The router could forge a self signed certificate even more easily than it could forge a certificate signed by a trusted CA.
Sure. As a form of TLS without authentication, one-time-use self-signed certificates would be worse than useless, offering only a false sense of security. The options here are "trust on first use" (betting against a malicious actor being present during initial setup) or offering some way to manually verify the fingerprint of the device's private key. Both approaches require authentication, not just encryption. For the second, more secure option, the cheapest way would be to permanently provision each device with a unique private key and print the key's fingerprint on the device's exterior. For more flexibility, store the key in a small removable HSM (similar to a SIM card) and print the fingerprint on the HSM itself and/or the accompanying documentation.
The only thing I can imagine here is a dedicated HSM chip... but that's overkill for a 10$ router?
2. Router coordinates with company server to get its own hostname like n-123123123.netgear.com (which points to 192.168.1.1 or whatever), generates private key and company server issues certificate for that key. HTTP requests to 192.168.1.1 return HTTP 302 to this address. It requires Netgear to operate CA or make some extended agreement with existing CA to let them issue certificates. I know that Plex does something similar, so it should be possible.
3. Company server creates hostname which points to a public router IP address and router uses letsencrypt to get a certificate for that hostname. But that requires public IP and some providers are using NAT nowadays, so it won't work universally.
4. To configure a device you're talking with company server, rather than your device and your device is talking with that server as well. It requires working Internet, so not a complete solution as well.
I don't believe that HSM chip would work. You would just extract that chip and use it as a private key and that's about it. You won't be able to extract private key bits, but you don't need them. May be it might work with very secure device like iPhone, where everything is signed, but yeah, $10 device and someone will still hack it, causing bad situation.
Everything is a hard problem if it’s any steps at all.
Although, one could argue that setting up a router really shouldn't be left to people who don't understand how they work...
The risk with #2 is then an attacker could potentially probe the firmware to figure out that process, and phish from that domain? The other is we need to trust Netgear to run HA infrastructure to support that. (lol)
Facebook, for example, has a deal with their preferred CA which says that all certificates for names under fb.com and facebook.com are only to be issued with authorization from Facebook's central security team.
So even if you can fulfil that CA's normal checking perfectly for fake-bank.facebook.com, if you haven't got someone on the inside of the Facebook security team you can't get a working certificate for your fake bank social media scam.
This would be categorically worse than hardcoding the SSL key.
Plex do it with a wildcard and generate a new one for each server: https://blog.filippo.io/how-plex-is-doing-https-for-all-its-...
1) An extension to PKI that browsers support. If you browse a service with a certain type of cert on a certain TLD, a) it won't work except on private networks, b) it will prompt the user to enter a key, such as a key printed on the backside of a router (this is already used to get consumers onto WPA routers, so we know it's reasonable). The device with the cert creates the cert itself. If the key is right, you can browse the service, and your browser caches the key with the cert. If an attacker learns the key they can try to make a new fake cert, but the browser would know it's not the same as the old cert and block it. Replacing these should be quite infrequent, as often as people replace their home network devices.
2) A TLD that browsers will only ever respond to if its names resolve to private address ranges. If a service wants to serve HTTPS privately, it will request a cert be signed by a device that is auto-detected on the local network. The device will return signed certs for a particular host on the local-only TLD. The device can have any logic it wants to restrict what can get a cert and how. The trick is how to make the client trust the device, which I think 1) would solve. This device can, for example, only ever create a given FQDN's cert once; if an attacker tries to generate a duplicate, it will fail. If the cert ever needs to be refreshed, the memory/flash of the device needs to be reset. This makes it possible to have a captive portal request a cert from a local router, and no attacker can generate an identical cert unless the router was reset and they triggered the cert generation before the captive portal did. (Also for captive portals, the admin of the router could just hard-code what clients could request what certs, so the captive portal could always request particular local certs for itself, while no attacker could). It's still tricky to determine trust on first connection in the case of things like coffee shops, but maybe you could receive a public key via WPA or something (then again that's kinda kicking the can down the road to WPA)
Not confusing users is why the fiasco with the https certificate problem here was initially created. And to be honest I do not have any idea how all needs (users have a trusted HTTPS certificate, a domain name and no "insecure" oe other warnings in the browser, while hackers don't get access to the certificate, and all of it works without internet uplink) can be met...
Constructing a scheme where NSA is an active agent in the threat model was not an original requirement :)
You are welcome to introduce any way to produce any part of a router or a PC for that matter that would protect from NSA, it seems that the biggest players in the field are still working out and it is very much a work in progress. When you have an adversary that is able to intercept hardware in transit and spend endless amounts of dollars on devising clever hacks or undetectable hardware exploits, then yes, you're right, some TLS scheme, regardless of where the certs are, is not going to be enough.
but this requires internet access so barely a solution.
I don't think that the fact that private keys for routerlogin.net are bundled with the router are an issue.
It's very logical to do so, and BETTER than plain HTTP in most real-life scenarios.
routerlogin.net is a local server (when you are using the router) and not a remote website so it's expected that you can trust the content as much as trustable your local network is.
The fallback would have to be self-signed TLS with random keys for initial local router configuration, which is hopefully done over an Ethernet connection rather than WiFi.
You don't need a secure enclave or HSM to solve this problem. It just requires caring about security and provisioning enough time to engineer and test the obvious solution. From what I can tell, that doesn't line up with Netgear's MO.
It prevents a bored kid who dumps the private key from one router from being able to reliably impersonate thousands of consumers' routers, the world over.
That's a meaningful security benefit.
The idea is that each router would get a certificate signed with a distinct subdomain unique to that router. So you still can't impersonate other devices. It wouldn't be a generic domain or wildcard certificate.
The remote signer would furthermore suspend suspicious connections (i.e. from multiple public IP addresses).
Even if you disagree with that, you should have first reported Key Compromises to Entrust and Comodo before publicly posting the private keys. They are bound by BRs and their own CPS to revoke certificates such as this one - and they would have done so promptly.
This is not what you should do as a security researcher - delete the gist until the CAs have a chance to revoke it via OCSP.
Of course it is, if you didn't get a response in the first 48 hours you're not going to, that's the way things like this tend to operate. The security contact for all companies is either a) inactive or b) very active. It's a detriment to everybody that for some reason it has been normalized that people should wait months or years for disclosing bugs. Full disclosure is the only way things get fixed.
> This is not what you should do as a security researcher - delete the gist until the CAs have a chance to revoke it via OCSP.
Github is archived in real time for the most part. Especially you'll notice that if you post a private key for a cryptocurrency address or credentials for AWS, it'll be stolen and used within seconds. There's some really good sets of information out there like https://www.gharchive.org/ which give you an idea of the sheet amount of data that github produces on a daily basis, and that's just the metadata and things like comments rather than actual git repository contents.
The idea that you could delete something from there and have it actually "removed from the internet" is amusing.
Though the CAs in question should probably have an automated challenge-response system to which a timed signed reply of a given message of their choice causes a revocation of the key that signed the message. (As sufficient proof of "this is vuln, kill it now, ask questions after".)
Netgear security is paid to manage security. They failed by not responding to these legitimate communications requests.
The researcher are not paid. They did what could be done to really fix the problem. Of course you can always do better as a researcher, always, but consider time available and paid vs. pro bono time. Also consider all the people who probably found this before and may have sold it on black market, you’re attacking the wrong people.
They were reached out to and didn’t respond. It’s their own fault and they have to make do with what they’ve been granted. Now they get to pick up the pieces.
My guess is that people before the ones in the gist have tried to contact Netgear about this (since it's not hard to find) and chose to ignore it.
I’m curious to what does this mean? They got a response form Bugcrowd but not Netgear themselves? Or they got through to Netgear but didn’t like what Netgear said so they went public? Something else?
Now it's an open race with the bad guys, and surely a lot of them all at once; hardly an advantageous scenario for end users.
On the other hand, making private keys publicly available is obviously far from ideal.
Damned if you do, damned if you don't. Again, I don't envy Netgear...
Maybe instead generate a password for each router and print it on the box along with the device's unique SSH Key fingerprint.
Technical users can then SSH in and bootstrap the system (also generate/upload their own TLS certs if they want to use a browser and connect over HTTPS, etc).
Non-technical users get an app and they can scan a QR code on the box which has the SSH user, password, and fingerprint for verification.
The App connects to the router over SSH to do router setup.
If an App is involved you wouldn't technically NEED to use SSH with the App but it was the first thing that came to mind.
Ship a USB stick with the device with installer software on it.
The installer software sets up an SSH tunnel to the device, so it includes an SSH client and a script (batch) file that effectively does:
SSH -Llocalhost:<some random local port>:localhost:<installer initial port> setup@<ip address of device>
It has the known SSH signature of the device in its known_hosts, and of course the user has to type the password from the sticker on the device.
Then it fires up the browser with the url http://localhost:<local port used in previous step> which has access to the full setup of the device via the browser. It is HTTP, therefore there are no initial setup TLS issues with the browser, as you are going via an encrypted tunnel set up via SSH using HTTP encapsulated within that encrypted tunnel (effectively a temporary VPN) rather than using TLS.
As part of the setup process, the device generates a self-signed cert, which you can put in your computers/browsers trusted cert store for future use. Also as part of the setup, once the self-signed keys and certs are generated, it disables this local HTTP webserver that is being forwarded to via SSH, so in future the user can use a standard TLS connection to the device using the generated certificates. If a factory reset of the device is done, it blats its config and re-enables the 'setup' webserver (that is only running on localhost therefore can only be access locally,i.e. via SSH tunneling).
So now I need an already-working smartphone to set up my internet connection.
If a certificate is valid beyond 825 days Chrome will not honor it. So if you generate a key and cert at the factory for every device you run into the issue of the end-user getting a device with an expired cert.
If you generate a unique key and self-signed cert users will complain and call support about the "insecure" site warning.
Communicating upstream to a third party to get a cert won't work for all users (e.g. those that need to configure a static IP from their ISP.)
The reporters should have asked the CAs to revoke the certificate. If the CA doesn't do that within 24 hours, it's considered a violation of the rules that the CA needs to follow to stay in browsers.
I have submitted a report - they are obligated to revoke within 24 hours.
If other users reported it and they failed to act, that's a BR violation and they're required to provide an incident reports to the root programs via Mozilla's mechanisms.
If they don't, and you have evidence of those reports, you should provide them to the root programs via the mozilla.dev.security.policy mailing list/group. There's already a thread about this issue, though not claiming that EnTrust was notified.
(Based on timestamps, it's also possible they revoked in response to that thread).
This "any CA can authenticate any domain" system is ridiculous. I manage my own CA, which is fine for my own self-hosted sites, but the issue is that it doesn't protect me from a cert made valid by some other CA.
Is there any way I can whitelist my own CA such that IT ALONE will work for my domains? (note: this is a me-only thing. Don't need anyone to be able to validate these domains).
PS, I've asked this question elsewhere  before. I did not find an adequate answer.
This is very interesting, I will surely take a look. Only issue I see is that it seems to use DNS, and that's not exactly secure either.
In my mind I am thinking of something that would be more theoretically secure from any remote attack (closer to a form of key-pinning maybe?)
This is easy to inspect because the FW is Debian-based, and enabling inbound SSH is easy.
Checking on mine, it has a cert only for the "nas.local"-domain, and the cert is stored in /etc/ssl/certs/ssl-cert-snakeoil.pem and /etc/ssl/private/ssl-cert-snakeoil.key respectively.
Have to admit I love the naming of those files :)
Edit: Obviously it is a different firmware. Mine is a NAS, this was a router. And “snakeoil” seems to be default debianism.
So what about all the routers that were already sold and the customers that bought them?
Are we ok with leaving all devices that are already in operation or that are currently being sold unconfigurable for non-technical users, just to protect against a highly hypothetical scenario of abusing routerlogin.net?
Sure, Entrust is required to revoke it. It is bound by BRs. If it refuses, Entrust will get itself blacklisted by browsers. On the other hand, having the cert revoked will cost Netgear dearly. As a result, I wonder if Entrust might be dragging its feet on the revocation.
Responsible disclosure ?
How about reposonsible security pracitces first.
Really. Please educate the world with your ingenious insight. I’ll be waiting.
The unavoidable truth is: You have to stuff the key in there somehow. There’s no way around that.
Either browsers have to start accepting “local” certs more easily, or end-user network deployed appliances can’t have ssl.
Take a pick.
It is definitely a hard problem because there's no easy way to authenticate the router to the client, but on the other hand I'm not sure how important all of this is when it's on a local link anyway.
A "best possible practice" solution would be to have the routers issued with individual certificates at factory programming time, and provide a rollover mechanism through the admin UI.
Certificate Authorities use the difference between 825 days and two years to offer certificate renewal weeks prior to the expiry date while keeping your "extra" days. e.g. your certificate expires 17 February 2020 but you renew today for two years, they can issue a cert which expires 17 February 2022 because that's less than 825 days in the future, if the rule was a hard two years they couldn't do that.
As to how you'd fix this: One option is firmware updates. After all a device which goes two years without firmware updates is also not in good shape, so you could arrange with a CA to bake certificate renewal into the firmware update process. At scale it would totally make sense for a CA to offer to issue renewals at say 10¢ per device renewal for up to ten years from inception date.
The certificate issuance does not require knowledge of the private key, that stays on each individual device, renewal just issues a newer cert periodically to the same device for use with its existing key.
That sounds like the worst of both worlds.
- Self signed certificates. Users will learn to acknowledge the error and an attacker can just present their own self-signed cert. This what most vendors do.
- Plain HTTP. Worst option, no security, not even against a passive listener (but no bad PR because of "security disclosures"... hooray!).
- Something similar to Keyless SSL, as suggested on GitHub and in this thread. Will at best slightly inconvenience the attacker, who can still extract any device key and impersonate the device. It also introduces a single point of failure. Server is down, nobody can log into their devices (and HN would not be happy).
- Secure enclaves and hardware key management for the shared key. This would be workable and reasonably secure, but the implementation costs likely won't fit the threat model.
- Issue an individual SSL cert for each domain and print the domain on the box. That would work, but be very complex to manage (plus certificate costs, and plenty of failure cases around renewal).
- Reverse proxy through vendor servers. Single point of failure, and the most likely time to log into your router is the internet is down.
Hardcoding a valid SSL cert for a single-purpose domain (routerlogin.net) is similar to a self-signed cert in terms of security, provides a better user experience and does not teach them to blindly acknowledge TLS errors. The threat model requires an attacker doing a MitM attack on the local link. Also, compromising a router provides very little access to an attacker in today's TLS-by-default world.
Anyone downvoting the parent post should offer a better solution.
I have to agree with the calls for allowing warning-less HTTP for LAN connections.
Edit: Add last sentence.
Completely flawed assumption, see the following for details:
Things to do instead of shipping TLS certs and static private keys to consumer-grade routers you sell by the thousand:
Generate a unique keypair per device.
Use this keypair to communicate upstream, in a similar fashion to CloudFlare's Keyless SSL.
This keypair you generate on the device would need to be preloaded at the factory, unique per device, and the public key would need to be stored in a database.
When a TLS handshake comes in, you verify the entire request is signed by that keypair before signing it.
Delegated online signing for a TLS handshake isn't even in the top 10 engineering challenges for cryptography in the 2020s.
Oh, and, if you don't have connectivity to upstream and your need users to connect to configure something?
Fallback to HTTP reachable via port 80 on the private IP for the server.
Same level of security, and a lot of extra complexity and cost.
How, exactly, is that better than using a common cert? What threat do you propose that applies to HTTPS with a shared cert, and not to HTTP?