Wow, I normally am all about how google handles security issues (they get dragged through the mud for some Project Zero stuff), but this def did not get handled well.
Super unclear communication, starting with "you're just using it wrong", more than six month turn around, and even then at the end no clear explanation of what went wrong with someone who was collaborating with you? That's amateur hour security.
I'd pay to see Bryan Cantrill's reaction [0][1] to this: A seemingly mysterious firmware bug of a secure element / trusted-execution-environment but there's no knowing if there are more bugs (or, shudder backdoors).
Since the source code isn't available for scrutiny (though Google has promised firmware transparency [2]), it is kind of difficult to tell what really went wrong in the current reported case and what else possibly could go wrong given the use-cases for it are far-reaching and sensitive: Google has advocated StrongBox as a trustable companion that could be used to attest user actions on medical devices [3], for instance; or for use as an Identity verificafion for documents such as Driving Licenses and Passports.
My reaction might not be worth the price of admission, because it's exactly as you'd expect: this is great, diligent, responsible work -- and if the Titan M firmware had been open, it seems likely that the author would have been able to get to root cause (or much closer to it). This in turn would have in turned tightened Google's response time, led to a better fix faster, etc.
That said, there are several reasons for optimism.
1. OpenTitan.[1] On the one hand, this is not about opening up extant Titan implementations so much as developing next-gen Titan in the open -- but it is nonetheless a laudable and important effort and it is increasingly reasonable to expect that the hardware roots-of-trust of the future will be entirely open.
2. Open firmware more generally. The Open Source Firmware Conference[2] this past fall was truly inspiring in terms of the broad interest from the industry: while there is much work to be done, there is more reason than ever to believe that it's attainable.
3. Rust. It's hard to speculate without knowing what the root cause of this issue actually was, but to the degree that memory corruption was at root here, the emergence of Rust for firmware is an incredibly important development. Speaking personally, if there was any doubt in my own mind about the appropriateness of Rust at this lowest level of software, it has been erased by our own experiences at Oxide over the past few months: Rust is unequivocally the right language for firmware, and it will yield higher quality artifacts.[3]
More troubling to me than the closed source firmware is that the bug in TFA seems like something that the most basic of a test suite should be catching.
It’s reminiscent of Apple’s “goto fail” lack of certificate checking - another easily testable case that simply wasn’t.
The test authors don’t even need to be on the same team/manager. They can just write black box tests to the spec, like the author of this post did.
I’m not even some big TDD guy. It just seems to me that in these core security-critical libraries/functions that should be pretty side-effect-free that you should have some basic “receive x, produce y” functional tests to make sure the API is doing what it claims to do on the tin.
the most basic of a test suite should be catching.
A most basic test suite is not likely to wait some arbitrary amount of time (2 seconds, as the author found by trial and error) between calls to the HSM.
The images in the 'Digging deeper' section suggest otherwise. They appear to show a successful run followed by one that fails because the 'encrypted' value is garbage. Where am I missing the instantly reproducible failure?
In what way? 'Temporal' fuzzing to an eon-like range of two seconds seems, naively at least, entirely impractical.
Edit: a somewhat different way of putting this concretely - what is a practical stochastic testing regime that can reasonably be expected to find this bug?
Well designed systems have a mockable clock and timer subsystem.
They could easily test "Do X, wait 100 years, do Y".
You find all kinds of wired bugs when you do that - things that poll, for example Cron daily, will have to be run 36500 times. Certificates expire. Counters overflow. Date systems can't convert the date to and from strings. Logfiles get too big. Etc.
The premise was that basic automated testing by some other team at Google (to whom the HSM is a black box) would catch this. I don't see how that's obvious. Then you're all 'but fuzzing!' and I'm like 'wat' and now you're asking me if I know you can set the clock on the HSM? I don't think I know that. It's a black box.
I think Google's handling of security issues in Android has been badly broken for quite a while now. Here's a timeline from another security bug in the same (security critical!) subsystem:
- 2014-02-24 AM - Discovery.
- 2014-02-24 PM - Vendor notification.
- 2014-02-24 PM - Vendor acknowledgement and confirmation.
- 2014-02-26 - Attempt to setup coordinated disclosure (no response).
- 2014-04-07 - Public disclosure.
- 2014-10-17 - Response from Android security team offering line in Android security - acknowledgements.
- 2014-11-03 - Verified fixed in Android Lollipop.
(The impact of this bug was keystore material leakage between apps. No CVE was assigned!)
That was a long time ago, and of course two anecdotes aren't data. But it makes it all the more interesting to read Project Zero's frustration with poor disclosure practices by others.
"Comments on a number of StackOverflow questions have pointed out that a fault address of deadd00d indicates a deliberate VM abort."
That ending up in the ciphertext multiple times seems to point to some memory corruption issue. It's also a good argument for using magic numbers that stand out.
In what way did they violate their own rules? Google didn't prevent the researcher from disclosing and the researcher could have disclosed - the timeline describes requests, not demands. For reference, Project Zero's disclosure FAQ:
There are several cases in which deadlines were extended way beyond 90 days. And in the post itself, the researcher points out they could (and, in hindsight, feel they should) have imposed a hard 90 day deadline.
Yep I fully agree, you can't hold others to a standard you're not willing to hold yourself to. I specifically remember Microsoft begging for more time on a bug.
Also if I understood it correctly, it seems as though some devices may require a factory reset to apply the new firmware? If so, for a lot of devices this still isn't fixed.
Author here. I should have made this clear in the blog post, but I'd be interested in seeing boot logs from Pixel 3 (or newer) devices. If the firmware update failed on more devices than just mine, it would be good to know about that. If you'd like to help, first make sure you're at least on the December 2019 security update. To capture the log plug your phone into a computer, run "adb reboot ; sleep 1 ; adb logcat | grep -i citadel" and turn the phone back on. Wait for it to boot, unlock the SIM card and unlock the screen. This should yield the version information of the firmware of Titan M.
The most important takeaway is to stick to the 90-day disclosure policy. The deadline is only credible when it's enforced, and on a number of occasions Google have stated they believe so themselves. [1]
The fact they still haven't gone around to open-sourcing the Titan M firmware is, honestly, what worries me the most about this. It's probably going through some heavy internal auditing now, which is not the best approach, all things considered.
If I were an Android user: I'd feel good with how Google managed this one.
Regarding NSA vs Google (seems like after I commented on a corona virus thread a few times, I was rate limited - editing existing comments still works though):
@baybal2:
"NSA infiltrates links to Yahoo, Google data centers worldwide"
PRISM doesn't give the government additional access to anything. It simply ingests data that the FBI has already collected by wiretapping individuals under investigation.
Edit (responding too fast):
> It by default gives government access without anyone at Google or anywhere else granting that access at time of use.
Where did you get that idea? All the documents show that it ingests data the FBI already has, for individuals the companies already manually approved after potentially fighting about it with a judge. You simply made up an illegal system out of whole cloth that wouldn't last a minute in court if anybody challenged it, unlike the phone metadata program, which went through two courts to conclude its illegality.
Edit 2:
> Page five lists the companies and page six lists the per company agreement date. Unless you're trying to argue that Google didn't respond to wiretapping requests from the FBI at all before 2009.
The FBI has to set up a system for canonicalizing and routing data from each different company. Those dates list when the FBI did that for each company. Since almost nobody (including suspected terrorists, apparently) uses Apple's email service, their system was the lowest priority to support.
This is well documented, both in Snowden's documents and in documents the government later declassified. Once again, if PRISM were as you described it, it would be flagrantly illegal and shut down long before the phone metadata program.
Edit 3:
iMessage was launched near the end of 2011, and FBI's DITU handles content collection via wiretaps. When are you going to address the fact that the program from your fever dreams is insanely illegal and that it doesn't match any of the documents? If you would like me to respond normally, upvote my comments, so I don't get rate-limited.
It's automated systems for those requests. It by default gives government access without anyone at Google or anywhere else granting that access at time of use.
It does record the request though which is why NSA tried to exceed the bounds of that with MUSCULAR.
Edit to respond to your edit: Page five lists the companies and page six lists the per company agreement date. Unless you're trying to argue that Google didn't respond to wiretapping requests from the FBI at all before 2009.
Edit 2 since apparently this is how we're doing this:
> Since almost nobody (including suspected terrorists, apparently) uses Apple's email service, their system was the lowest priority to support.
There's a fuck ton of metadata that iMessage reports back up; PRISM isn't just about email. And yes, iPhones are the most common smartphone in the world. I guarantee you that Apple isn't last because they were a low priority, that's absolutely absurd.
Edit 3:
Your argument that "it would be illegal and shutdown like the other illegal programs documented here if it were actually illegal" has to be one of the hottest takes I've heard.
And the PRISM collection was part of what the Supreme Court dismissed not because it isn't illegal, but because you can't prove that affected the claimant personally without a breach to national security, so they can't prove they have standing, so the case had to be dismissed. https://www.aclu.org/files/assets/amnesty_v_clapper_scotus_o...
The plaintiffs in Clapper v Amnesty would have standing if the program worked as you described. No documents have ever been released saying the program works as you described, including the documents Snowden leaked after that case. If such docu6were released, the case would be relitigated. Here is an article describing how it actually works, linking to multiple sources: https://www.cnet.com/news/no-evidence-of-nsas-direct-access-...
> "it would be illegal and shutdown like the other illegal programs documented here if it were actually illegal"
How did one illegal program turn into multiple "illegal programs"? How do you come up with this stuff?
> The plaintiffs in Clapper v Amnesty would have standing if the program worked as you described.
No, because the way the system works is that information makes it's way to the NSA on the presence of certain search terms and is prefiltered before it ends up in their hands. The ruling by the supreme court in the case of PRISM is that amnesty international can't prove that they were among the search terms ever searched for, so they can't prove that they standing. Only if there was a leak of the actual keys slated for collection (or if the NSA agreed to release that, which would never happen), then they could relitigate.
This is in contrast to the bulk call data, where, because the NSA was collecting from everyone who made calls, standing could be confirmed.
> How did one illegal program turn into multiple "illegal programs"?
I'm bundling it up with the other programs Snowden leaked.
That's not how the system works. The system allows collection of data to/from specific non-Americans outside the US. Amnesty International didn't know that it was for specific individuals at the time they filed their suit, but Snowden's leaks and later the DNI confirmed it.
> I'm bundling it up with the other programs Snowden leaked
Once again, only one of them (phone metadata collection) was illegal. The other programs he leaked, including PRISM, are so legal that nobody with any sense would attempt to challenge them.
The NSA's slides disagree with you. Here is the slide explaining how the data flows. https://imgur.com/setOJIm
PRISM is simply a data integration system that gets data from the FBI's Data Intercept Technology Unit, which is the group that handles Internet communication wiretaps on specific individuals under investigation.
If you paint with that broad brush, the users whose data PRISM consumes are also part of the PRISM program. That's a fairly useless definition. What people are interested when talking about PRISM is whom the NSA integrates with.
Google also isn't listed in NSA documentation as collaborators.
Google also didn't integrate with the PRISM program.
Same.
Edit (responding too fast):
> Literally page five lists the companies
Page 5 doesn't say they are "collaborators."
> and page six lists the per company agreement dates
Page 6 doesn't say there was an "agreement" with those companies. It simply lists the dates that the FBI made data they have from these companies available for ingestion.
Stop pretending words exist in the documents that don't. That's conspiracy theory nonsense by the exact same method as Pizzagate.
They're automated systems in support of the 702 program. The companies in question had to have done work to assist, by the nature of how the program works.
The PRISM slides list per company agreement dates spanning over five years. Microsoft started first in 2007, Google was added in 2009, with Apple being one of the last in 2012. If it was a system that was simply ingesting FBI data, why would it wait until 2012, unless you're suggesting that Apple didn't respond to FBI requests at all until late 2012?
Just because the NSA had a program to exceed even what had been negotiated via PRISM, doesn't mean that PRISM didn't involve collaboration.
@alexbakker: could you please make code in the snippets wrap at least when viewed on a phone? I gain nothing from them being non-wrapping, while having to scroll back and forth to read. In each editor I've used, code soft-wraps at screen edge, so I don't even understand where this trend came from.
PRISM involved automated systems created by the various companies to comply with FISA requests that could originate from the NSA but would be served by the FBI.
No. The FBI issues Section 702 data requests for individual users to the companies, whose lawyers manually review the requests and may dispute them before a FISC judge. Only after the company approves the request do they start sending data to the FBI's DITU. PRISM consumes the data from DITU's servers.
The system that sends new communications with the monitored individual to the FBI is definitely automated, but configuring an account to be surveilled is a manual process controlled by the company, not the FBI, and certainly not the NSA. The reason you cannot provide documents that say otherwise is that they don't exist. The reason those documents don't exist is that the program that you've described is a conspiracy theory fiction.
Can you give an example of a PRISM request being appealed?
Also, that article is entirely around a quote from Clapper's office that the NSA "does not unilaterally obtain information from the servers of U.S. electronic communication service providers", which we found out literally weeks after that article was openly a lie, at least because of the MUSCULAR program. Like Clapper has openly perjured himself on the specifics of some of these programs.
> Also, that article is entirely around a quote from Clapper's office
No, that quote is merely an update to the article. The bulk of the article quotes people who worked on the Secrion 702 data integrations at the tech comanies, quotes from a former government official who made the requests, quotes from a New York Times article that interviewed other people who worked on the system, quotes from the heads of the companies involved, quotes from former lawyers of the companies, and quotes from investigative joirnalists who specialize in national security. It also mentions that multiple government officials, including Senator Wyden who has long been concerned with government surveillance and has been read into all the programs, confirmed phone metadata collection but none confirmed Greenwald's ridiculous misreading of the PRISM slides. It also gives a layman's description of how Section 702 works, which is what enables this collection. Notably, Section 702 does not enable the government to do what you claim PRISM does. Finally, the article also answers your previous question by giving an example of a company fighting one of these requests. Your characterization of the article is mind-bogglingly inaccurate.
MUSCULAR cannot take arbitrary data off the companies' servers. At best, it can intercept their communications off unencrypted international WAN links. According to Snowden's leaks, this was used to collect email metadata for connection chaining (not email contents), a program that Snowden's documents said that Obama shut down.
> More recent article about Clapper's perjury
That's an opinion piece about a single incident. That incident involved a series of questions asking Clapper if the NSA built dossiers on Americans. It doesn't. Eventually, the questions loosened to whether the NSA collects any data on Americans. They do, but by that time, Clapper had been repeatedly saying No for some time and had not realized that any data also includes what he referred to as metadata. Notably, the phone metadata could only be queried in some fixed set of ways according to Snowden's documents, and tying that data to an individual required a separate request, so the phone metadata couldn't be reasonably thought of as constituting dossier information.
But the bigger issue is that you have absolutely no documents saying he lied about PRISM, and you once again exaggerated by saying he perjured on specifics of multiple programs.
I got to know that by (1) reading the documents that Snowden actually leaked instead of just the interpretations of those documents from computer illiterates like Greenwald, (2) knowing people who worked on the major email services at the time, and (3) reading the laws behind these programs.
Yes, Qwest (the telecom, formerly US West) resisted and fought tooth and nail to not cooperate.
Their CEO was prosecuted to hell and back for daring to do this, and the company was forced to sell to a competitor. Nobody even remembers his name anymore, few people even remember Qwest.
His name is Joseph Nacchio. If this story is true (and it has been around for many years), he is a hero for standing up for transparent governance, and the privacy of Qwest's customers.
I don't know anything about it, but I think there may still be related litigation going on.
Edit: maybe not. Wikipedia says he was convicted in 2007.
However, I was given an NDA regarding something that sounded related at some point after that, so...
You skipped the part where he acquired a competitor under false accounting, contributing to monopolization, as dumping his stock with irregular sales while to profit from value NSA contracts before the public knew they were canceled. Should a CEO make a fortune selling stock while the company loses 90% of its market cap?
Yeah it's funny how he was the only corrupt CEO in all of corporate America during that time. So weird how nobody else besides him and Martha Stewart ever got prosecuted for insider trading despite it happening ALL THE TIME EVERY DAY EVERYWHERE.
> Former Qwest CEO Joseph Nacchio, alleged in appeal documents that the NSA requested that Qwest participate in its wiretapping program more than six months before September 11, 2001. Nacchio recalls the meeting as occurring on February 27, 2001. Nacchio further claims that the NSA cancelled a lucrative contract with Qwest as a result of Qwest's refusal to participate in the wiretapping program. Nacchio surrendered April 14, 2009 to a federal prison camp in Schuylkill, Pennsylvania to begin serving a six-year sentence for an insider trading conviction. The United States Supreme Court denied bail pending appeal the same day.
He was happy to get paid hundreds of millions of dollars for spying on his customers, but didn't want to spy on his customers? What did he think those contracts were for?
I don't know what the lesson is after reading his wikipedia page. His refusal to cooperate with the NSA seems like a non-sequitur as a response to the whole insider trading and fraud thing. I mean, if the prosecution was revenge, well, ok, but I can't connect the dots to how that makes him not guilty.
Super unclear communication, starting with "you're just using it wrong", more than six month turn around, and even then at the end no clear explanation of what went wrong with someone who was collaborating with you? That's amateur hour security.