
CryptoCat iOS Application Penetration Test [pdf] - secalex
http://isecpartners.github.io/publications/iSEC_Cryptocat_iOS.pdf
======
kaeporan
Hi, I'm the lead developer for Cryptocat. I strongly urge you all to please
read our blog post regarding this audit:
[https://blog.crypto.cat/2014/04/recent-audits-and-coming-
imp...](https://blog.crypto.cat/2014/04/recent-audits-and-coming-
improvements/)

This audit document alone does not give enough context. This audit was
commissioned by us and concerns a pre-release version of Cryptocat for iPhone.
Many of the bugs it found are due to the fact that it was reviewing a
prototype with debugging features (such as NSLog) turned on. While this audit
definitely does find some vulnerabilities and room for improvement, none of
the critical bugs in this audit ever made it to Cryptocat for iPhone's
release.

It's very unfortunate that this audit is being taken out of context like this
and used to attack our effort. I'd appreciate it if you could please upvote
this comment and help me contextualize this audit. Again, please, read the
blog post for context (and also for the results of another audit we
comissioned in parallel.) We've done our best to address these issues and are
working towards an open discussion on how to improve accessible encryption.
[https://blog.crypto.cat/2014/04/recent-audits-and-coming-
imp...](https://blog.crypto.cat/2014/04/recent-audits-and-coming-
improvements/)

The blog post's last section ("On the Significance of Audits") discusses why
it is that Cryptocat has seen more audits published about it than other
encryption projects. Please, dare to discern. Read what we're doing to improve
the security of accessible encryption and our reasoning for publishing these
audits. I'll be grateful for you taking the time to read on what we're doing
and I am more than happy to discuss with you and answer your questions.

~~~
tptacek
Was it commissioned by you? The audit I saw had the Open Technology Fund's
logo on it. OTF is a US Government effort driven by Radio Free Asia and the
Broadcast Board of Governors.

OTF, again (smartly) using US taxpayer dollars, funds audits of a variety of
privacy technologies.

For instance, they also funded a good-sized chunk of the Truecrypt audit.

~~~
napoleond
I can't tell if you actually don't know who commissioned it or if this is your
way of suggesting that the parent comment is a lie. It seems like information
you would have access to, considering your connection with iSEC, no? (I'm not
trying to stir up shit, just genuinely curious.)

~~~
nikcub
Another option is that OTF approached CryptoCat, from where their options
would be:

a) don't do an audit - and the public would ask what they are hiding

b) agree to an audit being done, meaning you 'commissioned' it.

So it is just as important to know who approached who in this situation.

~~~
tptacek
One answer, given downthread, might be that the audit was a requirement
attached to grant funding from OTF.

------
primitivesuave
This is most alarming.

 _CryptoCat 's OTR implementation on all platforms allows a chat peer to
change their OTR key during a chat session without user notiﬁcation. An
attacker performing a man-in-the-middle attack against the client's XMPP or
HTTPS stream can inject their own OTR key in the discussion after a user has
authenticated their peer's OTR ﬁngerprint. This permits the attacker to
decrypt all messages that follow, and no user would have reason to suspect the
compromise._

~~~
kaeporan
Fixes and improvements to this, and more, are covered in our blog post. I
strongly urge you to read it. This audit alone doesn't give enough context.
[https://blog.crypto.cat/2014/04/recent-audits-and-coming-
imp...](https://blog.crypto.cat/2014/04/recent-audits-and-coming-
improvements/)

~~~
anologwintermut
What's your fix for the man-in-the-middle attack on all platforms (including
deployed ones) the audit identifies and your blog acknowledges?

In your blog post there seems to be little context that can excuse such a
mistake and nothing that explains how you fix it? Am I correct in reading your
blog post that right now there isn't a fix? I.e. it's an open attack assuming
someone compromises a CA or a cryptocat server?

Isn't this a rather big issue since the only point of cryptocat is to protect
against that kind of an attack. If you just wanted security only against
eavesdroppers(i.e. you trusted the chat server), xmpp over TLS would work
fine.

~~~
sneak
> In your blog post there seems to be little context that can excuse such a
> mistake

This is a recurring theme with cryptocat. Stay away for 5-10 years until they
get their act together.

------
rancor
When I saw one of the main CryptoCat developers present in 2012, I came away
with the impression that nobody on the core team understood crypto, security,
or software engineering. This audit is another rock on the mountain of
evidence I've seen supporting this impression in the following years.

A really nice job by iSec, though.

~~~
sentenza
It is also important to always remember that a crypto app isn't like other
apps. If the protesters in Turkey rely on shoddy crypto today, it might cost
them their lives a few months from now. I sincerely hope it doesn't come to
that but it is a realistic example.

Always be sure of what you are doing, rely on external review and never, I
repeat, _never_ overstate the security of your crypto system.

~~~
rancor
This point cannot be emphasized enough. In the extreme case, which Cryptocat
marketing materials have employed, secure communications software is life
safety critical on a level similar to medical or aviation software. As such,
the admit-your-mistakes-and-fix-them-later model of development isn't agile,
open, or any of those buzzwords. It's a way to get people killed.

~~~
kaeporan
Cryptocat has always provided ample warnings that no software can ever be
trusted with your life. These warnings appear every time you launch Cryptocat,
on the website and in various guides and blog posts.

~~~
rancor
And I agree, that's a bare minimum warning for all such software. I appreciate
your efforts to make strong crypto more accessible to the general public.

That being said, you and I both know that people are using Cryptocat in
dangerous situations. And having worked on both medical imaging and secure
messaging systems, I have a healthy respect for the consequences of
implementation failure. As such, I feel that your disregard for these
consequences in broadly releasing such broken software would displease any
professional review board, and I frankly doubt you'd ever attain such a
license given such a history of poor professional judgment.

In short, I take my profession damn seriously, and jokers like you are why
nobody trusts software.

~~~
kaeporan
It's important to note that this audit was commissioned to evaluate a
prototype build before release. It was expected to find bugs, and all bugs
were fixed before release. I believe I take my job very seriously when I
commission such audits on a bi-annual basis and transparently discuss the
results. Independent individuals who find bugs (such as "Decryptocat") are
also listened to and rewarded for their effort. I believe that I and my team
have been competent, honest and hard-working. If all encryption projects were
as transparent as us, you would realize that this kind of issues happens
everywhere.

Please make sure to read our blog post and Github discussions to see the kind
of open discussion we're hoping to lead so that our software can benefit.

That being said, I suppose comments like yours are why I've been having
recurring suicidal thoughts for the past two years. I don't know what else to
say at this point.

~~~
rancor
I read the blog post reacting to this batch of audit results quite carefully,
in point of fact. In general, when I read vendor responses to such devastating
findings, I'm looking for a concrete plan to improve the threat modeling and
development practices deficiencies which are inevitably the root cause of the
class of issues uncovered by the iSec and Least Authority audits. Without such
changes, saying that you're going to keep getting audited is precisely
equivalent to saying that you're going to to keep writing security bugs and
hope someone finds them before the actual red team owns you.

While I agree that the degree of openness your team has maintained is highly
desirable, repeatedly shipping bugs which adherence to industry best practices
such as "don't use fixed IVs" or "always use constant-time compares" would
have avoided makes it difficult to believe that your team possesses the
competence you claim as well as undermining the credibility of your
communication about such issues. Thus my failure to be impressed by a post
which only proposes band-aids and completely fails to apologize for the lapses
in judgment which led to this state of affairs.

I don't take using this level of harshness in a public forum lightly, and I'm
truly sorry to contribute to your unhappiness as a result. Please do talk to
somebody, even if it's not a professional, I've found it always helps.

~~~
kaeporan
I disagree; I don't think your summary is accurate. This is an audit of a pre-
release prototype. All the bugs were fixed before release, and our blog post
at [https://blog.crypto.cat/2014/04/recent-audits-and-coming-
imp...](https://blog.crypto.cat/2014/04/recent-audits-and-coming-
improvements/) does not discuss mere band-aids. It discusses, at length, real
solutions to complex problems that many encryption apps face. It resolves
pitfalls that even companies like Apple commit on a much wider scale and on a
much more dangerous level.

For example. We didn't simply "re-use fixed IVs". We know not to do that. The
resulting bug was the series of a much more complicated and hard to spot issue
with the re-keying mechanism. Understand you might not have the full picture
here.

Simply put, I refuse the assertion that Cryptocat's team has not dealt with
its software development in a competent, professional, responsible and honest
fashion.

I want to discuss this further with you. I want to convince you of my point of
view. Please email me at nadim@nadim.cc so I can have the opportunity to
discuss with you and hopefully convince that your perspective isn't exactly
right on this.

~~~
rancor
I made no criticisms of how you responded to individual issues raised by the
audit, and in fact it's encouraging to see many of the long-standing contact
authorization issues finally being addressed as well as what I would generally
consider an acceptable approach to handling individual security issues. But I
don't think either of these points addresses my concerns regarding security
conscious development practices.

I appreciate your willingness to continue this discussion, dropped you an
email.

------
natdempk
Also of interest is their blog post about how they plan to handle the issues
described in this report: [https://blog.crypto.cat/2014/04/recent-audits-and-
coming-imp...](https://blog.crypto.cat/2014/04/recent-audits-and-coming-
improvements/)

Reading that, I still am not sure why anyone would use CryptoCat especially
with things like TextSecure on the market that seem to take crypto far more
seriously. The only reason I can see for that is that they have clients on
more platforms, but if this is similar to the state of all of them, then
what's the point?

~~~
kaeporan
Thanks for linking to the blog post. This audit concerns a pre-release version
of Cryptocat for iPhone. Many of the bugs were due to debugging code and were
fixed before release.

~~~
tptacek
Which of the bugs were due to debugging code?

~~~
daira
Findings iSEC-RFACC0114-1 and iSEC-RFACC0114-3. (2 out of the 17
vulnerabilities found by iSec, of varying severity.)

------
venomsnake
There is a "many ways to skin a cat" joke here somewhere. It is actually
terrible - the hmac timing attack requires around 3 minutes of google
searching to avoid and is basic public domain knowledge. The other are much
worse.

~~~
sliverstorm
I wish I knew how to find my way into security as a hobby. Such a fun topic.

~~~
tptacek
Why find your way in as a hobby? Are you a professional developer now? Do you
like low-level code? Are you OK with jumping directly into the deep end of the
pool and maybe drowning a little bit? Why not reach out and talk to us about
working on a professional security team?

We have gotten very, very good at taking low-level devs and turning them into
terrifying killing machines, and if you don't mind having all your flesh
removed and your skeletal musculature replaced by pistons and servomotors,
we'd be happy to do the same to you.

[http://matasano.com/careers](http://matasano.com/careers)

~~~
ph0rque
... _if you don 't mind having all your flesh removed and your skeletal
musculature replaced by pistons and servomotors_...

Ah, but have you pen-tested the pistons and servomotors to make sure they're
secure against attacks?

~~~
tptacek
We only care about their ability to inflict harm, not their ability to
withstand it. :)

------
lincolnq
This is awesome. I'm sad that CryptoCat is getting slammed for this for being
one of the brave few to post this online. I am sure there are an infinite
number of "security-critical" apps which would fail an audit like this, but
who never even thought to GET an audit -- much less post it online. The
software development community is much stronger for being able to see
professional stuff like this posted.

Does anyone know how much these audits typically cost, if you're not being
subsidized?

~~~
rdl
Generally 10-50k is a good starting point for this level. Most firms are full
up on work most of the time, but will often try to get interesting new
companies or projects even if they're less profitable since 1) they can grow
into better stuff 2) good for reputation and for retention of their own
employees.

Compliance-only is usually cheaper; you can buy rubber stamps for <$10k.

~~~
m0nastic
It's important to distinguish that the places offering the <10k rubber stamps
aren't really offering the same service (even if you concede that it's only
for compliance).

Places like that are just running a scanner against your website. In the case
of an app like this (which was an iOS app), you might find a cheap place to
run it through a source code analyzer (either through a cloud-hosted service
like Veracode, or by running an app like AppScan).

Assuming you wanted to hire a "respectable" firm to actually perform a real
application assessment, I'd say it's closer to 30-50k.

If you look at the report, it was scoped at 3 man-weeks of testing (which in
this case looks like it was 3 engineers for 1 week). Even if you don't include
any additional overhead (like hours for the report generation, or project
management hours), you're looking at ~24k just for the engineers effort (if
they just priced it t&m, which hopefully they don't).

To be fair, this is a pretty exotic application though, compared to what a lot
of other people might be working on. The scope for a project to test a more
"normal" app would be less. Maybe even half that.

~~~
rdl
Right -- a $10k _real_ audit is a "deal" of some kind -- either a firm trying
to win future business with a discount, or an individual doing it directly
(especially from overseas, or as a side job).

You can also get a better deal if 1) you're open source, and the audit becomes
part of someone's portfolio 2) you provide really clear documentation,
security model, etc. to make the process more efficient 3) your app is already
well-architected so the security-critical part is small, and you only audit
that part.

There's probably a 100x bigger pool of people capable of doing "IT audits" and
"hosting environment audits" well vs. appsec for webapp or mobile app (or
especially desktop app).

------
zooko_LeastAuth
Here is our blog post about our audit of Cryptocat, which was also announced
today: [https://leastauthority.com/blog/](https://leastauthority.com/blog/)

------
imkevinxu
Didn't realize how vulnerable even a simple NSLog was... I wonder how many
websites have sensitive information they console.log but forgot to take out
for production

~~~
pokstad
Remember when the iPhone was under fire for logging everywhere a user went?
That was due to careless devs logging location data. Also, logging data is a
blocking operation, so if done too often it slows the performance of an app.

CORRECTION: the 3rd party apps were part of it, along with Apple's own
logging, when reported in the news.

------
jnbiche
Wow, remind me to never have an audit done by iSec. "Extremely thorough" would
have been tough but appropriate, but "brutal" seems just gratuitously
provocative. Was that what you were going for?

Edit: OK, apologies to iSec for my mistake. I thought he was still affiliated
with them. In any case, it looks like a top-notch report, so it would have
been a shame to detract from that accomplishment.

~~~
tptacek
Alex (the submitter) doesn't work for iSEC; he's the CISO of Yahoo now. He
left iSEC last year to start Artemis.

I agree that the title is bad, and (belatedly) asked 'dang to revert it.

------
secfirstmd
I for one think that Nadim and the Open Tech Fund are to be applauded for
opening up their review to the public. I have to wonder how many other
commercial and non-profit organisations would ever consider doing this?
(Especially those which are in the field of communications and are relied on
by people for their lives).

Many people on HN seem to be reading the review without actually looking at
Nadim's response on the Cryptocat blog - which I urge everyone to read first
before commenting.

[https://blog.crypto.cat/2014/04/recent-audits-and-coming-
imp...](https://blog.crypto.cat/2014/04/recent-audits-and-coming-
improvements/)

As far as I understand the username for Nadim (Kaeporan) was also blocked from
HN last night so probably he isn't able to continue responding to the comments
up here.

------
lawnchair_larry
Huh, this was apparently submitted by Alex Stamos, a co-founder of iSec
partners (who did this audit). And he editorialized the title, "Brutal
Professional Audit of CryptoCat Published."

Your former company did an audit for a customer, then you posted it to HN
calling it "Brutal"? Really?

~~~
fakedavidthiel
I was certainly not comfortable with that either; however, Alex is his own man
now, free to use whatever inflammatory adjectives he chooses. Not much we can
do about it, but also not unusual for him to follow the news on a company he
put so many years into.

I also do wish that the original post had pointed at CryptoCat's blog post -
aside from CryptoCat's context about problems and resolutions, people should
also be aware of the separate report by Zooko's team.

------
oafitupa
The good thing about CryptoCat is that everyone wants to trash it, so it's
becoming better.

~~~
marshray
The wisest words I've ever seen office administrative staff post over the FAX
machine are:

 _Everyone 's life has a purpose. Consider the possibility that yours is to
serve as a warning to others._

------
Wintamute
@secalex Dumb post man. Way to de-contextualise something, cause a drama and
damage reputations needlessly.

~~~
ris
Reputations? What reputations, exactly? The only reputation CryptoCat
developers have is for writing incredibly poorly thought out software full of
holes. Combined with their taste for publicity I would rather call them a
public menace.

