Hacker News new | past | comments | ask | show | jobs | submit login
Curl Security Audit (haxx.se)
283 points by NeutronBoy on Nov 24, 2016 | hide | past | favorite | 39 comments



Mario and cure53 team are a bunch of great people. I learned a lot by reading a lot of reports from Cure53 published on their website[1]. Their repos on Github are a timesaver when looking for XSS. Filedescriptor[2] is a part of cure53 and has some amazing reports on hackerone.

Mario is as humble as they get in the security scene.Tweet to him or just follow him generally and you will pick up on how to wreck browser apps.

[1]https://cure53.de/#publications [2]http://innerht.ml/


Very impressive results from Cure53: https://cure53.de/pentest-report_curl.pdf

    CRL -01-001 Malicious server can inject cookies for other servers ( Medium)
    CRL -01-002 ConnectionExists () compares passwords with strequal () ( Medium)
    CRL -01-005 OOB write via unchecked multiplication in base 64_ encode () ( High)
    CRL -01-007 Double - free in aprintf () via unsafe size _t multiplication ( Medium)
    CRL -01-009 Double - free in krb 5 read _ data () due to missing realloc () check ( High)
    CRL -01-011 FTPS TLS session reuse ( Low)
    CRL -01-013 Heap overflow via integer truncation ( Medium)
    CRL -01-014 Negative array index via integer overflow in unescape _ word () ( High)
    CRL -01-021 UAF via insufficient locking for shared cookies ( High)
    Miscellaneous Issues
    CRL -01-003 Ambiguity in curl _ easy _ escape () argument ( Low)
    CRL -01-004 Metalink provides an oracle ( Info)
    CRL -01-006 Potentially unsafe size _t multiplications ( Medium)
    CRL -01-008 % n is supported in format strings ( Low)
    CRL -01-010 Slashes and .. are decoded in file URIs ( Low)
    CRL -01-012 Only the md 5 of the SSH host key fingerprint is checked
    CRL -01-015 Default Compile - time options lack support for PIE and RELRO ( Low)
    CRL -01-016 Unchecked snprintf () calls ( Low)
    CRL -01-017 Permit disabling ( insecure ) fallbacks ( Low)
    CRL -01-018 Null pointer dereference in the RTSP protocol ( Low)
    CRL -01-019 nss _ init _ sslver uses version info from NSS header ( Info)
    CRL -01-020 dup _ nickname () doesn't check for memory allocation failure ( Low)
    CRL -01-022 polarssl _ connect _ step 1() lacks matching unlock ( Info)
    CRL -01-023 ssl _ thread _ setup () leaves mutex buffer partially uninitialised ( Info)


I just want to thank Daniel for volunteering his project for this level of scrutiny, and then putting in the very long hours to address the results of that process. I know software development is sometimes a thankless task, and I know he got some rather heavy handed response from Apple when he announced all these fixes at such "short notice".


Can you provide more info on the Apple thing? Can't imagine the stance of a company using a library (for free?) to the library author that would allow any sort of offence as the "short notice".


Awesome to see heavily used (and often embedded) OSS software getting a sponsored audit like this. I wonder the same could happen for sqlite and others.

Given the number of vulnerabilities that "state sponsored" folks are likely to know about, this seems like a very useful defense and a way to increase confidence in our building blocks.


It's great that these bugs have been found and fixed. I'm frightened by this report however. If a well managed open-source project like Curl has this number of pretty serious bugs that can be found by skilled auditors, what hope do we have of securing ourselves against state sponsored attacks? Really no hope, I think. Given the amount code on a modern computer, there must be no shortage of "zero-day" holes they can use.

Maybe this is not news to most informed people. It shocks me though. I know the situation is bad, just didn't fully grok how bad.


While you are right, you can approach security from graded risk perspective. eg non-validated user input on an internet facing interface is going to be a greater risk than a buffer overflow bug in cron (for example). So when hardening your systems the first thing you do is limit the amount of access the outside world has to your OS and application and ansure that all components that do access the outside world are up-to-date (curl is one of those libraries which would communicate to the outside world). So firewalling, OS updates, decent bit lenghs on your encryption keys, etc. Then you ensure all your applications are sandboxed so even if they are exploited the scope of data they can do is limited.

But as you said, a sufficiently competent and determined hacker will likely find a way into many a "secure" system. But if you can limit the code that is exposed to the internet then you at least limit the amount of code that can be exploited (or rather force an attacker to use more than one attack in conjunction to gain system access). However if an attacker has physical access to your system then I'd just give up now as you've already lost.


This is one of the reasons that people are excited about Rust; it promises to eliminate a whole class of security bugs that are often found in C applications and libraries. Without having to resort to managed code.


> Without having to resort to managed code.

Which was already partially possible in the last decades by the lines of Algol family of languages starting on the Mesa/Cedar and Ada branches.

What is more exciting about Rust is that a new generation of coders is rediscovering the ways of system programming before C took over the IT industry.

Also how their efforts to adopt Cyclone regions are influencing Swift, C++, D, ParaSail and Pony designs.


> Without having to resort to managed code.

Even better: gradually replacing parts of the current C code with Rust code is possible, while keeping the same API and ABI. IIRC, someone is already trying it with librsvg.



Completely agree. The first issue in the report is about reading lines from a file. Those kinds of bugs can and should be eliminated for good.


Creating 100% secure software is not realistic. If you can't win, change the game.

I'm not saying you should give up or that audits are a waste. They are a good thing, though you should also work on other approaches to the problem besides "work harder on security bugs".


There's pretty much no good reason that the most widely used command line HTTP fetcher should be a giant C project.


For libcurl - easy linking to any C/C++ code is very good reason. And standalone curl is lighweight with minimal system dependencies.


I understand libcurl well (I've used it before on projects), and don't dispute its utility. I also understand the value of a very small C-based HTTP fetcher for system bootstrapping (I'm not sure that's really what curl is).

What I don't get is why the most widely used general purpose command line HTTP fetcher is a giant C program that we're still finding UAFs in in 2016.

I respect the enormous effort that goes into maintaining curl (though I have my differences with the way security is handled). But it's past time curl is replaced by a Rust or Go program.


I love the delicious irony that the install instructions for Rust on linux (from https://doc.rust-lang.org/book/getting-started.html) specify :

"curl -sSf https://static.rust-lang.org/rustup.sh | sh"


That's not ironic. Tools bootstrap other tools all the time.


A statically linked curl for ARM64 is 205kb.

A statically linked hello world in Rust for ARM64 is 680kb.

Curl is used for a ton of embedded hardware, and is probably in your router. Until Rust can solve its massive binaries, that is somewhere on their roadmap, it just isn't appropriate for use in the embedded world, where kilobytes still count.


> Until Rust can solve its massive binaries, that is somewhere on their roadmap, it just isn't appropriate for use in the embedded world, where kilobytes still count.

A demo written in Rust [1] won a 64kB demo competition against C and C++ competitors. It is already solved if the binary size is a problem for you.

You can easily get down to 160kB without a whole lot of effort (i.e. avoiding libstd) [2].

[1]: https://www.reddit.com/r/rust/comments/597hhv/logicoma_elysi...

[2]: https://lifthrasiir.github.io/rustlog/why-is-a-rust-executab...


2 very big things.

1.

> How big was it before compression? - CyberDiablo

> Just over 200kb iirc - yupferris

2. Avoid the standard library is a whole lot of effort.

With C, and code elimination with GCC or clang, using fairly standard optimisations that require next to no knowledge of the native code being produced, I can have the following libraries, without bloating the executable:

<assert.h>

<complex.h>

<ctype.h>

<errno.h>

<fenv.h>

<float.h>

<inttypes.h>

<iso646.h>

<limits.h>

<locale.h>

<math.h>

<setjmp.h>

<signal.h>

<stdalign.h>

<stdarg.h>

<stdatomic.h>

<stdbool.h>

<stddef.h>

<stdint.h>

<stdio.h>

<stdlib.h>

<stdnoreturn.h>

<string.h>

<tgmath.h>

<threads.h>

<time.h>

<uchar.h>

<wchar.h>

<wctype.h>

Though you are incredibly unlikely to use all of the above at once, and occassionally some are unavailable on more limited embedded hardware (e.g. threads), C and its current compilers are light years ahead of rustc in terms of optimisation, even more so when you look at the massive list of supported architectures: GCC [0], Clang [1]

I love Rust, and I do look forward to seeing more of it, but avoiding libstd, or using a different malloc, and being forced to use a nightly, or any of the other little things you can do to mitigate large binaries are just not something that is easy compared to C, or easy to pipeline when making embedded software.

Relying on executable compression not directly supported by the compiler, is asking for trouble.

You can do the same insane things like no stdlib in C too [2], but it doesn't mean its a good idea. (See the IOCCC [3] for some truly crazy manipulations of the C language!)

[0] https://gcc.gnu.org/backends.html

[1] http://llvm.org/docs/doxygen/html/classllvm_1_1Triple.html

[2] http://ptspts.blogspot.com.au/2013/12/how-to-make-smaller-c-...

[3] http://www.ioccc.org/years.html


> Just over 200kb iirc - yupferris

So just to be clear, is your position that the demo could have been better if it were written in C? That flies in the face of all the evidence (that, you know, that it won against all its competitors, and that 200kB isn't even large by C standards), but if so, you should be up front about that.

> C and its current compilers are light years ahead of rustc in terms of optimisation,

"Light years"? Come on! The backends are the same! This is totally contrary to my experience.

Bringing up extra architectures supported by GCC/Clang is irrelevant to the discussion. Especially since rustc uses the same backend as Clang.

And header files are not libraries! You get essentially all of these with libcore. Citing things like stdbool is especially irrelevant, because bool is builtin in Rust. Assert is a macro in Rust that expands to nothing in the binary. Most of math.h is covered by LLVM intrinsics, which you get with libcore.

Using a nightly Rust is as easy as using GCC. Really. Just use rustup and switch to the nightly channel. In fact, compared to, say, setting up a GCC cross toolchain for embedded development, it's a breeze. Xargo makes cross platform package management incredibly easy; in fact there's nothing like it in C land.

Using a different malloc is as easy as "extern crate alloc_system". I do it when testing Servo when I want to make use of the platform native malloc instrumentation. It's far easier than in C, in fact.

Not using the standard library is hardly "insane". A lot of projects do it. It's a first-class thing.

All of these problems are total non-issues. There are zero good reasons why, in 2016, a command line Web fetching tool should be written in C.


> So just to be clear, is your position that the demo could have been better if it were written in C?

No. But that abusing a language to create compression, doesn't address the day to day case in any way or shape.

> Bringing up extra architectures supported by GCC/Clang is irrelevant to the discussion.

Hardly. I made more than one point. Size was not only it. C comes with the flexibility that it will compile for a lot more embedded hardware than Rust will.

LLVM is rustc's backend, and I am aware of that, but Rust has also had a lot of teething problems with actually compiling to many of those architectures. Its young enough that they don't matter yet, because Rust isn't targeted to every platform that LLVM supports yet.

Things like zinc and rfs are experimenting in this area, to see how feasable it is to have Rust replace C for embedded, but they're still experiments right now.

> And header files are not libraries!

No, they just allow you to link against them.

And the C PreProcessor is (probably, there's some debate) Turing Complete. So "header files are not libraries" is really not clear when it comes to C. Sometimes they are. Sometimes they aren't. Header-only libraries are in common use.

For example, NULL is a very useful C macro, that is used just about everywhere, that is usually accessed via <stdlib.h>

You could just add:

    #define NULL (void *)0
But there isn't a need to, because of dead code elimination.

The same dead code elimination that applies to C's inbuilts, isn't yet applied against Rust's.

Which means that C gets the flexibility to use any code, and in Rust you shouldn't use the standard library?

That being a first class thing is entirely a sign that Rust is still a young language.

> Using a different malloc is as easy as "extern crate alloc_system". I do it when testing Servo when I want to make use of the platform native malloc instrumentation. It's far easier than in C, in fact.

    #include <jemalloc/jemalloc.h>
And add -ljemalloc to the compile list.

I really don't see where its hard to link out to another malloc implementation.

So I can do it if I need to... But I don't for most use cases. Especially not to reduce executable size.

> Just use rustup and switch to the nightly channel.

Awesome. Try selling that to anyone developing on embedded hardware's manager, when Rust Nightly is... A nightly.

> Unstable features are only available on nightly Rust.

It comes with nearly a guarantee that something will be unstable.

> Xargo makes cross platform package management incredibly easy; in fact there's nothing like it in C land.

So its a package manager that hooks into LLVM's cross-compilation targets?

Which C does have as well, through clang.

Xargo and Cargo are nicer than make or cmake. They are. More modern technology, and they have better adoption because the language has used them since really early days. That is fantastic, and I do wish C had gone that way at some point, but its an old hat now.

My cross-platform stack is more complex:

* musl

* gcc with multiple backends (the most complex part!)

* make

But right now, today, it produces executables that:

* Are smaller than Rust's equivalents, by a large margin.

* Aren't always possible for Rust to produce.

> There are zero good reasons why, in 2016, a command line Web fetching tool should be written in C.

I'm sorry, but I highly disagree.

Maybe wget could be rewritten in Rust, for those distros where the platforms align with what Rust can produce.

But wget is just a commandline web fetching tool.

Curl isn't.

It's a library.

Used by manufacturers of devices that use SOCs and a ton of other bare-metal systems.

I'm not arguing against Rust here.

I'm arguing against Rust, today.

The progress of the language is great.

The premise of the language is great.

The tooling of the language is great.

It is fast becoming what we need, but it isn't there today.

It will be, soon.

My prediction?

All I've just said will be irrelevant in 2 years, maybe less.

And that'll be a good thing.

I'll be happy when it happens.

But today?

I can't accept Rust Nightly, I can't accept not using the tools the language gives me, and I can't accept binaries that the SOCs I've used can't even load into memory because of their size.


> The same dead code elimination that applies to C's inbuilts, isn't yet applied against Rust's.

That is false. LLVM uses the exact same dead code elimination in Rust as it does in C. In fact, Rust's tends to be better, as larger modules mean fewer exported global symbols.

If you read the blog post I linked, then you'll see that most of the size is libbacktrace and libjemalloc. You can omit both. If you were to write a C app that used libbacktrace and libjemalloc, then you'd have the same "bloat"; they are not dead code. The problem is not that C has some magic optimization that Rust doesn't have.

You can't just ignore the fact that Rust produced a demo that was size competitive. You can't just handwave it away as "abusing a language". You can't successfully argue that you can't produce small binaries in Rust, because we have empirical proof of it right there.

Rust was a fine language in which to write that demo. For the same reasons, Rust would be a fantastic language in which to write a libcurl replacement.


Don't forget, "losing the standard library" doesn't mean losing everything; it means losing libstd, but not libcore. A lot of those headers you're showing off would be in libcore, not in libstd, in Rust terms.

> C and its current compilers are light years ahead of rustc in terms of optimisation,

Given that we use LLVM, which is the part of clang that does most of the optimizing, we should be equivalent here, roughly.

(That said, I agree that there's more we can do to make this easy.)


I really don't like that I lose anything, but sure.

Rust isn't yet equivalent to clang in terms of LLVM, because of the overhead of some of the libraries that it needs. Like its own malloc implementation.

These are acceptable costs to get security.

I hope I haven't come across as disliking Rust, because I love it.

Unfortunately, when it comes to embedded, bytes matter, and those overheads aren't easy to limit or abolish in a simple manner. So you lose out on the security, and have strange compression scripts, or you lose out on the security and use C.

For now, the second is an easier sell to a manager.

I am expecting there will be progress here, from a few of the issues I've seen open on GitHub.

And I really like your work.


Rust doesn't need its own malloc implementation.


But there's also no reason why the most widely used HTTP fetcher in embedded software should be the same piece of software as the most widely used command line HTTP fetcher on workstations and servers, most of which do not care much about shaving kilobytes.


And no reason that the most widely deployed kernel in embedded and server worlds should be the same. But it is, and written in C. Its called GNU/Linux.


This is in fact a strong argument against your hard-line "every kilobyte matters" stance above. Linux is not a particularly small kernel. If code size in the embedded world were really as paramount as you claim, we'd see a lot less Linux and a lot more random RTOSes.


Not taking a side on the whole Linux vs. GNU/Linux thing, but the kernel is decidedly not called GNU/Linux; it's just Linux.


You're a pretty smart guy. Why not just knock out that little baby yourself?


I'm hoping someone else already is.


You might want to consider OpenBSD's ftp. It does a lot more than FTP.


So, when are you getting started?


There's a historic reason: curl was started in 1997. At that time it made perfect sense to use C.


I hope we see much more of this. https://www.opentech.fund/ funds open source security audits, or did a few years ago when I worked on some of them. They focused on problems like private chat and secure drops for whistleblowers.

On a smaller scale, https://defuse.ca/ set a good example with some one-person audits (under Research >> Audits on the site). Nothing's stopping anyone from learning by doing, the way most of us learned to program.


I've consistently wondered why cURL is so popular in languages with a solid web client(almost everything these days) . I've used it for quick hacks but definitely nothing in production.


Very interesting, the report PDF makes for interesting and informative reading.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: