Hacker News new | comments | show | ask | jobs | submit login
FFmpeg and a thousand fixes (googleonlinesecurity.blogspot.com)
263 points by abraham 1260 days ago | hide | past | web | 145 comments | favorite



It is interesting that YouTube isn't mentioned in this blogpost, despite there being good evidence that ffmpeg has been used there[1].

The fuzz testing they mention is based around constructing malformed (or at least "exotic") input files and then monitoring for failures... ie simulating exactly the kind of attack someone might use against YouTube's transcoding infrastructure.

[1] http://multimedia.cx/eggs/googles-youtube-uses-ffmpeg/


I'm not proud of this legacy code, but... it exists because it was a real world issue that would cripple a conversion server in production envs when fed certain files with timing/syncing errors as part of an automated upload and conversion process. When it would crash it would consume 100% of the cores and eat up enough RAM to force swap.

This cron has been keeping ffmpeg in check for over 4 years (not 6, whoopsie) in a production environment at this point... it processes thousands of videos a day using a custom queuing and reviewing system.

  1 <?php
  2 /*
  3 **  Cron responsible for detecting failed/hung ffmpeg
  4 **  instances and killing them.
  5 */
  6 require_once '/lib/class/dbmysqli.php';
  7 
  8 $output = shell_exec('ps -aeo pid,etime,args | grep ffmpeg | grep -v grep');
  9 
 10
 11 
 12 
 13 preg_match_all("/^[ ]{0,}([0-9]*)[ ]{0,}(.*?) .*?([0-9]*?)\-[0-9]*?\.flv /m", $output, $preg_out, PREG_SET_ORDER);
 14 
 15 if (!empty($preg_out)) {
 16 
 17     $db = new DBmysqli('dbmaster');
 18 
 19     foreach ($preg_out as $process) {
 20         $pid = intval($process[1]);
 21         $etime = $process[2];
 22         $queue_id = intval($process[3]);
 23 
 24         $etime = intval(str_replace(':','',$etime));
 25 
 26         // elapsed time >= 60:00 (1 hour)
 27         if ($etime >= 6000) {
 28             $fail_sql = "DELETE FROM media_upload_queue WHERE queue_id = $queue_id";
 29             shell_exec("kill -9 ".$pid); // kill hung ffmpeg process
 30             $db->query($fail_sql); // remove file from DB
 31             // debate if life is worth living...
 32         }
 33     }
 34 }
 35 ?>
Yes I know there are better ways to do this now at the OS level, but it was a quick hack over a half decade ago and continues to work... ain't broke, don't fix it kinda deal.


"I'm not proud of this legacy code,"

"This cron has been keeping ffmpeg in check for over 6+ years in a production environment at this point... it processes thousands of videos a day using a custom queuing and reviewing system."

why do we programmers always feel we need to apologize for something that we did quickly, but has been running without incident for a number of years.

take a bow my friend. that was an awesome patch you came up with all those years ago.


> why do we programmers always feel we need to apologize for something that we did quickly

Because if we don't, our programming brethren will assume that we thought the fix was correct and optimal. And we'll get bashed for it.

Haven't you ever heard of the quip (summarized): "The quickest way to get help from a technical crowd is to claim what you're doing is correct. People will bend over backwards to prove you wrong."


> why do we programmers always feel we need to apologize for something that we did quickly, but has been running without incident for a number of years.

The script is only treating the symptom, not the problem.

It would be better to detect the files before they waste an hour of time, so you could tell the user instead of having them silently disappear. Maybe there is something you could do to fix the files. The programmer part of me says there is a real fix that needs writing.

It's obviously a great script if it's worked that long. Who knows how long it would take to track down and fix the bug(s) causing the issue. If they haven't needed to fix the bug in all these years just writing that script was obviously a good decision.


> If they haven't needed to fix the bug in all these years just writing that script was obviously a good decision.

Exactly this. There are an unbounded number of bugs which will cause this single symptom. I think detecting the symptom is exactly the right solution.


In theory. In reality, after treating one or two causes, the occurrence rate may fall to once a month or year.

I agree that mitigating the problem is a good first step though.


Tell the truth, it's only been 4 years.


damn... should have checked svn logs first, busted.

r8645 | * | 2010-02-09 18:11:35 -0500 (Tue, 09 Feb 2010) | 2 lines


Did you really write that comment on line 31 four years ago? :)


Google has never officially acknowledged that YouTube uses ffmpeg. I seem to recall it was because of legal reasons they are not allowed to. So you will never hear a Google employee mention anything about YouTube and ffmpeg ;)


We do at least know that they use x264: http://x264licensing.com/adopters


The wonders of C:

- NULL pointer dereferences,

- Invalid pointer arithmetic leading to SIGSEGV due to unmapped memory access,

- Out-of-bounds reads and writes to stack, heap and static-based arrays,

- Invalid free() calls,

- Double free() calls over the same pointer,

- Division errors,

- Assertion failures,

- Use of uninitialized memory.

But hey, any good programmer always writes perfect C code.


Yeah, and you know, the Rust guys are actually doing some great work on making a language that solves some of these problems without giving up the advantages that make C programmers choose C.

Meanwhile, you are just being snarky on a message board while using a whole stack of software that was built by the programmers whose work you are criticizing.


Notably, and relevant to this post, the number of media codecs implemented is Rust is, I believe, zero. So perhaps the "some of these problems" is perhaps a bit spun.

Rust looks great for apps and middleware. I don't see ffmpeg or x264 (both of which are implemented with significant amounts of architecture-specific machine code -- an environment where neither C nor Rust is going to be able to help avoid the kind of bugs we're talking about) jumping ship any time this decade.


> Notably, and relevant to this post, the number of media codecs implemented is Rust is, I believe, zero.

I'm not claiming it's production quality, but I did write an MP2 decoder in an early version of Rust: https://github.com/pcwalton/fempeg

> Rust looks great for apps and middleware. I don't see ffmpeg or x264 (both of which are implemented with significant amounts of architecture-specific machine code -- an environment where neither C nor Rust is going to be able to help avoid the kind of bugs we're talking about) jumping ship any time this decade.

I grepped through the commit log for bugs marked "j00ru", as suggested in the post, and the first 20 I found were all mostly-architecture-independent, straightforward C code (often bitstream parsing code). This is the kind of problem that Rust is generally good at securing and making performant. None of the 20 patches I looked at touched any assembly code at all.


My (micro-) point was more that memory bugs are going to be inherent in a codec implementation because of the nature of hte problem, obviously not that all of the bugs are going to come from the assembly.

But the macro point is just empirical: the proof that Rust is an appropriate tool for codec development needs to come in the form of a production-quality codec implementation of some non-trivial standard (VP9, say), not a HN post.


> My (micro-) point was more that memory bugs are going to be inherent in a codec implementation because of the nature of hte problem, obviously not that all of the bugs are going to come from the assembly.

I don't understand this. If you write in a safe language, the compiler rules out the memory safety bugs, regardless of the problem.

> But the macro point is just empirical: the proof that Rust is an appropriate tool for codec development needs to come in the form of a production-quality codec implementation of some non-trivial standard (VP9, say), not a HN post.

Of course, no argument there. I am reasonably confident that we can get there with tuning though, based on our benchmarks so far. :)


> I don't understand this. If you write in a safe language, the compiler rules out the memory safety bugs, regardless of the problem.

For performance sensitive code, you sometimes need to deliberately step out of the strongholds of the compiler.


Rust allows this via `unsafe` blocks, which essentially have C semantics (and allow the use of inline assembly too).


> For performance sensitive code, you sometimes need to deliberately step out of the strongholds of the compiler.

...or just add more primitives to the language.


Why make the language larger when you can write identical things in libraries? (For Rust, you can use `unsafe` blocks to write very low-level primitives in normal user code and then expose a safe interface above them.)


It depends on what you guys are talking about.

If you mean "constant" primitives, i.e. things which don't require to expand the language grammar or type system, then it's a question of future prevalence of usage of said primitives. If they become almost ubiquitous to any project, then they obviously need to be included in the standard libs. If you're talking about a new language feature, that depends on how much that new feature may help design and write future libraries, and also what are the aims of said language. I understand that rust aims to become a safer system language, so any feature helping the design of low level, performance critical code should be welcome.


Yeah I don't think it's likely either. But it's a lot more likely than Modula-2, Ada, or any of the other languages that the OP would be arguing of in favor of C.

OP is criticizing without a plausible alternative; Rust is at least more plausible of an alternative. But I also suspect that C+ASM will continue to rule the day here, especially now that these fuzzing techniques are become more widespread and industrial-strength.


Fuzzing has found plenty of security vulnerabilities, to be sure, but I'm skeptical that it makes static analysis obsolete. Even if it were possible to write a theoretical fuzzer that would find every possible security vulnerability, it'd still be a whole lot cheaper, and easier, to just have a tool that enforces that you write the code correctly in the first place than to spend time writing it in an unsafe language and then developing a world-class fuzzer to find all the vulnerabilities.


Totally agree that static analysis is better than fuzzing for preventing memory-safety vulnerabilities. If ffmpeg can be written in Rust without any disadvantages compared to C, then that's an easier way to ensure memory safety than fuzzing. I would be surprised if Rust can compete against C anytime soon for writing codecs, but will be pleasantly surprised if it can!


This list of criticism is copied verbatim from the article's list of fixed bugs, not composed directly by the OP.


The snark, however, is wholly original.


Considering it's entirely predictable and I've seen ones just like it a million times before, no, it's not. Relevant xkcd, esp. panel 2: https://xkcd.com/568/

"You will never find a programming language that frees you from the burden of clarifying your ideas."


Criticism is good if valid, and it is in this case.


I don't think the poster was being snarkey, they were merely pointing out how hard it is to write correct C code. And there was no criticism of the developers, in fact the post implied that the bugs were the fault of C and not the developers.


Really breaking new ground here. What should FFmpeg have been written in? I can think of a few candidates, but I think they share a number of those "wonders".


On the 30C3 there was a talk about a C Compiler (actually an llvm optimization plugin) that can eliminate nearly every memory management related vulnerability by adding memory checks. The penalty is only a 100% increase in runtime.

> If you're a C programmer and somebody says like: I've that optimization that makes your program 3% faster you go: wow! Then you come and say: well, now I make it half as fast you go: w00t?! But then you remember that people actually use ruby to serve web pages.

I'd say this would be actually worth it for a far more secure FFmpeg.

https://www.youtube.com/watch?v=2ybcByjNlq8


The difference is that FFMpeg is very, very CPU-bound, taking four hours instead of two to encode a movie is a big hassle. On the other hand, not many people use a world-facing FFmpeg instance to submit jobs to. I don't see why security should trump speed here.



For realtime playback locally sure. I don't think there are many/any public-facing systems backed by VLC. Presumably VLC prioritizes speed over security.


Oh, that was the point being made. My view is that VLC essentially is a public facing system, in the same sense that my browser is a public facing system. Sure, it is not as easy to do a targeted attack against my VLC as against my mail client (or against the nginx on my server). But it is fed a lot of untrusted data. The same goes for Chrome, which also uses FFmpeg according to the article.


or plex, which i believe is ffmpeg based for real-time transcoding


There's also AT&T Labs' memory-safe dialect of C called Cyclone: http://cyclone.thelanguage.org/

It's been around since 2006, but I haven't really heard of anyone using it. I wonder why.


Cyclone has been dead since years (the last release is dated 5 May 2006)

It never compiled on 64bit architectures, and to get it to compile with 4.0 or 4.1 (newer versions of gcc won't work) you need to fetch it from SVN (last commit is dated May 2009)... and obviously: no distribution still ship such an old version of the build tools... you'd need something like debian lenny, which does not even receive security updates anymore


"that can eliminate nearly every memory management related vulnerability"

This is like saying "my bulletproof vest will stop nearly every round fired at it". Too bad it only takes ONE...


So I suppose you'd rather not wear any bulletproof vest?


Before C escaped UNIX, there used to be safer system programming languages like Modula-2, just to cite an example.

But Modula-2 had Lilith to offer, while C had UNIX.


There are other things to consider as well.

Compiler support across various platforms: are there compilers available and do they generate good code?

Mindshare: how big is the intersection of people who know the language and have the domain knowledge to contribute to the project?


When C was UNIX only, the generated code quality also had a lot to be desired.


Sorry, I don't get what you're arguing here. Are you saying the C-compilers generated poor code when the FFmpeg project was started back in 2000? Are you arguing that if the FFmpeg project had used Modula-2, this would have sparked more work on Modula-2 compilers?

Or are you perhaps just wishing that history had taken a different turn?


The initial release of FFmpeg was in 2000, well after that stage in C's life.


> What should FFmpeg have been written in?

There'll be a HN topic A FFmpeg reimplementation in JavaScript soon enough ...


Process videos in browser using FFmpeg compiled to JavaScript (56 days ago) https://news.ycombinator.com/item?id=6739582


The only options I could think of that're mature enough are OCaml and Haskell. I'm aware of video transcoders in both, though nothing with anything like the format support of ffmpeg.


OCaml I know doesn't support SIMD acceleration. Not sure about Haskell. You need SIMD support to get anything decently fast in the field of video/audio manipulation.


Haskell's GHC is working on it [1]. It's not primetime (not even close, really), but it does work with modified libraries and compiling to llvm.

There is also Intel Lab's HRC (Haskell Research Compiler) that does have SIMD capabilities [2]. Unfortunately, there isn't a public release of HRC yet.

---

[1]: https://ghc.haskell.org/trac/ghc/wiki/SIMD

[2]: http://dorchard.wordpress.com/2013/10/14/automatic-simd-vect...


That is a compiler issue, not language.

ISO/ANSI C also don't support SIMD, you have to go down to Assembly when writing portable code across C compilers.


if we're willing to posit the existence of unwritten software that makes simd work from ocaml or haskell, we could also just assume the existence of C that checks memory accesses for zero cost

also, I think you may be able to do reasonably well with icc and not much assembly


Ya, it's definitely a language issue; barring super-intelligent compilers that can vectorize things on their own (which GCC does for certain cases), you need a means to express vectorizable operations.

You don't need to work at the assembly level for this; you just need a language that can express vector operations in a way that doesn't require a compiler to solve the halting problem, and you need a compiler that knows how to compile such operations to SIMD.

Yes, ANSI C doesn't have such a construct, but GNU C (which I'd bet 95% of open-source code uses anyway) does.


Nearly all SIMD in ffmpeg/libav/x264 is just written in assembly. Back in the day, gcc's code generation for even x86 SIMD intrinsic functions was too poor to consider using it.

(Plus x86's intrinsics are just plain ugly. It's easier to read asm.)

As for the architecture-independent SIMD in GNU C, it's something, but not quite flexible enough. Even C is really not a good enough language to consider implementing this in, because it has an overly-conservative memory model when it comes to char *, which aliases all other memory. For a library dedicated to parsing things which come in the form of bytestreams and 8-bit pixels, this means that most compiler optimizations are defeated right in the inner loops where you'd need them.


Isn't that what the restrict keyword is for?


gcc mostly ignores restrict, though it's got a little better in recent versions. icc uses it a little more.

But restrict is only good in specific situations - if you have three pointers, and #2 can alias #1 but not #3, there's no way to express that.

It also tends to be used on function arguments, not at some higher up point of declaration. So when compilers inline the function and can see more global info about it, the restrict gets lost and optimization gets worse.

I think some kind of stronger 'typedef' would be better.


Yes, but how long does this exist? My guess would be that it isn't really used (yet) in ffmpeg.


Since C99 so 15 years, which is older than ffmpeg.


What I meant is how long is it supported by compilers. I know its by GCC, but I think it's not by MSCV. Is it possible to compile ffmpeg with MSVC? But ok, one could use it and just #define it away for crappy compilers.


According to MSDN, Visual C++ has supported the equivalent __restrict keyword since at least 2005 [1], and according to the ffmpeg documentation [2], "FFmpeg can be built with MSVC 2012 or earlier using a C99-to-C89 conversion utility and wrapper, or with MSVC 2013 and ICL natively."

[1] http://msdn.microsoft.com/en-us/library/5ft82fed(v=vs.80).as...

[2] http://www.ffmpeg.org/platform.html


Well, expressions like these look to me very clearly like a pattern that can be implemented using SIMD instructions:

    map (uncurry (+)) (zip [1,2,3] [4,5,6])
I guess because of these standard functions and the fact that everything in Haskell is immutable and side effect free it should be comparably easy to do SIMD optimizations. It's only a guess, though.


Not if [1,2,3] and [4,5,6] are actually lists, or aren't aligned properly, or aren't the correct bitwidth, or the expression can't saturate/wrap, etc.

The compiler needs to be able to see that the data conforms to all those restrictions (possibly more), and that's not a trivial thing to do.


Shouldn't the compiler be able to see this? After all, Haskell is statically typed. Ok, maybe the compiler (optimizer) needs to look across function boundaries, but it should be possible, shouldn't it?


In the simplest cases, sure. But inferring that amount of information in general programs is very much an open problem. Even with GCC, which incorporates what is roughly (at least within an order of magnitude) state-of-the-art autovectorization, you have to be careful how you write your loops and structure your arrays or the compiler won't pick up on it.

Here's an example in Haskell: consider the case that the list of numbers was generated from a list of some other structure; something like "map x coordinates" to get all the X values from a list of X-Y-Z coordinates. For vectorization to work, those values need to be packed together in memory; however the structure isn't organized as such (at best, you'd have XYZXYZXYZ). That means a nontrivial memory copy, which will cost much more than the vectorization will gain you. So the structure needs to be reorganized, but then that might cause other pessimizations.


I see.


Clearly the whole project should be scrapped and rewritten in Ruby ;)


Ada would have been a good one, and mature enough. :)


A) Ada isn't sexy and unproven B) Most people who do mad algorithmz don't give a fuck if they introduce security vulns (even if they do, they don't know that other languages exist other than C), nor do they care about other languages (probably are put off by the facts that 1. other mainstream languages don't support mad algorithmz (Ada isn't mainstream) 2. if you look up any mainstream programming language you get nothing but dogma)


And yet, FFmpeg couldn't really have been written in anything else. A 5% difference in performance could be the difference between a 1080p video decoding in real time on a low powered device like a phone or a netbook and the video decoding slightly slower than real time. Improvements to FFmpeg made my old netbook work as a video player. I don't think I would have been able to use an FFmpeg-based player at all if it had the overhead of being written in a more memory safe language.


Those 5% are written in Assembly anyway. C does not expose SIMD, non portable compiler specific extensions do.

For the remaining 95% of the code, since 1978 one could use Modula-2, Ada, Pascal dialects, SafeC and many other system programming languages.

Back in the early 80's, C compilers were as crap as any other system programming languages when generating code for home computers.


You forgot:

- One hell of an awesome encoder/decoder library that can hit more formats than I knew existed.

:)


That's not a bug, that's a feature (people just don't realize the POTENTIAL LOL).


From one of the linked blogs containing details:

> There were lots of dynamic allocations

This surprises me; a lot of codecs are specified in such as way as to not require dynamic allocation at all.

I hypothesise that if it was written in another language, we would just see bugs appear in a different form.


I hypothesise that if it was written in another language, we would just see bugs appear in a different form.

Exactly! Programming is hard to get right, no matter what the language. I hate these people who come along saying "hurr, durr, C sucks, project X should have been written in language Y!" All the while ignoring the facts that a) they can go and rewrite X in Y any time they like, and b) if it was written in Y, people would be bitching about issues with Y (eg, speed). Shortsightedness and lack of will to put your code where your mouth is are two of the most despicable traits a programmer can have.


Troll.


Would you really want to try and write a high-quality video encoder in any other language? Now surely you aren't recommending switching to hand-tuned assembly! :)


C++11


programmers are people


Programmers are people, but how many python programmers run into double-free bugs?

Language has a lot to do with what sort of issues programmers will have to tackle.


At the same time, how many video players have been written in pure python? Assuming the number is non-zero, how many get any use?

Using python as an example here makes little sense, since the constraints of the problem domain make it a particularly poor choice. Try choosing a language that doesn't suffer as much in performance (future Rust, maybe?).


Oh, I agree. I think that C was the right choice for FFmpeg, at least at the time, and I do not think that python would be the right choice for it today.

I'm just making a point that language is relevant when discussing the type and frequency of errors that are surfacing in a project.


I guess the correct solution is to use Java? Then we can bitch about run away garbage collection. How about Python then? It's really "fast". Or we can program it in snide remarks on forums, then we don't even have to write any working code to feel superior.


Modula-2, Ada just to cite two possible examples.


This is the first time I've ever seen anyone (seriously?) recommend Ada for a project like this.


If Java is somehow a problem, you can simply make a language without stack smashing but also without GC. Nobody has done this so far because nobody gives a shit about the fact that every program we use is full of remote code vulns and will never be fixed.


You haven't been reading all the Rust release announcement threads, have you?


The thing is, those languages do exist since C does.

But the OSs created with them weren't as fancy as UNIX was at the time, or used hardware more costly than the PDP-11.


I wasn't even taking Rust into consideration. One could make a language the way I said with no problem in a small amount of time. I'm talking no novel stuff. Arrays bound checked at runtime. No attempt at code correctness even has to be offered, get rid of remote code execution vulns, and suddenly you watch HD videos without fearing for your life. It simply isn't done because nobody cares. It's not sexy enough. It's been 30 years now.


this is why Java exploded in the enterprise backend niche


This is cool. I wonder how many of the bugs led to code execution?

If you managed to execute code, what privileges do you have in Chrome? I hope that Chrome was using OS sandboxing for video playing. After all, if they found 1000 bugs, there are probably a few more zero day's available.

Does playing Flash video in Chrome/Firefox end up using ffmpeg? I'm not that knowledgeable about how video works, but there are probably at least two execution contexts: playing .mp4 natively in Chrome, or playing it via a Flash container.


> I wonder how many of the bugs led to code execution?

Quite a few. We're often affected with VLC, and code execution is easy to get to. But with VLC, you're "only" in userland.


In other words, you're "only" dealing with a bug that would allow anything you downloaded from BitTorrent to own up your personal account on your computer. :)

(I know you know that, I'm just spelling it out for people).


Yep, and that's quite dangerous.

Especially since you can see often fake videos circulate around, that tell you to download a special software or codec to play them (with malware, of course). WMP also had a scripting/runtime system that could be abused greatly.

Be careful what you download :)


Chrome is also in userland - and it has a sandboxing system. Assuming they're sandboxing ffmpeg, these bugs are more risky for VLC users than Chrome users. Plus, Chrome is more diligent with security updates and the auto-update mechanism is fully automatic.

A sandboxing system provided by either ffmpeg or VLC would be a very good idea, though it would be some work... encoded data in, decoded frames out via shared memory. Negligible performance impact.


Yeah definitely. Really there is a gross violation of the principle of least privilege here. A video player is a great thing to sandbox because all it needs is a video output, very limited file access, and very limited gui input. It doesn't need to read all your files, open network connections, or start processes.

Given that there are apparently thousands of bugs in the video parsing code, it seems like a no-brainer.

Section 5.2 in this DJB paper talks about (portable) isolation of plain transformations. Video playing is already close to "pure" or could be made pure pretty easily.

http://cr.yp.to/qmail/qmailsec-20071101.pdf


> A video player is a great thing to sandbox because all it needs is a video output, very limited file access, and very limited gui input. It doesn't need to read all your files, open network connections, or start processes.

Not sure if serious or sarcasm, to be honest, since this seems very far from what we see.

A media player is not simple to sandbox, (as the MacOS X sandbox showed us for example), because:

- you need to open files by yourself, without user interaction, to support playlists,

- you need to open connections by yourself to support video protocol like RTSP, RTMP, RTP,

- you need raw device access to support Webcams, Capture devices, DVDs, DVB tuners,

- you need to access GPU buffers for direct rendering, and/or shaders to do fast filtering or just plain chroma-conversions,

- you need to be able to access the audio output, at low-level, for libsync which is not always doable with the simplified APIs,

- and I don't understand what you mean by "very limited gui input"; how is that less than other programs?

Sure, it can be done, with performance costs but it's clearly not a "no-brainer".


I'm talking about a multiprocess architecture like Chrome.

http://www.chromium.org/developers/design-documents/multi-pr...

The entire app isn't sandboxed -- just the code that does video parsing, i.e. with the thousands of bugs and hundreds of remote code execution exploits (!).

See my other comment on this topic. ffmpeg is already very modular, and used in many video players (user interfaces), so this separation is more than natural -- it already exists in the codebase.

BTW, some people seem to be unfamiliar with the multiprocess/Unix design approach (usually people with a Windows background, which I came from as well). I recommend http://www.catb.org/esr/writings/taoup/ for a great intro to this design philosophy.


> The entire app isn't sandboxed -- just the code that does video parsing, i.e. with the thousands of bugs and hundreds of remote code execution exploits (!).

The exploits are usually on the protocol (access) level, the demuxer (format) level but also the decoder level.

While in theory the first 2 are what you call parsing, many security issues appear also at the decoder level.

And if you want to split the video decoder from the rendering, you need to introduce an additional memcpy (or two) of full decoded frames, which has an important impact.

I agree it would be nice to try, with a correct, separated processes architecture, but it means changing also a bit the usual architecture, where there is a direct-rendering between the decoder and the output.


What you do is split the file I/O from the rendering. Drop privs on the renderer, and feed it data from another process that handles the file access, setup, and other privileged operations.

You can open your X11 socket (or whatever your GUI uses) in the I/O process and hand it off to the renderer before dropping privileges, and hand it off without any loss of performance. Another thought -- although I don't generally touch 3d rendering, or know enough to know if it's possible off the top of my head -- is something like cross-process pbuffers: you render into one, and composite the result to the screen from another process. Since your OS already does this sort of thing (X11 compositors, for example, do this sort of cross-process compositing), this can't be too expensive.

You might need to use the XACE extension to restrict the renderer to just the rendering output window in case it gets exploited, so that you don't have access to the rest of the UI.

The places where vulnerabilities can do damage are when communicating with the rest of the OS: File I/O allows you to get user data, permanently install malware, and delete things. The window system allows you to snoop keystroke/mouse data, synthesize input, and watch what the user does. Drop privs on those, and your app is fairly effectively sandboxed.


This bit made me gasp a bit: "[...] all it needs [...]"

Off the top of my head:

>video output

Meaning: RW access to /dev/dri/cardX on desktop. In an embedded system, replace that with /dev/fbX. In most cases, we're rendering to GL buffers. Do you trust the GL drivers to be completely bug free? (Hint: from some embedded drivers I've dealt with I find it remarkable they work at all.)

>doesn't need to [...] open network connections

How about RTMP streaming? Or DLNA discovery and playback?

Some of the cases may be protected with sandboxing, but I have serious doubts about getting anything near complete coverage.


The network stuff should be replaceable with a file descriptor and a dead simple app-specific protocol. Just like Chrome has a little protocol for communicating between the UI and renderer processes.

I think you could also provide less than full access to the graphics driver. You could have a file descriptor or shared memory protocol, with the sandboxed process outputting video frames, and the parent process actually communicating with the driver.

In addition to being more secure, it's also better software architecture. VLC/ffmpeg are already way more modular than say the Microsoft equivalents.


> Negligible performance impact.

Memcpying complete video frame in HD at 30 or 60 fps is not negligible performance impact. But I agree it would be a good idea.


You snipped out the part where I suggested shared memory :)

It might get hairier if the platform graphics api wants to supply you a buffer to render into, and you can't pass it a shared memory buffer yourself (that is accessible by that sandboxed process).


I'm sure some people are running VLC under root, still in userland but can be very dangerous.


I'm sure there are some people using telnet and using "password" as their root password. At a certain point it becomes more than reasonable to say "Well don't do that."


This is one of the reasons why you have to use a wrapper for the VLC binary to execute it within a root environment. Said wrapper is usually in an extra package on Linux and not provided on Windows and OS X (it can be compiled manually of course). Like 10 people per year complain about not being able to run VLC as root, but then again, why should you?


> I wonder how many of the bugs led to code execution?

The linked blog article [1] suggests 10% - 20%:

  Our personal feeling is that between 10% and 20% of the 
  problems could be considered easily exploitable security 
  issues; however, the estimation has not been formally  
  confirmed in any way.
[1] http://j00ru.vexillium.org/?p=2211


Playing .mp4's natively uses FFmpeg, but Flash video does not. Both plugins are sandboxed in a "privilege separation" environment that they mention in the post.


Until we can declare both projects "fuzz clean" we recommend that people refrain from using either of the two projects to process untrusted media files. You can also use privilege separation on your PC or production environment when absolutely required.

Well, it's not like I've got a whole lot of options for video processing. Note that they didn't recommend alternatives that they say are fuzz clean.

I use FFmpeg quite extensively and will continue to do so.


Interesting that they are pushing fixes for both FFmpeg and the libav fork. I guess they decided not to pick sides in that war.


That is good. But this reminds me of the first time I installed 'ffmpeg' on Ubuntu only to have it not work with my well-tested parameters and also declare ffmpeg 'deprecated'. Only after some head scratching did I figure out it had installed libav and an ffmpeg wrapper that tried but failed to be compatible with the real ffmpeg interface.

I wasn't even aware of the split at the time, but this shenanigan definitely gave me a strong negative initial impression of libav.


For sure. The story of FFmpeg and libav is full of sneaky shenanigans like that.


A good entry on this, for those interested: http://blog.pkh.me/p/13-the-ffmpeg-libav-situation.html


That is written by the FFmpeg side. Do you have a complementary view?


The mpv developers have published an overview of the situation from their independent perspective which (FWIW) is very accurate from my vantage point as well: https://github.com/mpv-player/mpv/wiki/FFmpeg-versus-Libav


I thought that shenanigan was entirely the result of actions by a DD/UD and not upstream?

I really miss ffprobe, I can never get avprobe to work on the first or second attempt.


libav tries to ship a compatible wrapper called ffmpeg and a redone one called avconv. But ffmpeg itself just ships ffmpeg and puts incompatible changes in there, so you'd actually be worse off.

What issues were you having, and with what file types?


Incompatible? Do you mean with libav?


Backwards incompatible with previous versions of itself.


Most of the old options have been mapped to the new interface (see opt_old2new() for instance). So most of the old commands are still compatible. Libav also has dropped ffmpeg forked copy since a long while, so FFmpeg is actually more retro compatible.


Once again I am happy that ‘Segmentation fault. Core dumped.’ is a perfectly reasonable reaction to malformed user input in my field of work, which only increases my respect for those who write real-world software. Thanks! :)


So my paranoia has finally taken over. My first read through and I only take in these features:

    * I presume all this infra is used for more than just ffmpeg so why this software specifically? Im conscious its included in everthing from web browsers to games. If this was an attack vector, its a BIG vector.
    * 1121 (or whatever it was) is a curious number to make a post about. Why not at the 1k mark or 1250?
    * Explicitly calling out use of an unpriv user to a readership likely already well versed in basic security practices but notorious for casually ignoring on personal machines
Google, blink twice if theyve got ya tied up in some legal requirement that you cant disclose.


A thousand, and all the categories they list are things the computer should catch for you. How many more will it take before we switch to better languages?


Be aware that these are open-source projects. I don't see you starting a fork in a better language.


I find this, 'don't complain, fork/fix' mentality ridiculous. Why should he? If his goal is get people off C, communicating/persuading in a developer community comments is probably more effective than him creating a project in some other language that no one would likely ever use.


In what way is asking someone, somewhere, to move a project to another, unnamed language, going to be more effective than actually starting a project and putting code into it? Sure, "go do it yourself" is not a pleasant response, but "go take the tens of thousands of lines of code that mostly works, and spend person-years rewriting it in another language so that a certain set of bugs aren't an issue anymore, instead of fixing the mostly working code" is in no way a reasonable request. "What can be done to help reduce the bug count, by a casual user?" might be. Contacting Coverity or another static analysis company that occasionally runs their tools on open source programs to help the world (and get the free press out of it...), might result in a huge list of subtle (and hideously obvious) bugs getting squashed.


> communicating/persuading

Snarky, whiny, bitching comments do not a discourse make.


I get a kick out of the fact that they recommend not using FFMPEG but they say that Chrome uses it.


Chrome is probably using a subset that represents 2% of FFMpeg. I'd guess they have thoroughly audited the code paths that are actually used to decode about 10 media different that chrome supports.

For instance, one fix mentioned was a fix the demuxer for .4xm files. 4xm is a proprietary container format for a now defunct company. Chrome would never have to worry about this bug, because it doesn't need to support playback of .4xm files.


They aren't just flat out not recommending using FFmpeg, just that you need to take appropriate measures when dealing with untrusted media sources.


> I get a kick out of the fact that they recommend not using FFMPEG

More specifically Google is recommending you don't use it to "process untrusted media files"


I don't trust any media files... Especially ones from Google who are doing most of the research... seriously though, media zero-days are unheard of, probably because they would be extremely valuable.


This must be one of the reasons it seems like there's a new update to VLC every time I launch it...


Yes. That's sad, but we need to update often, because of that.

And our update mechanism suck. We're working on improving the process because of that...


Bleeding edge double entendre.


> At Google, security is a top priority - not only for our own products, but across the entire Internet.

Hm, privacy was not on top, I guess - or we're toasted.


Can't deliver privacy guarantees unless you have a secure system in the first place. That's not to say Google does care about privacy, but security being top makes sense.


"a top priority" not "the top priority"


Bit off topic but I may as well bring this up here.

  1. flvrunner uses ffmpeg
  2. biggest result for flvrunner searches is how to remove flvrunner (including browser toolbars)
  3. doubleclick runs webads for flvrunner.com (google own doubleclick)
  4. such ads even run on onlinebehavior.com, which is owned by Google's analytics guy.
There's a lot of malware on the net that google could do more to reduce.

Someone more conspiratorially minded than me might make other deductions.


The software testing course in Udacity is an excellent introduction to the fuzzy testing techniques that the article talks about. Here is the link: https://www.udacity.com/course/cs258


Years ago (I think it was ffmpeg) it would crash constantly on tv input, but it ran in a different process and restarted automatically.


It sounds like about the right level of effort required to maintain something developed by Fabrice Bellard.


"Until we can declare both projects "fuzz clean" we recommend that people refrain from using either of the two projects to process untrusted media files."

BAHAHA What a joke.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: