On OSX, I can accept this argument, because Quicktime is the primary codec... repository(for lack of a better word) for basically everything on the OS.
However, I don't think it's the case for Windows or Linux. On those systems, it should just be another codec/browser plugin. Playing Quicktime videos should be the same as any other codec, so I don't really understand why it's not possible to sandbox it like Flash.