actually, there is a reason. because in IE6 it breaks depending on context. if you have an incorrect content header and don't prefix http:// on your url, then having an html 5 doctype makes your page try to "download" instead of display.
Whereas having a doctype that matches the tags you actually use, (xhtml or html4) will work fine without the http:// prefix. As most people don't use that prefix when typing your url, this can have a drastic effect on your page views.
afraid not, as I recently came across it and have fixed it by adjusting the content headers. it's a quick fix, but can be detrimental as a lot of web servers will present the incorrect content headers and developers will be completely unaware since browsers will obey doctypes over content headers (I think). In this particular case IE6 will obey the content header since it doesn't understand the html5 doctype and will try to "download" the content instead.
It should be trivial to setup a test case if you want. Setup a virtual host with an empty content-type html header, and a html5 doctype and open it in IE6 without putting http:// before the url. You should see it try to download the page.
Not that I'm doubting you, but it doesn't really make sense to me.
Browsers sniff the doctype to change rendering mode, but that's after they've already decided that it's HTML that they're going to display.
IE6 & 7 do content type sniffing, where they ignore what the server tells them it is and try and figure it out themselves, by looking at various bits of info including the start of the file.
This could lead to HTML being downloaded if it happened to look like a RAR file to IE, but generally it has the opposite effect as in something being sent as plain text that happens to include tags being rendered as HTML.
If anything, shortening the doctype should make something look more like HTML since it works without any doctype at all and the shorter the doctype the more room for HTML tags.
On the other hand content-sniffing is always going to give you unexpected results. Is there anything about the particular file you had that makes it atypical or likely to look like a binary file if you only consider the start of it?