Browsers sniff the doctype to change rendering mode, but that's after they've already decided that it's HTML that they're going to display.
IE6 & 7 do content type sniffing, where they ignore what the server tells them it is and try and figure it out themselves, by looking at various bits of info including the start of the file.
This could lead to HTML being downloaded if it happened to look like a RAR file to IE, but generally it has the opposite effect as in something being sent as plain text that happens to include tags being rendered as HTML.
If anything, shortening the doctype should make something look more like HTML since it works without any doctype at all and the shorter the doctype the more room for HTML tags.
On the other hand content-sniffing is always going to give you unexpected results. Is there anything about the particular file you had that makes it atypical or likely to look like a binary file if you only consider the start of it?