"A number of problems resulting from the use of the text/html MIME type in conjunction with XHTML content are discussed. It is suggested that XHTML delivered as text/html is broken and XHTML delivered as text/xml is risky, so authors intending their work for public consumption should stick to HTML 4.01, and authors who wish to use XHTML should deliver their markup as application/xhtml+xml."
Note the date (Sep 2002) and the fact that the author is now in charge of the HTML5 spec.
Some people argue that forcing web developers to follow rigid XML rules, plus an appendix full of hints on how to mangle your XML so that actual browsers wouldn't choke on it (http://www.w3.org/TR/xhtml1/guidelines.html) somehow got people in the right mindset to do semantic, accessible HTML. The prevailing view is that this was mostly cargo cult behaviour and neither those following or recommending this course of action really understood the supposed benefits or drawbacks.