No it's not getting preprocessed to actual HTML in React. Depending on the transform function, it can be turned to an object tree or a function calls tree (like in React) or actually HTML like you said but it's not a common use case AFAIK.
It's not a text templating language like classic PHP, it's an object/function calls hierarchy DSL (more like vb.net XML literals). I think most people critisizing react for putting html in JS havn't tried it or don't remember why "html" in code was considered bad in the first place.