In short - perfect reflector preserves etendue, but imperfect does not.
Thanks for pointing this out; I seem to have learnt of a new phenomenon I wasn't previously aware of: https://en.wikipedia.org/wiki/Opposition_surge
At this point I almost think an experimental measurement of temperature equilibrium would be publishable from one of the old re/frac/flectors you can put your head into.
The idea is that you can "organize" or "revert" any ray bundle from a system of non-absorbing lenses and specular reflectors, but if your reflector has billions of tiny irregularities it's not viable to build such a system (it's equivalent to an ideal diffuser, in which light is isotropically reflected). The ideal diffusion process is clearly not reversible by itself: if you shine a beam onto a diffuser it spreads the light; if you expose it to the same light (with reversed directions), it again diffuses it instead of reverting to the original beam. In theory again the physical laws of electromagnetism are time reversible, but in practice the effort to revert some systems might be too demanding (you can even do better -- see Maxwell's demon); manipulation of physical apparatus and information acquisition/manipulation itself has a cost that surpasses any gains.
In practice, any reflector is imperfect, and therefore has nonzero emissivity, and therefore goes towards thermal equilibrium with its environment.