You still need the photons to hit the surface of the object and deposit their energy. Talking about the temperature of the surface of the sun is just a simpler way to refer to the energy output. What you are saying is a very high level view of the actual physics involved. The physics just works out that we can use temperature to talk about this.
