RTX is meant to be linked to "raytracing".
Raytracing in general computes a scene by computing how photons are affected by matter - e.g. a full reflection by an absolutely smooth non-absorbing surface or a partial reflection&path_divergence done by liquids, etc... .
In the simulation, the photon that "bounces off" a surface is then "rebounced" by another surface and so on, and this creates a picture similar to the one we use to see in the real world.
RTX maybe cuts the whole "rebouncing" and generation of photons a bit short, meaning that there isn't really any new next-gen technology but it's just a bit more processing power that is available to do some additional parallel short/semi-pure raytracing stuff, but which does not work when your scene is complex and it does need many reflections "rebounded" many times.
Again: this is just my initial understanding.